Announcement

Collapse
No announcement yet.

Problem Importing a BIG PN pbpBB database

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Problem Importing a BIG PN pbpBB database

    I am importing a post nuke PHPBB 2 database with 8,000 users and 230,000 posts.

    It has been going for HOURS and now it's stuck on the very last page...

    Importing 71 posts

    From : 230500 :: To : 230571



    Normall it takes about 20 seconds per 500. This one is the very last page (only 71 posts left!) and it just hangs there for 10 minutes...then it says:

    CGI Timeout

    The specified CGI application exceeded the allowed time for processing. The server has deleted the process.


    If I click on import / export again, it just starts the process up again and tries to do the 71 posts again. It's tried 3 times, everytime I get the CGI timeout.

    What is wrong? How can this be fixed? This is very important, if anyone knows please post! Thanks.

  • #2
    The final stage of any post import is updating the parent id's so if the posts don't have valid threaded id's they are all updated to have the parent id of the first post in the thread.

    On 230K posts that is going to take a little while, though I've done it on 1mill+ boards, so it is possible.

    Clicking on the import / export link will reload the session at that page i.e. "get the last 71 post and if finished, update the parent id's".

    10 mins sounds exceptional though, is it exactly 10 mins, or just seems like for ever ?

    This final parent id step can be bypassed temporarily, though it will have to be done for the board to be viewed correctly.

    Do you have remote access to the MySQL server ? i.e. can you connected to it from another server ?
    I wrote ImpEx.

    Blog | Me

    Comment


    • #3
      I have remote desktop access to the Windows 2000 server which is running IIS, PHP, & MySQL.

      I tried it 3 times, it's not EXACTLY 10 minutes, it just seems like forever, and all 3 times it timed out.

      Comment


      • #4
        Also, after the third time it timed out, I just went to the update counter section and had it update the threads. I can see them now but it's saying I have more posts than I really do on the counter in the statistics and some other weird things...

        Do I need to start over at this point? It tooks 3 hours to import all the posts the first time, I was hoping not to have to do it again.

        Also, what is causing it to time out?

        Comment


        • #5
          Also, I don't know if it matters, but here is what I can see in the MySQL Database...

          The postid and threadid's match the importthreadid and importpostid's.

          For example:

          Post ID, Thread ID, Import Post ID, Import Thread ID
          1, 1, 1, 1
          2, 1, 2, 1
          3, 1, 3, 1

          etc.

          Comment


          • #6
            vBulletin database:

            Post: 216,433
            Thread: 22,641


            PHPBB database:

            Posts: 230,571
            Topics: 22,641



            What vBulletin says on the statistics at the bottom of the forum page:

            Threads: 22,641, Posts: 238,760


            Now I'm not exactly sure how it works, but here is what I could tell just from the numbers.

            If vBulletin adds threads + posts and then calls it posts in the statistics it would be 22,641 + 216,433 = 239,074.

            If it just displayed the number of posts on the statistics it would be 216,433.

            So the 5 million dollar question is...where the hell did 238,760 come from?

            I really don't care as long as everything works, but the numbers not matching suggests something isn't working.

            Comment


            • #7
              Ok it's looking like some posts got duplicated. I don't know if this is a coincidence or not, but I refreshed that last page (with 71 posts left) exactly 3 times, and there are exactly 3 duplicate posts in one of the threads I just looked at.

              This still doesn't account for there being 314 extra posts. Hitting an extra 2 times would only cause 142 extra posts if this were true.

              Still very strange. If you disapeared for the night that would really suck . Hope you're still out there!

              Comment


              • #8
                Originally posted by RealityDesign
                I have remote desktop access to the Windows 2000 server which is running IIS, PHP, & MySQL.
                Windows is going to make this harder.

                Originally posted by RealityDesign
                I tried it 3 times, it's not EXACTLY 10 minutes, it just seems like forever, and all 3 times it timed out.
                Is the MySQL database on the same server ?

                Originally posted by RealityDesign
                Also, after the third time it timed out, I just went to the update counter section and had it update the threads. I can see them now but it's saying I have more posts than I really do on the counter in the statistics and some other weird things...
                That because you restarted the import during a page it will grab more posts every time you click the link when its working.

                Originally posted by RealityDesign
                Do I need to start over at this point? It tooks 3 hours to import all the posts the first time, I was hoping not to have to do it again.
                Yes you will, though if you are importing into a fresh database I would suggest doing it on a more capable server, a P4 2.4 gig with 1 gig of ram should do that in 15-20 mins.

                Originally posted by RealityDesign
                Also, what is causing it to time out?
                As I explained in the first reply, the updating of the parent id's.

                Originally posted by RealityDesign
                Also, I don't know if it matters, but here is what I can see in the MySQL Database...

                The postid and threadid's match the importthreadid and importpostid's.
                That sounds about right if its a clean vBulletin install.

                Originally posted by RealityDesign
                vBulletin database: Post: 216,433 Thread: 22,641

                PHPBB database: Posts: 230,571 Topics: 22,641

                What vBulletin says on the statistics at the bottom of the forum page:

                Threads: 22,641, Posts: 238,760

                Now I'm not exactly sure how it works, but here is what I could tell just from the numbers.

                If vBulletin adds threads + posts and then calls it posts in the statistics it would be 22,641 + 216,433 = 239,074.
                That is not how it reports post.

                Originally posted by RealityDesign
                If it just displayed the number of posts on the statistics it would be 216,433.

                So the 5 million dollar question is...where the hell did 238,760 come from?
                You, by clicking the import / export link while it was still running.

                Originally posted by RealityDesign
                I really don't care as long as everything works, but the numbers not matching suggests something isn't working.
                It sounds like it is working fine, its just not being allowed to finished.

                Originally posted by RealityDesign
                Ok it's looking like some posts got duplicated. I don't know if this is a coincidence or not, but I refreshed that last page (with 71 posts left) exactly 3 times, and there are exactly 3 duplicate posts in one of the threads I just looked at.
                Once again, that is because you clicked the import / export link while it was running.

                Originally posted by RealityDesign
                This still doesn't account for there being 314 extra posts. Hitting an extra 2 times would only cause 142 extra posts if this were true.
                It depends on where it was in the pass.

                Originally posted by RealityDesign
                Still very strange. If you disapeared for the night that would really suck . Hope you're still out there!
                I am here, though I would suggest reading my replies and answering before posting 5 times !!

                You need a server or access to a server where the parent id's can be updated


                Originally posted by me
                Do you have remote access to the MySQL server ? i.e. can you connected to it from another server ?
                I wrote ImpEx.

                Blog | Me

                Comment


                • #9
                  The MySQL Database is on the same server. In other words yes, I can connect to it with phpmyadmin or using the command prompt.

                  I can't just go out an buy a new server...a whole bunch of websites are on this server and I'm not going to buy a new dedicated server for one forum.

                  The specs aren't that hot, it's fairly old, but it shouldn't be a huge problem...

                  AMD Athlon XP 2000+, 512mb ram...

                  You're telling me a few hundred MHZ and another stick of ram will make it run 10 times faster? That doesn't seem right.

                  As for the time out, I understand it's being caused by it trying to do the thing with the import ID's, but how do I make it NOT time out.

                  Comment


                  • #10
                    Originally posted by RealityDesign
                    The MySQL Database is on the same server. In other words yes, I can connect to it with phpmyadmin or using the command prompt.
                    That will help. Can you run php scripts from the command line ?

                    Originally posted by RealityDesign
                    I can't just go out an buy a new server...a whole bunch of websites are on this server and I'm not going to buy a new dedicated server for one forum.
                    You don't have to, I meant to get the import done.

                    Originally posted by RealityDesign
                    The specs aren't that hot, it's fairly old, but it shouldn't be a huge problem...

                    AMD Athlon XP 2000+, 512mb ram...
                    That is powerful enough, the major crunch with any system is current load and configuration.

                    Originally posted by RealityDesign
                    You're telling me a few hundred MHZ and another stick of ram will make it run 10 times faster? That doesn't seem right.
                    I've seen well set up servers go faster than that, especaily if the are *nix vr windows and well set up.

                    Originally posted by RealityDesign
                    As for the time out, I understand it's being caused by it trying to do the thing with the import ID's, but how do I make it NOT time out.
                    Do you have a mysqldump of the PNphpBB database available ?
                    I wrote ImpEx.

                    Blog | Me

                    Comment


                    • #11
                      I am doing the import using a mysql dump of the phpbb db. Let me better explain the situation.

                      There is website which we have recently purchased and it is hosted at a seperate hosting provider. When the time comes we plan to have him close down the boards, run a mysql dump, and transfer the dns. While the dns is transfering, we want to take that dump, import it to our mysql databse, and then convert it to vbulletin.

                      Right now we are just in the testing phase. We are using a mysql dump he made yesterday. We imported it to our database, then installed a fresh copy of vbulletin, then used the import feature.

                      I'm not sure if I can run PHP from the command line, I've never tried. If you could give me some instructions or tell me what you're thinking about doing, that would be great.

                      Right now I am using my computer at home using the vB control panel. I can always remote desktop to the server and try to execute the scripts in the command line like you said, I just don't know how.

                      As for the server load, it's very minimal right now. CPU usage is at 2%, unless I'm doing the import . I don't know about the configuration, I didn't build the server, but it's worked great for over a year with the 45 sites we host from it.

                      Comment


                      • #12
                        Originally posted by RealityDesign
                        I am doing the import using a mysql dump of the phpbb db. Let me better explain the situation.

                        There is website which we have recently purchased and it is hosted at a seperate hosting provider. When the time comes we plan to have him close down the boards, run a mysql dump, and transfer the dns. While the dns is transfering, we want to take that dump, import it to our mysql databse, and then convert it to vbulletin.

                        Right now we are just in the testing phase. We are using a mysql dump he made yesterday. We imported it to our database, then installed a fresh copy of vbulletin, then used the import feature.
                        Testing and planing rules the day !

                        Originally posted by RealityDesign
                        I'm not sure if I can run PHP from the command line, I've never tried. If you could give me some instructions or tell me what you're thinking about doing, that would be great.
                        Open a command line on the server and type :

                        Code:
                        php -v
                        Originally posted by RealityDesign
                        Right now I am using my computer at home using the vB control panel. I can always remote desktop to the server and try to execute the scripts in the command line like you said, I just don't know how.
                        Can you PM me a URL to download the mysqldump from so I can test an import just to rule out any bugs or dB issues ?
                        I wrote ImpEx.

                        Blog | Me

                        Comment


                        • #13
                          Sent you that PM. Going to try the php -v now.

                          Comment


                          • #14
                            'php' is not a recognized internal or external command.

                            I'm sure I'm typing it in the wrong place. I know with MySQL I had to use one of the exe files in the bin folder.



                            EDIT:

                            Ok, I had to change the directory to c:\php and then typed PHP -v and it works. Stupid me.

                            Also looks like you're downloading that dump file now. Server jumped to 600k/s steady bandwith lol

                            Comment


                            • #15
                              Originally posted by RealityDesign
                              Ok, I had to change the directory to c:\php and then typed PHP -v and it works.
                              Well if it comes to it we can stop impex doing the parent id update, do the post import then do the parent id update via the command line then carry on with the import. Its usualy a lot faster that way.

                              Originally posted by RealityDesign
                              Also looks like you're downloading that dump file now. Server jumped to 600k/s steady bandwith lol
                              "Alll your mysqldump file are belong to us"
                              I wrote ImpEx.

                              Blog | Me

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...
                              X