Announcement

Collapse
No announcement yet.

Problem Importing a BIG PN pbpBB database

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    Out of curiousity, would it be worth me setting up apache, php, and mysql on this computer? It's an AMD Athlon FX-55 with 4gb of ram, liquid cooled...I'm sure it could tear up the import process in a few minutes. I don't know if all that setup is worth it though.

    Asuming I did do that, I could do the import locally, and then dump the SQL file and then import that SQL file on the server, right?

    Comment


    • #17
      Originally posted by Jerry
      Well if it comes to it we can stop impex doing the parent id update, do the post import then do the parent id update via the command line then carry on with the import. Its usualy a lot faster that way.
      Asuming we tried this approach, how should I undo the import? Do I have to empty the users, posts, and threads tables and then reset the session? Or is there more to it than that.

      EDIT: Also, not that it is very relevant at this point, but asuming we get all this imported properly, is there a way to just import a few days worth of new data without doing the whole thing again? Right now that SQL dump is everything up until March 28th. The move is probably going to take place in a few days. So is there a way to just import the 3 or 4 days of new data without doing the 2 years of old stuff?

      On a side note, I realized just how screwed up this is now. I didn't even notice before, but the newest posts on the board (should be March 29th 2005) are January 3rd 1970. Forums in 1970 would have been a trip...lol

      Now if you actually click through to the post, it shows the correct date and time.
      Last edited by RealityDesign; Wed 30 Mar '05, 10:57pm.

      Comment


      • #18
        Originally posted by RealityDesign
        Out of curiousity, would it be worth me setting up apache, php, and mysql on this computer? It's an AMD Athlon FX-55 with 4gb of ram, liquid cooled...I'm sure it could tear up the import process in a few minutes. I don't know if all that setup is worth it though.
        Well with that hardware localy opposed to the server, I'd definatly set it up localy, even with moving the database "mysqldump -Q --opt".zip about I would guess it would be faster, just depends on attachements.

        Though you can move an import between modules.

        Originally posted by RealityDesign
        Asuming I did do that, I could do the import locally, and then dump the SQL file and then import that SQL file on the server, right?
        Yes.

        Originally posted by RealityDesign
        Asuming we tried this approach, how should I undo the import? Do I have to empty the users, posts, and threads tables and then reset the session? Or is there more to it than that.
        ImpEx does it, the link as at the top of ever ImpEx page, database clean up, remove all imported data and session.

        Originally posted by RealityDesign
        EDIT: Also, not that it is very relevant at this point, but asuming we get all this imported properly, is there a way to just import a few days worth of new data without doing the whole thing again? Right now that SQL dump is everything up until March 28th. The move is probably going to take place in a few days. So is there a way to just import the 3 or 4 days of new data without doing the 2 years of old stuff?
        That’s a “differential import” and is planned for ImpEx2, so at the moment, I’m sorry but no.

        Originally posted by RealityDesign
        On a side note, I realized just how screwed up this is now. I didn't even notice before, but the newest post on the board (should be March 28th 2005) are January 3rd 1970. Forums in 1970 would have been a trip...lol
        This is because the update counters haven't been run, please read the HOWTO.

        http://www.vbulletin.com/forum/showthread.php?t=116398

        Though that is irrelevant till the posts are imported correctly.
        I wrote ImpEx.

        Blog | Me

        Comment


        • #19
          OK, well it would definitely be EASIER for me atleast, to just do everything on the server. If it's only going to save an hour of time, I would rather not have the hassle of setting up a web server on this machine and all that.

          Have you had a chance to test that dump file? It seems to me that everything is fine, and that I just need to re-do the entire import and just do that last part using the command line (if you show me how).

          Comment


          • #20
            Originally posted by Jerry
            This is because the update counters haven't been run.
            Ya, I read the howto and ran the counters, I just did them out of order I guess. I did forums then threads. I just did forums again and it looks great. Now if we could just get all my threads working properly we would be golden .

            Comment


            • #21
              Still loading it, as its a straight dump.

              Next time you dump a database to move it I'd suggest using.

              -Q --otp

              http://dev.mysql.com/doc/mysql/en/mysqldump.html
              I wrote ImpEx.

              Blog | Me

              Comment


              • #22
                Like I said, I didn't dump it That is the file the person selling the website gave to me from his hosting company.

                It's safe to say that they don't know what they are doing.

                I mean, why else would they use PHP Nuke? LOL

                Comment


                • #23
                  Originally posted by RealityDesign
                  Like I said, I didn't dump it That is the file the person selling the website gave to me from his hosting company.

                  It's safe to say that they don't know what they are doing.
                  Well just so you know for when you do.
                  I wrote ImpEx.

                  Blog | Me

                  Comment


                  • #24
                    It's been a whole 30 minutes and no reply lol . Just letting you know I'm still here, I'll be on all night. Hope everything is going ok with that sql dump file.

                    Comment


                    • #25
                      Sniffle, I think you left for the night I still need to know how to solve the problem using the command prompt.

                      Comment


                      • #26
                        34 mins so I was a little out, though that includes all the threads and forums counter updates.

                        I would defiantly get the dev site sorted on localhost and move the database about as your server must be crippled some how.
                        Attached Files
                        I wrote ImpEx.

                        Blog | Me

                        Comment


                        • #27
                          Out of curiousity, why did it take 14 minutes to import 8000 users, but only 17 minutes to do 230,000 posts???

                          That just doesn't seem right.

                          Comment


                          • #28
                            Because there is a lot more SQL going on for user and if they have remotely hosted avatars they are imported so ImpEx has to wait for the PHP fopen() result and the download of the image if the link is valid.

                            Turning off the avatar import gets it down to 28 seconds.
                            I wrote ImpEx.

                            Blog | Me

                            Comment


                            • #29
                              22,641 threads in 27 seconds.

                              This would suggest that you are managing to import 800 threads per second. How is that possible? What setting are you using? I used the default 500 per page and 1 page certainly doesn't go by 1 in second using an AMD 64 3200+ with 2gb of ram.

                              You said it took you 27 seconds it took me more like 10 minutes, so what's the catch? I haven't tried it on the FX-55, I'm using the 3200+ because its a spare computer, but there is no way your computer is 20 times faster...so what am I doing wrong?

                              Specs:
                              AMD 64 3200+, 2gb pc-3200 ram, 160gb SATA raid-0.
                              Apache 2, MySQL 4.1, PHP 4

                              Modules so far:
                              001 - Check & Update - 5 seconds
                              002 - skipped
                              003 - Import Users - 15 minutes
                              004 - Import Forums - 1 second
                              005 - Import Thread - 12 minutes
                              006 - Import Smilie - 1 second
                              007 - Import post - still going, on posts 29,000 to 29,500 right now. Timed with a stop watch it takes about 14 seconds per 500. 230k / 500 = 460 x 14 seconds = 6,440 / 60 seconds = 107 minutes.

                              ~110 mins versus 17.

                              On a side note, the server you are now calling "crippled" did it in about 140 minutes (only 30 minutes slower than this computer). I know for a fact this computer isn't crippled...it gets 300 fps in Counter-Strike:Source, got amazing benchmarks sandra and 3dmark, so...is there some magical way to configure the apache / mysql server? I don't see how setting it up one way versus another would make it 20 times faster. You were saying yesterday that it would be caused by configuration and load...well there is zero load since it's a local computer, so what is wrong with my configuration? Do you have any ideas? This doesn't make much sense.
                              Last edited by RealityDesign; Thu 31 Mar '05, 7:48pm.

                              Comment


                              • #30
                                On yet another side note (yes, I'm just full of side notes) that computer imported the 400mb sql file dump in 8 minutes using the command prompt.

                                So lets see:

                                SQL through command prompt, importing the entire database, including the 8000 users, 230,000 posts, etc. takes 8 minutes but the import script is going to take more like 2 to 2.5 hours.

                                It seems to me that this would suggest there is nothing wrong with my computer and that you are either doing it a different way or have a different script than I do. This script is clearly what is causing the inefficiency.

                                Either way, coming on here and offering help and then finally after 24 hours posting a picture and saying "your server sucks" is not exactly a solution. I just hope it doesn't time out at post #230,071 again.
                                Last edited by RealityDesign; Thu 31 Mar '05, 7:59pm.

                                Comment

                                widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                                Working...
                                X