Announcement

Collapse
No announcement yet.

Null fields

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Null fields

    www.ScubaBoard.com is a huge forum and we are doing a clean install on a different server and then importing these files. I am almost finished with the Thread Module, and it's taken 12 hours (189,000 threads). How long will it take to import my 2.5 million posts? I am simply astounded at the length of time it's taking!
    1. How long is each module supposed to take?
    2. Is there anything I can do to speed things up?
    3. Also, can I do an incremental merge? IOW, the DB that I am pulling from is about two weeks old. I want to get all of my templates up and working on the new version and then do a db dump of all records that have been created or modified since the last dump, restore that as it's own db and import from there.

  • #2
    Null fields

    I lost about 1600 users in a recent DB import due to them having an empty e-mail field in their user profile. Can I still import these, or do I have to put in some bogus data in there to facilitate this?

    Comment


    • #3
      From what software are you importing (name, version) to which vb version?

      Comment


      • #4
        I've done 250,000 threads in less then 6 hours - so shouldn't be taking this long. But with millions of posts .. I can imagine it will take you a good day.

        Comment


        • #5
          email is a mandatory field in vBulletin ImpEx will not import a user without one as this would break vBulletin's database.
          I wrote ImpEx.

          Blog | Me

          Comment


          • #6
            As for the speed I would need to know some of this :

            http://www.vbulletin.com/forum/showthread.php?t=161907

            Two main things are set this in ImpExConfig :

            PHP Code:
            define('dupe_checking'false); 
            That will trade off the checking for speed.

            PHP Code:
            define('shortoutput'true); 
            Will lower the amount of data sent back to the browser, so overhead.

            Second thing is to raise the PHP memory limit a lot i.e. 256M, and then run a lot more items per page, as well as running ImpEx stand alone.

            The ImpEx dev sever is a Opteron 144 1.8Mz with 2 gig of ram and that will import 200,000 threads in an hour or so. With 500M for PHP and doing 20,000 threads per page running standalone.

            I just did a vB 3.6.8 to vB 3.6.8 import here (testing I'm working on) and it took this :

            Import thread 33723 0 35 sec(s)
            Import post 450307 220 22 min(s)

            and that was with a database restore going on in the back ground.
            I wrote ImpEx.

            Blog | Me

            Comment


            • #7
              Originally posted by Floris View Post
              From what software are you importing (name, version) to which vb version?
              We are moving from 3.5.7 to 3.6.8. We seem to have resolved this by running a simple query in MySQL and replaced the null fields with [email protected].

              Comment


              • #8
                • Type of import: CLEAN
                • Source board system and version. 3.5.7
                • Target version. 3.6.8
                • Module issue occurred on. Latest version of Impex
                • Any SQL error reported. None.
                I am going to check to see how close we are. The 2.5+ million posts appears to be needed 70 hours to accomplish. We are hoping for sometime tomorrow.

                The Server is a 3 gHz Dual Xeon with 4 Gigs of memory. It's running CentOS and mySQL 5.1

                Comment


                • #9
                  Originally posted by Jerry View Post
                  Two main things are set this in ImpExConfig :

                  PHP Code:
                  define('dupe_checking'false); 
                  That will trade off the checking for speed.

                  PHP Code:
                  define('shortoutput'true); 
                  Will lower the amount of data sent back to the browser, so overhead.

                  Second thing is to raise the PHP memory limit a lot i.e. 256M, and then run a lot more items per page, as well as running ImpEx stand alone.
                  All this has been done, and I am retrying the import process. Thanks!

                  Comment


                  • #10
                    WHOA! It is FLYING now. Importing the threads, which took over 12 HOURS took just over 13 minutes! That's an amazing throughput.

                    Comment


                    • #11
                      I would seriously suggest running an upgrade instead of using impex to import. Importing will reset and change all of your user and post ids/

                      Comment


                      • #12
                        Originally posted by NetDoc View Post
                        WHOA! It is FLYING now. Importing the threads, which took over 12 HOURS took just over 13 minutes! That's an amazing throughput.
                        I do 20,000 per/page stand alone usually, what are you running at ? I dare say 70 hours is a bit of an over estimate with a server of that capacity.
                        I wrote ImpEx.

                        Blog | Me

                        Comment

                        widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                        Working...
                        X