Announcement

Collapse
No announcement yet.

Refresh Issue on Large Import

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Refresh Issue on Large Import

    • Type of import: clean
    • Source board: Invision Power Board 2.3.1
    • Target version: 3.8
    • Module issue occurred on: Several
    • Any SQL error reported: none

    I'm importing a gigantic board (13milllion posts) and in my testing, i've been able to get it to work just fine, but I've moved to my actual development environment and things are all wacky. I know that impex doesn't officially support vb 3.8, but I doubt that the changes in vb would make impex not import in this fashion.

    1. The module pages keep refreshing, but not moving on to the next batch of data. It just keeps loading the same page over and over. When I did imports before, it would print everything out as I did it, but now it seems to work for a while before even loading the page. I think it has something to do with output_buffering, so i turned it off in my php.ini

    2. Sometimes during the import, the page will go back to the "start page" (where it lists each module that can be ran). When I click on the module that was running, it does work though.

    I think that the session isn't being saved properly or something, but does anyone have any ideas, why the refresh wouldn't work properly?
    Last edited by kaanon; Wed 28 Jan '09, 12:03am. Reason: more info

  • #2
    Originally posted by kaanon View Post
    I know that impex doesn't officially support vb 3.8, but I doubt that the changes in vb would make impex not import in this fashion.
    3.8 is supported, where did you read that it isn't ? ImpEx always supports the latest version of vBulletin.

    With a board of that size, PHP memory and the page speed would be the main things to me, I'd likely give it a page speed of 5-10 and as much memory as I could.

    Imports of that size have been done, I've done a few.

    Is it a local machine import or a remote server ?

    What memory does PHP have ? (I'm hoping that the time out is turned off).
    I wrote ImpEx.

    Blog | Me

    Comment


    • #3
      It's a local machine import (vbulletin & invision are on same box, but the mysql db is on a seperate machine. Both databases are on the same mysql box though)

      Here are my relevant settings:
      max_execution_time = 0
      max_input_time = 60
      memory_limit = 1024M

      When u say set page speed to 5 or 10 is that the setting in ImpExConfig.php?

      Comment


      • #4
        Originally posted by kaanon View Post
        It's a local machine import (vbulletin & invision are on same box, but the mysql db is on a seperate machine. Both databases are on the same mysql box though)
        Local in the sense that they aren't accessed over a network, i.e. on your local machine.

        All the big imports I do (that have issues) I typically do on my local development machine (which is a linux box that would put servers to shame) and then tarz a mysqldump and move that around.

        Originally posted by kaanon View Post
        Here are my relevant settings:
        max_execution_time = 0
        max_input_time = 60
        memory_limit = 1024M
        Same as mine

        When u say set page speed to 5 or 10 is that the setting in ImpExConfig.php?
        Yes the page speed, is a setting for the amount of seconds the JavaScript refresh waits before reloading, it allows the dB or network lag to recover before loading the next page.
        I wrote ImpEx.

        Blog | Me

        Comment


        • #5
          Fantastic, I'll give that a shot.

          Comment


          • #6
            With this done :

            http://www.vbulletin.com/docs/html/med_large_import

            I usually import 20,000 items per page.

            I would expect that many to take 6-8 hours.
            I wrote ImpEx.

            Blog | Me

            Comment


            • #7
              Yeah, my post table took 14 hours, for a total of 17 hours for the boards. You must have a really powerful desktop. This is way better than the several days it took before, though. Are there any other things I could try for speed? I took out the index and did 20,000 per page load. The only thing I could think of is removing the other indexes on the post table as well. Turns out I forgot to change the pagespeed, so that ought to help too.


              Clearly this is still too much downtime for boards, so I'm going to try and hack impex so that the *startat variables reflect the auto_increments in the table and see if I can get a proper delta to the imported data. This way, the only thing I'll lose is updates, which can be just be communicated to our users.

              Thanks for the help.

              Comment


              • #8
                Originally posted by kaanon View Post
                Clearly this is still too much downtime for boards, so I'm going to try and hack impex so that the *startat variables reflect the auto_increments in the table and see if I can get a proper delta to the imported data. This way, the only thing I'll lose is updates, which can be just be communicated to our users.

                Thanks for the help.
                Edit impex/systems/ipb2/007.php around line 88 is :

                PHP Code:
                $post_start_at $sessionobject->get_session_var('poststartat'); 
                Change that to :

                PHP Code:
                #$post_start_at = $sessionobject->get_session_var('poststartat');
                $post_start_at X
                Edit ImpExConfig :

                PHP Code:
                define('step_through'true); 
                Then you can run the post module for one page which will seed the startat point, then remove the first edit and restore the $sessionobject->get_session_var() which will allow ImpEx to carry on as normal.

                After performing that end, change the ImpEx config step_through back to false, so the page auto advances. Then click the button to advance the page in ImpEx (which will off been sitting there while you do the edits) to get it going again.

                From that point it will carry on as per normal.
                I wrote ImpEx.

                Blog | Me

                Comment


                • #9
                  Fail.

                  Apparently redoing a module clears out all users, so redoing the user module is now clearing out the user table and it's related tables.

                  Luckily, i still have the tarball of the db, so I'm going to comment out that line in 004.php and try again.

                  Comment


                  • #10
                    Ah yes ....... eek.

                    Line 38 of impex/systems/004.php

                    PHP Code:
                    if ($this->restart($sessionobject$displayobject$Db_target$Db_source,'clear_imported_users')) 
                    to

                    PHP Code:
                    if (true
                    Will be the same for other modules to stop them clearing out.
                    I wrote ImpEx.

                    Blog | Me

                    Comment

                    widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                    Working...
                    X