No announcement yet.

Impex Import super slow

  • Filter
  • Time
  • Show
Clear All
new posts

  • Impex Import super slow

    I am importing from a custom MSSQL based forum using the fusetalk template for reference. The data is importing, but at an extremely slow rate. It currently processes 10 users/min. I have 316,000 users and 13,000,000 posts to import. At this rate it will take me 2 months to import. I tweaked the php memory limit to 1024M and MySQL buffers but it is still slow. The load on the servers is less than .1% so I have resources to spare. Any Help/Guidance is appreciated.

  • #3
    Web Server (Nginx /php-fpm):

    OS: Red Hat Enterprise Linux Server release 6.3 (Santiago)
    CPU: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz x2
    Memory: 4 GB RAM
    NIC: VMware VMXNET3 Ethernet Controller

    MySQL Server (5.5.22)

    OS: Ubuntu 12.04 LTS
    CPU: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz x2
    Memory: 4 GB RAM
    NIC: VMware VMXNET3 Ethernet Controller


    • #4
      Is there any documentation on the minimum data that is required for importing to VBulletin? I want to try and break down the process into manageable chunks as the import process seems to trip up on itself, and I get no debug info even though the flag is set to true. I end up with the same users getting imported more than once, the first time they are correct, the second their usernames end up with "Imported" pre-pended to them.


      • #5
        Before you run the import again, you should clear the imported data.

        When you say it trips up, where exactly is it stopping? Is it saying it's failed or is it simply stopping mid-process?
        Vote for:

        - *Admin Settable Paid Subscription Reminder Timeframe*
        *PM - Add ability to reply to originator only*
        - Add Admin ability to auto-subscribe users to specific channel(s)
        - Highlight the correct navigation tab when you are on a custom page
        - "Quick Route" Interface...
        - Allow to use custom icons for individual forums


        • #6
          The problem is the process never seems to stop. It cycles through all the users in the origin db and then appears to start cycling through them again. It like it gets stuck in an infinite loop. In the vBulletin db I have the correct username (I.e. Croeder) at first and then I end up with another username of Imported_Croeder after it starts cycling again. I know it is doing something strange as I only have 400,000 users and it starts counting higher than that number.


          • #7
            I seem to be encountering something similar with Impex. I'm importing from phpBB v3 to vBulletin 4.2. I have 3.5 million user records and it seems that Impex is actually getting through them at a decent rate but then bombs out. The numbers in the web interface stop incrementing but the process never completes. Apache continues to run a high load with no traffic indefinitely until I can go into impex and delete the session.

            About to start my 3rd attempt on the users import after tweaking down the max memory in php.ini (it was set to more than the machine has available). Hopefully this time it can go through.

            Support - how can I get logging of what Impex is doing if this next run doesn't work?


            widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.