Announcement

Collapse
No announcement yet.

Importing 400,000 posts - taking forever

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Importing 400,000 posts - taking forever

    What is the recommended batch size for importing 400,000 posts? I've tried doing 25,000 at a time and it is just too sluggish and I'd rather not have to sit here and click "next" a million times.

    A good feature would be to automatically refresh to the next page, and a way to turn off the data dump to your browser window.
    Last edited by Verbose; Mon 22 Jan '07, 1:14pm.

  • #2
    also... I purposely closed my browser to try to killl the posts import module because it said there was some mysql error where it couldn't reach the server for a minute. So i closed my browser window to redo the import - well, when i logged back in the import was still actively running.

    Is there a kill switch somewhere or do you just have to let it run it's course for 30+ minutes?

    Comment


    • #3
      Originally posted by Verbose View Post
      What is the recommended batch size for importing 400,000 posts? I've tried doing 25,000 at a time and it is just too sluggish and I'd rather not have to sit here and click "next" a million times.

      A good feature would be to automatically refresh to the next page, and a way to turn off the data dump to your browser window.
      In ImpEx config :

      PHP Code:
      define('shortoutput'true);  // Depending on your source system, only some Tier 1 supported at the moment.
      define('step_through'false); 
      As for speed, that depends mainly on the PHP memory limit and the power of the database, 25,000 seems fine, I would expect that to import in about 15 mins depending on the over all power of the server.
      I wrote ImpEx.

      Blog | Me

      Comment


      • #4
        Originally posted by Jerry View Post
        In ImpEx config :

        PHP Code:
        define('shortoutput'true);  // Depending on your source system, only some Tier 1 supported at the moment.
        define('step_through'false); 
        As for speed, that depends mainly on the PHP memory limit and the power of the database, 25,000 seems fine, I would expect that to import in about 15 mins depending on the over all power of the server.
        Hi Jerry, thanks for your response

        I set the batch import size to 10,000 posts at a time - this took roughly 3 hours to import 400,000 posts. We've spoken to tech support and have our serving settings pretty high with fairly powerful hardware, so I'm not sure what else to try.

        I am doing the live import today, and will try it with the output turned off. FYI, our source board is UBB.threads 6.5.2 - I think that is Tier 1, so hopefully this feature is supported?

        Comment


        • #5
          I've just updated the 007.php file in the ubb_threads system so it has the short output, if you download and over write that file it will have it now.
          I wrote ImpEx.

          Blog | Me

          Comment

          widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
          Working...
          X