Announcement

Collapse
No announcement yet.

Import freezing at same point

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Jerry
    replied
    Good old indexing

    Leave a comment:


  • baker011
    replied
    The index seems to have worked. We have had two successful tests of the thread module. We are now running posts, so I may be back again

    Thanks Jerry for all your help so far, you have been incredibly helpful!

    Leave a comment:


  • baker011
    replied
    Thats on the Threads module. Could you also further clarify that last post? I noticed vb_35 and assume you mean vb_36?

    Leave a comment:


  • Jerry
    replied
    And that is during the post module ?

    That could be updating the poll id's for threads, just to check that comment out line 173 of impex/systems/vb_35/006.php.

    Also add an index to thread.importthreadid if it isn't already there, after the 001 module, but before the thread module.

    Leave a comment:


  • baker011
    replied
    I just got this email from our Server guy after the latest set of test imports:


    After restarting it, it looks to be trying to run the same section of thread inserts over and it gets to this one...

    131 Query INSERT INTO thread
    (
    forumid, title, importforumid,
    importthreadid, firstpostid, lastpost,
    pollid, open, replycount,
    postusername, postuserid, lastposter,
    dateline, views, iconid,
    notes, visible, sticky,
    votenum, votetotal, attach, similar,
    hiddencount, deletedcount
    )
    VALUES
    (
    '154',
    'RCTCC Survivor',
    '421',
    '557336',
    '8302002',
    '1168495448',
    '0',
    '1',
    '11',
    'aoshi',
    '71233',
    'Bitter Jeweler',
    '1168493355',
    '66',
    '0',
    '',
    '1',
    '0',
    '0',
    '0',
    '0',
    '',
    '0',
    ''
    )
    ... Then starts running a bunch of inserts...
    070111 1:14:45 129 Query SELECT threadid FROM thread where importthreadid = 77172
    131 Query SELECT threadid FROM thread where importthreadid = 77265
    070111 1:14:46 129 Query SELECT threadid FROM thread where importthreadid = 77231
    131 Query SELECT threadid FROM thread where importthreadid = 77971
    070111 1:14:47 129 Query SELECT threadid FROM thread where importthreadid = 77265
    131 Query SELECT threadid FROM thread where importthreadid = 79124
    070111 1:14:48 129 Query SELECT threadid FROM thread where importthreadid = 77971
    070111 1:14:49 131 Query SELECT threadid FROM thread where importthreadid = 80142
    070111 1:14:50 129 Query SELECT threadid FROM thread where importthreadid = 79124
    131 Query SELECT threadid FROM thread where importthreadid = 80465
    070111 1:14:51 129 Query SELECT threadid FROM thread where importthreadid = 80142
    131 Query SELECT threadid FROM thread where importthreadid = 82199
    070111 1:14:52 129 Query SELECT threadid FROM thread where importthreadid = 80465
    131 Query SELECT threadid FROM thread where importthreadid = 82445
    070111 1:14:53 129 Query SELECT threadid FROM thread where importthreadid = 82199
    131 Query SELECT threadid FROM thread where importthreadid = 83025
    070111 1:14:54 131 Query SELECT threadid FROM thread where importthreadid = 83066
    129 Query SELECT threadid FROM thread where importthreadid = 82445
    070111 1:14:55 131 Query SELECT threadid FROM thread where importthreadid = 83798
    129 Query SELECT threadid FROM thread where importthreadid = 83025
    070111 1:14:56 131 Query SELECT threadid FROM thread where importthreadid = 83879
    129 Query SELECT threadid FROM thread where importthreadid = 83066
    070111 1:14:57 131 Query SELECT threadid FROM thread where importthreadid = 84956
    129 Query SELECT threadid FROM thread where importthreadid = 83798
    070111 1:14:58 131 Query SELECT threadid FROM thread where importthreadid = 85512
    129 Query SELECT threadid FROM thread where importthreadid = 83879
    070111 1:14:59 131 Query SELECT threadid FROM thread where importthreadid = 86339
    070111 1:15:00 129 Query SELECT threadid FROM thread where importthreadid = 84956
    131 Query SELECT threadid FROM thread where importthreadid = 86532
    070111 1:15:01 129 Query SELECT threadid FROM thread where importthreadid = 85512
    070111 1:15:02 131 Query SELECT threadid FROM thread where importthreadid = 87237
    129 Query SELECT threadid FROM thread where importthreadid = 86339
    070111 1:15:03 131 Query SELECT threadid FROM thread where importthreadid = 87296
    129 Query SELECT threadid FROM thread where importthreadid = 86532
    070111 1:15:04 131 Query SELECT threadid FROM thread where importthreadid = 88375
    129 Query SELECT threadid FROM thread where importthreadid = 87237
    070111 1:15:05 131 Query SELECT threadid FROM thread where importthreadid = 88421
    129 Query SELECT threadid FROM thread where importthreadid = 87296
    070111 1:15:06 131 Query SELECT threadid FROM thread where importthreadid = 89624
    129 Query SELECT threadid FROM thread where importthreadid = 88375
    070111 1:15:07 131 Query SELECT threadid FROM thread where importthreadid = 89721
    129 Query SELECT threadid FROM thread where importthreadid = 88421
    070111 1:15:08 131 Query SELECT threadid FROM thread where importthreadid = 89838
    129 Query SELECT threadid FROM thread where importthreadid = 89624
    070111 1:15:09 131 Query SELECT threadid FROM thread where importthreadid = 89894
    129 Query SELECT threadid FROM thread where importthreadid = 89721
    070111 1:15:10 131 Query SELECT threadid FROM thread where importthreadid = 89945
    070111 1:15:11 129 Query SELECT threadid FROM thread where importthreadid = 89838
    131 Query SELECT threadid FROM thread where importthreadid = 91687
    070111 1:15:12 129 Query SELECT threadid FROM thread where importthreadid = 89894
    131 Query SELECT threadid FROM thread where importthreadid = 91822
    070111 1:15:13 129 Query SELECT threadid FROM thread where importthreadid = 89945
    070111 1:15:14 131 Query SELECT threadid FROM thread where importthreadid = 93187
    129 Query SELECT threadid FROM thread where importthreadid = 91687
    131 Query SELECT threadid FROM thread where importthreadid = 94411
    070111 1:15:15 129 Query SELECT threadid FROM thread where importthreadid = 91822
    131 Query SELECT threadid FROM thread where importthreadid = 95533
    070111 1:15:16 129 Query SELECT threadid FROM thread where importthreadid = 93187
    131 Query SELECT threadid FROM thread where importthreadid = 95551
    070111 1:15:17 129 Query SELECT threadid FROM thread where importthreadid = 94411
    070111 1:15:18 131 Query SELECT threadid FROM thread where importthreadid = 96532
    129 Query SELECT threadid FROM thread where importthreadid = 95533
    070111 1:15:19 131 Query SELECT threadid FROM thread where importthreadid = 96946
    129 Query SELECT threadid FROM thread where importthreadid = 95551
    070111 1:15:20 131 Query SELECT threadid FROM thread where importthreadid = 96958
    129 Query SELECT threadid FROM thread where importthreadid = 96532
    070111 1:15:21 131 Query SELECT threadid FROM thread where importthreadid = 100549
    129 Query SELECT threadid FROM thread where importthreadid = 96946
    070111 1:15:22 131 Query SELECT threadid FROM thread where importthreadid = 102470
    129 Query SELECT threadid FROM thread where importthreadid = 96958
    070111 1:15:23 131 Query SELECT threadid FROM thread where importthreadid = 103453
    070111 1:15:24 129 Query SELECT threadid FROM thread where importthreadid = 100549
    131 Query SELECT threadid FROM thread where importthreadid = 103595
    070111 1:15:25 129 Query SELECT threadid FROM thread where importthreadid = 102470
    ...that take a long time (each one takes a second or so). It times out while running these selects. All in all, it seems like the SQL isn't very optimized. If we could just have it spit out the SQL we could run it from the command line and not have to worry about timeouts. I've rerun it several times and it does the exact same thing every time. I don't know enough about the importer or vBulletin to debug this.

    Leave a comment:


  • baker011
    replied
    Ok we ran the import with that update to the script, it stopped at this thread:

    http://www.ataricommunity.com/forums...d.php?t=555553

    I dont see a lot wrong with it, I have asked our tech guy to run it once again to see if its consistantly that thread.

    Leave a comment:


  • baker011
    replied
    Pretty normal amount of redirects to be honest, I will have that line of code added and run another test import.

    Thanks for the help so far, it really is appreciated!

    Leave a comment:


  • Jerry
    replied
    Originally posted by baker011 View Post
    Ok, we have tried another import, upped the thread limit to 5000 a time and it stalled on 225,000-230,000 (almost exactly the same point it froze at when doing 2,000 a time. Then the PHP logs were checked and there were no reported errors. So back at square one
    True, though we have eliminated one avenue of investigation

    Do you have any forums with a large amount of redirects ?

    On line 107 of impex/systems/vb_36/006.php add this after the brace :

    PHP Code:
    echo "<br>Importing - {$thread_id}"
    That will show you which one its working on, so which one it dies on.

    Leave a comment:


  • baker011
    replied
    Ok, we have tried another import, upped the thread limit to 5000 a time and it stalled on 225,000-230,000 (almost exactly the same point it froze at when doing 2,000 a time. Then the PHP logs were checked and there were no reported errors. So back at square one

    Leave a comment:


  • baker011
    replied
    Sorry I was getting mixed up with the db logs that you choose to enable for the Impex. I will get our server admin to look through the PHP logs.

    Leave a comment:


  • Jerry
    replied
    Depends on the OS version and lay out, the sysadmin that built it should know exactly where the logs are, I can't think of any good reason to turn logs off, they are created for very good reasons.

    Check the php.ini to see what and where, see if this is set :

    Code:
    error_log =

    Leave a comment:


  • baker011
    replied
    If I remember rightly our Server guy upped it to 64mb. When it first acted up it was set to 8mb but it has failed at the same point from the very first try, so I am not so sure its down to the memory limit. I have got the server guy trying another import today with this database and I have requested he search for any error reports, but each time he has checked it has come back with none. Could you confirm where these error logs would be just to make sure we are checking the correct place.

    Thanks

    Leave a comment:


  • Jerry
    replied
    No error logs for PHP ? Eeek, thatís going to make it tricky !!

    On something that size, its probably memory or the database. PHP is going to use a huge amount of memory holding all the threadids.

    Have you turned off the PHP memory limit all together ?

    Leave a comment:


  • baker011
    replied
    No, absolutely no error logs. It just refuses to go past that stage then after Impex is completely locked up and refuses to load. I did all the suggested script changes too for large forums as this is a 4,000,000+ post forum.

    Leave a comment:


  • Jerry
    replied
    Anything in the PHP error logs ?

    Leave a comment:

widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
Working...
X