Announcement

Collapse
No announcement yet.

Problem Importing a BIG PN pbpBB database

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Jerry
    replied
    Taken offline to reslove.

    Leave a comment:


  • RealityDesign
    replied
    Is there any way I could just pay you to do the import with a new dump file tomorrow? If you can just import it and send me the .sql file I would need to make it work on my server that would probably be the best option. If it really only takes you 34 minutes, that would be the best way to go.

    I am sorry I got somewhat hostile, this process is just very aggrivating. I have 8,000 people giving me **** around the clock waiting for their new forums and I want the downtime for the move to be as short as possible.

    Leave a comment:


  • RealityDesign
    replied
    Could you please just tell me what is involved in doing this:

    "Well if it comes to it we can stop impex doing the parent id update, do the post import then do the parent id update via the command line then carry on with the import. Its usualy a lot faster that way."

    This is all I need to know. I was waiting and waiting last night for that answer but instead you decided to take it upon yourself to import my database, convert it, post a picture, and then say "your server is crippled." I'm sorry if I thought you saying my server is crippled implied that it sucks...

    I still don't understand why your import is going 20 times faster than mine. You haven't really shed any light on this . I know for a fact it's not the computer so what kind of changes can I make to the configuration of mysql or php??

    Leave a comment:


  • Jerry
    replied
    Originally posted by RealityDesign
    Either way, coming on here and offering help and then finally after 24 hours posting a picture and saying "your server sucks" is not exactly a solution. I just hope it doesn't time out at post #230,071 again.
    I have done nothing of the sort, I have attempted to reply to a lot of messages and answer your questions and explain your observations and assumptions, also it was 10 hours not 24 and that was due to my need to sleep as a side effect of being human.

    I did an import, which isn't a service or a typical offering to show that an import of that size doesn't take 3+ hours, I have explained different ways that you can achieve your import.

    I at no time said "your server sucks".

    SQL through command prompt, importing the entire database, including the 8000 users, 230,000 posts, etc. takes 8 minutes but the import script is going to take more like 2 to 2.5 hours.
    A CLI SQL load is a completely different operation, I have already tried to explain this to you.

    This script is clearly what is causing the inefficiency
    We are using exactly the same import scripts, I know because I wrote them and I released them.

    Leave a comment:


  • RealityDesign
    replied
    On yet another side note (yes, I'm just full of side notes) that computer imported the 400mb sql file dump in 8 minutes using the command prompt.

    So lets see:

    SQL through command prompt, importing the entire database, including the 8000 users, 230,000 posts, etc. takes 8 minutes but the import script is going to take more like 2 to 2.5 hours.

    It seems to me that this would suggest there is nothing wrong with my computer and that you are either doing it a different way or have a different script than I do. This script is clearly what is causing the inefficiency.

    Either way, coming on here and offering help and then finally after 24 hours posting a picture and saying "your server sucks" is not exactly a solution. I just hope it doesn't time out at post #230,071 again.
    Last edited by RealityDesign; Thu 31 Mar '05, 7:59pm.

    Leave a comment:


  • RealityDesign
    replied
    22,641 threads in 27 seconds.

    This would suggest that you are managing to import 800 threads per second. How is that possible? What setting are you using? I used the default 500 per page and 1 page certainly doesn't go by 1 in second using an AMD 64 3200+ with 2gb of ram.

    You said it took you 27 seconds it took me more like 10 minutes, so what's the catch? I haven't tried it on the FX-55, I'm using the 3200+ because its a spare computer, but there is no way your computer is 20 times faster...so what am I doing wrong?

    Specs:
    AMD 64 3200+, 2gb pc-3200 ram, 160gb SATA raid-0.
    Apache 2, MySQL 4.1, PHP 4

    Modules so far:
    001 - Check & Update - 5 seconds
    002 - skipped
    003 - Import Users - 15 minutes
    004 - Import Forums - 1 second
    005 - Import Thread - 12 minutes
    006 - Import Smilie - 1 second
    007 - Import post - still going, on posts 29,000 to 29,500 right now. Timed with a stop watch it takes about 14 seconds per 500. 230k / 500 = 460 x 14 seconds = 6,440 / 60 seconds = 107 minutes.

    ~110 mins versus 17.

    On a side note, the server you are now calling "crippled" did it in about 140 minutes (only 30 minutes slower than this computer). I know for a fact this computer isn't crippled...it gets 300 fps in Counter-Strike:Source, got amazing benchmarks sandra and 3dmark, so...is there some magical way to configure the apache / mysql server? I don't see how setting it up one way versus another would make it 20 times faster. You were saying yesterday that it would be caused by configuration and load...well there is zero load since it's a local computer, so what is wrong with my configuration? Do you have any ideas? This doesn't make much sense.
    Last edited by RealityDesign; Thu 31 Mar '05, 7:48pm.

    Leave a comment:


  • Jerry
    replied
    Because there is a lot more SQL going on for user and if they have remotely hosted avatars they are imported so ImpEx has to wait for the PHP fopen() result and the download of the image if the link is valid.

    Turning off the avatar import gets it down to 28 seconds.

    Leave a comment:


  • RealityDesign
    replied
    Out of curiousity, why did it take 14 minutes to import 8000 users, but only 17 minutes to do 230,000 posts???

    That just doesn't seem right.

    Leave a comment:


  • Jerry
    replied
    34 mins so I was a little out, though that includes all the threads and forums counter updates.

    I would defiantly get the dev site sorted on localhost and move the database about as your server must be crippled some how.
    Attached Files

    Leave a comment:


  • RealityDesign
    replied
    Sniffle, I think you left for the night I still need to know how to solve the problem using the command prompt.

    Leave a comment:


  • RealityDesign
    replied
    It's been a whole 30 minutes and no reply lol . Just letting you know I'm still here, I'll be on all night. Hope everything is going ok with that sql dump file.

    Leave a comment:


  • Jerry
    replied
    Originally posted by RealityDesign
    Like I said, I didn't dump it That is the file the person selling the website gave to me from his hosting company.

    It's safe to say that they don't know what they are doing.
    Well just so you know for when you do.

    Leave a comment:


  • RealityDesign
    replied
    Like I said, I didn't dump it That is the file the person selling the website gave to me from his hosting company.

    It's safe to say that they don't know what they are doing.

    I mean, why else would they use PHP Nuke? LOL

    Leave a comment:


  • Jerry
    replied
    Still loading it, as its a straight dump.

    Next time you dump a database to move it I'd suggest using.

    -Q --otp

    http://dev.mysql.com/doc/mysql/en/mysqldump.html

    Leave a comment:


  • RealityDesign
    replied
    Originally posted by Jerry
    This is because the update counters haven't been run.
    Ya, I read the howto and ran the counters, I just did them out of order I guess. I did forums then threads. I just did forums again and it looks great. Now if we could just get all my threads working properly we would be golden .

    Leave a comment:

widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
Working...
X