Failed: Merging Million Post Forums

[Typhon]

I've been trying to do a test merger of two vB 3.5.4 forums for about a week now ... and it just keeps on failing (On posts).

I'm running ImpEx 1.82 on standalone, and have my resources set at:
max_execution_time = 480
max_input_time = 240
memory_limit = 32M

There's 4GB of RAM on the box (Intel \ 2.4 GHz 1066FSB - Conroe \ Xeon 3060).

This is actually part of a 4 step process (merging 3x million post forums into another million post forum). I've pruned all the data I can ... but it just won't go. Is there a fool proof method of doing this with minimal downtime?

BTW, I'm importing posts at 500/page.

Thanks!

[Jerry]

and it just keeps on failing (On posts).

In what manner ? What is the error ?

[Typhon]

No error ... it just stops. The page heading is still there, but it's not loading anything. Refreshing the page doesn't start it back up. Not to mention that the process is horrible slow (probably in part due to the size of the post table).

Thank you in advance for any assistance.

[Typhon]

Tried another run last night on my test databases and it just keeps on hanging at the post import. I'd prefer not to run another test (since it takes about 2 hours to get to the post import section) unless I can make a change to the system that would increase the performance.

Would shutting down all other websites server down and giving 100% of the processing power to the import job complete the task (and make it run faster?). I'm keeping the production sites up and running during the import, but I get load spikes during the import process.

Any assistance would be greatly appreciated.

[Marco van Herwaarden]

In your ImpExconfig.php, find the following:

define('dupe_checking', true); 

and change to:

define('dupe_checking', false); 

This might have a strong positive effect on the performence and resources used.
You might also try to raise the pagespeed (ie. make slower):

$impexconfig['system']['pagespeed']  = 1; 

Try setting it to 2 or more second to give the server a bit time in between pages.

[Typhon]

Nothing ... my mysql (4.1.2) goes crazy and the cpu usage ends up hitting 100% for mysqld; eventually resetting the daemon (unrelated to the above changes ... just been watching my top more closely). I've tuned MySQL to the best of my ability using http://hackmysql.com/mysqlreport , and can post the output of that if needed.

Here's my my.cnf file ... maybe there's something in there that it doesn't like. My websites runs fine normally ... maybe I need a separate my.conf for running ImpEx?

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
back_log = 75
skip-innodb
max_connections = 500
key_buffer = 384M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1800
thread_cache_size = 384
wait_timeout = 7200
connect_timeout = 10
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 64M
max_connect_errors = 1000
read_rnd_buffer_size = 524288
bulk_insert_buffer_size = 8M
query_cache_limit = 4M
query_cache_size =128M
query_cache_type = 1
query_prealloc_size = 65536
query_alloc_block_size = 131072
default-storage-engine = MyISAM

[mysql.server]
user=mysql
basedir=/var/lib

[mysqld_safe]
nice = -5
open_files_limit = 8192
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 64M
sort_buffer = 64M
read_buffer = 16M
write_buffer = 16M 

[Marco van Herwaarden]

Importing a large board can use a lot of resources. Other then my advice above, i don't see any options to reduce that from an ImpEx P.O.V.

You could post in the server configuration forum, providing the information requested in [Server Optimisation]: Required info for optimisation requests [January 15, 2007] if you need MySQL tuning.

[Jerry]

There some caching that is due to be added to the importer that will help with this, I'd expect that to be in a version in the next week or so.

[Typhon]

There some caching that is due to be added to the importer that will help with this, I'd expect that to be in a version in the next week or so.

Excellent! I'll put the migration on hold until then. Thanks so much for providing outstanding support for such a great product!

[Jerry]

Excellent! I'll put the migration on hold until then. Thanks so much for providing outstanding support for such a great product!

It will be in 1.83 this week

[Typhon]

Great! Are we looking at before the weekend? I'd like to get in a couple test runs before I import my production data. Can't wait to get everything imported and upgraded to the latest and great version of vB! Project, Blog ... what's not to love?!?

[Jerry]

Great! Are we looking at before the weekend? I'd like to get in a couple test runs before I import my production data. Can't wait to get everything imported and upgraded to the latest and great version of vB! Project, Blog ... what's not to love?!?

Yes, I am planning for a Wednesday (tomorrow) release and the 3.5.x caching (which should help no end) is in there as of from yesterday.

[Typhon]

Jerry,
any word on this release? Anxiously waiting to finally get my boards merged. Thanks!

[Zachery]

Are you really using MySQL 4.1.2 alpha? You may want to consider upgrading from a non aphla release.

[Typhon]

I'll have to check. I'm hosted at ThePlanet, and that's the default set-up they provided me. They also do all the server management for me, so I'll see if they can upgrade it to a more current version.

I don't see where it says "alpha" anywhere though: http://www.realmworx.com/phpinfo.php

I know that ThePlanet doesn't support upgrade unless it's fed through the RHEL4 up2date package ... and I have no idea if that's even kept current anymore now that RHEL5 has switched over to yum.

EDIT: I just noticed that RHEL4 has an RPM for MySQL 5.0.45; so I'm going to make a request to get upgraded to that.