Announcement

Collapse
No announcement yet.

Failed: Merging Million Post Forums

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Failed: Merging Million Post Forums

    I've been trying to do a test merger of two vB 3.5.4 forums for about a week now ... and it just keeps on failing (On posts).

    I'm running ImpEx 1.82 on standalone, and have my resources set at:
    max_execution_time = 480
    max_input_time = 240
    memory_limit = 32M

    There's 4GB of RAM on the box (Intel \ 2.4 GHz 1066FSB - Conroe \ Xeon 3060).

    This is actually part of a 4 step process (merging 3x million post forums into another million post forum). I've pruned all the data I can ... but it just won't go. Is there a fool proof method of doing this with minimal downtime?

    BTW, I'm importing posts at 500/page.

    Thanks!
    Last edited by Typhon; Wed 18 Jul '07, 8:22pm.
    ::Typhon::
    www.realmworx.com

  • #2
    and it just keeps on failing (On posts).
    In what manner ? What is the error ?
    I wrote ImpEx.

    Blog | Me

    Comment


    • #3
      No error ... it just stops. The page heading is still there, but it's not loading anything. Refreshing the page doesn't start it back up. Not to mention that the process is horrible slow (probably in part due to the size of the post table).

      Thank you in advance for any assistance.
      ::Typhon::
      www.realmworx.com

      Comment


      • #4
        Tried another run last night on my test databases and it just keeps on hanging at the post import. I'd prefer not to run another test (since it takes about 2 hours to get to the post import section) unless I can make a change to the system that would increase the performance.

        Would shutting down all other websites server down and giving 100% of the processing power to the import job complete the task (and make it run faster?). I'm keeping the production sites up and running during the import, but I get load spikes during the import process.

        Any assistance would be greatly appreciated.
        ::Typhon::
        www.realmworx.com

        Comment


        • #5
          In your ImpExconfig.php, find the following:
          PHP Code:
          define('dupe_checking'true); 
          and change to:
          PHP Code:
          define('dupe_checking'false); 
          This might have a strong positive effect on the performence and resources used.
          You might also try to raise the pagespeed (ie. make slower):
          PHP Code:
          $impexconfig['system']['pagespeed']  = 1
          Try setting it to 2 or more second to give the server a bit time in between pages.
          Want to take your board beyond the standard vBulletin features?
          Visit the official Member to Member support site for vBulletin Modifications: www.vbulletin.org

          Comment


          • #6
            Nothing ... my mysql (4.1.2) goes crazy and the cpu usage ends up hitting 100% for mysqld; eventually resetting the daemon (unrelated to the above changes ... just been watching my top more closely). I've tuned MySQL to the best of my ability using http://hackmysql.com/mysqlreport , and can post the output of that if needed.

            Here's my my.cnf file ... maybe there's something in there that it doesn't like. My websites runs fine normally ... maybe I need a separate my.conf for running ImpEx?

            PHP Code:
            [mysqld]
            datadir=/var/lib/mysql
            socket
            =/var/lib/mysql/mysql.sock
            back_log 
            75
            skip
            -innodb
            max_connections 
            500
            key_buffer 
            384M
            myisam_sort_buffer_size 
            64M
            join_buffer_size 
            1M
            read_buffer_size 
            1M
            sort_buffer_size 
            2M
            table_cache 
            1800
            thread_cache_size 
            384
            wait_timeout 
            7200
            connect_timeout 
            10
            tmp_table_size 
            64M
            max_heap_table_size 
            64M
            max_allowed_packet 
            64M
            max_connect_errors 
            1000
            read_rnd_buffer_size 
            524288
            bulk_insert_buffer_size 
            8M
            query_cache_limit 
            4M
            query_cache_size 
            =128M
            query_cache_type 
            1
            query_prealloc_size 
            65536
            query_alloc_block_size 
            131072
            default-storage-engine MyISAM

            [mysql.server]
            user=mysql
            basedir
            =/var/lib

            [mysqld_safe]
            nice = -5
            open_files_limit 
            8192
            err
            -log=/var/log/mysqld.log
            pid
            -file=/var/run/mysqld/mysqld.pid

            [mysqldump]
            quick
            max_allowed_packet 
            16M

            [myisamchk]
            key_buffer 64M
            sort_buffer 
            64M
            read_buffer 
            16M
            write_buffer 
            16M 
            Last edited by Typhon; Fri 20 Jul '07, 11:30am.
            ::Typhon::
            www.realmworx.com

            Comment


            • #7
              Importing a large board can use a lot of resources. Other then my advice above, i don't see any options to reduce that from an ImpEx P.O.V.

              You could post in the server configuration forum, providing the information requested in [Server Optimisation]: Required info for optimisation requests [January 15, 2007] if you need MySQL tuning.
              Want to take your board beyond the standard vBulletin features?
              Visit the official Member to Member support site for vBulletin Modifications: www.vbulletin.org

              Comment


              • #8
                There some caching that is due to be added to the importer that will help with this, I'd expect that to be in a version in the next week or so.
                I wrote ImpEx.

                Blog | Me

                Comment


                • #9
                  Originally posted by Jerry View Post
                  There some caching that is due to be added to the importer that will help with this, I'd expect that to be in a version in the next week or so.
                  Excellent! I'll put the migration on hold until then. Thanks so much for providing outstanding support for such a great product!
                  ::Typhon::
                  www.realmworx.com

                  Comment


                  • #10
                    Originally posted by Typhon View Post
                    Excellent! I'll put the migration on hold until then. Thanks so much for providing outstanding support for such a great product!
                    It will be in 1.83 this week
                    I wrote ImpEx.

                    Blog | Me

                    Comment


                    • #11
                      Great! Are we looking at before the weekend? I'd like to get in a couple test runs before I import my production data. Can't wait to get everything imported and upgraded to the latest and great version of vB! Project, Blog ... what's not to love?!?
                      ::Typhon::
                      www.realmworx.com

                      Comment


                      • #12
                        Originally posted by Typhon View Post
                        Great! Are we looking at before the weekend? I'd like to get in a couple test runs before I import my production data. Can't wait to get everything imported and upgraded to the latest and great version of vB! Project, Blog ... what's not to love?!?
                        Yes, I am planning for a Wednesday (tomorrow) release and the 3.5.x caching (which should help no end) is in there as of from yesterday.
                        I wrote ImpEx.

                        Blog | Me

                        Comment


                        • #13
                          Jerry,
                          any word on this release? Anxiously waiting to finally get my boards merged. Thanks!
                          ::Typhon::
                          www.realmworx.com

                          Comment


                          • #14
                            Are you really using MySQL 4.1.2 alpha? You may want to consider upgrading from a non aphla release.

                            Comment


                            • #15
                              I'll have to check. I'm hosted at ThePlanet, and that's the default set-up they provided me. They also do all the server management for me, so I'll see if they can upgrade it to a more current version.

                              I don't see where it says "alpha" anywhere though: http://www.realmworx.com/phpinfo.php

                              I know that ThePlanet doesn't support upgrade unless it's fed through the RHEL4 up2date package ... and I have no idea if that's even kept current anymore now that RHEL5 has switched over to yum.

                              EDIT: I just noticed that RHEL4 has an RPM for MySQL 5.0.45; so I'm going to make a request to get upgraded to that.
                              Last edited by Typhon; Thu 26 Jul '07, 7:02am.
                              ::Typhon::
                              www.realmworx.com

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...
                              X