Announcement

Collapse
No announcement yet.

vBulletin 4.x upgrade process for large vBulletin forums

Collapse
X
Collapse
  •  

  • vBulletin 4.x upgrade process for large vBulletin forums

    Basically, you upgrade on a copy of your live database so as to leave live database untouched in case you mess up the upgrade.

    eva2000's upgrade process

    My vB 3.8.x to vB 4.1.x upgrade process which I personally use myself as well as for private paying clients is as follows and assumes:
    1. you have ssh2 telnet access
    2. your vB 3.x forums are only using MySQL default MyISAM/Memory tables with NO InnoDB tables
    3. you have enough free disk space to hold all the backups approximately 3x to 6x times the size of your forum database/files/attachments in free disk space.
    4. public root directory = /home/username/public_html
    5. live database name = livedbname
    6. new databasename name = newdbname



    Step 1. Tune php and mysql settings specifically for backup, restore and upgrade process. Tune mysql to ensure enough memory is allocated to buffers to speed things up, increase timeout values for mysql to ensure no mysql timeouts and for php. Ensure mysql server has InnoDB support disabled otherwise you’ll have to deal with InnoDB tuning parameters before you start as vB4 automatically converts 6 tables to InnoDB if mysql server has InnoDB support enabled. Read the reasons of how and why you’d want InnoDB support disabled in my blog post at http://vbtechsupport.com/675/ and if you do decide to use InnoDB storage engine for some database tables, I’d also look at upgrading your MySQL version to a better performing MySQL version such as MySQL 5.5.12, Percona 5.5.10-20.2 or my recommendation MariaDB 5.2.x series.

    Step 2. Backup live forums

    Code:
    cd /home/username/
    mysqldump --opt -Q -u mysqlusername -p livedbname > /home/username/vb3backup.sql
    cp -a public_html public_html_oldvb38
    if you have attachments outside of public_html i.e. /home/username/attachments

    Code:
    cd /home/username
    cp -a attachments attachments_oldvb38
    Step 3. Create new empty database called newdbname and import backup vb3backup.sql into newdbname

    mysql -u mysqlusernae -p newdbname < /home/username/vb3backup.sql

    You could compress these vBulletin forum sql, attachment and public_html backups, but for large vBulletin forums it justs adds additional time to the upgrade task and if you have to revert to one backup it adds more time in uncompressing the vBulletin backups. So I usually don’t compress these backups until after the vBulletin 3.x to 4.x upgrade process is completed fully.

    Step 4. Get new vB version files uploaded to /home/username/public_html. Two ways of doing this - I prefer 2nd method if I am on the clock and don't have much time.

    Method #1
    • Empty out /home/username/public_html and upload new vB version files from zip folder /upload


    Method #2
    • On local computer, extract vB new version zip file and in /upload rename config.php.new to config.php and populate all the settings pointing to step #3's newdbname database which now has a copy of your live database imported into newdbname.
    • Also add to vB new version config.php following 2 lines to disable plugins and mail for upgrade process
      PHP Code:
      define("DISABLE_HOOKS"true);
      define('DISABLE_MAIL'1); 
    • If you use vB datastore caching in config.php either APC, Xcache, File or Memcached datastore caching, disable it temporarily for the upgrade by commenting it out with double forward slashes.
    • Then zip up /upload folder i.e. upload.zip and upload that to /home/username/ via FTP
    • Then ssh2 telnet into server as same username as your vB forum site account (this is so extracted files retain the correct file owner/group permissions).
    • Once in ssh2 go to /home/username directory and extract upload.zip you'll have a folder at /home/username/upload now which contains all the new vB version's php files etc

      Code:
      unzip upload
    • Still in ssh2 telnet, copy all files in /home/username/upload and overwrite all files in /home/username/public_html

      Code:
      cd /home/username/upload
      unalias cp
      cp -Rfp * /home/username/public_html


    Now you should have overwritten all files in /home/username/public_html with new vB version files

    At this stage, if you are using vBSEO addon and have MaxCDN or any other CDN hosted vB php/js/css files served externally, please flush and purge the CDN caches from your MaxCDN or CDN provider control panels to ensure you're upgrading using the uploaded new files and not cached CDN files.

    Step 5. vB 3.6.x to vB 3.8.x upgrades.
    • Run upgrade to 3.7.x and then repeat Step 2 and backup 3.7.x forums.
    • Run upgrade from 3.7.x to 3.8.x and then repeat Step 2 and backup 3.8.x forums.


    Step 6. vB 3.8.x to vB 4.0.9 upgrades.
    • Similar to large imports mentioned here http://www.vbulletin.com/docs/html/med_large_import, drop the post table's fulltext index on title BEFORE running vB 3.8.x to vB 4.1.x stage as vB 4.x no longer uses it. Run query on newdbname database (copy of your live forums). In ssh2 telnet type following to enter mysql command line:

      Code:
      mysql -u mysqlusername -p newdbname
    • Then run query to drop fulltext indices in post, thread, groupmessage and socialgroup tables. If you get error saying the key doesn't exist you can ignore it and move on.

      Code:
      ALTER TABLE `post` DROP INDEX `title`;
      ALTER TABLE `thread` DROP INDEX `title`;
      ALTER TABLE `groupmessage` DROP INDEX `gm_ft`;
      ALTER TABLE `socialgroup` DROP INDEX `name`
    • Now begin upgrade process from vB 3.8.x to 4.0.9


    Step 7. vB 4.0.9 to vB 4.1.x latest - utilise the new command line upgrade process oulined at http://www.vbulletin.com/docs/html/upgrade_cli
    • ssh2 telnet into server as same username as vB site account and run from /home/username/public_html/install directory this command
      Code:
      php -f upgrade.php


    Step 8. vB 4.x upgrade tasks and backup
    • Once upgrade complete, I like to make another 2 backups by repeat Step 2 and backup 4.1.x forums BEFORE vB 4 upgrade tasks are done and another backup AFTER vB4 upgrade tasks are done.
    • vB4 upgrade tasks are those notice prompts when you first log into vB4.x admincp, you'll be prompted to do the following: rebuild search index, adding keywords to threads and rebuilding thread information.


    Rebuilding Search Index

    Rebuilding search index can be a time consuming task, so vBulletin included a way of doing this rebuild via SSH2 telnet via PHP command line. In the downloaded vB zip file folder called do_not_upload there’s a file called searchindex.php. You upload searchindex.php to the root directory of your vBulletin forum installation. In this example it would be uploaded to /home/username/public_html/searchindex.php. You then log in via SSH2 telnet as the same user as the forum domain/site user and run the following two commands:

    Code:
    cd /home/username/public_html
    php searchindex.php
    You’ll be prompted to enter the path to your vBulletin installation in this case = /home/username/public_html.

    Rebuilding Threads

    Rebuilding threads can also be time consuming for large vB forums. vBulletin developer, Michael Anders unofficially posted a PHP command line script for such task as well called threadcli.php (download link for registered vB members only as it isn’t included in official vB4 zip downloads). Works pretty much the same way as searchindex.php but asks for starting thread id and number of threads to update per pass.

    Code:
    cd /home/username/public_html
    php threadcli.php
    If threadcli.php doesn’t work for you, try Paul K’s cli.php script instead which has more options.

    Code:
    cd /home/username/public_html
    php cli.php
    Code:
    php cli.php
    > Checking vB installation
    > Installation seems to be fine, continuing
    #################################################
    > Please select an option
    1) Quit
    2) Update User Titles and Ranks
    3) Rebuild Thread Information
    4) Rebuild Forum Information
    5) Add Add Missing Thread Keywords
    6) Update User Names
    7) Update Post Counts
    8) Fix Broken User Profiles
    9) Rebuild Post Cache
    10) Empty Signature Cache
    11) Delete Duplicate Threads
    Step 9. Revert your php and mysql settings back to everyday 24/7 values and remove from you config.php the lines

    PHP Code:
    define("DISABLE_HOOKS"true);
    define('DISABLE_MAIL'1); 
    Then re-enable vB datastore caching in config.php if you had disabled it for the upgrade.

    Step 10. Update YUI version – by either updating to vB 4.1.3 PL2 or higher or you can edit includes/class_core.php and update the version of YUI used from v2.7.0 to v2.9.0. Then after vB 4.x is upgraded, in AdminCP -> Options -> Server Settings and Optimization Options -> Use Remote YUI = select Google

    from

    Code:
    define('YUI_VERSION', '2.7.0'); // define the YUI version we bundle
    to

    Code:
    define('YUI_VERSION', '2.9.0'); // define the YUI version we bundle
    So basically, you do the upgrade on a copy of your live database in database named newdbname – leaving live database untouched in case you have to revert to a backup if upgrade messes up. You’ll end up with a few backups at different stages so it’s easier to revert to a particular stage rather than completely start over from scratch if upgrade messes up somewhere between.

    You’ll have vB 3.6.x backup, vB 3.7.x backup, vB 3.8.x backup and vB 4.1.x BEFORE upgrade tasks done and AFTER upgrade tasks done. If your database is very large then disk space could be an issue, so you could be selective as to how many backups you do, I’d at least have 3 backups, original live database backup, vB 3.8.x backup and vB 4.1.x backup.

    Largest vB forum I have tried this process on was aproximately 38GB size. If you have problems or vBulletin errors during upgrade, best to check out vBulletin 4 Installations and Upgrades support forums.

    Once you have upgraded to vBulletin 4.x, you can check out Shawn/Digitalpoint.com’s vBulletin 4 Optimization Guide.

    You can discuss the article here.

    Update: June 9th, 2012
    One year anniversary from original blog post. Added youtube video for steps 1 to 4 outlined above as an example of upgrading from vB 4.2.0 to 4.2.0 PL1. Creating this video was a bit awkward as for years now, I normally use multiple monitors for different apps and screens so having to condense this video to single monitor recording was a bit of a challenge.

    Video best viewed in HD 720P mode full screen.



    Updated Notes:
    If upgrading vBulletin 3.x with vBulletin 3.x API installed to vBulletin 4.2.x, you may get an error at vBulletin 4.1.10 Alpha 1 part of the upgrade script as follows:

    Code:
    Upgrading to 4.1.10 Alpha 1
    ----------------------------------
    Step 1 - Updating setting table
    Step 2 - Updating settinggroup table
    Step 3 - Database error in vBulletin 4.1.9:
     
    Invalid SQL:
    DELETE FROM navigation WHERE productid = 'vbapi';
     
    MySQL Error : Table 'databasename.navigation' doesn't exist
    Error Number : 1146
    Request Date : 
    Error Date : 
    Script : http://
    Referrer :
    IP Address :
    Username :
    Classname : vB_Database
    MySQL Version : 5.1.61-log
    This bug is fixed in upcoming vBulletin 4.2.1

    • drdoc
      #8
      drdoc commented
      Editing a comment
      I don't know why VB allows members to just click update button so that everything will get updated in seconds ( they can charge for upgrades) just like we see JAVA & ADOBE etc...I wish VB team will bring this kind of fast and quicker upgrade against old method what we are following now. it will be very difficult for non tech people like me. If VB introduce such feature then everyone will purchaseVB

      I mean VB needs simple methods for upgrades and updates but not so complicated for people like us.Still i love VB than any other software and wish they get all new features asap

      Thanks..
      Last edited by drdoc; Mon 8th Oct '12, 12:18am.

    • sdi_employee
      #9
      sdi_employee commented
      Editing a comment
      Don't know if anyone will see this here or not, but I've followed this process and have everything up to running the search index script completing in about 2.5 hours, which is pretty reasonable.

      That being said, the search index script is taking *days* to finish, and I can't find the thread script to run anywhere so I am just running that in the browser and it also takes a very long time. At this point, the only solution I can find without taking my site down for days is to let my users know that they are going to lose posts for the upgrade period, leave the site up, and upgrade a backup. Replacing the running site with the backup and losing the posts for those couple of days that the post-upgrade tasks were running is definitely not an ideal solution, but it's all I have right now.

      For context - I'm running a site with a few thousand active users at any one time, and 20 million posts currently in the database. (Also, it seems like the database size itself balloons quite a bit after the upgrade to 10Gb from around 3-4 before the upgrade, and that has me a bit worried from a backup prospective, but I don't think there's anything to be done about that.)

    • George L
      #10
      George L commented
      Editing a comment
      Originally posted by dr.reddy
      I don't know why VB allows members to just click update button so that everything will get updated in seconds ( they can charge for upgrades) just like we see JAVA & ADOBE etc...I wish VB team will bring this kind of fast and quicker upgrade against old method what we are following now. it will be very difficult for non tech people like me. If VB introduce such feature then everyone will purchaseVB

      I mean VB needs simple methods for upgrades and updates but not so complicated for people like us.Still i love VB than any other software and wish they get all new features asap

      Thanks..
      Unfortunately, vb upgrades can be complicated by modifications directly to code and add ons etc. I would love such a system

      Originally posted by sdi_employee
      Don't know if anyone will see this here or not, but I've followed this process and have everything up to running the search index script completing in about 2.5 hours, which is pretty reasonable.

      That being said, the search index script is taking *days* to finish, and I can't find the thread script to run anywhere so I am just running that in the browser and it also takes a very long time. At this point, the only solution I can find without taking my site down for days is to let my users know that they are going to lose posts for the upgrade period, leave the site up, and upgrade a backup. Replacing the running site with the backup and losing the posts for those couple of days that the post-upgrade tasks were running is definitely not an ideal solution, but it's all I have right now.

      For context - I'm running a site with a few thousand active users at any one time, and 20 million posts currently in the database. (Also, it seems like the database size itself balloons quite a bit after the upgrade to 10Gb from around 3-4 before the upgrade, and that has me a bit worried from a backup prospective, but I don't think there's anything to be done about that.)
      thread script is already linked to in above article ?

      As to size of db after vB 3x to 4.x upgrade, I wrote an article of my personal blog at http://vbtechsupport.com/675/ explaining why the size increases due to InnoDB tables etc and why and how you can move back from InnoDB to MyISAM. But for such large forums, you're better off with InnoDB tables as they are now for performance.
    Posting comments is disabled.

About the Author

Collapse

George L Find out more about George L

Article Tags

Collapse

advanced (5) android (2) api (29) array (17) attachment options (1) beginner (17) blog (4) calendar (2) center (1) cms (2) forum (3) forums (4) Intermediate (7) iphone (3) mapi (30) methods (10) mobile (34) post (1) security (2) ssl (1) style (2) threads (4) tutorial (1) vb5howto (5) vBulletin (5)

Latest Articles

Collapse

  • Managing CMS Section and Content
    by Don Kuramura
    Here's a quick Visual Guide on how Manage Sections in the new CMS.

    1. Editing a Section: If you have permissions to manage a Section, as you hover over the Section title, a pencil icon will display.

    After clicking the pencil icon, you will be taken to the Section Edit page. Here's what you will see:

    2. Section Name:
    Enter the Section Name
    3. SEO URL Alias: This is the SEO Friendly URL. By default, if this is blank, the system will automatically copy the section title.
    4. Section Layout: For each section you can define an individual section layout.
    ...
    Tue 11th Mar '14, 3:04pm
  • How to Create a New Article
    by Don Kuramura
    Here's a quick visual guide on how to create a new article with the CMS.

    1. Create New Article Button: Navigate to the section you want the article to be published in. Click on the “Create a New Article” button. This will open an article form.


    2. Article Title: In the "Add/Edit Article" screen, enter the title of your article in the “Title” textbox.
    ...
    Tue 11th Mar '14, 3:04pm
  • Promoting Articles from the Forums
    by Don Kuramura
    One of the innovative new features on vBulletin 4.0 Publishing Suite is the cross-publishing "Promote to Article" functionality. In the spirit of promoting content discovery, we wanted to create features that will allow content to flow between Forums, Blogs, and Articles. For example, there might be a really popular post in the forums, but unfortunately it is buried inside page 13 of a thread. We will now have an option (if you have permissions) to “promote” a forum (or blog) post as a new artic...
    Tue 11th Mar '14, 3:04pm
  • Welcome to the new CMS. Read me first.
    by Don Kuramura
    Welcome to the new CMS. Here's a quick guide of the different areas of this page.

    1. Section Navigation Widget. This widget allows you to go to different sections. The "plus icon" means that this section has sub-sections. Clicking on the "plus icon" will display the sub-sections. ...
    Tue 11th Mar '14, 3:04pm
  • Recovering a hacked vBulletin Site
    by TheLastSuperman
    Symptoms

    Having your site hacked is something no owner wants to experience due to the fact it takes quite a bit of effort to fix, often times requiring payment to a experienced third-party when a owner feels inundated by the sheer amount of work required to troubleshoot and remedy the situation.
    • Site has been defaced - This is the most blatant form of hacking where your sites pages have been replaced with a very disruptive and sometimes rude message from the hacker themselves.
    ...
    Tue 12th Jun '12, 2:37am
  • Issues After Upgrading
    by TheLastSuperman
    If your running a completely default forum upgrading is a very simple process. Many sites however run third-party add-ons and even use a custom style therefor when things change between older and newer
    Thu 24th May '12, 1:02pm
Working...
X