Importing HOWTO

[Jerry]

Latest build version and updates

HOWTO ImpEx (Import / Export)

Here are some things to think about when doing an import and steps to follow.

First there are some things to realise :

• Testing an import before shutting down the source forum is the best way to go.
• There will probably be some residual HTML from the content that will need cleaning up afterwards, which is part of the import task and process, not a function of the support team.
• The links within the forum to other threads/posts/users will be imported, and link to the old board. A 301/404 solution will need to be put in place if that is a concern.

Before the import

Firstly, if you have a board up and running leave it running until you are comfortable with the import process.

Once again .......

LEAVE THE ORIGIONAL BOARD RUNNING TILL YOU HAVE READ THIS AND DONE SOME TEST IMPORTS

"I've turned my board off and done an import and it didn't work 100% corretly first time, my users are upset, fix it now"

Isn't really the best way of getting a board imported.

Right, on we go.


Secondly, make a back up of your source board and do the initial trail run import from that, not the live board.

ImpEx only ever reads from a source board it will never alter source data. Though running an import against source data that is being updated can have some strange outcomes for the import as well as the source board, also it will cause a lot of load on the source board.

Make sure you have the latest download.

Setting up

Set up a board for practicing an import into.

The vBulletin license allows you to have a development board installed for the purposes of development and testing. Practicing an import is a good example of this.

Copy your source data to the same machine that your development vBulletin install, be that a database (i.e. phpBB) or the files (i.e. ubb.classic).

Attachments

Ensure that you have your storage type set up how would like it if it is a fresh install, i.e. if you don't want your attachments in the database set vBulletin to use the file system before the import.

ImpEx will import using what ever the current settings for you board is.

Getting ImpEx

ImpEx can be downloaded from the members area, http://www.vbulletin.com/members/import.php

For futher details on installing, configuring and running ImpEx, please refer to the ImpEx section of the online manual: http://www.vbulletin.com/docs/html/impex

When uploading ImpEx, you only need to up load the dirs and file in the impex/systems folder that you actually need. i.e. if you are importing from phpBB just upload the impex/systems/phpBB folder with the core files.

Uploading them all won't do any harm, though ImpEx will use more memory when starting for the first time as it checks them all and loads all the modules.

Config

ImpEx has a configuration file, this allows the user to configure the database details.

The [ Target Database ] settings will the vast majority of the time be the same as the config.ini.php settings for vBulletin itself. By default impex will read the details from includes/config.php so you don't need to set the target database details, impex will display a message at the top of every page to say when it is using the config.php file settings.

If you are importing from a database opposed to a file based system edit sourceexists = false to sourceexists = true.

Then, edit the rest of the [ Source Database ] setting to match the database of the source board.

Two things to double check here, firstly the table prefix.

In this example I am using a phpBB board. I would have to check the phpBB database to see that the tables are all phpbb_ then the tables name. Therefour phpbb_ is the table prefix.


Running an import

If ImpEx is installed and configured you will be able select it via the admin control panel under "Import -> Import".

From there you can select the system you want to import from the drop down list.


The initial modules

The first module of nearly all the importers updates the vB3 database to include importid's, so ImpEx can keep the association between the users and the data during the import.

The second module is for associating users, this is a manual way of linking imported users to existing users. This will only need to be done if you have existing users you would like to associate to. NEVER associate with an vBulletin admin user during an import, wait till the import is finished and all is well, then use the admincp to merge the users.

Also the association only needs to be performed if the users you are trying to associate have a different email address, during the user import module ImpEx can be told to merge users who have the same email address, this is a lot faster and easier than manually associating them.

If the importer you are using doesn't not have the auto associate function in the user import module, check the notes on the status thread, if it is not mentioned in the development notes at the bottom of the post, make a request for it to be retro fitted to the importer you want to use, if it hasn't been already.


The rest of the modules are self explanatory and have dependencies on each other which is evident when you try to run a module. For instance to import the posts you need the threads etc.


Re-running modules

When you re-run a module, for what ever reason it will clean up any previous data imported of that type.

For instance if you run the import post module for our phpBB import, it will import all the post from the phpBB board and place them in the vBulletin database. All the posts will have a importpostid which is the original post id of the post in the phpBB board. Any original post from the vBulletin board before the import will have no importpostid, as they are original.

So when the module is run for a second time (another practice run, you change some setting, updated a parser, got more posts, etc) it will delete all posts from the vBulletin board that have an importpostid.

This is use full for getting timings or re-running modules that had issues that you have changed or updated.

WarningWarningWarning

If you have associated or merged users, the original vBulletin users will now have a importuserid, re-running the import user module will delete these users, if you don't have any associated users this isn't an issue.

This is why we make backups of databases

WarningWarningWarning

The end game

Once you have completed all the modules that you wish to run, there are still several steps that you need to take.

Firstly, in the "Import & Maintenance -> Update Counters" section of the admin control panel update the threads counters then the forums, also you will need to rebuild the search index and possible the user details to clean up after an import depending on the system you are importing from.

Secondly, check the forums are ordered correctly and are being displayed, you may need to re-order them in the admin control panel forum manager.

Thirdly, most importers will import the usergroups or create a default import usergroup, this is so you can check the users being imported and manage them accordingly, to move users to the default "Register" group in vBulletin, delete the import group they are in.

Finally, check the permissions of your moderators and the forums. As all boards different to varying degrees, only so much information can be imported as values in vBulletin will not exist in other boards so they will be left to default.

A good example of this is the COPPA setting for the users, this many be need to be updated for all users.

An import will as a rule of thumb err on the side of caution, if an age can't be ascertained for instance, it will make a user COPPA then let the admin correct it later.

Rebuild the search index if you want to be able to search on the imported posts.

Now what?

Now you have done an import, the main ImpEx page will give you an idea of how long it takes, plus you re-configuring and post install setup.

Try the import a few times to make sure you are happy with the process and know what to do when the time comes.

This is the time to ask question here about things that you don't understand or believe are going wrong, not when you do the final live import !!

Have some of your trusted users or admin/moderators look around the test import boards to help with finding any issues and pointing out permissions that need to be updated etc.


Ok, I've done it and its live

Now would be the time to use vBulletin's 'Email all Users' functionality to inform your users that you board is now open again and running on vBulletin and that they will need to update their profiles in their User Control Panel to ensure that all their details are correct.

Also a good time to update their signatures using BB code and removing HTML if the legacy source board allowed it or to reset their passwords with your boards reset password link.


Happy importing

Once again .......

LEAVE THE ORIGIONAL BOARD RUNNING TILL YOU HAVE READ THIS AND DONE SOME TEST IMPORTS

No really, I'm serious.

Finally

Remove the ImpEx folder and scripts till they are next needed, if the import is done and live you don't need them. If you do need to do another import from another board it is lightly that there will be another version available then anyway.

[Jerry]

How do I remove import id's and do second imports ?

ImpEx relies on import id's during an import so that it can match one set of data to another.

For instance a user will have a userid and a importuserid once they are imported. The userid is the vBulletin assigned userid and will be what ever is next in the line and available as per the database.

The importuserid is the userid from the source board.

This is imported and used to match the user to their imported posts, attachments PM's etc.

The same goes for posts for instance threadid has a importthreadid so that ImpEx knows where to put the imported posts.

When you have done an import and are completely finished and wish to do another import you will have to set all these import id's to 0.

This is so that you finalized the data in the board, that is remove its legacy id's so that it permanently becomes part of the target board, after this is done, there is no way back (apart from restoring the back up you made before starting).

As you can import into an empty board or an existing one, this has some serious implications. e.g.

You have a source board and your vBulletin target board.

The vBulletin target board has a lot of content, once you have performed an import of the source board you can still reverse the operation as the source database has importsomethingid's for all the imported data, therefore ImpEx can tell what is original and what is imported data, this is needed for the re-running of modules also.

As soon as you finishing an import, the import id's are no longer needed and serve no purpose, with one exception that I have come across, that being :

3rd party applications.

If you have an application that uses the user data from your source board, it will no doubt be associated with the userid from the source board, if you wish to keep that, update the 3rd party application to use the importuserid in the vBulletin database.


So with the exception of 3rd party applications there is no more use for the import id's.

They can be ignored for all purposes save Importing consecutive boards and general tidiness (also indexes are created that may slow down huge boards slightly).

You MUST remove the import id's before performing a 2nd import, this is because ImpEx will not know what is original imported data and what is just imported etc.

If you are importing boards that use the same user data, just use the auto-email associate to link up the userids automatically between board imports.

To remove the import id's use the clean database found at the top of the ImpEx page, as show in the attached images.

[Jerry]

Example importer for a custom user table

Ok, here is an example of how to use the ImpEx API to create an import for yourself.

The details here come from this thread.

I'm assuming that if you are going to read this or modify the code etc to get it working with your own user table you have some coding and PHP knowledge

-------------------

In the attachment there are 4 files in a directory, which follow the ImpEx naming schema, that being :

<foldername>/000.php
<foldername>/001.php
.....
etc

Each of the files contain a class relative to the <foldername> and filenumber, so in this case the <foldername> is patricksumner so its :

patricksumner/000.php >> class patricksumner_000 extends ImpExModule
patricksumner/001.php >> class patricksumner_001 extends patricksumner_000
patricksumner/002.php >> class patricksumner_002 extends patricksumner_000
patricksumner/003.php >> class patricksumner_003 extends patricksumner_000

The <foldername>/000.php must extend ImpExModule to be able to use all the functionalilty in ImpEx, also it acts as a function repository for the importer, parsing, database access, date manipulation, etc.

All the other classes, 001,002, etc layer under 000 to inherit the functions and gain access to ImpEx.

In 000 you will notice all the function for getting the details back from the database, though that could be a flat file, LDAP, mail server, carrier pigeon, etc.

001, is the check module, it uses var $_valid_tables in 000 to list off the tables in the dB its connected to to show the user what it has to work with. This is also the place, where extra index's are created, dB connection, file paths, importid's are added to the tables.

In this example you can see from line 116 to 128 I've created a usergroup to import the users into.

002, I put in by default for all importers in case any one wants to do manual association.

003, init() just asks questions about how many users to import per page and if you'd like to match them on email if importing into a populated database, then sets up the module variables and counters.

resume() is where it does the work starting on line 92, the details are coming back from 000.php in $user_details and between 160 and 184 they are put into the relative vBulletin variables.

The line :

// Check for page end 

Checks to see if the array has returned less that was requested, this is 99% of the time the indication that you have selected the last of the users so you can closed down the module and update the users statistics.

[Jerry]

Invision Free

If you are importing from invision free you will need to get a back up of your data, if you don't have a IPB 1.3 database to restore the mysqldump into I have created a schema here that has the tables that you need populated.

If you create the tables in a database with the attached SQL, you can then restore the invision free SQL into it with ignoring the INSERT's for the tables that ImpEx doesn't use in a IPB1.3 import thus :

mysql -f -u <user> -p invision_free_database_name < your_invison_free_data.sql