Announcement

Collapse
No announcement yet.

Migrating people and the archive of their e-mail to vBulletin

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Migrating people and the archive of their e-mail to vBulletin

    I just bought vBulletin and am ready to get started. I have experience running a vBulletin system somebody else set up, but now I have my own. I've read the ImpEx documentation and a bunch of threads about it, but I need to understand the "big picture" for my import/migration requirement.

    I have an e-mail archive in flat-file Unix mail format. It is from a retired listserv. Many listserv members will be migrating to the vBulletin forum along with their old messages. Brand new members will join too. I have no more access to the listserv databases.

    Conceptually, there are three parts to my setup process, and I'd like to understand how they fit together:
    Part 1. Importing the user list. Users would be identified by their name and e-mail address in the From fields in the e-mail archive, e.g, John Smith <[email protected]> would become User Name John Smith with e-mail address [email protected]. If it would help, I have a list of all e-mail addresses from the mail archive as a tab-delimited file, e.g., lines of the form John Smith (tab) [email protected].

    Part 2. Importing the e-mail archive into vBulletin. I'll split the flat file into one file per month and import them into matching threads, one for each month, with each e-mail message becoming one post in that month's thread.

    Part 3. Having the former listserv users "sign up" for the vBulletin forums.
    My questions:
    Question 1: Do I need to do an ImpEx import for Part 1 and then a separate ImpEx import for Part 2, or can I use ImpEx to do Part 1 and Part 2 in one pass? Remember that I'm doing Part 2 in monthly sections and that each user may be represented many times throughout the set of mail files.

    Question 2: Is this type of import (from flat file mail files with users identified by the From field) a standard feature of ImpEx or do I need to write custom code?

    Question 3: How do people who are represented in the e-mail archive move to the forums? I can imagine two possibilities but don't know which one makes the most sense. Making it easy for users is more important than making it easy for me.

    Possibility #1:
    I tell each person to go to the Lost Password page and use it with the e-mail address that matches their forum posts, which should cause a message to be sent to them to reset their password. Then they "become" that pre-defined user by setting the password to what they want and logging in as that pre-defined user.
    Possibility #2:
    I globally change my flat file ahead of time, or use prefixes, so that during importing John Smith's automatically-defined account will have User Name archive-John Smith and e-mail address [email protected]. I tell each person to register for the forums as a brand new user (e.g., User Name John Smith and e-mail address [email protected]).

    Then I use admincp's Merge Users function to merge user archive-John Smith into user John Smith for each member who registers with an e-mail address that matches an archived version. The number of such merges will be manageable so this choice, although manual, is practical.
    Thank you for your guidance!

  • #2
    Well there is no importer for a *nix mail file, the closet thing is the yahoo groups importer.

    You could strip out all the email addresses, and use the CSV importer to create your user base.
    I wrote ImpEx.

    Blog | Me

    Comment


    • #3
      Thanks, Jerry. Too bad there's no direct support for mailfile import. I can indeed import my user names and e-mail addresses from a CSV file, since that would be easy to produce.

      That leaves the mailfile messages (which are to become posts). I've seen the list of supported systems. I'm not familiar with those other BBS. Is there any text file format from which ImpEx can import, or are all imports from databases?

      If there is any supported text file format:
      I can study that format, convert the mailfile to that format, and import it to populate my threads/posts. I'll just need to know where to find documentation on that format.

      If there is no supported text file format:
      It looks like I'll need to convert my mailfile to a MySQL database following some schema that ImpEx will import, i.e., mimicking enough of one of the supported formats to get the posts loaded. If this is the case, can you suggest a good candidate for me to consider? I know I'll have legwork, to research the details of its database layout and write some code to load databases from my mailfiles, but I could use an educated guess as to which system to mimic.
      Thanks again!

      Comment


      • #4
        Originally posted by Tomorrow View Post
        Thanks, Jerry. Too bad there's no direct support for mailfile import. I can indeed import my user names and e-mail addresses from a CSV file, since that would be easy to produce.
        There isn't a constant mail format that could be supported, also the threading of email replies to threads with out correct replyto's would be very problematic, and basically more effort that its worth.

        Originally posted by Tomorrow View Post
        That leaves the mailfile messages (which are to become posts). I've seen the list of supported systems. I'm not familiar with those other BBS. Is there any text file format from which ImpEx can import, or are all imports from databases?
        About 25% are from flat file systems, it doesn't matter to ImpEx the nature of the source as long as valid data is passed to be imported.

        If there is any supported text file format:
        I can study that format, convert the mailfile to that format, and import it to populate my threads/posts. I'll just need to know where to find documentation on that format.

        Originally posted by Tomorrow View Post
        If there is no supported text file format:
        It looks like I'll need to convert my mailfile to a MySQL database following some schema that ImpEx will import, i.e., mimicking enough of one of the supported formats to get the posts loaded. If this is the case, can you suggest a good candidate for me to consider? I know I'll have legwork, to research the details of its database layout and write some code to load databases from my mailfiles, but I could use an educated guess as to which system to mimic.
        Thanks again!
        I would try to create 3 tables, one with users, one with threads (or the initial message), then a third with the replies.

        Then they could be mapped to users, threads and posts via ImpEx fairly simply.
        I wrote ImpEx.

        Blog | Me

        Comment


        • #5
          I appreciate your helpful responses.

          My third question above was about how to connect humans with their pre-loaded user identities.

          Assuming random passwords are generated (my data contains no passwords), I have to make sure that the passwords get (privately) to the right people. I can't send everyone e-mail to tell them their new password because not all people will be interested in migrating from listserv to forums, so it would be spam. Instead, interested people need to follow directions I give them to "sign up again."

          I have to prevent them from geting access to somebody else's membership simply by telling me somebody else's e-mail address. So it has to involve sending them a password or link by e-mail. That's where my idea to use the Lost Password system came from. My other idea, to have them sign up new and then merge users manually, seems strange but I don't know why it wouldn't work.

          Perhaps there's a more straightforward solution that I haven't thought of. How do other people setting up a vBulletin forum handle the assignment of new passwords when they load from an old BBS, and how do they get people their passwords?

          Thanks!

          Comment


          • #6
            Originally posted by Tomorrow View Post
            I appreciate your helpful responses.

            My third question above was about how to connect humans with their pre-loaded user identities.
            For the yahoo groups importer I had to match the users to their profile by the email address of the from field.

            Originally posted by Tomorrow View Post
            Assuming random passwords are generated (my data contains no passwords), I have to make sure that the passwords get (privately) to the right people. I can't send everyone e-mail to tell them their new password because not all people will be interested in migrating from listserv to forums, so it would be spam. Instead, interested people need to follow directions I give them to "sign up again."

            I have to prevent them from geting access to somebody else's membership simply by telling me somebody else's e-mail address. So it has to involve sending them a password or link by e-mail. That's where my idea to use the Lost Password system came from. My other idea, to have them sign up new and then merge users manually, seems strange but I don't know why it wouldn't work.

            Perhaps there's a more straightforward solution that I haven't thought of. How do other people setting up a vBulletin forum handle the assignment of new passwords when they load from an old BBS, and how do they get people their passwords?

            Thanks!
            This should take care of that :

            http://www.vbulletin.com/docs/html/impex_passwords

            As long as they are the only ones with access to their own email address then they can get the password, when using ImpEx to import the users you can set a random password so no one knows it and they will have to use the lost password link, as long as their email address is ok, then its fine.
            I wrote ImpEx.

            Blog | Me

            Comment


            • #7
              Good, so my "lost password" idea wasn't a bad idea. I'll just consider it "resetting passwords" rather than "losing" them. I can use the sample message you pointed me to:
              Hello $username,

              We have recently moved our forum to vBulletin, part of moving to this amazing software is that you are required to reset your password, if you follow this link and enter your email address you will be emailed directions to resetting your password.

              http://www.example.com/forum/login.php?do=lostpw

              Thanks,
              Webmaster of example.com
              All I have to do is rename my domain to example.com!

              Thank you for the advice.

              Comment


              • #8
                Yes, it was a fine idea.

                The only real requirement for a user is an email address, if they are willing to have their username as their email address. Having a username is obviously better, and everything else can be generated or set to default.
                I wrote ImpEx.

                Blog | Me

                Comment


                • #9
                  The filter I wrote use John Smith as the user name if the email address is John Smith <[email protected]> and uses johnsmith as the user name if the email address is simply [email protected] with no personal name.

                  I'll check it for duplicate user names before I load it.

                  I think that's about the best I can do to give everyone a suitable user name. I'll change user names on request afterwards.

                  Comment


                  • #10
                    Originally posted by Tomorrow View Post
                    The filter I wrote use John Smith as the user name if the email address is John Smith <[email protected]> and uses johnsmith as the user name if the email address is simply [email protected] with no personal name.

                    I'll check it for duplicate user names before I load it.

                    I think that's about the best I can do to give everyone a suitable user name. I'll change user names on request afterwards.
                    Yup, the best route, basically what I had to do with the yahoo groups importer.

                    Check out impex/systems/yahoogroups_text/000.php thats where I parse their file.
                    I wrote ImpEx.

                    Blog | Me

                    Comment


                    • #11
                      I have written a program that parses email and stores it in three MySql tables: one table for users, one table for threads, and one table for posts. I picked column names based on vBulletin's tables, but I can easily make them any names at all.

                      I want to use ImpEx to import this home-grown database into vBulletin. But I need to tell ImpEx my Source Board Format, as if my source is a particular other forum system and I don't know which one to pick.

                      I don't care what source vBulletin thinks I'm using, as long as my tables will match whatever the importer expects to find. And I can change my tables (renaming columns or adding columns with constant data) as necessary since I'm building them myself.

                      For example, if my tables happen to match a subset of what UBB.threads uses, I could specify Source Board Format as UBB.threads; if my tables happen to match a subset of what phpBB2 uses, I could specify Source Board Format as phpBB2; etc.

                      I could even even import from a system that uses flat files, and have my program output the data in a matching format. Note that I have no documentation on any other forum system, so I have no prior information about how to mimic any of them. Note also that I do not want to write an importer; I need to use an existing one, and make my data match what it expects.

                      Here are the tables and fields I currently have:

                      Table "user":
                      userid int(10) unsigned
                      username varchar(100)
                      password varchar(32)
                      email varchar(100)
                      joindate int(10) unsigned

                      Table "thread":
                      threadid int unsigned(10)
                      title varchar(250)

                      Table "post":
                      postid int(10) unsigned
                      threadid int(10) unsigned
                      username varchar(100)
                      userid int(10) unsigned
                      title varchar(250)
                      dateline int(10) unsigned
                      pagetext mediumtext

                      My specific questions:
                      1. What Source Board Format choice should I make in ImpEx?
                      2. What changes should I make to my tables and columns to match that source?
                      3. Do I need the username column in the post table, or can vBulletin fill it in automatically if I have only the userid?
                      4. All threads I'm importing are to go into a single forum, and I have no forumid in my thread table, although I could add a constant one. Do I need to? What determines which forum the threads will be added to during the import?

                      Thanks for your advice!

                      Comment


                      • #12
                        It occurs to me that one of the forum systems I could mimic is vBulletin itself. My current database has a very small subset of the tables and columns of a real vBulletin system. Can I import the database described above if I select "vBulletin 2" as my Source Board Format?

                        I'd be surprised if I'm the first person who ever tried to build a database from scratch and then import from it, but I couldn't spot a similar question (and answer) in these forums.

                        Comment


                        • #13
                          Since I didn't get any advice here, I experimented and improvised until I found a way to do the import.

                          For anyone who may attempt something similar, here's what I ended up doing:
                          1. I created an empty "import" database using phpMyAdmin.
                          2. I borrowed a .sql backup file from a completely unrelated phpBB board.
                          3. I extracted the "CREATE TABLE" commands to make my own .sql file.
                          4. I coded my email conversion program to output "INSERT INTO" lines that mimick the "INSERT INTO" commands in the backup file, but with my own data.
                          5. I used phpMyAdmin to import my new .sql files into the "import" database.
                          6. I used
                            ImpEx build version : 1.75
                            Source Board Format : phpBB2 2.0.21
                            Target Board Format: vBulletin 3.6.4
                            to import the "import" database into vBulletin.

                          It took me many tries on my "testvb" forum system to get the kinks worked out, but when it finally worked I used the same process on my live vBulletin forums, and at last I have completed the project I started in August.

                          I see a few odd after-effects but I started a separate thread to ask about them.
                          Last edited by Tomorrow; Tue 2nd Jan '07, 8:40pm. Reason: added thread link

                          Comment


                          • #14
                            Originally posted by Tomorrow View Post
                            I'd be surprised if I'm the first person who ever tried to build a database from scratch and then import from it, but I couldn't spot a similar question (and answer) in these forums.
                            Nope, most people who are importing custom systems on their own usually move the data into an existing source systems database, i.e. phpBB as you did or SMF, then use the default importer for ImpEx for that system.
                            I wrote ImpEx.

                            Blog | Me

                            Comment


                            • #15
                              I feel kinda bad that I wrote one-time email-to-forum conversion programs for exactly my particular data, and not something more general that I could share with others, but it was already a big project, and I couldn't expand it to handle email in general (and document it for others). Also, it wasn't entirely automated. I did some of the clean-up by hand in a text editor.

                              But I'll be happy to answer questions if and when anybody else faces the same kind of situation.

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...
                              X