Announcement

Collapse
No announcement yet.

Some questions before converting from phpBB

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Some questions before converting from phpBB

    Hi,

    I'm planning a conversion of our phpBB forum (21,000+ users, 760,000+ posts) to vBulletin.

    Obviously, I want to minimize the downtime, so I want to do as much pre-work as possible before shutting down the forums.

    I'm doing an import of the entire database on a different server now, to see how that goes. I find the import process of the posts especially painfull, since it takes forever. The PHP memlimit is currently at 16M, but I guess I'd better up it to 64M for the real conversion, because otherwise I can't import more than 1000 posts at a time. I guess there is no way to run the import from a SSH console through W3M, right? Because the downloading of all the pages of status-info takes valuable time over a DSL line... It's also too bad I can't cancel the posts-step, clear the imported posts and re-run that step without having to re-run everything (or is this possible?).

    What I would like to be able to do is the following:

    Using this test-import, I want to set up the forums, permissions, usergroups and general config of the board. Then when the time comes, I shut down the phpBB board, convert ONLY users, threads, posts, PMs, etc. This would be possible, because the forums and such will not change from the current import to the next.

    I don't believe this is possible with ImpEx out-of-the-box. I'm an experienced PHP programmer myself, so hacking the ImpEx source won't be a problem. I'd like to get some pointers on how to go about this and where to get started though.

    Any tips to reduce the amount of time it will take to import are also very much welcome. The server doing the real conversion is a dual Opteron 242 with 4GB of RAM and SCSI harddisks in hardware RAID 0, so it should be able to handle quite some load.

  • #2
    Originally posted by Matthijs
    I'm planning a conversion of our phpBB forum (21,000+ users, 760,000+ posts) to vBulletin.
    First, read all of this HOWTO

    Originally posted by Matthijs
    Obviously, I want to minimize the downtime, so I want to do as much pre-work as possible before shutting down the forums.
    The howto covers timings and migration plans.

    Originally posted by Matthijs
    I'm doing an import of the entire database on a different server now, to see how that goes. I find the import process of the posts especially painfull, since it takes forever. The PHP memlimit is currently at 16M, but I guess I'd better up it to 64M for the real conversion, because otherwise I can't import more than 1000 posts at a time.
    Yes, I would set it a LOT higher personally, just to get the import done.

    Originally posted by Matthijs
    I guess there is no way to run the import from a SSH console through W3M, right? Because the downloading of all the pages of status-info takes valuable time over a DSL line...
    Yes, a CLI interface to ImpEx is needed, known about and being planned, though not available atm.

    Originally posted by Matthijs
    It's also too bad I can't cancel the posts-step, clear the imported posts and re-run that step without having to re-run everything (or is this possible?).
    Yes, that is possiable, though would require some direct database intervention.


    Originally posted by Matthijs
    What I would like to be able to do is the following:

    Using this test-import, I want to set up the forums, permissions, usergroups and general config of the board. Then when the time comes, I shut down the phpBB board, convert ONLY users, threads, posts, PMs, etc. This would be possible, because the forums and such will not change from the current import to the next.
    Do a test import and import the usergroups and forums, then set the board up as you wish, don't remove the impex session or any of the import ids, then one you have the permissions done, take a back up.

    Then carry the import on from your phpBB board for a test to see how it goes, though I can't see anything being diffrent as there is no time dependancy, as long as there arn't new forums.

    Originally posted by Matthijs
    I don't believe this is possible with ImpEx out-of-the-box. I'm an experienced PHP programmer myself, so hacking the ImpEx source won't be a problem. I'd like to get some pointers on how to go about this and where to get started though.
    This can be done out of the box, i've done it a few times and phpBB->vBulletin3 is one of the most common imports so its been updated a lot and I've not had a bug report on it for a while.

    Originally posted by Matthijs
    Any tips to reduce the amount of time it will take to import are also very much welcome. The server doing the real conversion is a dual Opteron 242 with 4GB of RAM and SCSI harddisks in hardware RAID 0, so it should be able to handle quite some load.
    Set the PHP memory limit to 512Meg or something huge just to get the import done. Then experiment with the per-pages values to see what works best, for a board that size and with the physical resource you have I'd experiment with 3-5000 posts per page maybe higher.

    Also run ImpEx stand alone, i.e. not in the admincp, but in a root dir to your website and fill out ImpExConfig.php completely.
    I wrote ImpEx.

    Blog | Me

    Comment


    • #3
      Thank you for your fast answers, this will help me a lot and save me quite some time.

      I will be back here during my experiments when more questions arise.

      Comment


      • #4
        Okay, I've completed this first test-run and everything seems to have converted ok. There are some things though that I have noticed which are not making the process any easier. Let me explain:
        1. ID's of topics, posts and users are changed with the conversion. I understand that this is due to the nature of the importer and enables multiple imports and the merging of boards, but it is also very inconvenient for people like me who convert their entire board at once. I am seriously thinking of hacking the ImpEx source to preserve these IDs.
          Reasons:
          • No easy way to change all URLs that point to topics/posts/users on my board to the new location (a translation-table would have to be used)
          • All incoming URLs from other websites will no longer function
          • All URLs indexed by Google and other search engines will no longer function
        2. De-activated users (not banned, just de-activated) become registered users after the conversion. Moderators on my forum tend to de-activate useraccounts of irritating people, not ban them. This retains all their information/posts/etc. I think these users should be added to the "Unregistered / Not Logged In" usergroup, but ImpEx does not seem to do that. This is very inconvienent.
        3. URLs with an & in it (like http://xxx/forum/profile.php?mode=viewprofile&u=22) are overzealeously &-ed again, creating a broken link.
        4. Subscribed threads are not converted, even though this is pretty straightforward.
        These are my initial remarks on the conversion process. Please don't see them as criticism but rather pointers for improvement. If you (Jerry) could comment on my points, that would be very helpful.

        Also, maybe there are some tricks devised by others to overcome my main problem: the non-functioning links to phpBB functionality. If the IDs are retained, writing a couple of redirect-scripts for external links to topics and posts and altering the posts in the database to contain the new links is a piece of cake. Having to use a translation table is slightly more complicated and prone to errors.

        Thanks for any comments.
        Last edited by Matthijs; Thu 28 Jul '05, 5:11am.

        Comment


        • #5
          1.

          This isn't anything to do with the ImpEx primarily, is a result of the way that MySQL auto_inc works, have a search in this forum for "internal link parsing" for the explanations I've given before, phpBB was one of the first is was trailed on, so I've had it in beta before.

          Also the relationships between the import and new ID's is maintained, with the import id's, importuserid, importthreadid etc.

          It's not to facilitate multiple imports (the session and importid's do that), it's because ImpEx's primary goal is to protect vBulletin's database, if you try to drop the index's force ID's to maintain internal links, you will still have the wrong URL's and a broken database, hence why internal link parsing is the answer.

          I've been through this quite a few times now with various boards and people with addon's like Photopost etc.

          As for incoming URL's, the URL to a post/thread/users is always going to be different after an import, you'd have to use mod_rewrite to deal with that.

          2.

          Users should be mapped to their phpBB usergroups when they are imported, no user will be imported into the default Registered group.

          3.

          Well that's a bug I can look at.

          4.

          Nope never really bothered with them, though its a trivial thing to do, or for users to deal with afterwards.
          I wrote ImpEx.

          Blog | Me

          Comment


          • #6
            Thanks for your anwsers, I'll do some searching on changing the internal URLs.

            As for
            Originally posted by Jerry
            2.
            Users should be mapped to their phpBB usergroups when they are imported, no user will be imported into the default Registered group.
            I don't understand totally. I know that users are imported in a Imported group, I read that you have to delete that group to get everybody in the Registered users group. But what you say is that I need to manually put them in the correct usergroups afterwards? Why doesn't ImpEx do that, because it does convert the groups themselves...

            So I'll have to write a script to do that, I guess (no way I'm doing all these users manually)...

            Comment


            • #7
              Could you send me a copy of this internal link parser, without support. I can recode it to suit my needs. Because I can only find threads about it being pulled, but no downloadable code?

              Comment


              • #8
                Originally posted by Matthijs
                I don't understand totally. I know that users are imported in a Imported group, I read that you have to delete that group to get everybody in the Registered users group. But what you say is that I need to manually put them in the correct usergroups afterwards? Why doesn't ImpEx do that, because it does convert the groups themselves...
                It dosn't put them in groups, because for various reasons you have to keep the orgional and imported groups seperate if you want to manage the users within the board seprartly, i.e. if two clans formed one board, you'd need to have diffrent groups for each clan member etc
                I wrote ImpEx.

                Blog | Me

                Comment

                widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                Working...
                X