Announcement

Collapse
No announcement yet.

More questons for eve/groupee import...

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • More questons for eve/groupee import...

    Okay, my last thread is a little old, and I have a few questions that are in a different direction...

    I have 99% of the data properly imported from my original database export.

    Now, I have the following questions/issues....

    I'm hung on the last step still of my import, however, the info is all in the database, and I have been running maintenance scripts on the data to build the search index, update post counts, etc.... all is working fine...

    However, I'm curious about the passwords imported from eve/groupee...

    there is a hash stored for the user in the database, but the password isn't working when attempting to use a password created with eve/groupee...

    Is this a known thing? Will I have to send every user a password reset email? is that possible? Or is it supposed to import, and work properly from eve?

    Also, Since I'm satisfied with the data from the import even though I've had problems with the last step or two, how do I clear out the import system without deleting all the new data?

    And lastly, I will have to import again all the data from the original site that has been posted over the last week...

    This will include posts that are attached to threads that have already been imported, and also new threads and posts....

    How do I accomplish this with the impex system???

    Thanks for your help!

  • #2
    Originally posted by himerus
    Okay, my last thread is a little old, and I have a few questions that are in a different direction...

    I have 99% of the data properly imported from my original database export.

    Now, I have the following questions/issues....

    I'm hung on the last step still of my import, however, the info is all in the database, and I have been running maintenance scripts on the data to build the search index, update post counts, etc.... all is working fine...

    However, I'm curious about the passwords imported from eve/groupee...

    there is a hash stored for the user in the database, but the password isn't working when attempting to use a password created with eve/groupee...

    Is this a known thing? Will I have to send every user a password reset email? is that possible? Or is it supposed to import, and work properly from eve?
    Password can't be imported from eve do to the way they are stored in the source data.

    http://www.vbulletin.com/docs/html/impex_passwords

    Originally posted by himerus
    Also, Since I'm satisfied with the data from the import even though I've had problems with the last step or two, how do I clear out the import system without deleting all the new data?
    Can you explain what you mean by new data.

    Originally posted by himerus
    And lastly, I will have to import again all the data from the original site that has been posted over the last week...

    This will include posts that are attached to threads that have already been imported, and also new threads and posts....
    You will have to re-do the whole import anyway, I think what you are asking is can a differential import be done, which it can't.

    http://www.vbulletin.com/docs/html/impex_terminology
    I wrote ImpEx.

    Blog | Me

    Comment


    • #3
      Okay, the password situation is easy enough to fix... I figured that'd be the case anyway....

      Now, about the data, Yes, from the terminology page, I believe I do mean a differential import...

      If I'm unable to accomplish this somehow... I don't know how to make it work, as it has taken HOURS upon HOURS between exporting from eve board, then ftp'ing to new server, importing to database, then running imports... with a million posts, after getting them all imported, and then having to update search index, post counts, thread info, etc... then it's at least 48 hours later, and thousands of posts more on the site...

      How can I work around this to only import the new data? you don't know of any way? without serious custom coding?

      Comment


      • #4
        Originally posted by himerus
        How can I work around this to only import the new data? you don't know of any way? without serious custom coding?
        Big imports are always going to be a pain as I'm sure you aware, though now that you have done it once, caught the changes that were needed, etc consecutive imports should be faster as you know the steps and the clean up etc.

        Usually there is a cut over where you take down the source, lock it, or say that after X hour the posts will be lost as its in transit. Move the SQL, set up the import, do it, then do the clean up that you practiced in the test imports.
        I wrote ImpEx.

        Blog | Me

        Comment


        • #5
          Okay, a couple more questions regarding import....

          As I've mentioned, I have a huge amount of data, and other sites to combine into the new converted site, but this is my problem...

          With my original import, I still have all of the data sitting in the import system as the last step failed, and I'm not sure how it's supposed to work at the end....

          Do the importpostid, importthreadit, importforumid, etc. stay in the database after the import is finished? Or are they supposed to be removed at the end?

          I am going to have to go my own way with the rest of the import, but need a little guideance on the systems functionality.... I believe I can convert some of the key import functions to my purpose, and import the latest export of my 1.5GB database, and only extract new posts and threads, and with the proper postids and threadids stored in the import fields, it shouldn't be too bad...

          I'm also going to have to convert several new boards into the current forums, and that will only be a problem if the importpostid's and threadid's match up with the data from the other forum... Here, I can work around by doing a check, and if it does exist, then I'll reassign the importthreadid, and keep it in record to be able to adjust it later, and not mess up the other posts...

          So my main question is,

          Is the import data supposed to be removed at the end? and no import id's are saved? or do they stay there forever? (I don't want the import system to remove them my accident if the first job is to clear anything with an import id)

          Comment


          • #6
            And how can I clear out the data that is in the impex system without removing the importid's????

            Comment


            • #7
              Originally posted by himerus
              With my original import, I still have all of the data sitting in the import system as the last step failed, and I'm not sure how it's supposed to work at the end....

              Do the importpostid, importthreadit, importforumid, etc. stay in the database after the import is finished? Or are they supposed to be removed at the end?
              This depends on if you are going to do consecutive imports, if you are you have to remove them.

              http://www.vbulletin.com/docs/html/impex_multi_import

              Originally posted by himerus
              I am going to have to go my own way with the rest of the import, but need a little guideance on the systems functionality.... I believe I can convert some of the key import functions to my purpose, and import the latest export of my 1.5GB database, and only extract new posts and threads, and with the proper postids and threadids stored in the import fields, it shouldn't be too bad...
              Go your own way ? I have offered all the support I can, I'm not sure what you mean by that.

              Originally posted by himerus
              I'm also going to have to convert several new boards into the current forums, and that will only be a problem if the importpostid's and threadid's match up with the data from the other forum... Here, I can work around by doing a check, and if it does exist, then I'll reassign the importthreadid, and keep it in record to be able to adjust it later, and not mess up the other posts...
              I have covered that above, impex can remove the importid's to allow another import, this is a very common action.

              Originally posted by himerus
              So my main question is,

              Is the import data supposed to be removed at the end? and no import id's are saved? or do they stay there forever? (I don't want the import system to remove them my accident if the first job is to clear anything with an import id)
              I'm guessing your asking about the multi import process, its in the link above.

              Originally posted by himerus
              And how can I clear out the data that is in the impex system without removing the importid's????
              Err ..... the importid's are part of the imported data, if you remove the imported data you will remove everything you've just imported, you can just remove the import id's from the imported data though.
              I wrote ImpEx.

              Blog | Me

              Comment


              • #8
                Jerry, thanks for all your help...

                By saying I was going to have to go a different way, I mean that because of the boards I have to import, and combine to threads/users/forums, none of the importers you have were made exactly for this situation...

                I don't have the ability to take a final export from the old site, and completely re-run the import and update all the counters during a time-frame where neither site is fully functional...

                I mean that I'm going to have to take the impex system, and modify it to my needs to be able to appropriately import data at a later time, and match new posts to threads already in the database, and new threads to forums that have already been converted, and new users and threads from other sites into custom forums....

                I was curious that just simply running any impex function would clear the previous import data...

                This is where part of my problem is.... I'm going to have to have the ability to start a new import, and compare it to data that was previously imported...

                It's the way I have to get it done via the instructions from the boss, so it's going to take a super hack from me to make it work the way I need it to...

                Thanks!!!

                Comment


                • #9
                  Yes big sites are troublesome at the moment, the ideal is a Differential import .

                  You could work this into impex at the moment by checking the importthreadid and importpost id for each one to see if its already there, you could use get_threads_ids() to see if the groupee id is already there etc.

                  There are plans for differential import's being supported in impex, though core development is on hold as I'm working on other things atm.
                  I wrote ImpEx.

                  Blog | Me

                  Comment


                  • #10
                    I definitely understand the complexity of the impex system, and dealing with 4 billion types of forum software that may or may not be put together well, and for packages that are updated frequently, maintaining the system must be a NIGHTMARE

                    One last question.... I only have some small data (5000) posts I think from the differential import I need to complete... maybe 500 threads, 100 users....

                    On the AdminCP screen for the Impex system, when I click on

                    Database cleanup & restart, does that REMOVE the data from the import, or JUST the import-id's...

                    I need to run this import for the current data, and then I'll be able to remove the import id's, and use impex on my import of a second site that is MUCH smaller....

                    So, my only concern is that if I click on Database cleanup & restart it will remove all the data... I just want to clarify that it wont, and only removes the import ids that were saved from the previous board...

                    I may even rename the fields for the time being to continue to use them for my purposes, and allow impex to recreate new import fields on the next board to import... I know this'll have a impact on the database, but might be worth it for the first few weeks so that if I run into problems, I'll be able to use that info to find the problem...

                    Comment


                    • #11
                      If you follow that link onto the clean up page there are a few options, remove the import id's, remove all the data, remove the session etc.
                      I wrote ImpEx.

                      Blog | Me

                      Comment


                      • #12
                        Awesome...

                        I THINK I have found a way to trick the impex system without many modifications, and a handfull of queries ran directly on the system to perform a differential import...

                        I'll let you know if it works.... it's a little tricky, and I'm still trying it on my test board to make sure it works... If it does, it'd only be for the not very faint of heart, but should work for any users in my situation that have the experience to get it done...

                        You've been a huge help on this Thanks so much!!!

                        I've been a vb fan for a long time, and I'm trying to convert the new company I'm working for over to vb on some projects, so I hope I can bring you guys more business!!!

                        Comment


                        • #13
                          Originally posted by himerus
                          s... If it does, it'd only be for the not very faint of heart, but should work for any users in my situation that have the experience to get it done...
                          Groupee import of that size, differential one as well, brave man Good luck !

                          Originally posted by himerus
                          You've been a huge help on this Thanks so much!!!

                          I've been a vb fan for a long time, and I'm trying to convert the new company I'm working for over to vb on some projects, so I hope I can bring you guys more business!!!
                          Thanks, glad to hear your happy
                          I wrote ImpEx.

                          Blog | Me

                          Comment

                          widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                          Working...
                          X