Announcement

Collapse
No announcement yet.

Import posts (007) step failing half way (webwiz)

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Import posts (007) step failing half way (webwiz)

    I've run a test import with an old (years old) version of the database and that worked fine.

    I'm getting a very strange occurance when i used a backup which was 1 day old.

    I have approximately 6000 threads and a total of 376,000 posts.

    The 2 times i've run the import on step 7, its stopped at ~117,000 posts. There is no error returned in the error column or on screen.. it just stops as if it has reached the end of the table.

    eg. 007 - Import Posts - 116999 successful - 0 failed.

    I've checked the posts in the source table and it wasnt skipping records individually, they all are part of the 2nd half of the data (in sorted order).

    The first time i ran it with 2,500 at a time, the 2nd time 500 at a time, just to see if it was related to that.

    Any reason why this is occuring?

    I have run a test to see if there are any links threads missing from the posts, but there wernt any.

    Code:
    select *
    from tblThread t
    left join tblTopic tt on tt.topic_Id = t.topic_id
    where tt.topic_id is null
    Returned 0 results.

    The import for this stage takes approximately 30-40 mins to get to the point where it stops, so its hard to test every variation.
    Last edited by psylenced; Wed 11th Jun '08, 6:40am. Reason: typo

  • #2
    Further, is it possible to comment out the clear_imported_posts function and add a condition where postid > 117001 to the get_webwiz_post_details to avoid having to run the whole import again?

    Comment


    • #3
      What version of which database, the source or the target ?

      If the PHP process is stopping, and there is no ImpEx or SQL error, then the error will be in the PHP error logs.

      I don't understand if you are attempting to do two imports, or a merge, or a clean import.


      Originally posted by psylenced View Post
      Further, is it possible to comment out the clear_imported_posts function and add a condition where postid > 117001 to the get_webwiz_post_details to avoid having to run the whole import again?
      No.
      I wrote ImpEx.

      Blog | Me

      Comment


      • #4
        Sorry i'm attempting a single import. I attempted step 7 twice (with different buffer sizes).

        I've converted an access database into a temp mysql database.

        Then imported from tempmysql -> vb mysql using impex.

        Mysql version is: 5.0.45-community-nt

        I cant see any related errors in php_error.log (only relating to phpmyadmin).

        It doesn't look like the php process is stopping, it just looks like it has reached the end of the records it's trying to import.

        The strange thing is it's finished at the same point both times.

        Comment


        • #5
          Have done some further investigation tonight.

          I have limited the records of the posts import to only go between a certain range.

          eg.
          Code:
                      SELECT * FROM " .
                      $tableprefix."Thread
                      WHERE Thread_ID > 350558
                      and thread_id < 352758
                      ORDER BY Thread_ID
          When i swap the starting number below a certain thread_ID, the 500 record import for some reason will only import 499 records and stop. When i move it above a certain ID it starts jumping through and importing the next few batches.

          The reason it's stopping is due to the second condition in this:
          PHP Code:
                  // Check for page end
                  
          if (count($post_array) == OR count($post_array) < $post_per_page
          One record is being skipped for a certain reason and it's not importing the full page size, so the code is assuming that means it's the end of the data.

          I'm still not sure which record is causing that, and i'll investigate that now.

          Perhaps a suggestion is to change the exit code to only stop if 0 records have been imported which might mean 1 additional page refresh.

          Comment


          • #6
            Ok tracked the issue.

            For some reason there was duplicate ID's in the post table of the source database. Not sure how that could happen, but it did.

            Thread_ID Topic_ID Author_ID Message Date
            350554 1410 553 ################ 2005-11-14 19:23:30
            350554 1410 1205 <P><FONT color... 2005-11-14 19:28:34

            Comment


            • #7
              Originally posted by psylenced View Post
              For some reason there was duplicate ID's in the post table of the source database. Not sure how that could happen, but it did.
              Hmm, I remember a similar issue with phpBB2, I'll have to check that with this system as well.

              Did removing the duplicate ID solve the issue ?
              I wrote ImpEx.

              Blog | Me

              Comment


              • #8
                Yep removing it fixed it.

                The problem with it wasnt that the importing record failed, it was that the candidate list was less than the page size (due to being distinct), which kicked the import out early thinking it was on the final batch.

                Comment

                widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                Working...
                X