Various importing problems

[Gladius]

I'm importing from UBB.classic 6.7.1 to vB 3.6.8 and I'm wondering about this:

####
#
# DUPLICATE CHECKING
# Where unique import id's are aviable ImpEx can check for duplicates with some
# Tier2 systems this will need to be set to false.
#
# yahoo_groups, YaBB 2.1, Ikonboard 3.1.5 (for attachments)
#
####

define('dupe_checking', true);

---

I've read that setting this to false will speed things up (it's a large import, over 600k posts), but I'm wondering what exactly does setting this to false mean? Is it relevant to UBB.classic at all, etc. Thanks in advance for the info.

[Jerry]

Setting that to false, makes ImpEx skip checking for the importid before each save, i.e. just before it saves any piece of data it checks to make sure there isn't one with the same importid. (hence only working with source data with unique id's or where I can extrapolate a unique id, file number + line number etc).

As will all things it is a trade off, speed vr robustness. Though if an import runs fine and your not stopping and starting or refreshing the page a lot, i.e. it runs smoothly then you should be fine and it will run a lot faster with it off.

It was mainly put in for people who's connection dropped out mid page or who were prone to hitting refresh so reloading half a page of posts, users, threads etc and the dupe_checking would stop them being added again.

Good connection and lots of testing, you can turn if off for speed for sure, especially with a ubb.classic import, they need all the help they can get

[Gladius]

Ah, I see. I'm doing the import locally on a Windows box with XAMPP and I just let it run without interfering in any way. Oddly though, once the import is done, it shows it's imported nearly 6000 more posts than the UBB stats show. I haven't imported with dupe checking off yet to see if there's any difference, but considering this, I guess I'll have to. :/

Do you have any idea what could cause such a large discrepancy?

[Gladius]

Various importing problems

I'm having a hell of a time with my UBB.classic (6.7.1) import to vB 3.6.8 (Impex 1.83), so I guess I'll just make this thread for all the issues I encounter as I go to prevent opening threads for each of them...

The latest issues I've encountered were none of the Buddy/Ignore lists importing... Impex goes over all the users, but displays the equivalent of "this user has no one on the Buddy list" and "this user has no one on the Ignore list".

Any ideas why that would happen? I get total 0 imported on both buddy and ignore lists.

The same thing happens with moderators as well - 0 moderators import.

On top of it all, no private messages import either apparently... I get stacks of this error when attempting to import PMs:

Invalid ImpExData object, skipping. Failed on : fromuserid
Invalid ImpExData object, skipping. Failed on : fromuserid
Invalid ImpExData object, skipping. Failed on : fromuserid
Invalid ImpExData object, skipping. Failed on : fromuserid
...

---
Thanks for any help in advance...

[Jerry]

In my experience I would say un-deleted data from ubb.classic that is still there, i.e. posts that weren't deleted from threads that were deleted.

Better to have more than less data from an import I'd say !

[Jerry]

The latest issues I've encountered were none of the Buddy/Ignore lists importing... Impex goes over all the users, but displays the equivalent of "this user has no one on the Buddy list" and "this user has no one on the Ignore list".

Any ideas why that would happen? I get total 0 imported on both buddy and ignore lists.

The same thing happens with moderators as well - 0 moderators import.

Most likely it can't find the files or the source is in an unexpected format.

On top of it all, no private messages import either apparently... I get stacks of this error when attempting to import PMs:

Invalid ImpExData object, skipping. Failed on : fromuserid
Invalid ImpExData object, skipping. Failed on : fromuserid
Invalid ImpExData object, skipping. Failed on : fromuserid
Invalid ImpExData object, skipping. Failed on : fromuserid
...

---
Thanks for any help in advance...

Are 100% like that, or most or some ?

[Gladius]

In my experience I would say un-deleted data from ubb.classic that is still there, i.e. posts that weren't deleted from threads that were deleted.

Better to have more than less data from an import I'd say !

Well, yes and no... we don't delete on a whim, so posts that we've deleted should have been deleted. I also find it odd that UBB wouldn't have deleted them but only hidden them from view... as far as I know, that's not how it works.

Also, we haven't deleted anywhere near 6000 posts. Probably nowhere near 500 in all of the years the board has been up. That's the problem...

Most likely it can't find the files or the source is in an unexpected format.

That wouldn't be because Impex doesn't OK the relative path to the non-cgi dir, would it? For some reason I keep getting NOT OK on that path...

Are 100% like that, or most or some ?

From what I've seen, every single one of them. I'm pretty sure the import result was a total 0 there too...

[Gladius]

Can you help me out here, Jerry?

[Jerry]

Well, yes and no... we don't delete on a whim, so posts that we've deleted should have been deleted. I also find it odd that UBB wouldn't have deleted them but only hidden them from view... as far as I know, that's not how it works.

Also, we haven't deleted anywhere near 6000 posts. Probably nowhere near 500 in all of the years the board has been up. That's the problem...

Then going to have to keep an eye open in the target board for extra posts or duplicates, then they can be pruned or if found while still testing tracked back to see what's going on.

That wouldn't be because Impex doesn't OK the relative path to the non-cgi dir, would it? For some reason I keep getting NOT OK on that path...

That isn't going to help, though that should effect the posts, that path should be for the smilies etc.

From what I've seen, every single one of them. I'm pretty sure the import result was a total 0 there too...

Line 178 of impex/systems/ubb_classic/010.php is :

$_pmtext->set_value('mandatory', 'fromuserid', $vbuserid["$pm_from[$i]"]); 

Change that to :

$_pmtext->set_value('mandatory', 'fromuserid', "1"); 

It will appear that all PM's have come from the admin, though at least the users have the content.

[Gladius]

Then going to have to keep an eye open in the target board for extra posts or duplicates, then they can be pruned or if found while still testing tracked back to see what's going on.

I've actually compared the forum post/thread figures on UBB and vB after the import, and there was a difference of only a few dozen posts as far as the figures for every forum go, which made me suspect that the stats hack on UBB is wrong. This has turned out to be the case after a manual count of the post figures for all the forums... so that's one mystery cleared up.

That isn't going to help, though that should effect the posts, that path should be for the smilies etc.

Ah, that'd explain why so many smilies were broken... but at least that can be fixed afterwards.

It will appear that all PM's have come from the admin, though at least the users have the content.

I'll give it a shot. Any idea what I can do about the buddy/ignore lists? We haven't had any hacks installed for that so the files should be completely unchanged and as expected.

[Jerry]

I've actually compared the forum post/thread figures on UBB and vB after the import, and there was a difference of only a few dozen posts as far as the figures for every forum go, which made me suspect that the stats hack on UBB is wrong. This has turned out to be the case after a manual count of the post figures for all the forums... so that's one mystery cleared up.

Vindicated

Ah, that'd explain why so many smilies were broken... but at least that can be fixed afterwards.

Indeed, setting up the smilies manager and then using cleaner.php to switch over any old bbcode to new ones is a good way of doing that.

I'll give it a shot. Any idea what I can do about the buddy/ignore lists? We haven't had any hacks installed for that so the files should be completely unchanged and as expected.

With out seeing them and trawling though the code that does that with examples, not of the top of my head ! The main causes are that ImpEx can't parse the perl arrays to PHP.