[Forum] imported language phrases truncated at first special character

[BrianHoltz]

For all the language packs (french, spanish, italian, etc.) I've imported into VB (4.1.8), the phrases get truncated at the first special character in my beta and prod installations. However, no truncation happens in my dev environment. One difference I see there is that my dev env has "latin1" for four mysql settings (below) that are all utf8 in beta and prod. Is this the problem? Below is a simplified sample language pack that I've tested with.

character_set_client	latin1
character_set_connection	latin1
character_set_database	utf8
character_set_filesystem	binary
character_set_results	latin1
character_set_server	latin1
character_set_system	utf8

<?xml version="1.0" encoding="ISO-8859-1"?>
<language name="BHItaliano (BH)" vbversion="4.1.8" product="vbulletin" type="custom">
<settings>
<options><![CDATA[3]]></options>
<languagecode><![CDATA[bh]]></languagecode>
<charset><![CDATA[ISO-8859-1]]></charset>
<imagesoverride />
<dateoverride />
<timeoverride />
<registereddateoverride />
<calformat1override />
<calformat2override />
<logdateoverride />
<locale />
<decimalsep><![CDATA[.]]></decimalsep>
<thousandsep><![CDATA[,]]></thousandsep>
<phrasegroupinfo><![CDATA[a:1:{s:8:"ckeditor";a:1:{s:11:"maxdateline";s:10:"1321140706";}}]]></phrasegroupinfo>
</settings>
<phrasetype name="Advertising" fieldname="advertising">
<phrase name="ad_active_description" date="1260639964" username="Darkshenron" version="4.0.0 Release Candidate 2"><![CDATA[Se è attiva e nessun criterio è stato impostato, la pubblicità verrà mostrata a tutti.]]></phrase>
</phrasetype>
</language>

[BrianHoltz]

Answering my own question: the truncation problem is fixed by 1) translating the file via 'iconv -f iso-8859-1 -t utf-8' and 2) editing the encoding and charset to be "UTF-8".

This fixed my DE FR ES language packs. However, for IT and TR some of the special characters are garbled. In the case of IT, the truncation problem wasn't fixed by the above, and I tried converting from ISO-8859-2 instead. That solved truncation, but left me with garbling. Maybe the language packs I have just can't work with UTF-8 at all?

[Zachery]

If you want to use UTF8, everything needs to be utf8. Everything needs to match the first time around. You have data stored in another format, and you're not converting it voer, just changing the way its stored.

You may need to undo your changes, export the database, and then reimporting it making sure the system is set to use it as UTF8. You might also need to do data conversion at some point.