Announcement

Collapse
No announcement yet.

Error when saving a module with swedish characters in the title

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • pmquist
    replied
    I had the same problem with search and swedish characters. I ended up solving it with a Sphinx search installation and in the setup for Sphinx you could define what characters should be searchable. Works great now.
    I am not able to help with the install since I had Sphinx do it for me, but it works well.

    Leave a comment:


  • Mr B
    replied
    Originally posted by Wayne Luke View Post
    You can't until either VBV-1333 or VBV-17331 is implemented. You can probably use a custom Google Search for your site instead of the built-in vBulletin search.
    The first vbv is from 2012 and the second from 2018. When can we see a solution for this?

    Leave a comment:


  • Mr B
    replied
    Originally posted by Wayne Luke View Post
    You can't until either VBV-1333 or VBV-17331 is implemented. You can probably use a custom Google Search for your site instead of the built-in vBulletin search.
    What??? Can i help you to proceed with the vbvs? This is vital for my site. Google search is not an alternative.

    Leave a comment:


  • Wayne Luke
    replied
    You can't until either VBV-1333 or VBV-17331 is implemented. You can probably use a custom Google Search for your site instead of the built-in vBulletin search.

    Leave a comment:


  • Mr B
    replied
    Originally posted by Wayne Luke View Post
    Search Engine Issue

    https://tracker.vbulletin.com/vbulle...ssues/VBV-1333
    https://tracker.vbulletin.com/vbulle...sues/VBV-17331

    I need the software to fully support UTF-8 before I can push these issues more. That work is currently ongoing. The biggest issue is upgrades because vBulletin 3 and 4 tried to fake UTF-8 support by converting these characters to ASCII HTML Equivalents. So if you're upgrading, we'll need a tool to convert all these entities into actual characters. Not a small endeavor while maintaining database integrity. Every single post as has to be scanned, modified, and resaved.
    But i have done a fresh new install (not an upgrade) in /testvb/ folder to test if old data is the issue. No old data in that test. Only one test post containing åäö i the posts words. The same issue appears there. How can we proceed?

    Leave a comment:


  • Wayne Luke
    replied
    Search Engine Issue

    https://tracker.vbulletin.com/vbulle...ssues/VBV-1333
    https://tracker.vbulletin.com/vbulle...sues/VBV-17331

    I need the software to fully support UTF-8 before I can push these issues more. That work is currently ongoing. The biggest issue is upgrades because vBulletin 3 and 4 tried to fake UTF-8 support by converting these characters to ASCII HTML Equivalents. So if you're upgrading, we'll need a tool to convert all these entities into actual characters. Not a small endeavor while maintaining database integrity. Every single post as has to be scanned, modified, and resaved.

    Leave a comment:


  • Mr B
    replied
    Originally posted by Wayne Luke View Post
    Yes, keep your database as Latin1 and make sure that the new database uses the same character set and collation as the old database. I can't guarantee that you'll be able to use anything other than English though.

    If you had a support contract, we could log into your server and do the conversion for you. It takes about an hour to complete. If you migrate to vBulletin Cloud, we will do the conversion as part of the Migration process.
    I think that we have support contract but we have problems to give telnet access etc.

    I did a clean test install of vBulletin 5.4.4 with utf8mb4 and utf8mb4_swedish_ci and the problem with åäö in module names is gone.

    But, the search function still not working! Actually you have the same problem here on the vBulletin forum. If i post some Swedish fish names here like gädda gös färna etc they will not be searchable. If you exclude the åäö from the searchword like gdda gs frna they will be searchable.

    In the DB table "words" they will appear like gdda gs frna (they missing the åäö). The posts shows the correct characters (the node data seems ok) but the indexing of the search words do not work. Why is the searchindex missing the åäö parts of Swedish words?

    See attached images:

    Leave a comment:


  • Wayne Luke
    replied
    Yes, keep your database as Latin1 and make sure that the new database uses the same character set and collation as the old database. I can't guarantee that you'll be able to use anything other than English though.

    If you had a support contract, we could log into your server and do the conversion for you. It takes about an hour to complete. If you migrate to vBulletin Cloud, we will do the conversion as part of the Migration process.

    Leave a comment:


  • Mr B
    replied
    Originally posted by Wayne Luke View Post
    Yes. You're using UTF-8 characters so you should be using a UTF-8 character set and collation in your database. We recommend using the UTF8MB4 character set and corresponding collation (utf8mb4_general_ci or utf8mb4_swedish_ci in your case). If the vBulletin installer creates the database for you, it will use UTF8MB4 with the utf8mb4_genera_ci collation. UTF8MB4 normalizes the number of bytes per character at 4 each and allows new UTF-8 functionality such as the Emoticons popular on mobile devices.

    The character set controls how characters appear and are sent from the database. The collation affects sort orders when running queries and therefore searches in the software. In addition to the database, you need to set your Language options in the AdminCP to they use UTF-8 (new installs should do this) as opposed to ISO-8859-1 (upgrades from older versions) and you may have to set the Locale to a UTF8 compatible Locale under Languages & Phrases -> Language Manager -> Your Language -> Edit.
    Hmm, it seems to be a VERY complex task to convert the DB, tables and columns from latin1 to utf8mb4. Our forum have been upgraded over a lot of years and latin1 have always worked. Is this solvable in another way? We also have issues searching for words or phrases with åäö in them. All words/phrases with åäö is missing in "words" database table so they cant be searched…

    Is it some easy fix for this? Or can the conversation from latin1 to utf8mb4 be done in some easy way? Please help with this.

    Leave a comment:


  • Mr B
    replied
    Originally posted by Wayne Luke View Post
    I recommend rebuilding the search engine. You should use the searchindex.php file located in the do_not_upload folder. You run it from the command line and let it go. It doesn't need to be monitored and won't time out.

    If that still takes too long, you should look into using Sphinx search. The instructions are in the do_not_upload folder.
    I ran into major issues when trying to make a testvb installation to test the solution for the search problem. Please also take a look at https://www.vbulletin.com/forum/foru...in-site-broken

    I do not dare to make the database changes if i dont have tested it in an test environment before and the search issue is a major thing that must be solved.

    Leave a comment:


  • Wayne Luke
    replied
    I recommend rebuilding the search engine. You should use the searchindex.php file located in the do_not_upload folder. You run it from the command line and let it go. It doesn't need to be monitored and won't time out.

    If that still takes too long, you should look into using Sphinx search. The instructions are in the do_not_upload folder.

    Leave a comment:


  • Mr B
    replied
    Originally posted by Wayne Luke View Post
    Yes. You're using UTF-8 characters so you should be using a UTF-8 character set and collation in your database. We recommend using the UTF8MB4 character set and corresponding collation (utf8mb4_general_ci or utf8mb4_swedish_ci in your case). If the vBulletin installer creates the database for you, it will use UTF8MB4 with the utf8mb4_genera_ci collation. UTF8MB4 normalizes the number of bytes per character at 4 each and allows new UTF-8 functionality such as the Emoticons popular on mobile devices.

    The character set controls how characters appear and are sent from the database. The collation affects sort orders when running queries and therefore searches in the software. In addition to the database, you need to set your Language options in the AdminCP to they use UTF-8 (new installs should do this) as opposed to ISO-8859-1 (upgrades from older versions) and you may have to set the Locale to a UTF8 compatible Locale under Languages & Phrases -> Language Manager -> Your Language -> Edit.
    Thanks! I will put up a testvb tomorrow and test it out. Do you know if i need to rebuild search index afterwards? The process takes around 2 days on my board.

    Leave a comment:


  • Wayne Luke
    replied
    Yes. You're using UTF-8 characters so you should be using a UTF-8 character set and collation in your database. We recommend using the UTF8MB4 character set and corresponding collation (utf8mb4_general_ci or utf8mb4_swedish_ci in your case). If the vBulletin installer creates the database for you, it will use UTF8MB4 with the utf8mb4_genera_ci collation. UTF8MB4 normalizes the number of bytes per character at 4 each and allows new UTF-8 functionality such as the Emoticons popular on mobile devices.

    The character set controls how characters appear and are sent from the database. The collation affects sort orders when running queries and therefore searches in the software. In addition to the database, you need to set your Language options in the AdminCP to they use UTF-8 (new installs should do this) as opposed to ISO-8859-1 (upgrades from older versions) and you may have to set the Locale to a UTF8 compatible Locale under Languages & Phrases -> Language Manager -> Your Language -> Edit.

    Leave a comment:


  • Mr B
    replied
    Originally posted by Wayne Luke View Post
    I cannot recreate this issue using the latest version of vBulletin 5 and a UTF8MB4 database. Can you provide more information including all steps to recreate?
    Click image for larger version

Name:	2018-10-16_11-00-31.png
Views:	66
Size:	118.9 KB
ID:	4400564


    Our database is a "latin1 - default collation" InnoDB running on MySQL 5.7.20, PHP 7.2.7 and IIS 10 running.

    Im adding a search module and call it "Köp & sälj" and try to save it. Then the error message appears.

    Can it be the database collation that is faulty? I also having another åäö problem related to ordinary search. Ive posted in this thread about it:
    https://www.vbulletin.com/forum/foru...dish-charactes

    Maybe related?

    Leave a comment:


  • Wayne Luke
    replied
    I cannot recreate this issue using the latest version of vBulletin 5 and a UTF8MB4 database. Can you provide more information including all steps to recreate?
    Click image for larger version

Name:	2018-10-16_11-00-31.png
Views:	66
Size:	118.9 KB
ID:	4400564



    Leave a comment:

Related Topics

Collapse

Working...
X