Announcement

Collapse
No announcement yet.

strange encoding problem after migrating to new server

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • [Forum] strange encoding problem after migrating to new server

    Hi,
    I've just moved forum to another server.
    Files & database files was copied 1:1 but there is strange behavior...

    When I create new post, everything is ok:

    nooo i nie zapomnij jeszcze o czymś na odwagę
    but when I use quick reply (ajax submitted reply) the same message is saved in database as:

    nooo i nie zapomnij jeszcze o czym& # 347 na odwag & # 243
    ś is being replaced with & # 347
    ó is being replaced with & # 243

    please, can you help me with this very strange behavior???

  • #2
    I could be completely mistaken, but what language is the database written in? UTF-8?

    Comment


    • #3
      Do they display correctly on the forums? Behavior isn't really strange as we don't completely support UTF-8 yet and there has to be some changes to data to maintain display integrity. Encoding characters as HTML Entities is one such method.
      Translations provided by Google.

      Wayne Luke
      The Rabid Badger - a vBulletin Cloud demonstration site.
      vBulletin 5 API - Full / Mobile
      Vote for your favorite feature requests and the bugs you want to see fixed.

      Comment


      • #4
        Do they display correctly on the forums? Behavior isn't really strange as we don't completely support UTF-8 yet and there has to be some changes to data to maintain display integrity. Encoding characters as HTML Entities is one such method.

        they are displayed in same way - first (new thread text) is ok. reply text - malformed...

        ok, but on the old server everything was fine...
        where should i look?

        database is in latin2_general_ci as it allways was....
        on old server was ok - after movement this is very strange...

        like I said - in new thread encoding is ok.
        when I use reply (AJAX-based) the same texts are malformed :/

        maybe it's php/mysql connection settings? I can insert here mysql & php settings from old & new server (both are working now)

        btw: you can see it here: http://forum.vw-passat.pl/threads/114619-test
        login: test password: tester123

        Comment


        • #5
          What is the character encoding for your language?

          Languages & Phrases -> Language Manager.

          Click the Edit link for your language.

          Do you have multiple languages on your forums?
          Translations provided by Google.

          Wayne Luke
          The Rabid Badger - a vBulletin Cloud demonstration site.
          vBulletin 5 API - Full / Mobile
          Vote for your favorite feature requests and the bugs you want to see fixed.

          Comment


          • #6
            only one language - the same as on old server.
            Click image for larger version

Name:	lang.jpg
Views:	1
Size:	34.5 KB
ID:	3686551

            i found some differences, on old server i have locale -a (debian):
            Code:
            C
            C.UTF-8
            en_US.utf8
            pl_PL
            pl_PL.iso88592
            polish
            POSIX
            new server locale -a (ubuntu)
            Code:
            C
            en_AG
            en_AU.utf8
            en_BW.utf8
            en_CA.utf8
            en_DK.utf8
            en_GB.utf8
            en_HK.utf8
            en_IE.utf8
            en_IN
            en_NG
            en_NZ.utf8
            en_PH.utf8
            en_SG.utf8
            en_US.utf8
            en_ZA.utf8
            en_ZW.utf8
            POSIX
            this may be the cause??

            Comment


            • #7
              i've just upgraded to 4.1.10 ut nothing changed

              Comment


              • #8
                This isn't a situation that upgrading wouldn't fix.

                The change in locale and hence a change in encoding can cause this. Unfortunately, without manual updating I don't have a quick fix for you as you're on different operating systems. You can attempt to convert the old database to UTF-8 on a debian server and import it as UTF-8 on the new server. Then you have to coax vBulletin into using UTF-8 by editing the config.php file, your language encoding and using the MySQLi class library. You can't convert on your new server though because it would just try to convert the HTML entities.
                Translations provided by Google.

                Wayne Luke
                The Rabid Badger - a vBulletin Cloud demonstration site.
                vBulletin 5 API - Full / Mobile
                Vote for your favorite feature requests and the bugs you want to see fixed.

                Comment


                • #9
                  hmm... but forum is running > 24h in new server and there is about 1k new posts...
                  can you tell me what should cause this type of problems?

                  Comment


                  • #10
                    Originally posted by jedrus View Post
                    hmm... but forum is running > 24h in new server and there is about 1k new posts...
                    can you tell me what should cause this type of problems?
                    Yes. As I said in the first post, vBulletin doesn't completely support UTF-8. Since your local changed, letters can be represented differently. In order to try to maintain the display, unrecognized letters can be changed to HTML Entities. It doesn't necessarily work all the time but works most of the time. Until vBulletin supports UTF-8 on a 100% level or you started your forum under UTF-8, you can have problems like this when you move servers.
                    Translations provided by Google.

                    Wayne Luke
                    The Rabid Badger - a vBulletin Cloud demonstration site.
                    vBulletin 5 API - Full / Mobile
                    Vote for your favorite feature requests and the bugs you want to see fixed.

                    Comment


                    • #11
                      Ok. I just never had problems with the forum encoding and I can't understand it. But can you, please, explain to me one more thing:
                      I've just set option "disable AJAX...." to "Disable all AJAX functions" and everything is ok - texts are fine. But this increases the server load.

                      Comment


                      • #12
                        AJAX processes submissions through Javascript and can do so without reloading the page. This allows the text to be processed in the PHP and has better chances of being presented correctly. Turning off AJAX results in higher server loads because it reloads pages and all the HTML contained in them.
                        Translations provided by Google.

                        Wayne Luke
                        The Rabid Badger - a vBulletin Cloud demonstration site.
                        vBulletin 5 API - Full / Mobile
                        Vote for your favorite feature requests and the bugs you want to see fixed.

                        Comment


                        • #13
                          Huh. I know how it works... But why turning of AJAX functions causes encoding to works fine? Where's the trick? Classic form submission encodes chars correctly - Ajax submission causes errors...

                          Comment


                          • #14
                            Like I said, we don't completely support UTF-8 at the moment. It could be that the AJAX and Javascript methods need to convert it so it gets submitted correctly. Within PHP we have access to things like ICONV that allow us to convert things if needed.
                            Translations provided by Google.

                            Wayne Luke
                            The Rabid Badger - a vBulletin Cloud demonstration site.
                            vBulletin 5 API - Full / Mobile
                            Vote for your favorite feature requests and the bugs you want to see fixed.

                            Comment


                            • #15
                              ok, because I really have to solve this problem I browsed the vbulletin source code step by step... and I think the problem is in function convert_urlencoded_unicode.
                              encoded value of "ó" is:
                              Code:
                              %u00F3
                              (fiddler & firebug says that)

                              in old server, this line:

                              Code:
                              $return = preg_replace(
                              		'#%u([0-9A-F]{1,4})#ie',
                              		"convert_unicode_char_to_charset(hexdec('\\1'), \$charset)",
                              		$text
                              	);
                              returns:
                              Code:
                              ó
                              in new server, same line for same string returns:
                              Code:
                              
                              
                              yes - empty string/null or sth

                              i think the problem is maybe here:
                              Code:
                              convert_unicode_char_to_charset
                              
                              when doing:
                              Code:
                              $output = @iconv('UTF-8', $charset, convert_int_to_utf8($unicode_int));
                              the test is failing and this function returns:
                              Code:
                              return "&#$unicode_int;";
                              and it saves to database in this stupid way.

                              please, tell me if usage of iconv can cause this strange behavior?

                              on old server(PHP Version 5.3.9-1) iconv is:
                              Code:
                              iconv
                              iconv support enabled
                              iconv implementation glibc
                              iconv library version 2.13
                              iconv.input_encoding ISO-8859-1 ISO-8859-1
                              iconv.internal_encoding ISO-8859-1 ISO-8859-1
                              iconv.output_encoding ISO-8859-1 ISO-8859-1
                              on new server (PHP version 5.3.2-1) iconv is:
                              Code:
                              iconv
                              iconv support enabled
                              iconv implementation glibc
                              iconv library version 2.11.1
                              iconv.input_encoding ISO-8859-1 ISO-8859-1
                              iconv.internal_encoding ISO-8859-1 ISO-8859-1
                              iconv.output_encoding ISO-8859-1 ISO-8859-1

                              maybe there are some known issues? please, help me with this irritating error

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...
                              X