Announcement

Collapse
No announcement yet.

Truncated posts and intermittent garbage characters (possibly UTF8 problem)

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • alfreema
    replied
    Originally posted by Wayne Luke View Post
    The HTTP header shouldn't matter as long as it says UTF-8 in the code itself. What is the character causing the truncation?
    Is it possible that clearing the CACHE table could have fixed this? Is it possible that the CACHE or CACHEEVENT table had a hardcoded reference to ISO-8859-1? That's the only thing I have done since before testing when it wasn't working.

    Leave a comment:


  • alfreema
    replied
    OMG ... I just copied a string in that was broken, and it worked. So I quickly viewed the headers and it's saying UTF-8. And I didn't do anything since giving up and researching on this issue yesterday.

    Leave a comment:


  • Wayne Luke
    replied
    The HTTP header shouldn't matter as long as it says UTF-8 in the code itself. What is the character causing the truncation?
    Last edited by Wayne Luke; Fri 20th Oct '17, 9:52am.

    Leave a comment:


  • alfreema
    replied
    Ok I searched and nothing came back. I am starting to think that the mysql export and import process may be causing this. We did not explicitly set any charsets when we exported/imported so it's quite possible that that has put our database in a weird state.

    Leave a comment:


  • plongeur.com
    replied
    MAybe you should search "ISO-8859-1" your template to make sure it is not hardcoded somewhere.

    Leave a comment:


  • alfreema
    replied
    Whoa. When I hit the https://shockernet.net/admincp/ the response comes back UTF-8! There is just something lingering from the vb4 to vb5 upgrade causing the ISO-8859-1 to UTF-8 setting to not work on the main site, though it does appear to work in AdminCP.

    Leave a comment:


  • alfreema
    replied
    I exported the production database that is not working properly and imported it into the test VM that was working properly, in order to see if the problem is some setting or something in the database. Sure enough, after importing the database and running the tools.php to change the site URL and cookies, the ISO-8859-1 problem has moved with the database. So while mysql is showing everything is set to utf at the table and field levels, I am guessing there is some field in one of the tables that is tripping vBulletin5 up. And that field is still set to ISO-8859-1 because the old database was set that way (so it wasn't overwritten in the upgrade process). Any ideas what tables I could look in?

    Leave a comment:


  • alfreema
    replied
    I installed a brand new install of vb5 (no upgrade) on a brand new bitnami lampstack. IT was UTF-8 by default. When I set HTML Language Charset to ISO-8859-1, it indeed start spitting out ISO-8859-1 like you would expect. Then I set it back to UTF-8, and it started spitting out UTF-8 just like you would expect. So the behavior of a new installation is different than an upgrade from 4.1.7 (which was originally ISO-8859-1). So the new database is utf8, but setting the HTML Language Charset doesn't work after an upgrade, but it does if it's a brand new install.

    Leave a comment:


  • alfreema
    replied
    Originally posted by Wayne Luke View Post
    What is the code in your header template?
    If you are referring to the test code I wrote to hit the database directly, there is no header template. It's the simple code I pasted in this post.

    If you are referring to my vb 5.3.3 headers, where would I look? (which file do you want me to dump?)

    Leave a comment:


  • Wayne Luke
    replied
    What is the code in your header template?

    Leave a comment:


  • alfreema
    replied
    Originally posted by plongeur.com View Post
    Well, if i am on the right site, it keeps sendig "<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">"
    So i guess there is an issue somewhere.
    Yes, you are on the right site. Nothing I do can switch the charset from ISO-8859-1 to UTF-8 when I call a vBulletin PHP. If I create my own PHP it works perfectly. For example hit this:

    https://shockernet.net/hello.php

    That shows UTF-8. It's a PHP I wrote.

    Then try this:

    https://shockernet.net/a.php

    That shows UTF-8 and it's a PHP I wrote that hits a table in the vBulletin database.

    So I can rule out database problem and PHP/Apache problem. Something inside of vBulletin is forcing ISO-8859-1, but it isn't the HTML Language Charset, because as you can see I have it set to UTF-8.

    Leave a comment:


  • plongeur.com
    replied
    Well, if i am on the right site, it keeps sendig "<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">"
    So i guess there is an issue somewhere.

    Leave a comment:


  • alfreema
    replied
    Originally posted by plongeur.com View Post
    As we already said, the two stuff making the charset are :
    - the html language charset setting in the language options
    - the $config['Mysqli']['charset'] in the php.config file

    I took the liberty of going on your site, the text encoding is still latin1 as the screenshot state.
    This means the html language charset setting is not set to UTF-8 (or not taken into account).

    Click image for larger version

Name:	19-10-2017 16-42-01.jpg
Views:	88
Size:	113.8 KB
ID:	4379394
    Right, that's exactly the symptom. So far, nothing I have changed makes any difference. Maybe I am misunderstanding something you want me to change? Here are my current language settings and config.php settings:

    Click image for larger version

Name:	language_settings.PNG
Views:	84
Size:	56.4 KB
ID:	4379396
    Note: I also tried en.UTF-8 in the HTML Character Set, but nothing changed.


    Click image for larger version

Name:	config_php.PNG
Views:	82
Size:	13.9 KB
ID:	4379397

    Did I screw something up or misunderstand something you want me to change?

    Leave a comment:


  • plongeur.com
    replied
    As we already said, the two stuff making the charset are :
    - the html language charset setting in the language options
    - the $config['Mysqli']['charset'] in the php.config file

    I took the liberty of going on your site, the text encoding is still latin1 as the screenshot state.
    This means the html language charset setting is not set to UTF-8 (or not taken into account).

    Click image for larger version

Name:	19-10-2017 16-42-01.jpg
Views:	88
Size:	113.8 KB
ID:	4379394

    Leave a comment:


  • alfreema
    replied
    I just did another test. I created this test.php file:

    PHP Code:
     <?php
    $servername 
    "localhost";
    $username "xxx";
    $password "xxxx";
    $dbname "xxx";

    // Create connection
    $conn mysqli_connect($servername$username$password$dbname);
    // Check connection
    if (!$conn) {
        die(
    "Connection failed: " mysqli_connect_error());
    }

    $sql "SELECT * FROM userlist";
    $result mysqli_query($conn$sql);

    if (
    mysqli_num_rows($result) > 0) {
        
    // output data of each row
        
    while($row mysqli_fetch_assoc($result)) {
            echo 
    "id: " $row["userid"]. " - Name: " $row["type"]. " " $row["friend"]. "<br>";
        }
    } else {
        echo 
    "0 results";
    }

    mysqli_close($conn);
    ?>
    ... and when I ran it I got the following output ...

    PHP Code:
    HTTP/1.1 200 OK
    Date
    Thu19 Oct 2017 13:48:50 GMT
    Server
    Apache
    X
    -Frame-OptionsSAMEORIGIN
    X
    -Powered-ByPHP/7.0.23
    Vary
    Accept-Encoding
    Connection
    keep-alive
    Transfer
    -Encodingchunked
    Content
    -Typetext/htmlcharset=UTF-8

     id
    384 Nameignore no<br>id58 Namefollow yes<br
    So the plot thickens. Hitting mysql with PHP and leaving vBulletin out of the loop, I get the proper result. The only time I get ISO-8859-1 is when I call vBulletin5 PHPs. So I think I can now definitively say that there is something specific to vBulletin 5 causing the default UTF-8 charset to be overwritten with ISO-8859-1.

    Any hints on where to look for that?

    Leave a comment:

Related Topics

Collapse

  • -=|zami|=-
    default UTF-8
    by -=|zami|=-
    friends, please: in admincp>>language & phrases>>language manager>>edit & settings there is the "HTML Character Set" option with the description "This value tells...
    Fri 19th Apr '19, 9:54am
Working...
X