Announcement

Collapse
No announcement yet.

New wave of database errors, now MySQL is offline

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • New wave of database errors, now MySQL is offline

    This past Friday we got a single database error notification from vBulletin. Our admittedly novice deciphering of it was that there was just a timeout. Today there have been 8 more errors, leading up to MySQL crashing and our site going offline. MySQL will not restart because it is crashing while rolling back transactions.

    We were bit hard by the exploits fixed in 5.6.1 PL1 because we did not upgrade immediately from 5.6.1, and the first signs of us having been exploited were similar database errors, so we are paranoid about what may be going on here. For the time being, we have taken all of vBulletin offline to prevent any malicious activity.

    We would greatly appreciate any and all assistance in determining what is going on with our server. The attached text file includes the full text of the database errors. I removed specific URLs from the errors, but they are otherwise unaltered (the missing line number at the end of the stack is in fact missing from the originals.)

    We're running vBulletin 5.6.1 PL1
    PHP 7.3.16
    MySQL 5.6.47

    vBulletinDatabaseErrors.txt

    Thank you for any help you can give.

  • #2
    Originally posted by Wayne Luke
    We have absolutely no way of helping without knowing what the actual error is.
    That makes perfect sense, which is why I attached the errors the errors that vBulletin recorded in my initial post. Is there something else specifically that you'd like me to supply?

    Comment


    • #3
      The queries are just basic caching queries. Your server can't handle them for some reason. Clearing the system cache in the ADminCP or truncating the cache and cacheevent tables may clear up the issue temporarily. Without information on the configuration of the server, it is difficult to know why. This is why we ask that these questions be answered in every single support request:

      https://forum.vbulletin.com/forum/vb...esting-support

      Your version of MySQL is pretty low by today's standards though.

      In addition to those questions, what TABLE ENGINE is every single table in your database using?

      P.S., when attaching non-image files, they will be more noticeable if they are not inserted inline.
      Translations provided by Google.

      Wayne Luke
      The Rabid Badger - a vBulletin Cloud demonstration site.
      vBulletin 5 API - Full / Mobile
      Vote for your favorite feature requests and the bugs you want to see fixed.

      Comment


      • #4
        Originally posted by Wayne Luke View Post
        The queries are just basic caching queries. Your server can't handle them for some reason. Clearing the system cache in the ADminCP or truncating the cache and cacheevent tables may clear up the issue temporarily.
        The errors clearly state the SQL is invalid. These are extremely similar to the invalid SQL errors generated by the injection attacks against 5.6.1 that lead to our site being compromised. Since vBulletin staff knows your database schema and SQL far better than we could ever hope to, we need you to look at these more closely.

        Originally posted by Wayne Luke View Post
        Without information on the configuration of the server, it is difficult to know why. This is why we ask that these questions be answered in every single support request:

        https://forum.vbulletin.com/forum/vb...esting-support
        .
        I saw that topic before posting and I provided 1 through 3 from that list. The others do not seem relevant, particularly the browser and template questions. #11 is not possible because MySQL is offline due to whatever happened. But I will do as you request:
        1. vBulletin Version - Viewable in the header of the AdminCP. 5.6.1 patchlevel 1
        2. PHP Version - Viewable on the AdminCP Home Page. 7.3.16
        3. MySQL Version - Viewable on the AdminCP Home Page. 5.6.47
        4. Any Addons installed. None
        5. Does the issue occur in a default style? Not display related
        6. Does the issue occur using the English language provided? We use English only
        7. Error message on the screen. None. Database errors are being emailed to us
        8. Browser and Browser version used. N/A
        9. Did you clear the browser cache and did the error continue? N/A
        10. A list of steps that can be used to recreate the issue. Unknown, we are asking for help in determining what is generating these errors
        11. Output of the error with vBulletin in Debug Mode. Unavailable
        12. If the issue is an Invalid Server (500 server error) response, the web server and PHP logs that correspond with its timestamp. N/A


        Originally posted by Wayne Luke View Post
        Your version of MySQL is pretty low by today's standards though.

        In addition to those questions, what TABLE ENGINE is every single table in your database using?

        P.S., when attaching non-image files, they will be more noticeable if they are not inserted inline.
        All tables are InnoDB.

        I only know how to insert inline.

        Comment


        • #5
          Originally posted by AlphaSoftware View Post
          The errors clearly state the SQL is invalid. These are extremely similar to the invalid SQL errors generated by the injection attacks against 5.6.1 that lead to our site being compromised. Since vBulletin staff knows your database schema and SQL far better than we could ever hope to, we need you to look at these more closely.
          Actually the error is:

          MySQL Error : MySQL server has gone away
          Error Number : 2006
          Request Date : Friday, June 19th 2020 @ 08:58:39 PM
          Error Date : Friday, June 19th 2020 @ 08:58:40 PM
          Script : <URL 1>
          Referrer :
          IP Address : 46.229.168.129
          Username : Guest
          Classname : vB_Database_MySQLi
          MySQL Version :
          It is the exact same error for every single query. This specifically means that the connection to MySQL was closed before a result was returned. This is often due to the server being overloaded on shared hosting providers or not being able to handle the query do to configuration reasons.


          I saw that topic before posting and I provided 1 through 3 from that list. The others do not seem relevant, particularly the browser and template questions. #11 is not possible because MySQL is offline due to whatever happened. But I will do as you request:
          1. vBulletin Version - Viewable in the header of the AdminCP. 5.6.1 patchlevel 1
          2. PHP Version - Viewable on the AdminCP Home Page. 7.3.16
          3. MySQL Version - Viewable on the AdminCP Home Page. 5.6.47
          4. Any Addons installed. None
          5. Does the issue occur in a default style? Not display related
          6. Does the issue occur using the English language provided? We use English only
          7. Error message on the screen. None. Database errors are being emailed to us
          8. Browser and Browser version used. N/A
          9. Did you clear the browser cache and did the error continue? N/A
          10. A list of steps that can be used to recreate the issue. Unknown, we are asking for help in determining what is generating these errors
          11. Output of the error with vBulletin in Debug Mode. Unavailable
          12. If the issue is an Invalid Server (500 server error) response, the web server and PHP logs that correspond with its timestamp. N/A
          Still troubleshooting steps. The App is a client/server app and a lot of information is stored within your browser that can cause additional stress on the page if it is corrupted somehow.


          All tables are InnoDB.
          This could very well be the problem. On your version of MySQL, the language, phrases, and usertextfield tables should never be INNODB. We simply have no idea what content is stored in these tables after installation. Though, I suspect that the problem is that your CACHE tables are larger than MySQL is configured to handle due to limitations. In most circumstances, MySQL should be configured to consume 80% of the server's resources. This is rarely the case.

          I only know how to insert inline.
          Don't click the "Insert" link when you upload the file using the Attachment in the Attachment panel.
          Attached Files
          Translations provided by Google.

          Wayne Luke
          The Rabid Badger - a vBulletin Cloud demonstration site.
          vBulletin 5 API - Full / Mobile
          Vote for your favorite feature requests and the bugs you want to see fixed.

          Comment


          • #6
            Do the errors go away if you clear the system cache as stated in my post above?
            Translations provided by Google.

            Wayne Luke
            The Rabid Badger - a vBulletin Cloud demonstration site.
            vBulletin 5 API - Full / Mobile
            Vote for your favorite feature requests and the bugs you want to see fixed.

            Comment


            • #7
              I will look closer at the table types. I don't have a functioning database instance at the moment so I did a quick scan of a semi-recent backup. I will make sure I double check language, phrases, and usertextfield in particular.

              I will also review the resources allocated to MySQL. This is a dedicated server, not a shared system, so I assume that we have relatively high resources avaialble, but I cannot say for sure at this time.

              Comment


              • #8
                Originally posted by Wayne Luke View Post
                Do the errors go away if you clear the system cache as stated in my post above?
                I can't currently start MySQL. It crashes while trying to roll back a transaction. I am working on restroing a backup of the database, then I can clear the system cache.

                Comment


                • #9
                  I'm sorry about the bad info on the table types. All I have to work with at the moment is a MySQL dump with create table statements so it is easier for me to check specific tables than to tell you about "all" tables. I had looked at the first few and saw only InnoDB previously.


                  language is MyISAM
                  phrase is MyISAM
                  usertextfield is MyISAM

                  Comment


                  • #10
                    We had to restrore a backup oif MySQL to get past a transaction that it could not roll back. We have now cleared the system cache. Unfortunately since I do not know what provokes this error and you were not able to tell either, I guess all we can do is wait to see if it happens again.

                    Comment


                    • #11
                      We've now upgraded MySQL to 5.7.30. We need to do some more testing before moving to 8. Are there any specific vBulletin related issues with moving to 8?

                      We will also be upgrading vBulletin to 5.6.2 tonight.

                      Comment


                      • #12
                        Just the password issue listed in this article: https://forum.vbulletin.com/articles...best-practices. Though I think it may be resolved in newer versions of PHP (PHP 7.3+). I haven't checked since the password method outlined in the article works.
                        Translations provided by Google.

                        Wayne Luke
                        The Rabid Badger - a vBulletin Cloud demonstration site.
                        vBulletin 5 API - Full / Mobile
                        Vote for your favorite feature requests and the bugs you want to see fixed.

                        Comment

                        Related Topics

                        Collapse

                        Working...
                        X