Announcement

Collapse
No announcement yet.

SEVERE 2.2.7 problem - crashed FreeBSD server

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • SEVERE 2.2.7 problem - crashed FreeBSD server

    I installed V2.2.7 tonight on our website. The site has been running stable on V2.2.6 for a long time. I'm sure I uploaded all of the 2.2.7 files and ran upgrade20.php (which executed fine). I finished the install of 2.2.7 around 6:00 pm (started around 5:00). Here's a copy of the uptime.log on my system for today a little while before the upgrade through when it brought the server to it's knees:

    4:00PM up 13 days, 17:57, 1 user, load averages: 1.05, 0.78, 0.49
    4:05PM up 13 days, 18:02, 2 users, load averages: 0.40, 0.70, 0.55
    4:10PM up 13 days, 18:07, 2 users, load averages: 0.36, 0.48, 0.48
    4:15PM up 13 days, 18:12, 2 users, load averages: 0.35, 0.36, 0.41
    4:20PM up 13 days, 18:17, 0 users, load averages: 0.26, 0.34, 0.38
    4:25PM up 13 days, 18:22, 0 users, load averages: 0.33, 0.40, 0.40
    4:30PM up 13 days, 18:27, 0 users, load averages: 0.29, 0.32, 0.35
    4:35PM up 13 days, 18:32, 0 users, load averages: 0.32, 0.26, 0.30
    4:40PM up 13 days, 18:37, 0 users, load averages: 0.13, 0.23, 0.27
    4:45PM up 13 days, 18:42, 0 users, load averages: 0.21, 0.18, 0.23
    4:50PM up 13 days, 18:47, 0 users, load averages: 0.10, 0.17, 0.21
    4:55PM up 13 days, 18:52, 0 users, load averages: 0.29, 0.23, 0.21
    5:00PM up 13 days, 18:57, 0 users, load averages: 0.06, 0.15, 0.18
    5:05PM up 13 days, 19:02, 0 users, load averages: 0.45, 0.26, 0.20
    5:10PM up 13 days, 19:07, 0 users, load averages: 0.57, 0.40, 0.27
    5:15PM up 13 days, 19:12, 0 users, load averages: 0.19, 0.33, 0.27
    5:20PM up 13 days, 19:17, 0 users, load averages: 0.70, 0.45, 0.32
    5:25PM up 13 days, 19:22, 0 users, load averages: 0.28, 0.44, 0.35
    5:30PM up 13 days, 19:27, 0 users, load averages: 0.34, 0.42, 0.36
    5:35PM up 13 days, 19:32, 0 users, load averages: 0.48, 0.44, 0.39
    5:40PM up 13 days, 19:37, 0 users, load averages: 0.36, 0.48, 0.42
    5:45PM up 13 days, 19:42, 0 users, load averages: 0.45, 0.43, 0.40
    5:50PM up 13 days, 19:47, 0 users, load averages: 0.43, 0.41, 0.40
    5:55PM up 13 days, 19:52, 0 users, load averages: 0.54, 0.43, 0.41
    6:00PM up 13 days, 19:57, 0 users, load averages: 1.08, 0.64, 0.49
    6:05PM up 13 days, 20:02, 0 users, load averages: 0.98, 0.76, 0.58
    6:10PM up 13 days, 20:07, 0 users, load averages: 1.24, 0.95, 0.70
    6:15PM up 13 days, 20:12, 0 users, load averages: 1.00, 0.83, 0.69
    6:20PM up 13 days, 20:17, 1 user, load averages: 1.44, 1.07, 0.83
    6:25PM up 13 days, 20:22, 0 users, load averages: 0.41, 0.91, 0.85
    6:30PM up 13 days, 20:27, 0 users, load averages: 0.33, 0.60, 0.72
    6:35PM up 13 days, 20:32, 0 users, load averages: 1.09, 0.78, 0.76
    6:40PM up 13 days, 20:37, 0 users, load averages: 0.96, 0.77, 0.75
    6:45PM up 13 days, 20:42, 0 users, load averages: 0.48, 0.59, 0.66
    6:50PM up 13 days, 20:47, 0 users, load averages: 0.55, 0.63, 0.65
    6:55PM up 13 days, 20:52, 0 users, load averages: 1.11, 0.81, 0.71
    7:00PM up 13 days, 20:57, 0 users, load averages: 2.12, 1.30, 0.94
    7:05PM up 13 days, 21:02, 0 users, load averages: 0.28, 0.97, 0.92
    7:10PM up 13 days, 21:07, 0 users, load averages: 0.79, 0.81, 0.86
    7:15PM up 13 days, 21:12, 0 users, load averages: 0.64, 0.84, 0.86
    7:20PM up 13 days, 21:17, 0 users, load averages: 1.31, 1.09, 0.95
    7:25PM up 13 days, 21:22, 0 users, load averages: 0.77, 0.81, 0.85
    7:30PM up 13 days, 21:27, 0 users, load averages: 1.99, 1.44, 1.11
    7:35PM up 13 days, 21:32, 0 users, load averages: 1.45, 1.38, 1.16
    7:40PM up 13 days, 21:37, 0 users, load averages: 0.88, 1.15, 1.12
    7:45PM up 13 days, 21:42, 0 users, load averages: 1.00, 1.01, 1.05
    7:50PM up 13 days, 21:47, 0 users, load averages: 1.24, 1.14, 1.09
    7:55PM up 13 days, 21:52, 0 users, load averages: 1.31, 1.28, 1.17
    8:00PM up 13 days, 21:57, 0 users, load averages: 0.82, 1.20, 1.19
    8:05PM up 13 days, 22:02, 0 users, load averages: 0.96, 1.08, 1.13
    8:10PM up 13 days, 22:07, 0 users, load averages: 0.76, 1.10, 1.13
    8:15PM up 13 days, 22:12, 0 users, load averages: 2.06, 1.25, 1.14
    8:20PM up 13 days, 22:17, 0 users, load averages: 1.07, 1.21, 1.16
    8:25PM up 13 days, 22:22, 1 user, load averages: 1.01, 1.09, 1.12
    8:30PM up 13 days, 22:27, 1 user, load averages: 1.00, 1.03, 1.07
    8:35PM up 13 days, 22:32, 1 user, load averages: 4.29, 4.05, 2.64

    Previously we have never seen a server load above 2.0 and it's an extremely rare event to see it over 1.0 (usually only when I'm copying files around, backing up the database, etc). For now I've restored the 2.2.6 programs (although the templates still show 2.2.7 because I didn't push those back yet). Any ideas on what the problem may be? Has anyone else seen this with 2.2.7?
    -Steve St.Laurent
    Webmaster of http://www.turbodieselregister.com

  • #2
    System is working fine now with the 2.2.6 code back on it. Here's my current uptime.log since going back to the 2.2.6 code:

    10:00PM up 16 mins, 1 user, load averages: 0.62, 0.54, 0.33
    10:05PM up 21 mins, 1 user, load averages: 0.15, 0.37, 0.31
    10:10PM up 26 mins, 1 user, load averages: 0.27, 0.32, 0.30
    10:15PM up 31 mins, 0 users, load averages: 0.25, 0.33, 0.32
    10:20PM up 36 mins, 0 users, load averages: 0.71, 0.47, 0.37
    10:25PM up 41 mins, 0 users, load averages: 0.47, 0.42, 0.36
    10:30PM up 46 mins, 1 user, load averages: 0.30, 0.36, 0.33
    10:35PM up 51 mins, 0 users, load averages: 0.35, 0.36, 0.33
    10:40PM up 56 mins, 0 users, load averages: 0.20, 0.39, 0.36
    10:45PM up 1:01, 0 users, load averages: 0.66, 0.56, 0.44
    10:50PM up 1:06, 0 users, load averages: 0.29, 0.45, 0.41
    10:55PM up 1:11, 1 user, load averages: 0.40, 0.41, 0.40
    11:00PM up 1:16, 1 user, load averages: 0.65, 0.49, 0.43
    11:05PM up 1:21, 1 user, load averages: 0.50, 0.46, 0.42
    11:10PM up 1:26, 1 user, load averages: 0.53, 0.46, 0.42
    11:15PM up 1:31, 1 user, load averages: 0.52, 0.40, 0.40
    11:20PM up 1:36, 1 user, load averages: 0.58, 0.50, 0.43

    BTW, there are 222 users on the forums right now and the peak it was hitting when the site began to crash with 2.2.7 was around 140 users.
    -Steve St.Laurent
    Webmaster of http://www.turbodieselregister.com

    Comment


    • #3
      Anyone? I really want to get 2.2.7 installed to see if it helps out with the login issues that MANY of my users are having (I spent an 45 minutes on the phone with one of my users using compuserve today and tried everything and still can't get him logged on).
      -Steve St.Laurent
      Webmaster of http://www.turbodieselregister.com

      Comment


      • #4
        Honestly I don't see any reason why 2.2.7 would cause these server issues. Nor do I understand why your server was brought 'to it's knees' with those server loads. Except for the last entry they aren't that different from the 2.2.6 server loads.

        I'm afraid you'll have to wait for one of the Developers to look at this for you.
        Steve Machol, former vBulletin Customer Support Manager (and NOT retired!)
        Change CKEditor Colors to Match Style (for 4.1.4 and above)

        Steve Machol Photography


        Mankind is the only creature smart enough to know its own history, and dumb enough to ignore it.


        Comment


        • #5
          I agree, I can't see any reason why 2.2.7 would cause this either - I just know it did in my case. If you look at the loads on the server you'll see that the highest load was .71 (first column - that seems to be the one to peak first) since rolling back to 2.2.6 and that the user load was higher than the period after the upgrade. Between 6:00 and 8:35 there were 26 periods where the load was above .71 (or 81% of that time) and the user loads were lower during all of those times. The server was "brought to it's knees" at the 8:30 time period. I had users that complained about slow downs (which were probably around those 2.0+ numbers) but it completely shut down the site at the 8:30 time frame. Foretunately I was there at that time to see what was going on and jump on it. This is on a dual CPU server BTW, so fully loaded on that server is 2.0 - beyond that is stuff backing up.

          I'm going to be in Atlanta where the servers are hosted (I'm in Michigan) on thursday and plan on trying 2.2.7 again at that point (starting with fresh code to be sure there is nothing in common) to see what happens. That way I can reboot the servers myself as soon as I see things going astray. BTW, for a little more detail when the server loads went crazy and the site went down around 8:30 I was still able to ssh into the server. I shut down Mysql and Apache and the server loads still stayed high on the second and third columns, restarting them still would not bring the site back up. I rebooted the server and rolled back the software to 2.2.6 and got everything back up.

          -Steve
          -Steve St.Laurent
          Webmaster of http://www.turbodieselregister.com

          Comment


          • #6
            I notice a very very slight increase in server loads too - hard to know what the cause is though in case... especially because my board is so hacked. Nothing to cause a server crash though.
            Avatar Chat

            Comment


            • #7
              I don't see any changes to the main files (sessions.php, global.php, index.php, forumdisplay.php and showthread.php) that would cause any load increases. What you can do is put 2.2.7 back on and then replace the above files, one at a time, with the 2.2.6 versions to see what your load looks like over an acceptable time frame.

              Comment


              • #8
                have you at all checked your server apache, and mysql logs for any indications of warnings or error messages ?

                what versions of apache/php and mysql are you using with FreeBSD ?
                :: Always Back Up Forum Database + Attachments BEFORE upgrading !
                :: Nginx SPDY SSL - World Flags Demo [video results]
                :: vBulletin hacked forums: Clean Up Guide for VPS/Dedicated hosting users [ vbulletin.com blog summary ]

                Comment


                • #9
                  can you please provide the following

                  1. your server specs, such as mysql and php version
                  2. if possible how mysql was compiled/installed
                  3. your top stats
                  4. your mysql configuration variables located at /etc/my.cnf or c:\my.cnf if on Windows server if you don't have that file you need to log into telnet and as root user type

                  mysqladmin -u root -p variables

                  copy and paste output here

                  5. your mysql extended-status output either still telnet as root user type

                  mysqladmin -u root -p extended-status

                  copy and paste output here

                  or preferred is to installed extended-status output script which is either located

                  - in your vB 2.2.6 zip file's extra's folder, upload mysqlinfo.php script to your site or if you're on an pre vB 2.2.6 install go to
                  - http://vbulletin.com/forum/showthread.php?threadid=3477 and install that scrip making sure to edit $mysqllogin line with your own mysqlusername and password

                  and post url to that here

                  6. oh and is your vB the only thing on the server? or other scripts? sites?

                  7. how many average and max concurrent users on your vB forum ?

                  8. create a file named phpinfo.php and place this code in it and post the url/link to it from your web site

                  <?
                  phpinfo();
                  ?>

                  i.e. yourdomain.com/phpinfo.php

                  9. if you run Apache and you have your own dedicated server or access to your httpd.conf (apache configuration file) can you post the values you have set for the following :

                  KeepAlive
                  MaxKeepAliveRequests
                  KeepAliveTimeout
                  MinSpareServers
                  MaxSpareServers
                  StartServers
                  MaxClients

                  10. what version of vB are you running ?
                  :: Always Back Up Forum Database + Attachments BEFORE upgrading !
                  :: Nginx SPDY SSL - World Flags Demo [video results]
                  :: vBulletin hacked forums: Clean Up Guide for VPS/Dedicated hosting users [ vbulletin.com blog summary ]

                  Comment


                  • #10
                    I think you missed what I said above. Our site is running perfectly with V2.2.6 of the software. The problem only happened when I attempted to upgrade to V2.2.7 and within a couple of hours of installing it - you can see the typical server loads on my second post on this thread (which was with 222 users online running 2.2.6) and the problem loads on the first post (which was with a peak of 140 users running 2.2.7). There was no way to access the site at all through the web. I SSH'd into the server and I shut down MYSQL and Apache and the server loads still stayed sky high, although nothing was shown on the the top report as using a ton of CPU. I eventually had to reboot it, and then I restored the program files back to V2.2.6. The site has been running fine since that time with up to 304 users online simultaneously (window is set to 60 seconds) whereas the server crash running 2.2.7 was with only about 140 users on. Here is the info that you requested.

                    1. MYSQL is Ver 11.15 Distrib 3.23.47, PHP is version 4.1.1, FreeBSD is Ver 4.5

                    2. mysql was installed using FreeBSD's built in ports I believe (was done before I took over)

                    3. Here are the current top stats with 147 users online:

                    last pid: 49744; load averages: 0.47, 0.40, 0.37 up 5+07:45:02 19:15:59
                    96 processes: 1 running, 95 sleeping
                    CPU states: 2.6% user, 0.0% nice, 2.6% system, 0.0% interrupt, 94.7% idle
                    Mem: 422M Active, 421M Inact, 99M Wired, 48M Cache, 112M Buf, 13M Free
                    Swap: 3072M Total, 86M Used, 2986M Free, 2% Inuse

                    4. Here are my mysql configuration variables:
                    +---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
                    | Variable_name | Value |
                    +---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
                    | back_log | 50 |
                    | basedir | /usr/local/ |
                    | bdb_cache_size | 8388600 |
                    | bdb_log_buffer_size | 524288 |
                    | bdb_home | /data/mysql/ |
                    | bdb_max_lock | 10000 |
                    | bdb_logdir | |
                    | bdb_shared_data | OFF |
                    | bdb_tmpdir | /var/tmp/ |
                    | bdb_version | Sleepycat Software: Berkeley DB 3.2.9a: (December 27, 2001) |
                    | binlog_cache_size | 32768 |
                    | character_set | latin1 |
                    | character_sets | latin1 dec8 dos german1 hp8 koi8_ru latin2 swe7 usa7 cp1251 danish hebrew win1251 estonia hungarian koi8_ukr win1251ukr greek win1250 croat cp1257 latin5 |
                    | concurrent_insert | ON |
                    | connect_timeout | 10 |
                    | datadir | /data/mysql/ |
                    | delay_key_write | ON |
                    | delayed_insert_limit | 100 |
                    | delayed_insert_timeout | 300 |
                    | delayed_queue_size | 1000 |
                    | flush | OFF |
                    | flush_time | 0 |
                    | have_bdb | YES |
                    | have_gemini | NO |
                    | have_innodb | DISABLED |
                    | have_isam | YES |
                    | have_raid | NO |
                    | have_openssl | NO |
                    | init_file | |
                    Last edited by Steve St.Lauren; Tue 10 Sep '02, 3:42pm.
                    -Steve St.Laurent
                    Webmaster of http://www.turbodieselregister.com

                    Comment


                    • #11
                      | innodb_additional_mem_pool_size | 1048576 |
                      | innodb_buffer_pool_size | 8388608 |
                      | innodb_data_file_path | |
                      | innodb_data_home_dir | |
                      | innodb_file_io_threads | 4 |
                      | innodb_force_recovery | 0 |
                      | innodb_thread_concurrency | 8 |
                      | innodb_flush_log_at_trx_commit | OFF |
                      | innodb_fast_shutdown | OFF |
                      | innodb_flush_method | |
                      | innodb_lock_wait_timeout | 50 |
                      | innodb_log_arch_dir | |
                      | innodb_log_archive | OFF |
                      | innodb_log_buffer_size | 1048576 |
                      | innodb_log_file_size | 5242880 |
                      | innodb_log_files_in_group | 2 |
                      | innodb_log_group_home_dir | |
                      | innodb_mirrored_log_groups | 1 |
                      | interactive_timeout | 28800 |
                      | join_buffer_size | 2093056 |
                      | key_buffer_size | 33550336 |
                      | language | /usr/local/share/mysql/english/ |
                      | large_files_support | ON |
                      | log | OFF |
                      | log_update | OFF |
                      | log_bin | ON |
                      | log_slave_updates | OFF |
                      | log_long_queries | OFF |
                      | long_query_time | 10 |
                      | low_priority_updates | OFF |
                      | lower_case_table_names | 0 |
                      | max_allowed_packet | 1048576 |
                      | max_binlog_cache_size | 4294967295 |
                      | max_binlog_size | 1073741824 |
                      | max_connections | 500 |
                      | max_connect_errors | 10 |
                      | max_delayed_threads | 20 |
                      | max_heap_table_size | 16777216 |
                      | max_join_size | 4294967295 |
                      | max_sort_length | 1024 |
                      | max_user_connections | 0 |
                      | max_tmp_tables | 32 |
                      | max_write_lock_count | 4294967295 |
                      | myisam_max_extra_sort_file_size | 256 |
                      | myisam_max_sort_file_size | 2047 |
                      | myisam_recover_options | 0 |
                      | myisam_sort_buffer_size | 67108864 |
                      | net_buffer_length | 16384 |
                      -Steve St.Laurent
                      Webmaster of http://www.turbodieselregister.com

                      Comment


                      • #12
                        | net_read_timeout | 30 |
                        | net_retry_count | 1000000 |
                        | net_write_timeout | 60 |
                        | open_files_limit | 0 |
                        | pid_file | /data/mysql/turbodieselregister.pid |
                        | port | 3306 |
                        | protocol_version | 10 |
                        | record_buffer | 2093056 |
                        | record_rnd_buffer | 2093056 |
                        | query_buffer_size | 0 |
                        | safe_show_database | OFF |
                        | server_id | 1 |
                        | slave_net_timeout | 3600 |
                        | skip_locking | ON |
                        | skip_networking | OFF |
                        | skip_show_database | OFF |
                        | slow_launch_time | 2 |
                        | socket | /tmp/mysql.sock |
                        | sort_buffer | 3145720 |
                        | sql_mode | 0 |
                        | table_cache | 1024 |
                        | table_type | MYISAM |
                        | thread_cache_size | 512 |
                        | thread_stack | 65536 |
                        | transaction_isolation | READ-COMMITTED |
                        | timezone | GMT |
                        | tmp_table_size | 33554432 |
                        | tmpdir | /var/tmp/ |
                        | version | 3.23.47-log |
                        | wait_timeout | 7200 |
                        +---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+

                        5. Here is the extended status output (the url is http://www.turbodieselregister.com/f.../mysqlinfo.php - but you won't be able to run it because I have that directory password protected):

                        Tue Sep 10 19:07:44 GMT 2002


                        last pid: 49545; load averages: 0.28, 0.37, 0.40 up 5+07:36:48 19:07:45
                        67 processes: 2 running, 65 sleeping

                        140 mysql 2 0 463M 102M poll 1 918:22 16.70% 16.70% mysqld


                        Http processes currently running = 53
                        Mysql processes currently running = 4

                        Netstat information summary
                        1 FIN_WAIT_1
                        1 LAST_ACK
                        9 LISTEN
                        26 FIN_WAIT_2
                        42 ESTABLISHED
                        188 TIME_WAIT


                        +---------------------------+-----------------+
                        | Variable_name | Value |
                        +---------------------------+-----------------+
                        | Aborted_clients | 238 |
                        | Aborted_connects | 0 |
                        | Bytes_received | 2178289596 |
                        | Bytes_sent | 1887126601 |
                        | Com_admin_commands | 7614 |
                        | Com_alter_table | 0 |
                        | Com_analyze | 0 |
                        | Com_backup_table | 0 |
                        | Com_begin | 0 |
                        | Com_change_db | 982437 |
                        | Com_change_master | 0 |
                        | Com_check | 0 |
                        | Com_commit | 0 |
                        | Com_create_db | 0 |
                        | Com_create_function | 0 |
                        | Com_create_index | 0 |
                        | Com_create_table | 1 |
                        | Com_delete | 43433 |
                        | Com_drop_db | 0 |
                        | Com_drop_function | 0 |
                        | Com_drop_index | 0 |
                        | Com_drop_table | 1 |
                        | Com_flush | 0 |
                        | Com_grant | 0 |
                        | Com_insert | 71095 |
                        | Com_insert_select | 1006 |
                        | Com_kill | 0 |
                        | Com_load | 0 |
                        | Com_load_master_table | 0 |
                        | Com_lock_tables | 10 |
                        | Com_optimize | 0 |
                        | Com_purge | 0 |
                        | Com_rename_table | 0 |
                        | Com_repair | 0 |
                        | Com_replace | 3948 |
                        | Com_replace_select | 23 |
                        | Com_reset | 0 |
                        | Com_restore_table | 0 |
                        | Com_revoke | 0 |
                        | Com_rollback | 0 |
                        | Com_select | 8085362 |
                        | Com_set_option | 108 |
                        | Com_show_binlogs | 0 |
                        | Com_show_create | 108 |
                        | Com_show_databases | 41 |
                        | Com_show_fields | 140 |
                        | Com_show_grants | 0 |
                        | Com_show_keys | 23 |
                        | Com_show_logs | 0 |
                        | Com_show_master_stat | 0 |
                        | Com_show_open_tables | 0 |
                        | Com_show_processlist | 0 |
                        | Com_show_slave_stat | 0 |
                        | Com_show_status | 433480 |
                        | Com_show_tables | 264 |
                        | Com_show_variables | 150 |
                        | Com_slave_start | 0 |
                        | Com_slave_stop | 0 |
                        | Com_truncate | 0 |
                        | Com_unlock_tables | 1 |
                        | Com_update | 1299521 |
                        | Connections | 535492 |
                        | Created_tmp_disk_tables | 4977 |
                        | Created_tmp_tables | 337944 |
                        | Created_tmp_files | 0 |
                        | Delayed_insert_threads | 0 |
                        | Delayed_writes | 0 |
                        | Delayed_errors | 0 |
                        | Flush_commands | 1 |
                        -Steve St.Laurent
                        Webmaster of http://www.turbodieselregister.com

                        Comment


                        • #13
                          | Handler_delete | 73771 |
                          | Handler_read_first | 117620 |
                          | Handler_read_key | 877712918 |
                          | Handler_read_next | 829013589 |
                          | Handler_read_prev | 1339 |
                          | Handler_read_rnd | 72539951 |
                          | Handler_read_rnd_next | 406083409 |
                          | Handler_update | 659888004 |
                          | Handler_write | 135544548 |
                          | Key_blocks_used | 31168 |
                          | Key_read_requests | 467796996 |
                          | Key_reads | 23412 |
                          | Key_write_requests | 1586610 |
                          | Key_writes | 283015 |
                          | Max_used_connections | 500 | Max. connections reached

                          | Not_flushed_key_blocks | 0 |
                          | Not_flushed_delayed_rows | 0 |
                          | Open_tables | 1024 | 100% of table_cache in use
                          | Open_files | 867 |
                          | Open_streams | 0 |
                          | Opened_tables | 1198 |
                          | Questions | 11022914 |
                          | Select_full_join | 10 |
                          | Select_full_range_join | 0 |
                          | Select_range | 1752480 |
                          | Select_range_check | 0 |
                          | Select_scan | 1867323 |
                          | Slave_running | ON |
                          | Slave_open_temp_tables | 0 |
                          | Slow_launch_threads | 0 |
                          | Slow_queries | 1649 | (execution time > 10 secs)
                          | Sort_merge_passes | 0 |
                          | Sort_range | 1437034 |
                          | Sort_rows | 81338769 |
                          | Sort_scan | 1033330 |
                          | Table_locks_immediate | 20956808 |
                          | Table_locks_waited | 3250 |
                          | Threads_cached | 454 |
                          | Threads_created | 501 |
                          | Threads_connected | 47 |
                          | Threads_running | 2 |
                          | Uptime | 459373 | 5 days 7 hrs 36 mins 13 secs
                          +---------------------------+-----------------+


                          Key Reads/Key Read Requests = 0.000050 (Cache hit = 99.99995%)
                          Key Writes/Key Write Requests = 0.178377
                          Connections/second = 1.166 (/hour = 4196.527)
                          KB received/second = 4.565 (/hour = 16434.888)
                          KB sent/second = 4.012 (/hour = 14442.358)
                          Temporary Tables Created/second = 0.736 (/hour = 2648.389)
                          Opened Tables/second = 0.003 (/hour = 9.388)
                          Slow Queries/second = 0.004 (/hour = 12.923)
                          % of slow queries = 0.015%
                          Queries/second = 23.996 (/hour = 86384.029)

                          6. There are other scripts running on the server - nakenchat, e-classifieds, dclinks, Matt Kruse's calendar, a custom photo script, paFaq, dbman, and webalizer. None of those scripts have been changed and they are all running perfectly along with 2.2.6. Our site is the only site running on this server.

                          7. We usually have at least 130 users, 150-190 most of the time and hit peaks around 300 for two hours each night (with a 60 second window)

                          8. phpinfo.php is at http://www.turbodieselregister.com/forums/phpinfo.php

                          9. httpd.conf variables:
                          KeepAlive on
                          MaxKeepAliveRequests 100
                          KeepAliveTimeout 15
                          MinSpareServers 5
                          MaxSpareServers 15
                          StartServers 10
                          MaxClients 256
                          MaxRequestsPerChild 19999

                          10. currently running 2.2.6 - works great with 2.2.6 (except for LOTS of login issues, which is why I want to try to upgrade to 2.2.7), the problems happened with 2.2.7 which has been removed for now.
                          -Steve St.Laurent
                          Webmaster of http://www.turbodieselregister.com

                          Comment


                          • #14
                            Originally posted by Steve St.Lauren
                            I think you missed what I said above. Our site is running perfectly with V2.2.6 of the software. The problem only happened when I attempted to upgrade to V2.2.7 and within a couple of hours of installing it - you can see the typical server loads on my second post on this thread (which was with 222 users online running 2.2.6) and the problem loads on the first post (which was with a peak of 140 users running 2.2.7). There was no way to access the site at all through the web. I SSH'd into the server and I shut down MYSQL and Apache and the server loads still stayed sky high, although nothing was shown on the the top report as using a ton of CPU. I eventually had to reboot it, and then I restored the program files back to V2.2.6. The site has been running fine since that time with up to 304 users online simultaneously (window is set to 60 seconds) whereas the server crash running 2.2.7 was with only about 140 users on. Here is the info that you requested.
                            well i do see a few items in your configuration that will definitely increase your server loads unnecessarily including you enabling binary logging, that alone will drive your loads up, and reaching both your max_connections and table_cache limits set by your current mysql configuration

                            also it shows in mysql extended-status output that your mysql is using more than 500 max_connections (concurrent connections) probably due to mysql usage by other scripts other than vB in total. Since vB is not the only script using mysql (your other scripts use mysql too right ?), it's hard to compare loads and mysql usage based solely on vB's users online output

                            to illustrate

                            - vb user online doesn't necessarily = to mysql concurrent connection as there is a cookie timeout

                            - so that 250 users on vB could mean 125 connecting simultaneously and 125 guests or people who connected within 15 mins (hence not simultaneously)

                            so:

                            - vb226 could have 250 users utilising 125 concurrent mysql connections
                            - script2 using mysql could be utilising 50 mysql concurrent connections
                            - script 3 could be using 75 mysql connections
                            total = 250 max_used_connections (concurrent mysql connections)

                            versus

                            - vb 227 could have 200 users online but just that moment there was 150 concurrent mysql connections and only 50 users browsing within the 15 mins
                            - script2 could use 75 concurrent connections
                            - script3 could use 90 concurrent connections
                            total = 315 max_used_connections (concurrent mysql connections)

                            hence, you could see more users on vB 2.26 with lower loads than vB 2.27 because it ultimately comes down to mysql concurrent connections of which vB users online is not an accurate indicator of especially if you have other scripts utilising mysql and a default 15 min time out cookie...

                            things to do:

                            1. create or replace your /etc/my.cnf mysql config file with the below one and restart mysql

                            [mysqld]
                            set-variable = max_connections=750
                            set-variable = key_buffer=16M
                            set-variable = myisam_sort_buffer_size=64M
                            set-variable = join_buffer=1M
                            set-variable = record_buffer=1M
                            set-variable = sort_buffer=2M
                            set-variable = table_cache=1280
                            set-variable = thread_cache_size=256
                            set-variable = wait_timeout=9600
                            set-variable = connect_timeout=10
                            set-variable = max_allowed_packet=16M
                            set-variable = max_connect_errors=10

                            [safe_mysqld]
                            open_files_limit=8192

                            [mysqldump]
                            quick
                            set-variable = max_allowed_packet=16M

                            [myisamchk]
                            set-variable = key_buffer=64M
                            set-variable = sort_buffer=64M
                            set-variable = read_buffer=16M
                            set-variable = write_buffer=16M
                            2. i'd recommend eventually upgrading php to at least 4.1.2 to fix bugs in 4.1.1 and upgrade mysql to 3.23.52 to fix ALOT of bugs

                            3. also from your stats output you're reaching 500+ mysql concurrent connections which is by vB usage observations by me from handling lots of large vBs, it's time for you to split your server into 2 one for web server and one for a dedicated mysql database server (especially if you consistently hit 500+ concurrent mysql connections as a result of all your scripts (not only vB) utilising mysql.
                            :: Always Back Up Forum Database + Attachments BEFORE upgrading !
                            :: Nginx SPDY SSL - World Flags Demo [video results]
                            :: vBulletin hacked forums: Clean Up Guide for VPS/Dedicated hosting users [ vbulletin.com blog summary ]

                            Comment


                            • #15
                              We only went over our 500 connections because we recently ran a script that compares our subscription database to our vBulletin user database and goes through every single account and reset's their user group. That is a situation that comes up once every 3 months and we are aware that it will cause problems at that point. We have not gone over our 500 connections in the last 5 months. Here is my uptime.log during the time period when we were running that subscription script and went over 500 connections:

                              12:35PM up 1 day, 1:04, 1 user, load averages: 1.08, 0.99, 0.76
                              12:40PM up 1 day, 1:09, 1 user, load averages: 1.04, 1.05, 0.84
                              12:45PM up 1 day, 1:14, 1 user, load averages: 1.00, 1.03, 0.88
                              12:50PM up 1 day, 1:19, 1 user, load averages: 1.07, 1.04, 0.92
                              12:55PM up 1 day, 1:24, 1 user, load averages: 1.15, 1.09, 0.96
                              1:00PM up 1 day, 1:29, 0 users, load averages: 1.07, 1.08, 0.99
                              1:05PM up 1 day, 1:34, 0 users, load averages: 1.01, 1.03, 1.00
                              1:10PM up 1 day, 1:39, 0 users, load averages: 1.03, 1.03, 1.00
                              1:15PM up 1 day, 1:44, 1 user, load averages: 1.04, 1.03, 1.00
                              1:20PM up 1 day, 1:49, 1 user, load averages: 1.00, 1.00, 1.00
                              1:25PM up 1 day, 1:54, 1 user, load averages: 1.00, 1.00, 1.00
                              1:30PM up 1 day, 1:59, 1 user, load averages: 1.00, 1.00, 1.00
                              1:35PM up 1 day, 2:04, 0 users, load averages: 0.17, 0.55, 0.79

                              As you can see, even when we went over 500 simultaneous connections on mysql our sever load didn't go over 1.15

                              The reason I have binary logging enabled is because I am running two servers, this one is a primary server and is running mysql in a master mode, our secondary server is running mysql in a slave mode and replicating our database on a continuous basis in case we ever have a problem with the primary server. As I said above our timeout for make continuous users is set to 60 seconds so that user count is in a 60 second window, not 15 minutes. In the last 4 months since I have taken over as webmaster we have never seen a server load over 2.0. I had the server's split with one acting as the database server and the other as the web server but with the server loads being so low I decided to change them to back each other up so that if one went down the other could take over. If you'll look at the uptime logs at the top of this thread you'll see that the highest server load with nearly twice the number of users online running 2.2.6 was 0.66, 0.56, 0.44. With half the number of users online running 2.2.7 we hit 2.12, 1.30, 0.94 at 7:00pm; 2.06, 1.25, 1.14 at 8:15pm; and 4.29, 4.05, 2.64 at 8:35pm which was when it for all intensive purposes brought the web server down. 2.2.7 was installed at 6:00pm. The ONLY change to the system was to install 2.2.7 and I am 100% sure that the increase in the server loads was due to the install of 2.2.7 - otherwise it is an unbelievably (nearly impossible) coincedence that at the exact time 2.2.7 was installed that other software that was on the server that has been working flawlessly for 4 months (and ever since) along with up to nearly 3 times the user load on vB all of a sudden caused a problem. PLEASE look at the uptime logs above and you'll see exactly what I mean. Look at the first logs and notice that the loads all of a sudden start increasing dramatically at 6:00 pm - the exact time I installed the new scripts - and there was no increase in the user load at that time.
                              -Steve St.Laurent
                              Webmaster of http://www.turbodieselregister.com

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...
                              X