Announcement

Collapse
No announcement yet.

SEVERE 2.2.7 problem - crashed FreeBSD server

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • George L
    replied
    Originally posted by Steve St.Lauren
    I installed 2.2.7 again today. Thus far the loads are looking fine. I downloaded a new copy of 2.2.7 and started from scratch. I'll be monitoring things close for a while but it seems that there was either an error in the download or I made an error in converting the installed hacks from 2.2.6 to 2.2.7 (all hacks are installed at this point).
    sounds good... you should try running vB 2.2.7 unhacked for several weeks.. it could be an unoptimised hack and not vB itself causing the problem

    Leave a comment:


  • Steve St.Lauren
    replied
    I installed 2.2.7 again today. Thus far the loads are looking fine. I downloaded a new copy of 2.2.7 and started from scratch. I'll be monitoring things close for a while but it seems that there was either an error in the download or I made an error in converting the installed hacks from 2.2.6 to 2.2.7 (all hacks are installed at this point).

    Leave a comment:


  • Steve St.Lauren
    replied
    I'm going to attempt to reinstall 2.2.7 later in the week with a fresh copy and will try those suggestions then. I need to do it during the day when the co-location facility is staffed in case a reboot is needed (our servers have a reboot bug such that I can't reboot them remotely - what a pain). I'll get back to you on the results.

    Leave a comment:


  • George L
    replied
    Originally posted by freddie
    I don't see any changes to the main files (sessions.php, global.php, index.php, forumdisplay.php and showthread.php) that would cause any load increases. What you can do is put 2.2.7 back on and then replace the above files, one at a time, with the 2.2.6 versions to see what your load looks like over an acceptable time frame.
    also have you tried freddie's suggestion ?

    Leave a comment:


  • George L
    replied
    could you add your log-bin and server id back into the suggested my.cnf below

    [mysqld]
    set-variable = max_connections=750
    set-variable = key_buffer=16M
    set-variable = myisam_sort_buffer_size=64M
    set-variable = join_buffer=1M
    set-variable = record_buffer=1M
    set-variable = sort_buffer=2M
    set-variable = table_cache=1280
    set-variable = thread_cache_size=256
    set-variable = wait_timeout=9600
    set-variable = connect_timeout=10
    set-variable = max_allowed_packet=16M
    set-variable = max_connect_errors=10

    [safe_mysqld]
    open_files_limit=8192

    [mysqldump]
    quick
    set-variable = max_allowed_packet=16M

    [myisamchk]
    set-variable = key_buffer=64M
    set-variable = sort_buffer=64M
    set-variable = read_buffer=16M
    set-variable = write_buffer=16M

    and try it out with vB 2.2.7 ?

    Leave a comment:


  • Steve St.Lauren
    replied
    We only went over our 500 connections because we recently ran a script that compares our subscription database to our vBulletin user database and goes through every single account and reset's their user group. That is a situation that comes up once every 3 months and we are aware that it will cause problems at that point. We have not gone over our 500 connections in the last 5 months. Here is my uptime.log during the time period when we were running that subscription script and went over 500 connections:

    12:35PM up 1 day, 1:04, 1 user, load averages: 1.08, 0.99, 0.76
    12:40PM up 1 day, 1:09, 1 user, load averages: 1.04, 1.05, 0.84
    12:45PM up 1 day, 1:14, 1 user, load averages: 1.00, 1.03, 0.88
    12:50PM up 1 day, 1:19, 1 user, load averages: 1.07, 1.04, 0.92
    12:55PM up 1 day, 1:24, 1 user, load averages: 1.15, 1.09, 0.96
    1:00PM up 1 day, 1:29, 0 users, load averages: 1.07, 1.08, 0.99
    1:05PM up 1 day, 1:34, 0 users, load averages: 1.01, 1.03, 1.00
    1:10PM up 1 day, 1:39, 0 users, load averages: 1.03, 1.03, 1.00
    1:15PM up 1 day, 1:44, 1 user, load averages: 1.04, 1.03, 1.00
    1:20PM up 1 day, 1:49, 1 user, load averages: 1.00, 1.00, 1.00
    1:25PM up 1 day, 1:54, 1 user, load averages: 1.00, 1.00, 1.00
    1:30PM up 1 day, 1:59, 1 user, load averages: 1.00, 1.00, 1.00
    1:35PM up 1 day, 2:04, 0 users, load averages: 0.17, 0.55, 0.79

    As you can see, even when we went over 500 simultaneous connections on mysql our sever load didn't go over 1.15

    The reason I have binary logging enabled is because I am running two servers, this one is a primary server and is running mysql in a master mode, our secondary server is running mysql in a slave mode and replicating our database on a continuous basis in case we ever have a problem with the primary server. As I said above our timeout for make continuous users is set to 60 seconds so that user count is in a 60 second window, not 15 minutes. In the last 4 months since I have taken over as webmaster we have never seen a server load over 2.0. I had the server's split with one acting as the database server and the other as the web server but with the server loads being so low I decided to change them to back each other up so that if one went down the other could take over. If you'll look at the uptime logs at the top of this thread you'll see that the highest server load with nearly twice the number of users online running 2.2.6 was 0.66, 0.56, 0.44. With half the number of users online running 2.2.7 we hit 2.12, 1.30, 0.94 at 7:00pm; 2.06, 1.25, 1.14 at 8:15pm; and 4.29, 4.05, 2.64 at 8:35pm which was when it for all intensive purposes brought the web server down. 2.2.7 was installed at 6:00pm. The ONLY change to the system was to install 2.2.7 and I am 100% sure that the increase in the server loads was due to the install of 2.2.7 - otherwise it is an unbelievably (nearly impossible) coincedence that at the exact time 2.2.7 was installed that other software that was on the server that has been working flawlessly for 4 months (and ever since) along with up to nearly 3 times the user load on vB all of a sudden caused a problem. PLEASE look at the uptime logs above and you'll see exactly what I mean. Look at the first logs and notice that the loads all of a sudden start increasing dramatically at 6:00 pm - the exact time I installed the new scripts - and there was no increase in the user load at that time.

    Leave a comment:


  • George L
    replied
    Originally posted by Steve St.Lauren
    I think you missed what I said above. Our site is running perfectly with V2.2.6 of the software. The problem only happened when I attempted to upgrade to V2.2.7 and within a couple of hours of installing it - you can see the typical server loads on my second post on this thread (which was with 222 users online running 2.2.6) and the problem loads on the first post (which was with a peak of 140 users running 2.2.7). There was no way to access the site at all through the web. I SSH'd into the server and I shut down MYSQL and Apache and the server loads still stayed sky high, although nothing was shown on the the top report as using a ton of CPU. I eventually had to reboot it, and then I restored the program files back to V2.2.6. The site has been running fine since that time with up to 304 users online simultaneously (window is set to 60 seconds) whereas the server crash running 2.2.7 was with only about 140 users on. Here is the info that you requested.
    well i do see a few items in your configuration that will definitely increase your server loads unnecessarily including you enabling binary logging, that alone will drive your loads up, and reaching both your max_connections and table_cache limits set by your current mysql configuration

    also it shows in mysql extended-status output that your mysql is using more than 500 max_connections (concurrent connections) probably due to mysql usage by other scripts other than vB in total. Since vB is not the only script using mysql (your other scripts use mysql too right ?), it's hard to compare loads and mysql usage based solely on vB's users online output

    to illustrate

    - vb user online doesn't necessarily = to mysql concurrent connection as there is a cookie timeout

    - so that 250 users on vB could mean 125 connecting simultaneously and 125 guests or people who connected within 15 mins (hence not simultaneously)

    so:

    - vb226 could have 250 users utilising 125 concurrent mysql connections
    - script2 using mysql could be utilising 50 mysql concurrent connections
    - script 3 could be using 75 mysql connections
    total = 250 max_used_connections (concurrent mysql connections)

    versus

    - vb 227 could have 200 users online but just that moment there was 150 concurrent mysql connections and only 50 users browsing within the 15 mins
    - script2 could use 75 concurrent connections
    - script3 could use 90 concurrent connections
    total = 315 max_used_connections (concurrent mysql connections)

    hence, you could see more users on vB 2.26 with lower loads than vB 2.27 because it ultimately comes down to mysql concurrent connections of which vB users online is not an accurate indicator of especially if you have other scripts utilising mysql and a default 15 min time out cookie...

    things to do:

    1. create or replace your /etc/my.cnf mysql config file with the below one and restart mysql

    [mysqld]
    set-variable = max_connections=750
    set-variable = key_buffer=16M
    set-variable = myisam_sort_buffer_size=64M
    set-variable = join_buffer=1M
    set-variable = record_buffer=1M
    set-variable = sort_buffer=2M
    set-variable = table_cache=1280
    set-variable = thread_cache_size=256
    set-variable = wait_timeout=9600
    set-variable = connect_timeout=10
    set-variable = max_allowed_packet=16M
    set-variable = max_connect_errors=10

    [safe_mysqld]
    open_files_limit=8192

    [mysqldump]
    quick
    set-variable = max_allowed_packet=16M

    [myisamchk]
    set-variable = key_buffer=64M
    set-variable = sort_buffer=64M
    set-variable = read_buffer=16M
    set-variable = write_buffer=16M
    2. i'd recommend eventually upgrading php to at least 4.1.2 to fix bugs in 4.1.1 and upgrade mysql to 3.23.52 to fix ALOT of bugs

    3. also from your stats output you're reaching 500+ mysql concurrent connections which is by vB usage observations by me from handling lots of large vBs, it's time for you to split your server into 2 one for web server and one for a dedicated mysql database server (especially if you consistently hit 500+ concurrent mysql connections as a result of all your scripts (not only vB) utilising mysql.

    Leave a comment:


  • Steve St.Lauren
    replied
    | Handler_delete | 73771 |
    | Handler_read_first | 117620 |
    | Handler_read_key | 877712918 |
    | Handler_read_next | 829013589 |
    | Handler_read_prev | 1339 |
    | Handler_read_rnd | 72539951 |
    | Handler_read_rnd_next | 406083409 |
    | Handler_update | 659888004 |
    | Handler_write | 135544548 |
    | Key_blocks_used | 31168 |
    | Key_read_requests | 467796996 |
    | Key_reads | 23412 |
    | Key_write_requests | 1586610 |
    | Key_writes | 283015 |
    | Max_used_connections | 500 | Max. connections reached

    | Not_flushed_key_blocks | 0 |
    | Not_flushed_delayed_rows | 0 |
    | Open_tables | 1024 | 100% of table_cache in use
    | Open_files | 867 |
    | Open_streams | 0 |
    | Opened_tables | 1198 |
    | Questions | 11022914 |
    | Select_full_join | 10 |
    | Select_full_range_join | 0 |
    | Select_range | 1752480 |
    | Select_range_check | 0 |
    | Select_scan | 1867323 |
    | Slave_running | ON |
    | Slave_open_temp_tables | 0 |
    | Slow_launch_threads | 0 |
    | Slow_queries | 1649 | (execution time > 10 secs)
    | Sort_merge_passes | 0 |
    | Sort_range | 1437034 |
    | Sort_rows | 81338769 |
    | Sort_scan | 1033330 |
    | Table_locks_immediate | 20956808 |
    | Table_locks_waited | 3250 |
    | Threads_cached | 454 |
    | Threads_created | 501 |
    | Threads_connected | 47 |
    | Threads_running | 2 |
    | Uptime | 459373 | 5 days 7 hrs 36 mins 13 secs
    +---------------------------+-----------------+


    Key Reads/Key Read Requests = 0.000050 (Cache hit = 99.99995%)
    Key Writes/Key Write Requests = 0.178377
    Connections/second = 1.166 (/hour = 4196.527)
    KB received/second = 4.565 (/hour = 16434.888)
    KB sent/second = 4.012 (/hour = 14442.358)
    Temporary Tables Created/second = 0.736 (/hour = 2648.389)
    Opened Tables/second = 0.003 (/hour = 9.388)
    Slow Queries/second = 0.004 (/hour = 12.923)
    % of slow queries = 0.015%
    Queries/second = 23.996 (/hour = 86384.029)

    6. There are other scripts running on the server - nakenchat, e-classifieds, dclinks, Matt Kruse's calendar, a custom photo script, paFaq, dbman, and webalizer. None of those scripts have been changed and they are all running perfectly along with 2.2.6. Our site is the only site running on this server.

    7. We usually have at least 130 users, 150-190 most of the time and hit peaks around 300 for two hours each night (with a 60 second window)

    8. phpinfo.php is at http://www.turbodieselregister.com/forums/phpinfo.php

    9. httpd.conf variables:
    KeepAlive on
    MaxKeepAliveRequests 100
    KeepAliveTimeout 15
    MinSpareServers 5
    MaxSpareServers 15
    StartServers 10
    MaxClients 256
    MaxRequestsPerChild 19999

    10. currently running 2.2.6 - works great with 2.2.6 (except for LOTS of login issues, which is why I want to try to upgrade to 2.2.7), the problems happened with 2.2.7 which has been removed for now.

    Leave a comment:


  • Steve St.Lauren
    replied
    | net_read_timeout | 30 |
    | net_retry_count | 1000000 |
    | net_write_timeout | 60 |
    | open_files_limit | 0 |
    | pid_file | /data/mysql/turbodieselregister.pid |
    | port | 3306 |
    | protocol_version | 10 |
    | record_buffer | 2093056 |
    | record_rnd_buffer | 2093056 |
    | query_buffer_size | 0 |
    | safe_show_database | OFF |
    | server_id | 1 |
    | slave_net_timeout | 3600 |
    | skip_locking | ON |
    | skip_networking | OFF |
    | skip_show_database | OFF |
    | slow_launch_time | 2 |
    | socket | /tmp/mysql.sock |
    | sort_buffer | 3145720 |
    | sql_mode | 0 |
    | table_cache | 1024 |
    | table_type | MYISAM |
    | thread_cache_size | 512 |
    | thread_stack | 65536 |
    | transaction_isolation | READ-COMMITTED |
    | timezone | GMT |
    | tmp_table_size | 33554432 |
    | tmpdir | /var/tmp/ |
    | version | 3.23.47-log |
    | wait_timeout | 7200 |
    +---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+

    5. Here is the extended status output (the url is http://www.turbodieselregister.com/f.../mysqlinfo.php - but you won't be able to run it because I have that directory password protected):

    Tue Sep 10 19:07:44 GMT 2002


    last pid: 49545; load averages: 0.28, 0.37, 0.40 up 5+07:36:48 19:07:45
    67 processes: 2 running, 65 sleeping

    140 mysql 2 0 463M 102M poll 1 918:22 16.70% 16.70% mysqld


    Http processes currently running = 53
    Mysql processes currently running = 4

    Netstat information summary
    1 FIN_WAIT_1
    1 LAST_ACK
    9 LISTEN
    26 FIN_WAIT_2
    42 ESTABLISHED
    188 TIME_WAIT


    +---------------------------+-----------------+
    | Variable_name | Value |
    +---------------------------+-----------------+
    | Aborted_clients | 238 |
    | Aborted_connects | 0 |
    | Bytes_received | 2178289596 |
    | Bytes_sent | 1887126601 |
    | Com_admin_commands | 7614 |
    | Com_alter_table | 0 |
    | Com_analyze | 0 |
    | Com_backup_table | 0 |
    | Com_begin | 0 |
    | Com_change_db | 982437 |
    | Com_change_master | 0 |
    | Com_check | 0 |
    | Com_commit | 0 |
    | Com_create_db | 0 |
    | Com_create_function | 0 |
    | Com_create_index | 0 |
    | Com_create_table | 1 |
    | Com_delete | 43433 |
    | Com_drop_db | 0 |
    | Com_drop_function | 0 |
    | Com_drop_index | 0 |
    | Com_drop_table | 1 |
    | Com_flush | 0 |
    | Com_grant | 0 |
    | Com_insert | 71095 |
    | Com_insert_select | 1006 |
    | Com_kill | 0 |
    | Com_load | 0 |
    | Com_load_master_table | 0 |
    | Com_lock_tables | 10 |
    | Com_optimize | 0 |
    | Com_purge | 0 |
    | Com_rename_table | 0 |
    | Com_repair | 0 |
    | Com_replace | 3948 |
    | Com_replace_select | 23 |
    | Com_reset | 0 |
    | Com_restore_table | 0 |
    | Com_revoke | 0 |
    | Com_rollback | 0 |
    | Com_select | 8085362 |
    | Com_set_option | 108 |
    | Com_show_binlogs | 0 |
    | Com_show_create | 108 |
    | Com_show_databases | 41 |
    | Com_show_fields | 140 |
    | Com_show_grants | 0 |
    | Com_show_keys | 23 |
    | Com_show_logs | 0 |
    | Com_show_master_stat | 0 |
    | Com_show_open_tables | 0 |
    | Com_show_processlist | 0 |
    | Com_show_slave_stat | 0 |
    | Com_show_status | 433480 |
    | Com_show_tables | 264 |
    | Com_show_variables | 150 |
    | Com_slave_start | 0 |
    | Com_slave_stop | 0 |
    | Com_truncate | 0 |
    | Com_unlock_tables | 1 |
    | Com_update | 1299521 |
    | Connections | 535492 |
    | Created_tmp_disk_tables | 4977 |
    | Created_tmp_tables | 337944 |
    | Created_tmp_files | 0 |
    | Delayed_insert_threads | 0 |
    | Delayed_writes | 0 |
    | Delayed_errors | 0 |
    | Flush_commands | 1 |

    Leave a comment:


  • Steve St.Lauren
    replied
    | innodb_additional_mem_pool_size | 1048576 |
    | innodb_buffer_pool_size | 8388608 |
    | innodb_data_file_path | |
    | innodb_data_home_dir | |
    | innodb_file_io_threads | 4 |
    | innodb_force_recovery | 0 |
    | innodb_thread_concurrency | 8 |
    | innodb_flush_log_at_trx_commit | OFF |
    | innodb_fast_shutdown | OFF |
    | innodb_flush_method | |
    | innodb_lock_wait_timeout | 50 |
    | innodb_log_arch_dir | |
    | innodb_log_archive | OFF |
    | innodb_log_buffer_size | 1048576 |
    | innodb_log_file_size | 5242880 |
    | innodb_log_files_in_group | 2 |
    | innodb_log_group_home_dir | |
    | innodb_mirrored_log_groups | 1 |
    | interactive_timeout | 28800 |
    | join_buffer_size | 2093056 |
    | key_buffer_size | 33550336 |
    | language | /usr/local/share/mysql/english/ |
    | large_files_support | ON |
    | log | OFF |
    | log_update | OFF |
    | log_bin | ON |
    | log_slave_updates | OFF |
    | log_long_queries | OFF |
    | long_query_time | 10 |
    | low_priority_updates | OFF |
    | lower_case_table_names | 0 |
    | max_allowed_packet | 1048576 |
    | max_binlog_cache_size | 4294967295 |
    | max_binlog_size | 1073741824 |
    | max_connections | 500 |
    | max_connect_errors | 10 |
    | max_delayed_threads | 20 |
    | max_heap_table_size | 16777216 |
    | max_join_size | 4294967295 |
    | max_sort_length | 1024 |
    | max_user_connections | 0 |
    | max_tmp_tables | 32 |
    | max_write_lock_count | 4294967295 |
    | myisam_max_extra_sort_file_size | 256 |
    | myisam_max_sort_file_size | 2047 |
    | myisam_recover_options | 0 |
    | myisam_sort_buffer_size | 67108864 |
    | net_buffer_length | 16384 |

    Leave a comment:


  • Steve St.Lauren
    replied
    I think you missed what I said above. Our site is running perfectly with V2.2.6 of the software. The problem only happened when I attempted to upgrade to V2.2.7 and within a couple of hours of installing it - you can see the typical server loads on my second post on this thread (which was with 222 users online running 2.2.6) and the problem loads on the first post (which was with a peak of 140 users running 2.2.7). There was no way to access the site at all through the web. I SSH'd into the server and I shut down MYSQL and Apache and the server loads still stayed sky high, although nothing was shown on the the top report as using a ton of CPU. I eventually had to reboot it, and then I restored the program files back to V2.2.6. The site has been running fine since that time with up to 304 users online simultaneously (window is set to 60 seconds) whereas the server crash running 2.2.7 was with only about 140 users on. Here is the info that you requested.

    1. MYSQL is Ver 11.15 Distrib 3.23.47, PHP is version 4.1.1, FreeBSD is Ver 4.5

    2. mysql was installed using FreeBSD's built in ports I believe (was done before I took over)

    3. Here are the current top stats with 147 users online:

    last pid: 49744; load averages: 0.47, 0.40, 0.37 up 5+07:45:02 19:15:59
    96 processes: 1 running, 95 sleeping
    CPU states: 2.6% user, 0.0% nice, 2.6% system, 0.0% interrupt, 94.7% idle
    Mem: 422M Active, 421M Inact, 99M Wired, 48M Cache, 112M Buf, 13M Free
    Swap: 3072M Total, 86M Used, 2986M Free, 2% Inuse

    4. Here are my mysql configuration variables:
    +---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Variable_name | Value |
    +---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
    | back_log | 50 |
    | basedir | /usr/local/ |
    | bdb_cache_size | 8388600 |
    | bdb_log_buffer_size | 524288 |
    | bdb_home | /data/mysql/ |
    | bdb_max_lock | 10000 |
    | bdb_logdir | |
    | bdb_shared_data | OFF |
    | bdb_tmpdir | /var/tmp/ |
    | bdb_version | Sleepycat Software: Berkeley DB 3.2.9a: (December 27, 2001) |
    | binlog_cache_size | 32768 |
    | character_set | latin1 |
    | character_sets | latin1 dec8 dos german1 hp8 koi8_ru latin2 swe7 usa7 cp1251 danish hebrew win1251 estonia hungarian koi8_ukr win1251ukr greek win1250 croat cp1257 latin5 |
    | concurrent_insert | ON |
    | connect_timeout | 10 |
    | datadir | /data/mysql/ |
    | delay_key_write | ON |
    | delayed_insert_limit | 100 |
    | delayed_insert_timeout | 300 |
    | delayed_queue_size | 1000 |
    | flush | OFF |
    | flush_time | 0 |
    | have_bdb | YES |
    | have_gemini | NO |
    | have_innodb | DISABLED |
    | have_isam | YES |
    | have_raid | NO |
    | have_openssl | NO |
    | init_file | |
    Last edited by Steve St.Lauren; Tue 10 Sep '02, 4:42pm.

    Leave a comment:


  • George L
    replied
    can you please provide the following

    1. your server specs, such as mysql and php version
    2. if possible how mysql was compiled/installed
    3. your top stats
    4. your mysql configuration variables located at /etc/my.cnf or c:\my.cnf if on Windows server if you don't have that file you need to log into telnet and as root user type

    mysqladmin -u root -p variables

    copy and paste output here

    5. your mysql extended-status output either still telnet as root user type

    mysqladmin -u root -p extended-status

    copy and paste output here

    or preferred is to installed extended-status output script which is either located

    - in your vB 2.2.6 zip file's extra's folder, upload mysqlinfo.php script to your site or if you're on an pre vB 2.2.6 install go to
    - http://vbulletin.com/forum/showthread.php?threadid=3477 and install that scrip making sure to edit $mysqllogin line with your own mysqlusername and password

    and post url to that here

    6. oh and is your vB the only thing on the server? or other scripts? sites?

    7. how many average and max concurrent users on your vB forum ?

    8. create a file named phpinfo.php and place this code in it and post the url/link to it from your web site

    <?
    phpinfo();
    ?>

    i.e. yourdomain.com/phpinfo.php

    9. if you run Apache and you have your own dedicated server or access to your httpd.conf (apache configuration file) can you post the values you have set for the following :

    KeepAlive
    MaxKeepAliveRequests
    KeepAliveTimeout
    MinSpareServers
    MaxSpareServers
    StartServers
    MaxClients

    10. what version of vB are you running ?

    Leave a comment:


  • George L
    replied
    have you at all checked your server apache, and mysql logs for any indications of warnings or error messages ?

    what versions of apache/php and mysql are you using with FreeBSD ?

    Leave a comment:


  • Freddie Bingham
    replied
    I don't see any changes to the main files (sessions.php, global.php, index.php, forumdisplay.php and showthread.php) that would cause any load increases. What you can do is put 2.2.7 back on and then replace the above files, one at a time, with the 2.2.6 versions to see what your load looks like over an acceptable time frame.

    Leave a comment:


  • Erwin
    replied
    I notice a very very slight increase in server loads too - hard to know what the cause is though in case... especially because my board is so hacked. Nothing to cause a server crash though.

    Leave a comment:

widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
Working...
X