BTMash

A few years ago, Khalid wrote a wonderful script that would help make your apache server sane again after the opcode caching on the server started to throw segmentation faults (he aptly named it logwatcher). It was great because APC would crash for unknown reasons at the time and completely kill a website. It took care of an important issue (the one change I had made was to clear the apc cache instead of restarting the server but all in all, super ^_^).

A few years back during my time at zinc Roe, we were finding that the server would start to die due to spikes in traffic on the Zimmer Twins. In trying to figure out what was happening, we found that the limit on the number of connections that the db server would allow was in fact making the apache server spiral out of control. What happened is:

1. User does a page request (be it via form submit or clicking a link)
2. DB connections are all taken up
3. Apache starts writing to log the list of errors cropping up from the server
4. Repeat for thousands of users coming on around the same time

Part of the issue with the db server had to do some with bad queries (search was killing the site as it tried to find relevant results from over a million nodes). Users would get impatient and try to request a page again, and repeat apache writing to the disk and you get a server basically thrashing to death. What I did find is that *if* I was able to log over into the server and restart apache, the problem would resolve itself and people could actually start using the site sanely once again. However, when a server is getting out of control, the simple act of trying to ssh in can be very challenging.

As a result of this, I created a daemon script (well..2 scripts) called 'loadwatcher' which does something somewhat straightforward. It checks to see what the load average on the server is every 60 seconds (which you could change to be whatever interval you wanted) and if it passed a threshold (in my case I used 10.0), it would restart Apache (I then waited a while before starting to do the checks again). Really silly idea, right? Luckily, what the script did along with the assumption, I made took care of nearly all server uptime issues (the server remained up; there might be *some* site downtime but it was better than having to wholly reboot a server). It also gave some breathing room to figure out ways to optimize the search functionality and reduce site downtime even further. Its easy enough to add any other things you want done (like restarting mysql if that was a possible cause of the issue) or to change other pieces. I've been able to use the script across various servers over the years with similar results (keeping the server going when crap hits the fan) and I figured it would be something worthwhile to share.

The daemon script I had would be continually running on the server checking what needs to be done (also posted on pastebin).

<?php
 
define ("LOAD_AVERAGE_THRESHOLD", 10.0);
define ("APACHE_RESTART_THRESHOLD", 60 * 5);
define ("DEFAULT_LOAD_AVERAGE_PATH", "/proc/loadavg");
define ("APACHE_COMMAND_STOP", '/etc/init.d/apache2 stop');
define ("APACHE_COMMAND_START", '/etc/init.d/apache2 start');
define ("APACHE_COMMAND_RESTART", '/etc/init.d/apache2 restart');
define ("APACHE_COMMAND_STATUS", '/etc/init.d/apache2 status');
 
function get_current_load_average() {
  $load_average_file_name = DEFAULT_LOAD_AVERAGE_PATH;
  $fp = @fopen($load_average_file_name, 'r');
  if ($fp == null) {
    die("unable to open file at $load_average_file_name\n");
  }
  $load_averages = explode(" ", fread($fp, 4096));
  return floatval($load_averages[0]);
}
 
function restart_apache_server() {
  // Do some stuff in here!
  printf ("Server might be going through problems...restart the web server now!\n");
  system(APACHE_COMMAND_STOP);
  printf ("Stopped web server");
  sleep(10);
  system(APACHE_COMMAND_START);
  printf ("Started web server");
}
 
$allowed_restart_time = time();
 
while (1) {
  $current_load_average = get_current_load_average();
  printf("Current load - %f\n", $current_load_average);
  if ($current_load_average < LOAD_AVERAGE_THRESHOLD) {
    printf ("Server is doing pretty well!\n");
  }
  else if ($current_load_average >= LOAD_AVERAGE_THRESHOLD && time() > $allowed_restart_time) {
    restart_apache_server();
    $allowed_restart_time = time() + APACHE_RESTART_THRESHOLD;
  }
  else {
    printf ("Wait for some time to ensure server is getting back to normal.\n");
  }
  sleep(30);
}

I then wrapped this up with a shell script which would keep track of the pid and kill any old loadwatcher instances (also posted on pastebin):

#!/bin/sh
 
BASE_DIR=/path/to/base
SCRIPT=$BASE_DIR/scripts/logwatcher.php
PID_FILE=/var/run/loadwatcher.pid
EMAIL=btmash@gmail.com
 
# If there is an old process, kill it
kill `cat $PID_FILE`
# Make sure the file is clean
rm -f $PID_FILE
 
cd $BASE_DIR
nohup php $SCRIPT > /dev/null &
PID=$!
 
echo $PID > $PID_FILE

Naturally, there are better things that can be done to ensure the server stays sane and that you don't run into issues like these in the first place :) What do you do?

Update: Kevin Kaland told me about monit (http://mmonit.com/monit/) which seems to do a lot more than what I have been doing and I'll be looking into it for more work in the future.