There are countless tools and software stacks out there to monitor your servers out, though some are not the easiest to setup. I went with Nagios since it's straightforward to configure and easy to setup. For reference, I used Nagios 4.0.8 on Debian 7 using Debian 7 slaves. The Nagios master node already had lighttpd and PHP set up so I'll gloss over that.

Master node

Start by downloading Nagios and Nagios plugins:

 

wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-4.0.8.tar.gz
wget http://nagios-plugins.org/download/nagios-plugins-2.0.3.tar.gz

 

Extract, compile and install Nagios:

tar xzf nagios-4.0.8.tar.gz
cd nagios-4.0.8
./configure
sudo make all
sudo make install
sudo make install-init
sudo make install-config
sudo make install-commandmode

Extract, compile and install Nagios plugins

tar xzf nagios-plugins-2.0.3.tar.gz
cd nagios-plugins-2.0.3

./configure
sudo make
sudo make install

Everything should install nicely, with the config residing in /usr/local/nagios/etc/. Next we'll have to create a user with a password so we can login to Nagios:

 sudo htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

 

If you have a working mail server setup, edit /usr/local/nagios/etc/objects/contacts.cfg and change email to receive email alerts.

Now onto lighttpd configuration. You'll have to add the "mod_alias", "mod_auth", "mod_setenv" and "mod_cgi" to your server.modules in lighttpd.conf. Next we'll add the proper aliases and allow us to authenticate.

alias.url  += ("/nagios/cgi-bin" => "/usr/local/nagios/sbin")
alias.url  += ("/nagios/stylesheets" => "/usr/local/nagios/share/stylesheets")
alias.url  += ("/nagios" => "/usr/local/nagios/share")

$HTTP["url"] =~ "^/nagios/cgi-bin" {
        cgi.assign = ( "" => "" )
}
$HTTP["url"] =~ "nagios" {
  auth.backend = "htpasswd"
  auth.backend.htpasswd.userfile = "/usr/local/nagios/etc/htpasswd.users"
  auth.require = ( "" => (
    "method" => "basic",
    "realm" => "nagios",
    "require" => "user=nagiosadmin"
    )
  )
}
setenv.add-environment = ( "REMOTE_USER" => "nagiosadmin" )

Restart lighttpd to check our changes:

sudo /etc/init.d/lighttpd restart

At this point, be sure to check if http://your-server-address/nagios works.

Slave nodes

The nice thing is that once we configure our slave nodes once, we don't have to touch them again (as far as configuration goes really). On each of the nodes:

sudo apt-get install nagios-nrpe-server nagios-plugins

This will install a whack of dependencies, for me it totalled to just over 100MB. Once that is installed, we'll have to add our master node IP to each slave. Edit /etc/nagios/nrpe.cfg, find the following line and add your master node IP address:

...
allowed_hosts=127.0.0.1 master.node.ip.address
...

Once completed, restart nrpe-server

sudo /etc/init.d/nagios-nrpe-server restart

Monitoring configuration

We'll have to tell Nagios where our slave nodes configurations are, so edit /usr/local/nagios/etc/nagios.cfg and uncomment the following line:

cfg_dir=/usr/local/nagios/etc/servers

Create the directory which will hold our slave nodes

sudo mkdir /usr/local/nagios/etc/servers

The config for the slave nodes is located at /usr/local/nagios/etc/servers/clients.cfg. Fortunately, there are other tutorials that are more in depth than mine regarding monitoring. Currently, I'm using a basic config to see if the slave node is alive, and check for HTTP/SSH. See below for an excerpt:

define host{
use                             linux-server
host_name                       your_fancy_hostname
alias                           your_fancy_hostname
address                         slave.node.ip.address
max_check_attempts              5
check_period                    24x7
notification_interval           30
notification_period             24x7
}


define service {
        use                             generic-service
        host_name                       your_fancy_hostname
        service_description             SSH
        check_command                   check_ssh
        notifications_enabled           0
        }
define service {
        use                             generic-service
        host_name                       bonbon
        service_description             HTTP
        check_command                   check_http
        notifications_enabled           0
        }

And of course once we're done making changes, restart Nagios

sudo /etc/init.d/nagios restart

If all is well, you'll end up with something like this:

For me, all I'm looking for is basic application and uptime monitoring. Nagios is quite extensible, having a catalogue of plugins. There is of course more extensive documentation on monitoring configuration as well.

References: http://redmine.lighttpd.net/projects/1/wiki/NagiosRecipe http://library.nagios.com/library/products/nagiosxi/documentation/299-installing-the-nagios-ubuntu-and-debian-linux-agent