Nagios on a Pi, NagiosGraph on a Dedicated Web Server

Nagios and NagiosGraph running on separate servers.

Update: The Raspberry Pi 2 with it's quad-core CPU has enough grunt to run NagiosGraph on the device. It's better/easier to upgrade as your graphs will be live instead of delayed by the RRD file sync schedule.

To update a Pi 1 Raspbian install to support the Pi 2, update it with 'sudo apt-get update', 'sudo apt-get upgrade' and 'sudo apt-get dist-upgrade'.


After playing with Nagios in a VM on my home network, I decided to keep it around on a Raspberry Pi - there's no point it being a VM running on the machine it's monitoring.

Nagios runs nicely on the Pi, pings are a little higher but consistent. NagiosGraph however puts the load through the roof for 30-60s while generating graphs. During this time, host/service checks can time out triggering false alerts; a minute is also longer than my patience likes. I need NagiosGraph on something more powerful.

While setting up NagiosGraph, I'd seen the RRD files in /usr/local/nagiosgraph/var/rrd/ so I threw these at a separate web server with NagiosGraph on to see if they'd stick. It took a bit of fiddling due to architecture differences, but it worked. If the web server is down, the graphs are still available on the Pi at the original URL.

This guide explains how I set this up. Note that this guide is specific to a Raspberry Pi Nagios install and an x86/x64 NagiosGraph install. If you just want to run NagiosGraph on a separate web server of the same architecture as the Nagios server you should be able to rsync the RRD files directly.

Setting Up NagiosGraph on a Dedicated Server

The process below will export the RRD files from Nagios running on Raspbian on the Pi, these will be transferred to NagiosGraph running on CentOS 7 x64 with Apache.

Because the RRD files are binary, created on a 32-bit RISC system and need to be read on an x64 system, they need to be exported to XML on the Pi and restored to RRD on the web server.

To follow this guide, you'll need:

  • A Nagios system setup to collect performance data with NagiosGraph already setup and displaying graphs.
  • An Apache server with both Nagios and NagiosGraph installed.
    • Set this up in the same way you set it up on the Pi, except you do not need to configure or start the NagiosGraph or Nagios services. NagiosGraph on the Pi will be processing the performance data, Nagios needs to be present to install NagiosGraph.
    • You should be able to access the NagiosGraph CGI scripts on this server, for e.g. http://example.com/nagiosgraph/cgi-bin/export.cgi should return usage instructions.
  • The following tools installed on both the Pi and web server:
    • cron
    • rrdtool
    • rsync
    • ssh
  • The SSH server should be running on the web server and the firewall configured to allow connections from the Pi.

For simplicities sake, this guide will run the scripts as root as I don't want to confuse things messing with permissions. You should setup a user to do this securely.

Setup SSH Passwordless Login

I'm going to blast through this, if you need a more thorough guide, there are thousands out there.

On the Nagios server, create the public/private keys.

[root@nagios ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): leave blank
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
08:e2:e9:4c:d4:ba:d9:a0:af:33:05:48:4f:a3:a8:1b root@nagios.savnet.cjsavage.com
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
| . +             |
|+ * +            |
|++ = . .         |
|..*   . S        |
|E=.=             |
|.+= .            |
|+.               |
|.+.              |
+-----------------+

Copy the public key to the web server and add it to the authorised keys file.

[root@nagios ~]# scp /root/.ssh/id_rsa.pub root@savnet-www-001:~/.ssh/nagios.pub
root@savnet-www-001's password:
id_rsa.pub                    100%  417     0.4KB/s   00:00
[root@nagios ~]# ssh root@savnet-www-001
root@savnet-www-001's password:
[root@savnet-www-001 ~]# cat .ssh/nagios.pub >> .ssh/authorized_keys
[root@savnet-www-001 ~]# chmod 600 .ssh/authorized_keys
[root@savnet-www-001 ~]# exit

You should be able to (from the root account on the Nagios machine) ssh root@savnet-www-001 and get a terminal without being asked for a password. If this doesn't work, check your /etc/ssh/sshd_config file on the web server isn't preventing key-based or root logins if you're using root.

Scripts and Directories

Using rrdtool directly on the RRD files used by NagiosGraph seemed to upset it. To get around this, I copied the files to a temporary directory for processing, these then get rsync'd to the web server where another script will pick them up, convert them back to RRD and update the NagiosGraph files.

Create a working directory on both Nagios and the web server.

[root@savnet-www-001 ~]# mkdir /opt/nagiosgraph-import
...
[root@nagios ~]# mkdir /opt/nagiosgraph-export

Create the script files:

On the Nagios machine, create the file /opt/nagiosgraph-export/sync-rrds.sh with the following content:

#!/bin/bash

rrd_path="/usr/local/nagiosgraph/var/rrd/";
rrd_dest="/opt/nagiosgraph-export/xml/";

# Prepare the destination
cp -R ${rrd_path} ${rrd_dest}

# Get all of the RRD files in $rrd_path
cd ${rrd_dest}
rrd_files=`find -name *.rrd`;

for line in ${rrd_files}
do
    xml=${line%.rrd}.xml
    rrdtool dump ${line} ${xml}
done

rm ${rrd_files}
rsync -a /opt/nagiosgraph-export/xml/ root@savnet-www-001:/opt/nagiosgraph-import/xml/ 2>/dev/null
rm -Rf ${rrd_dest};

And on the web server: /opt/nagiosgraph-import/build-rrds.sh with the contents:

#!/bin/bash

rrd_path="/usr/local/nagiosgraph/var/rrd/";
xml_src="/opt/nagiosgraph-import/xml/";

cd ${xml_src}
xml_files=`find -name *.xml`

for line in ${xml_files}
do
    rrd=${line%.xml}.rrd
    rrdtool restore ${line} ${rrd}
    rm $line
done

rm -R ${rrd_path}*
mv -f * ${rrd_path}

Make both scripts executable:

[root@nagios ~]# chmod +x /opt/nagiosgraph-export/sync-rrds.sh
...
[root@savnet-www-001 ~]# chmod +x /opt/nagiosgraph-import/build-rrds.sh

Review both scripts, remember to change the location of the RRD files if different on your install and change the rsync destination to your server's hostname.

Run both scripts and ensure that NagiosGraph on the web server is displaying graphs from the Nagios server. You can do this by adding /cgi-bin/showhost.cgi?host=$HOSTNAME$&period=day&expand_period=day to the end of your NagiosGraph server URL replacing $HOSTNAME$ with the name of one of your monitored hosts. Worst case, view the graph on the Pi and amend the URL.

Setup the Cron Schedule

Once you've confirmed the above is working, all that's left to do is automate the process and a final tweak to the URL pointers.

The most important part of automating this is leaving enough time for each script to complete. You can run the scripts with the time command to get their execution time and plan accordingly. This will be hugely dependant on how many hosts and services you are monitoring.

I opted for a 15 minute update cycle, you can choose whatever interval you like based on resources. Create a line in /etc/crontab on each server to run the scripts, I'm running the Nagios export every 15 minutes and the web server will pick these up 5 minutes later (5 past, 20 past, 35 past, 50 past etc.).

On the Nagios machine:

0,15,30,45 * * * * root nice -n 19 /opt/nagiosgraph-export/sync-rrds.sh

and on the NagiosGraph machine:

5,20,35,50 * * * * root nice -n 19 /opt/nagiosgraph-import/build-rrds.sh

Restart the cron service on both machines:

[root@nagios ~]# service cron restart
...
[root@savnet-www-001 ~]# systemctl restart crond.service

Make sure the cron daemon is set to start automatically and is running.

Fixing URL Pointers

When setting up NagiosGraph, you will have specified an 'action_url' for some or all of your services/service groups within the Nagios configuration. You need to amend this URL to point to the new NagiosGraph server.

Finally, you may have noticed the host links in NagiosGraph no longer work as they also expect Nagios to be running on the same box. To fix this, on the web server, open '/usr/local/nagiosgraph/etc/nagiosgraph.conf' and look for the line starting: nagioscgiurl.

This setting will have a relative URL pointing to something like: /nagios/cgi-bin/, change it to the fully qualified absolute URL for your web server:

nagioscgiurl = //nagios.savnet.cjsavage.com/nagios3/cgi-bin

That is it, you should now have NagiosGraph running on a separate server to Nagios.