In our environment, we make decisions on the importance of hosts based on their specific role. Puppet is great and has gone a long way towards simplifying our standard build. Finding out it has built-in types for working with nagios was exciting, but the problem became – how can we have our hosts automatically get added into nagios through puppet, but easily augment those additions with human-decided service levels? Leveling is the technique we use to determine which hours of the day we’re alerted to problems, and through which medium. For example, level1 means “any hour, day or night, via both email and text message.” and level5 means “email me during work hours only.”

The initial solution, unsurprisingly, came from puppet exported resources. You can read more about them [here]. Briefly, putting a @ character in front of an object definition makes it virtual. That is, the object will exist, but will not get sent to the client. Putting two @ characters in front of a definition also exports the virtual resource, making it available to other hosts. A bit of initial work is needed to enable exported resources, mysql (or some other database) must be set up. So, to do that:

[root@puppet ~]# cat /etc/puppet/puppet.conf
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl

[puppetmasterd]
storeconfigs = true
dbadapter = mysql
dbuser = puppet
dbpassword = whatever
dbserver = localhost
dbsocket = /var/lib/mysql/mysql.sock
downcasefacts = true

Now that that’s out of the way, let’s see how we create nagios hosts for each puppet client. In our environment, my base standard-config class is called every-server. So, in everyserver.pp, we find this:

@@nagios_host { “${fqdn}”:
ensure => present,
alias => “${hostname}”,
address => “${ipaddress}”,
use => “level5”,
target => “/etc/nagios/dynamic.cfg”;
}

@@nagios_service { “${hostname}_check_ping”:
ensure => present,
host_name => “${fqdn}”,
notification_interval => 60,
flap_detection_enabled => 0,
service_description => “Ping”,
check_command => “check_ping!300.0,20%!500.0,60%”,
use => “level5”,
target => “/etc/nagios/dynamic.cfg”;
}

@@nagios_service { “${hostname}_check_ssh”:
ensure => present,
host_name => “${fqdn}”,
notification_interval => 60,
flap_detection_enabled => 0,
service_description => “SSH”,
check_command => “check_ssh”,
use => “level5”,
target => “/etc/nagios/dynamic.cfg”;
}

So what does this do? Every client that connects to puppet creates these objects. However, they do not exist on the client in /etc/nagios/dynamic.cfg because the @ characters mark the resource as virtual and exported. So basically, each host creates its own nagios host object and two nagios service objects. As you can see, every host and service defaults to level5, the least important level. Now, the question becomes, how do we get these objects into nagios, and more specifically, how do we override the level parameter before writing out these objects?

In my nagiosmonitor.pp file, I have this:

class nagios-monitor inherits every-server
{
# If the dynamic file changes, restart nagios so it picks up the new definitions…
file {
“/etc/nagios/dynamic.cfg”:
mode => 644,
notify => Service[“nagios”];
}

# Collect/overwrite the service level used by a specific host and service…
Nagios_service <<| title == "hostname1_check_ssh" |>> {
use => level3
}

# Collect everything else, which will be staying at the default level5…
Nagios_host <<||>>
Nagios_service <<||>>
}

This looks a lot more complex than it really is. The <<| |>> bit is the collection operator, which makes virtual resources into real resources which will now be sent to the host. Since this is only in the nagios-monitor manifest, only nagios hosts will receive the file /etc/nagios/dynamic.cfg, which is what we want. Also, since nagios-monitor inherits every-server, it will create nagios_host and nagios_service objects for monitoring itself. Now, in between the pipes of the <<| |>> operator, one can specify attributes for which resource to realize. And after the <<| |>>, one can specify curly braces if one wishes to override any of the parameters in the resource. So, first, I realize a specific service, titled in everyserver.pp as a hostname and a check command. When I realize the check_ssh service on host hostname1, I bump its level from level5 to level3, making it a more important service in a simple one-liner. This can be repeated as many times as you wish, and is fairly easy to augment and maintain since it is so compact.

What follows this overwrite section is the collection process for all nagios hosts and services for which we are not overwriting any attributes. This is why the area between the pipes is empty. The result of these manifests is a file on the nagios servers named /etc/nagios/dynamic.conf which contains definitions for all hosts currently in puppet, and checks ping and ssh for all of those servers. The checks are all at level5, except for the check_ssh check running against the host hostname1, which will be at level3.