Synchronizing Puppet generated Icinga configuration

The Basics
One of the features of Puppet is to have so-called Stored Configurations. With those it is possible that the Puppet clients publish some of their local data back to the Puppet master. For example, it is possible for the clients to report to the master which Icinga (Nagios) check it needs like shown in this example:

  @@nagios_service { "check_load_${hostname}":
      use => "generic-service",
      host_name => "$fqdn",
      service_description => "Load",
      check_command => "check_nrpe!check_load",
      }

In turn the Puppet master realizes this configuration with some Puppet code like this:

  Nagios_service <<||>> { }

This will really create a ready to use Icinga configuration. Assume a host whose name resolves to ‘myhost’, this Puppet code will create a file where you’ll find a service configuration like this:

define service{
	use                     generic-service
	host_name               myhost
	service_description     check_load_myhost
	check_command           check_nrpe!check_load	
	}

Pretty easy so far. If you now have Icinga running on the same host as the Puppet master, you just have to make sure to reload Icinga’s configuration and the new check will be in place.

Icinga is not on Puppet master
But what if Icinga is not running on the same host as the Puppet master? How to get the created configuration in the right place on the right host? Actually I had to switch from having Icinga and Puppet on one host to another scenario: I now had two Puppet masters, each serving several environments and one Icinga host for everything. All running on different boxes. See the next picture.

The basic problem

The basic problem: Synchronizing Puppet and Icinga

Asking Puppet User mailing list, I got some helpful hints which served as the basis for the solution I’m showing here.

Analyzing the problem turned out several basic things to do:

  1. Create the Icinga configuration with Puppet’s stored configurations on each Pupet master.
  2. Get the generated configurations from the Puppet boxes to the Icinga host.
  3. If a new configuration is available, check if it’s valid.
  4. If a new configuration is valid, reload Icinga’s configuration.
  5. Have everything done automatically without human intervention.
  6. Have everything configured with Puppet.

For several weeks now (at the time of writing this), I have established everything that makes the points of the above list run and I’d like to show the details here. First, see the next picture for a general overview of how it is implemented right now.

The overall concept

The overall concept of synchronizing Puppet and Icinga

Beware that I’ll show only how to get along with service checks. Of course this applies to all Icinga resources within Puppet.

Creating the configuration
Every time the Puppet agent, which runs on Puppet master’s host is executed, the stored configurations ‘should be’ (see problem below) regenerated. That is, the configuration files for Icinga are recreated. I did not want to have the default directory and I needed special access rights for the generated files, so I changed the Puppet configuration that realizes the resources to something like this:

  include icinga
  $conf_file_srvs="srvs_${hostname}_pupp_gen.cfg"

  Nagios_service <<||>> {
    target => "${icinga::baseconfigdir}/${conf_file_srvs}",
    before => File["${icinga::baseconfigdir}/${conf_file_srvs}"],
  }

  file { "${icinga::baseconfigdir}/${conf_file_srvs}":
    ensure => "file",
    owner => "puppet",
    group => "puppet",
    mode => "0644",
    backup => false,
    replace => false,
  }

I changed the target directory for the generated Icinga files (line 5 and lines 9 – 16). There’s one weird thing about this. Obviously the Icinga configuration doesn’t get recreated once they have been created the first time. Even if the stored configurations change, the content of the configuration files keeps the same. I found no solution for this so far – even with the help of Puppet’s user list, so I established a workaround with a cron job. It’s a bit ugly, but the only thing that really provides a reliable recreating of the files seems to have the Puppet agent, which runs on the Puppet master’s host, not running as a daemon process, but instead start it in regular intervals and before starting, delete the previously generated Icinga configuration files. So I’ve created a script execPuppetAgent.sh which is triggered by a cron job and looks like this:

#!/bin/bash

PUPPET_OPTS="--onetime --no-daemonize"
CFG_DIR="/tmp/puppet_pm1"

/bin/logger "$0: Preparing Puppet agent for execution"
/bin/rm -f ${CFG_DIR}/*
/bin/logger "$0: Deleted Icinga configuration files below ${CFG_DIR}"
/bin/logger "$0: Starting Puppet agent now"
/usr/bin/puppet agent $PUPPET_OPTS
/bin/logger "$0: Puppet run finished"

The ${CFG_DIR} variable of course points to the directory where Puppet creates the Icinga files. This script is now executed every 30 minutes on both of my Puppet masters.

Getting the configuration to Icinga – rsync
Now we need a way to get the configuration files from the Puppet masters to the Icinga host. An easy way to do this is to use rsync.

So I have a script in place running on the Icinga box the does basically:

  1. rsync the configuration from both Puppet masters.
  2. Check if the new configuration files are valid.
  3. Reload Icinga’s configuration if the files are valid.
  4. If any errors occur, the script sends an email or logs to syslog.
#!/bin/bash

ICINGA_BASE="/etc/icinga"
PM2_DIR="pm2"
PM1_DIR="pm1"
REMOTE_DIR="/tmp/puppet_generated"
PM1_FQDN="pm1.somedomain.net"
PM2_FQDN="pm2.somedomain.de"
RSYNC_ARGS="-avz -e ssh"
USER="puppet"
MAIL_ADDR="somemail@somedomain.de"
ERR_MSG="The rsync job failed!nIcinga configuration my not be up to date.nError message from rsync was:n"

# Function sends mail with the text given in $1.
function sendMail {

   /usr/bin/printf "%b" "$1" | /bin/mail -s "Icinga/rsync notification" $MAIL_ADDR
}

echo "Syncing configuration from PM2"
RESULT=`rsync ${RSYNC_ARGS} ${USER}@${PM2_FQDN}:${REMOTE_DIR}_pm2/ ${ICINGA_BASE}/${PM2_DIR} 2>&1`
EXCODE_PM2=$?
if [ $EXCODE_PM2 -gt 0 ]; then
  sendMail "PM2: ${ERR_MSG}${RESULT}"
fi


echo "Syncing configuration from PM1"
RESULT=`rsync ${RSYNC_ARGS} ${USER}@${PM1_FQDN}:${REMOTE_DIR}_pm1/ ${ICINGA_BASE}/${PM1_DIR} 2>&1`
EXCODE_PM1=$?
if [ $EXCODE_PM1 -gt 0 ]; then
  sendMail "PM1: ${ERR_MSG}${RESULT}"
fi

# Now let's check the new configuration
/usr/bin/icinga -v /etc/icinga/icinga.cfg
if [ $? -eq 0 ]; then
  if [ $EXCODE_PM2 -eq 0 -a $EXCODE_PM1 -eq 0 ]; then
    logger "New configuration from Puppet masters available. Going to reload Icinga's configuration now."
      /etc/init.d/icinga reload
  else
    logger "Did not restart Icinga due to previous rsync errors."
  fi
else
  sendMail "Icinga's configuration pre check failed after syncronizing from Puppet master!nPlease check manually."
fi

I’ve mentioned previously that the Puppet agent is restarted every half an hour and before restarting the configuration files for Icinga will be deleted. So I had to ensure that the synchronize script will not run at the same time Puppet is regenerating the configuration. This is basically very easy to accomplish by configuring the cron jobs accordingly.

This is the Puppet configuration for the cron job that triggers restarting of Puppet agent:

  cron {"execute-agent":
    command => "/root/bin/execPuppetAgent.sh",
    ensure => present,
    user => root,
    hour => "*",
    minute => "*/30",
    require => File["/root/bin/execPuppetAgent.sh"],
  }

And this is the Puppet configuration for the cron job that synchronizes the configuration files:

  cron { sync-stored-config:
    command => "/var/icinga/bin/sync-stored-config.sh",
    user => "icinga",
    hour => "*",
    minute => [15,45],
  }

As you can see, the configurations (and the restart of the Puppet agent) will
happen every 0th and 30th minute an hour, the synchronization runs every 15th and 45th minute an hour. So there will be no crossover.

Conclusion and Credits
Seems I’m the only one having the above mentioned problem concerning the not recreated configuration files. However, with this workaround, I have a pretty stable and reliable system now. As soon as a new host id added to Puppet’s configuration, it takes some 30 minutes until the new Icinga checks are in place. No additional work to do, which is very comfortable.
Thanks to Puppet User mailing list which gave me some helpful tips and of course thanks to Puppetlabs for providing this great tool for configuration management. For a general introduction into stored configurations you may refer to Puppet Labs’ introduction.

, ,

No comments yet.

Leave a Reply

* Copy This Password *

* Type Or Paste Password Here *