Ever wondered why switching the NRPE daemon to debug mode has no effect in any log files at all? I did, a lot of times. You may have a configuration something like this in your nrpe.cfg file:
log_facility=daemon pid_file=/var/run/nrpe.pid server_port=5666 nrpe_user=nrpe nrpe_group=nrpe allowed_hosts=some.ip.address.here debug=1
Line 1 indicates it should log everything to the syslog daemon. Depending on your system, this will most likely be the file /var/log/messages or something else depending on your log daemon. Line 7 should switch the NRPE daemon to debug mode and issue some more information than the usual standard output – but it does not.
My clients observed by Icinga were all running on RHEL6. So I decided to have a closer look at the syslog configuration. It turned out that rsyslog was running on my boxes. The configuration file for this log daemon is /etc/rsyslog.conf. Digging into the man pages and some ‘googling’ I found out that something like this was missing inside this file:
After adding this line and restarting the daemon with
[root@somehost ~]# service rsyslog restart
I got all the debug information I was missing so far in /var/log/debug.
Even though this might be a little help when trying to find errors when executing remote Icinga/Nagios checks, this may not be sufficient. Maybe you will also have a look at another post I made which covers a very special case here which covers a problem with failing remote checks caused by SELinux.