[from [email protected]] Using Netsaint with SNMPTT (scalable trap handling for Netsaint/Nagios eq N/N)

Dear Ladies and Gentlemen,

I am writing with a letter from Mr Alex Burger about using his SNMPTT
(SNMP Trap Translator) system to provide scalable support to
Netsaint/Nagios (N/N) for SNMP traps.

This thread comes about after Mr Brian Seppanens letter about this
matter and my recent interest in getting N/N to swallow traps with as
little effort on my part as possible (in particular I would like to
avoid

. adding another case to snmptrapd.conf and restarting snmptrapd

. 'writing' another traphandler to handle the new trap

)

Mr Burger saw fit to provide a detailed reply to my dumb questions about
integrating SNMPTT with Netsaint. 

He has been using Netsaint with SNMPTT successfully for some months.

If you have an interest in Netsaint trap handling, I think his comments
are helpful and encouraging, and will get you handling traps with
N/N quickly (provided you have read the N/N trap docco and have
Net-SNMP installed and running).

One of the many attractive aspects of SNMPTT is that it supports
distributed monitoring as well as N/N does. An SNMPTT can act as trap
collector and Netsaint passive check submitter as outlined in the N/N
docco (in this case the program SNMPTT EXECs ('submit_check_result')  
would invoke the ncsa_client command).

I have pruned some of my remarks quoted in Mr Burgers reply.

Yours sincerely.



----- Forwarded message from Alex Burger <[email protected]> -----


On Fri, 22 Nov 2002, Stanley Hopcroft wrote:

> It looks like a real good way of reducing the cost of supporting traps.
> I am interested in using SNMPTT as a shim between snmptrapd and
> Netsaint/Nagios.

I hope it works well for you!  If you find any problems, or have any
suggestions, please let me know.

I am actually using SNMPTT with Netsaint 0.7 right now, and have been
since I wrote SNMPTT back in April.  I was planning on adding some
documentation on how to integrate it into Netsaint, but have not got
around to it yet.  I'll try to add something to the next release's
documentation.

> Your code would comprehend the trap and call something else that would
> produce a Netsaint/Nagios PASSIVE_SERVICE_CHECK and feed it into the
> Netsaint/Nagios command Queue

I have configured all of my traps to EXEC the following command:

EXEC /usr/local/netsaint/libexec/eventhandlers/submit_check_result $r
TRAP_SYSTEM 1 "the trap message..."

The arguments for submit_check_result are:

#  $1 = host_name (Short name of host that the service is
#       associated with)
#  $2 = svc_description (Description of the service)
#  $3 = return_code (An integer that determines the state
#       of the service check, 0=OK, 1=WARNING, 2=CRITICAL,
#       -1=UNKNOWN).
#  $4 = plugin_output (A text string that should be used
#       as the plugin output for the service check)s

As you can see, $r is the hostname, TRAP_SYSTEM is the service, 1 is the
WARNING return code.

For each host, I have defined the following service:

service[serverxx]=TRAP_SYSTEM;1;none;1;1;1;cg_core;31536000;24x7;0;1;1;;check-host-alive

You have to provide some sort of a check for a service, so I use
check-host-alive (ping).  This also allows me to force a service check
which results in an OK to 'clear the trap'.  The 31536000 is the
notificaton interval.  If you set it to 0, you'll never get notified, so I
picked a real large number.  The other values should make sense based on
the N/N passive docs.

BTW, I call it TRAP_SYSTEM because I also defined a TRAP_APP to allow
application traps to be sent to a different contact group.  For example,
if a process has an exception error on a web server, I want the web people
to know also.  If that hard drive fails, it's my problem - not theirs.

> You may want to consider adapting SNMPTT to become or provide more
> support for N/N.
>

I'm not sure how I could improve SNMPTT for better Netsaint support.  I
know the way I am integrating SNMPTT into Netsaint is not perfect.  For
example:

-every host needs to have a TRAP service defined
-because there is only one TRAP service, all traps use that service.  This
is not good because the Netsaint status will only show the last trap
received.
-My Compaq snmptt.conf file contains hundreds of traps.  It would not be a
good idea to define 100 service definitions for each host - that would be
crazy
-If for example you had a Cisco router with 4 interfaces, how would you
define that in Netsaint?  One host, 4 'services'?  It would be tricky to
integrate a link up / down with SNMPTT and Netsaint.
-You could define a link up to send an OK on the exec line, and a link
down to send a WARNING, but there is still the above question.

> Netsaint for example may not recognise hosts by FQDN and in any case, a
> Netsaint service is labelled by Netsaint host name + Netsaint service
> description. Maybe a hash of records will do, or maybe the shim will
> read the Netsaint config file ...)

All of *my* host definitions in Netsaint are named after the host name, so
the server server01.domain.com is called 'server01' in Netsaint.  I do
not have a problem with FQDN because I use a host table on the Netsaint /
SNMPTT machine.  With the host file, snmptrapd will pass just the host
name instead of the FQDN to SNMPTT.  This results in the $r being just
the host name.  I could add a feature (new  variable) that strips the
host name off of the domain name, but I don't think it's really needed,
unless other systems do not work the same as mine for host name
resolution.

Someone else was doing the same thing before I mentioned the hosts file to
them.  I think they did what you are talking about.  I'll look for the
message.

> If not, and there is any remarks that my work with your code leads to I
> will let you know.

If you have any suggestions, I would love to hear them!

I recently added MySQL support to SNMPTT.  Someone else is planning on
using SNMPTT and have it store all the received traps in a SQL database.
He is then going to have a web site that reads the contents, and allows
techs to respond - flag as acknowledged etc.  This person is not using
Netsaint so they are basically writing their own logging / acknowledgement
system.

Due to the current design of SNMPTT and Netsaint, they do not mesh
together perfectly.  If you were to compare to something like HPOV
(limited knowledge), everything is based on traps.  If HPOV pings a
machine and it's down, it sends a trap to itself for 'Node Down' which is
then handled by EXECing a program, logging etc.  Basically, everything is
designed around the idea of traps.

If Netsaint supported 'dynamic' services such as allowing a passive
service check for a service that doesn't exist, then SNMPTT integration
would be much easier.  For example, a server sends 5 traps and SNMPTT
executes 5 EXEC statements with:

EXEC /usr/local/netsaint/libexec/eventhandlers/submit_check_result $r
TRAP_$N $s "text"

$r is the host name that matches the device / host name in Netsaint
$N is the event name (text after EVENT such as COMPAQ_11003)
$s would be the severity (which should be defined in the snmptt.conf
   file as 0=OK, 1=WARNING, 2=CRITICAL)

Dynamic services that stay until they are set to OK would allow us to see
many traps for a host.  They could be cleared by forcing an immediate
check which would result in an OK from the ping (or whatever).

Thanks

Alex






----- End forwarded message -----

-- 
------------------------------------------------------------------------
Stanley Hopcroft
------------------------------------------------------------------------

'...No man is an island, entire of itself; every man is a piece of the
continent, a part of the main. If a clod be washed away by the sea,
Europe is the less, as well as if a promontory were, as well as if a
manor of thy friend's or of thine own were. Any man's death diminishes
me, because I am involved in mankind; and therefore never send to know
for whom the bell tolls; it tolls for thee...'

from Meditation 17, J Donne.


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf