Purpose

Provide a SNMP trap handling solution that can scale to 300 traps per second.

Overview

This solution leverages snmptrapd to initially pull the trap off the wire, apply access  control, translate, then forward it to rsyslog.  rsyslog then puts the translated trap in a log file to be processed by opEvents.  opEvents then applies filtering, parsing and actions as appropriate. 

SNMP Trap Processing - Line Diagram
 
snmptrapd--> rsyslog--> /var/log/nmis/snmptrap.log --> opEvents --> Blacklist --> EventParserRules --> SnmpTrapParserPlugin.pm

Deployment Steps

Step #1 - Configure snmptrapd to forward traps to rsyslog

Edit /etc/sysconfig/snmptrapd

Below is an example of configuring snmptrapd to send traps to rsyslog.  The '-Ls' flag tells snmptrapd to send logging output to syslog.  Using '-Ls 2' specifies that snmptrapd will send it with the local2 facility value.  The facility value is what rsyslog keys on for routing decisions.  Please review the snmptrapd and snmpcmd man pages.

 OPTIONS="-n -Ls 2 -p /var/run/snmptrapd.pid -m ALL -M /usr/local/nmis8/mibs/traps"

Verify /etc/snmp/snmptrapd.conf

Verify there is not reference to a traphandle in /etc/snmp/snmptrapd.conf

Step #2 - Configure rsyslog to route traps into a specified log file

We need the traps to be placed into a specified log file that opEvents will process.  The following example states that all messages with a facility of local2 will be placed in the /usr/local/nmis8/logs/snmptrap.log file.  Please review the rsyslog.conf man page.

local2.*                /usr/local/nmis8/logs/snmptrap.log

Most likely we will not want these messages to also go to /var/log/messages.  We can edit /etc/rsyslog.conf to prevent this from happening.  An example for facility local2 follows; notice the 'local2.none' statement.

 *.info;mail.none;authpriv.none;cron.none;local2.none                /var/log/messages

Step #3 - Configure opEvents to process SNMP trap log file

Modify opCommon.nmis

We need to tell opEvents to process the newly created snmptrap.log file.  This is done in /usr/local/omk/conf/opCommon.nmis.  Be careful with this file; in reality it is a perl hash, any syntax error will render the OMK server dead.  After modifying this file check it for syntax errors with the following command 'perl -c /usr/local/omk/conf/opCommon.nmis'.  If you are not scared you should be

Something like the following example needs to be added to the opevents section of opCommon.nmis. 

 'opevents_logs' => {       
  'snmptraps' => [
    '<nmis_logs>/snmptrap.log'
   ], 

Modify EventParserRules.nmis

EventParserRules.nmis is where parsing generally occurs.  In this case we are anticipating some complex maneuvers; so we are going to tell EventParserRules to send this to an opEvents plugin where complexity is better dealt with.  Remember all that big bad syntax talk?  Same applies here.

Something like the following example needs to be added to EventParserRules.nmis

%hash = (
        'snmptraps' => {
                1 => {
                        IF => 1,
                        THEN => "plugin(snmpTrap)"
                },
        },

Install SNMP trap parser plugin

Install an opEvents parser plugin such as: snmpTrap.pm .  This perl module will be placed in /usr/local/omd/conf/parser_plugins. 

Step #4 - Restart associated daemons

Restart the following daemons:

• rsyslog
• snmptrapd
• opeventsd

Step #5 - Verify

• Use tcpdump to observe snmptraps being recieved by the server
• Use the ps command to ensure snmptrapd, rsyslog, omkd, and opeventsd are running with the proper options
• Tail /usr/local/nmis/logs/snmptraps.log file
• Tail /usr/local/omk/log/opEvents.log
• Via the GUI; check opEvents views-> raw logs
• Via the GUI; check opEvents views -> events