opEvents EventParserRules - Adding Rules For SNMP Traps and Syslog Messages

Overview

opEvents provides the ability for the adminstrator to customise an event's properties from a variety of inputs. For example, if a user wanted to set a specific priority for an event it can be done during the input parsing stages.  This article will provide a methodology for creating events from SNMP traps, via a generic extensible parser with EventParserRules.

The generic parser rules are defined in EventParserRules.json which  is found in the configuration directory /usr/local/omk/conf.  Please read the notes at the top of this file first as they are very informative as to what is possible in regard to the parser rules.

Evaluate The Traps To Be Processed

Create a list of SNMP traps that are required be processed by opEvents. 

Correlate Events Into Stateful Pairs

For this discussion we will assume that the concept of 'state' is desirable.  i.e. If there is a "down" event, there should be a corresponding "up" event, and opEvents should keep track of the state and ignore duplicate inputs. (It is possible that several "down" events could share a single "up" or clearing event.)

State

opEvents tracks state based on a tuple of three event properties.

  • node
  • element
  • stateful

This is a critical concept.  The node property will always be the same for any given node.  The element property will be somewhat dynamic, usually a regular expression will parse and 'capture' it.  The most comment element example would be an interface; gig0/0 versus gig0/1.  The stateful property is necessary because the same element may have different events; consider an interface down event versus an OSPF event on the same element (gig0/0).

If any of these three event properties are not set state will not function well. 

Consider a case where the element property is not set; thus being null.  In this case if a 'port down' for gig0/0 was received a 'port up' for gig0/1 would clear the g0/0 'port down' event.  Without the element being set opEvents cannot differentiate between interfaces.


Example parser rule for the element property.

            "53" : {
               "IF" : "IF-MIB::ifIndex\\.\\d+=(\\d+)",
               "THEN" : [
                  "capture(element)"
               ]
            },

Example parser rule for the stateful property

            "51" : {
               "IF" : "IF-MIB::linkDown",
               "THEN" : [
                  "set.event(Interface Down)",
                  "set.stateful(Interface)",
                  "set.state(down)",
                  "set.priority(3)"
               ]
            },


Create Parser Rules

opEvents will process the trap log file as specified on opCommon.json.  When parsing the traps, at least the following properties should be extracted:

  • date
  • host
  • trap
  • details
  • event
  • element
  • stateful
  • state
  • priority

The shipped version of EventParserRules.json has a traplog section that will extract the date, host, trap and details fields for most situations. 

This article focuses on situations where customers want customization for the remaining fields.

Set the Element

Review all the SNMP traps to determine which OID best describes what will become the element property.  Write a regular expression that matches this.

### This is observed the trap opEvents is receiving, and 
### is the best candidate to become the element property:

STARENT-MIB::starSlotNum=6

### Refering to the vendors mib file starSlotNum is found:

starSlotNum OBJECT-TYPE
    SYNTAX  Integer32(1..48)
    MAX-ACCESS  accessible-for-notify
    STATUS  current
    DESCRIPTION
        "The slot number"
    ::= { starSlotEntry 1 }

Based on this we can write the regular expression to set the element.

"2": {  
	"DESCRIPTION": "Set element for card number.",
    "IF": "(STARENT-MIB::starSlotNum=\d+)",
    "THEN": ["capture(element)"]
},

Notice the regular expression will catch an number of digits following the '=' character.  This rule 'captures' the element.  In this way we can dynamically assign event properties based on a regular expression.

Set Other Properties

Generally the other properties that we wish to set can be done with one rule. Consider the following trap received by opEvents.

2017-07-12T12:23:37   10.113.176.4    UDP: [10.113.176.4]:36570->[10.255.26.7]       
SNMPv2-MIB::sysUpTime.0=6:1:04:53.72    
SNMPv2-MIB::snmpTrapOID.0=STARENT-MIB::starCardTempOK   
STARENT-MIB::starSlotNum=1        
STARENT-MIB::starCardTemperature=40 degrees Celcius     
SNMPv2-MIB::snmpTrapEnterprise.0=STARENT-MIB::starentTraps

Evaluating this trap it's determined that a single rule can set the properties below. 

Notice that if  the author is creative and accurate with regular expressions the number of rules my be decreased.

"103": {
    "IF": "STARENT-MIB::starCardTempOK",
    "THEN": [
      "set.event(Card Temperature OK)", 
      "set.stateful(temperature)", 
      "set.state(up)", 
      "set.priority(2)"
    ]
},


Based on a match of "STARENT-MIB::starCardTempOK", the rule will take action.

  • event - "Card Temperature OK"
  • stateful - "temperature"
  • state - "up"
  • priority - "2"

Evaluate The Syslog Messages To Be Processed

Create a list of Syslog messages that are required be processed by opEvents. 

For example: 

Apr 01 16:38:29 CNOC-01 b102ogt: [SYSTEM]<6> Local authentication failed(user: admin): Admin password error.

Create Parser Rules for Syslog

opEvents will process the syslog log file as specified on opCommon.json. 

"opevents_logs" : {
         "traplog" : [
            "<nmis9_logs>/trap.log"
         ],
         "nmis_eventlog" : [
            "<nmis9_logs>/event.log"
         ],
         "tivoli_log" : [
            "<nmis9_logs>/tivoli.log"
         ],
         "cisco_compatible" : [
            "<nmis9_logs>/cisco.log"
         ],
         "syslog_message" : [
            "<nmis9_logs>/syslog.log"
         ],
         "winlogd" : [
            "<nmis9_logs>/winlogd.log"
         ]
      },

When parsing the syslog, at least the following properties should be extracted:

  • date
  • host
  • details
  • event
  • element
  • stateful
  • state
  • priority

The shipped version of EventParserRules.json has a syslog section that will extract the date, host and details fields for most situations. 

This article focuses on situations where customers want customization for the remaining fields. 

Base on the message that we select we need to create a regular expression to extract the date, host and event. 

Apr 01 16:38:29 CNOC-01 b102ogt: [SYSTEM]<6> Local authentication failed(user: admin): Admin password error.

"syslog_message" : {
    "10" : {
	 "IF" : "^(\\w+\\s\\d+\\s\\d+:\\d+:\\d+)\\s(\\w+[-_]\\w+)",
         "THEN" : [
            "capture(date,host)"
         ]
        },
       	"11" : {
          "IF" : "Local authentication failed",
          "THEN" : [
            "set.event(Authentication Failed)",
            "set.priority(8)"
         ]
      },
   }


After this we need to restart the opeventsd daemon then opEvents will create an event for Authentication Failed.