Event Actions and Escalation

opEvents provides the Event Action Policy as a flexible mechanism for reacting to events. This document briefly describes how to configure the service, the policy language and the currently supported actions.

Action Policy Language

The action policy is configured in conf/EventActions.nmis, primarily in the section named policy. The policy consists of any number of nested if-this-then-that clauses, which specify the  conditions an event must conform to and what actions to take in case of a match. Further configuration sections specific to particular actions can be present in the same file.

Here is a brief example policy snippet:

'policy' => {
 '1' => {
 	IF => 'node.customer eq "important"',
 	THEN => {
	  '10' => {
		 IF => 'node.roleType eq "core" and event.event =~ "Down"',
 		 THEN => 'log.disaster() AND escalate.twentyfourseven()',
		 BREAK => 'true'
 	  },
	 '20' => {
		 IF => 'node.roleType eq "distribution" and event.event =~ "Down"',
		 THEN => 'priority(+2) AND email(admin)',
		 BREAK => 'false'
	  },
 	BREAK => 'false'
 },
 '2' => ...
},

The overall structure is relatively straight-forward: a rule has a numeric identifier which controls the order of evaluation, precisely one set of IF and THEN clauses and an optional BREAK property.

The IF expression is basically any arbitrary Perl expression, but tokens of the form event.name or node.name are substituted with the respective event or node property value. The special wildcards event.any and node.any are replaced by a logical true value. Furthermore, tokens that match extdb.queryname.column will be substituted with the result of an external enrichment query.

The standard event properties are listed on this page, and the common node properties are documented here.

The THEN clause is executed if and only if the IF expression evaluates as true (ie. non-zero, non-blank, defined). The THEN clause contains either a nested sub-policy, or a single string that specifies any number of action invocations separated by the token " AND ". The order of action invocations is relevant, but the token " AND " is just a separator: all given actions in a THEN will be executed regardless of success or failure of prior ones. All action invocations follow the same patterns: actionname(argument), actionname.subtype() or actionname.subtype(argument). The empty set of parentheses must not be omitted.

Policy evaluation starts at the outermost policy level, and proceeds in order of the numeric rule identifiers. All rules on the same nesting level are evaluated one after the other, unless a successful rule has its BREAK option set to true: in this case the rules after the successful one are skipped. In the example above, rule 20 would be skipped if rule 10 succeeds, and policy evaluation would contine at rule 2. If rule 10's IF does not match, then its BREAK option has no effect. If the IF expression of rule 1 doesn't match, then the sub-policy 10/20 isn't considered at all.

Action Policy Timing

Normally all newly createded events are subject to policy actions immediately after having been created, but this can be fine-tuned and adjusted:

• No policy actions are performed for events with the property action_checked set to 1 or property action_required set to 0.
• If the configuration option opevents_no_action_on_flap is set to true in conf/opCommon.nmis, then no action is performed on a flap event. 
• Policy action handling is delayed by state_flap_window seconds for all stateful events, so that state flaps can be detected before any actions are performed.
• Policy action handling is delayed for synthetic events, if the event creation rule sets the property delayedaction.

Supported Policy Actions

Notes for watchdog and element_watchdog

The watchdog timer system does require a priming event to establish the timer in the first place, and if a timer is disabled using watchdog.disable() it is completely removed and forgotten.

The consequence of this design is that newly added nodes or elements are not subject to any watchdog timers until opEvent receives an event that causes the watchdog creation. This is normally not a problem, unless such a new node is not creating events because it is down for example. To create watchdog timers without or independent of an event, you can use opeventd's command line event creation facility.

Configuration for log.XYZ()

The log section of conf/EventActions.nmis controls the log types to be made available for actions. Here is an example:

'log' => {
 'tmp' => {
	format => 'text', # a simple textual log
 	file => '/tmp/opevents.log', # last-or-all messages go into this file
 	mode => 'append', # if not given, the log is overwritten
 },
 'machinelog' => {
 	format => 'json', # a machine-oriented json log
 	dir => "/tmp/opevents_json", # the directory to log into (created on demand)
 },
},

You can setup any number of log types; just make sure that your log.type() action call uses the name of a defined log type.

Two log formats are supported, text and json.

Text logs contain only the most essential event properties as a tab-delimited list, one event per line. If the mode argument is not present, then the log file is overwritten every time the action is executed; the more common mode would be append.

JSON logs on the other hand contain all event properties, one event per JSON file. You have to give a dir option which specifies where those logfiles will be created. The logfiles are named timestamp-number.json, timestamp being the UNIX timestamp and number being a running counter (the UNIX timestamp has a one second granularity, number differentiates between multiple events in a single second).

Configuration for script.XYZ()

The script action lets you execute a program of your choice, and optionally captures and saves that program's output with the event. As usual, the section script of conf/EventActions.nmis contains the required configuration directives:

'script' => {
	'traceroute_node' => {
		arguments => '--max-hops=20 node.host',
		exec => '/bin/traceroute',
		output => 'save'
	},
 	'ping_node' => {
		arguments => '-c 5 node.host',
		exec => '/bin/ping',
 		output => 'save'
 	},
 }

The path to the program file must be given in the exec option. Arguments can be passed to the program; simply add them to the arguments option. Any tokens of the form event.name or node.name will be replaced by the named event or node property, respectively. If the option output is set to save, then the output of the program execution is captured and saved with the event in question; otherwise the output is discarded.

Please note that opEvents currently does not support long-running programs in script actions; opeventsd blocks until the program terminates.

Configuration for email()

The action email is different from the  others in that its configuration is stored in separate files: conf/opCommon.nmis sets the global email parameters,  conf/Contacts.nmis contains the definitions of contacts that opEvents should know about, and conf/EventEmails.nmis defines which email template to use for a particular contact.

Here is an example mail section from opCommon.nmis:

'email' => {
 'mail_from' => 'yourmailname@yourdomain.com',
 # auth is attempted if both user and password are set
 'mail_user' => 'your_user_account@your_domain.com', 
 'mail_password' => 'your_password', 
 'mail_server' => 'smtp.yourdomain.com',
 'mail_server_port' => 25, # 487 is another common choice
 'mail_use_tls' => 'true', # use STARTTLS for encrypted smtp
},

At the very least you will have to set mail_server and mail_server_port to the appropriate values for your infrastructure; it is recommended that you use mail_use_tls so that emails (and username/password) are transmitted in encrypted form.

If your mail server requires smtp authentication for sending email, then set mail_user and mail_password to suitable values; It is also highly likely that you will need to adjust mail_from to a valid email address, which will be used as the sender's address.

The settings in Contacts.nmis are straight-forward and self-explanatory: a named contact section defines the name to use for the email action, and its Email attribute assigns one or more email addresses to this contact (multiple addresses must be given as a comma-separated string). At the current time opEvents uses only the Email part of Contacts.nmis.

Thus, to send event emails to contact xyz with email address abc@def.com, you have to specify the action as email(xyz), and add a contact section for xyz (with email abc@...) to Contacts.nmis.

Configuration for syslog.XYZ() and nmissyslog.XYZ()

Actions that involve syslog servers require that conf/EventActions.nmis contains a matching server definition in its syslog section, similar to this example:

'syslog' => {
 'server1' => {
	'facility' => 'local1',
 	'server' => 'localhost',
 	'protocol' => 'udp',
 	'port' => '514',
 },
 'server2' => {
 	'facility' => 'local1',
 	'server' => 'weth',
 	'protocol' => 'udp',
 	'port' => '514',
 },
},

The definition has to include the server name or address and the syslog facility to use; the port number defaults to 514, and at this time opEvents only supports syslog over udp protocol.

The syslog severity is computed from the event priority (see opEvents priority levels vs. NMIS and Syslog levels), or from the optional priority argument in the action call (e.g. syslog.someserver(7)).

Configuration for escalate.XYZ()

Escalation of open issues is handled flexibly in opEvents: you can specify which events should be potentially escalated, and you can formulate different policies for those escalations. Escalations in opEvents apply only to unacknowledged events.

Writing escalate.somepolicy() in a THEN clause marks the matched event for future escalation according to the escalation rules of somepolicy. An event can be subject to multiple escalation policies at the same time. All escalation policies that an event is marked for will be applied independently, and when a policy is unapplicable because of time and day restrictions, it is ignored - but only temporarily until the time and day match up again.

Only when an event is acknowledged does escalation for it cease. Events are normally acknowledged manually, but for stateful entities the "down" event is acknowledged automatically if the configuration option opevents_auto_acknowledge is enabled in conf/opCommon.nmis.

Escalation Policies

To formulate an escalation policy, you need to decide on your preferred escalation steps, their respective time thresholds and actions, and express that in section escalate of the config file conf/EventActions.nmis. Here is an example configuration fragment:

'escalate' => {
 'weekday' => {
 	'name' => 'weekday',
 	'IF' => {
 		priority => '>= 0',
 		days => 'Monday,Tuesday,Wednesday,Thursday,Friday',
 		begin => '9:00',
 		end => '19:00', 
 	},
 	'60' => 'log.problem() AND script.ping_node()',
 	'300' => 'email(operations)',
 	'1200' => 'email(operations_pager) AND script.disaster()',
 	'2400' => 'email(operations_manager)',
 	'3600' => 'email(it_manager)',
 },

 'afterhours' => {
...

Your escalation policy clearly needs a name; the example uses weekday and afterhours. The two other components of the escalation policy are the IF clause, which sets the scope of the policy, and the  list of escalation steps.

Escalation Time Restrictions

The IF clause is used to determine whether a particular escalation policy should be active at a given time, and for events of a given priority. The priority setting is required and contains a comparison operator, a space and a number; if your policy is to be unrestricted simply use >= 0 (priorities range from 0 to 10). The days setting is optional, and should contain a comma-separated list of weekdays when the policy should be active. begin and end set up the daily time range for this policy. The policy will be active in the interval between begin and end, if the begin time is earlier than end (like in the example above). To invert the interval meaning, ie. for outside work hours, simply swap begin and end over. For example, a policy with begin 18:00 and end 05:00 will work after 18:00 and before 05:00. If days are not given, then the policy works on all days. No begin means "starts at midnight" and no end is interpreted as "ends at midnight".

Escalation Step Definition

The remaining components of the escalation policy are the definitions of the escalation steps; these consist of the escalation threshold, and the actions to take. The escalation threshold (in seconds) specifies the minimum age of the unacknowledged event for this escalation step to activate, and the action part works the same as the THEN expression in the action policy.

When escalations are processed, the highest new escalation step is determined based on the age of the event, the associated actions are performed and the event state is updated. When escalations are processed next, only escalation steps higher than the most recently active one will be considered for this event. Please note that different escalation polices are applied independently and each has its own active highest escalation step.

With the example weekday policy above, an event would be acted  upon after 60 unacknowledged seconds, then again once it reaches 300 unacknowledged seconds and so on. Each action would be taken at most once: if the policy becomes active for the first time if the event is already 5900 seconds unacknowledged, then only the highest escalation step (3600) would be applied.

The action part of the step definition has the same syntax and interpretation as the THEN expressions of the main action policy described earlier in this document, except that action escalate.anypolicy() from within an escalation policy makes no sense and is therefore disabled.