Generally speaking the overall goal is to provide centralised logging services for the purposes of operations, compliance and audit. All systems should send log messages to the central server, where the logs will be kept in original form for the required period of time and the logs will be availble for event, incident and problem management purposes.
Centralised Logging Architecture
Architectural Considerations
There are several considerations for creating a centralised logging solution.
- What devices, and operating systems will be sending logs.
- What applications will be sending logs.
- What protocol will be used to send log messages.
- What software will be used to send the log messages.
- What timezone are each of the devices sending logs in.
- What criticality of logs is required.
Devices Sending Logs
You should complete a table as below to catalogue all the devices which will be sending logs.
Name | Purpose | Logging System |
---|---|---|
Windows Event Manager | Event and Audit | Windows Event Logging |
CentOS Linux 5.x | Event and Audit | syslog |
CentOS Linux 6.x | Event and Audit | syslog |
Cisco IOS Switches | Event and Audit | Cisco IOS syslog |
Cisco IOS Routers | Event and Audit | Cisco IOS syslog |
Applications Sending Logs
The following applications logs need to be send centrally.
Application Name | Purpose | File | Device |
---|---|---|---|
Monkey Auth System | Audit | C:\Program Files\MAS\logs\monkeyauth.log | Windows 2008 Servers |
Elephant Financial | Audit | /data/elefin/log/app.log | CentOS Linux 6.8 |
Logging Protocol
syslog has proven to be a very robust protocol for large scale log management.
TCP should where reliable logging is required, UDP works very well, 99.99% of the time.
Centralised Logging and Archiving Solution
The following diagram shows how a system can be implemented using syslog and Opmantek opEvents.
Centralised Logging Design
Syslog Method and Transport
The following table summarises how the logs will get from the source into the opEvents server. Three basic techniques exist, native syslog transport supported by the operating system, a logging agent which monitors for data and sends the resulting data/events/log as a syslog.
There are several good choices for Windows, but NXLOG has proven to meet all the requirements, almost all other systems include embedded syslog systems
Source | Method and Transport |
---|---|
Windows 2003 Servers | nxlog monitoring Windows Event log, transport over syslog |
Windows 2008 Servers | nxlog monitoring Windows Event log, transport over syslog |
Windows 2012 Servers | nxlog monitoring Windows Event log, transport over syslog |
CentOS Linux 5.x | rsyslog 3.x |
CentOS Linux 6.x | rsyslog 7.6 |
Cisco IOS Switches | Native IOS syslog |
Cisco IOS Routers | Native IOS syslog |
Monkey Auth System | nxlog running on Windows. |
Elephant Financials | rsyslog running on Linux |
Logging Severity Levels
The requirement is to send level 6 and above.
https://en.wikipedia.org/wiki/Syslog#Severity_level
Value | Severity |
---|---|
0 | Emergency |
1 | Alert |
2 | Critical |
3 | Error |
4 | Warning |
5 | Notice |
6 | Informational |
7 | Debug |
syslog Facility
The best reference is: https://en.wikipedia.org/wiki/Syslog#Facility
We are primarily concerned with the facilities localX facilities. Logs will also grow at different rates and having them in separate files will allow for more granular control. The following table summarises which log files will end up in which files.
Device Type | syslog facility | Log file |
---|---|---|
| local0 | /data/log/local0.log |
Log server to log server (future) | local1 | /data/log/local1.log |
Application logging e.g. MonkeyAuth | local2 | /data/log/local2.log |
Windows servers (nxlog default) | local3 | /data/log/local3.log |
Cisco ASA default (VMware ESXi default) | local4 | /data/log/local4.log |
| local5 | /data/log/local5.log |
Linux syslog | local6 | /data/log/local6.log |
Cisco Routers and Switches | local7 | /data/log/local7.log |
Alternate file naming can be supported if required, e.g. cisco.log instead of local7.log.
Centralised Logging Implementation
Sample Configuration for rsyslog 7.6
The following is a config which sends all syslog over severity 6
Example Topology
In the example above all syslog messages received with a facility of local7 will be forwarded to the master server at 10.215.1.5. When this message is forwarded from the poller to the master, the poller will insert its own timestamp into the message.
### syslog message from the node to the poller server 02:23:37.250516 IP 10.10.1.1.58415 > 10.215.1.7.syslog: SYSLOG local7.notice, length: 100 E....Y.....+ .. ..../...l./<189>90: *Feb 1 11:23:35.623: %SYS-5-CONFIG_I: Configured from console by hero on vty0 (10.215.1.5) ^C ### syslog message from the poller server to the master server 11:23:37.273514 IP 10.215.1.7.35902 > 10.215.1.5.syslog: SYSLOG local7.notice, length: 126 E.....@.?.#. ... ....>.....j<189>Feb 1 02:23:37 10.10.1.1 90: *Feb 1 11:23:35.623: %SYS-5-CONFIG_I: Configured from console by hero on vty0 (10.215.1.5)
If the servers/nodes are in differnt time zones or the clocks are not correct opEvents may not by default accept the syslog message. Setting the opEvents debug option to '1' will show the following message in /usr/local/omk/log/opEvents.log if this is the case.
[Wed Feb 1 09:08:49 2017] [debug] worker[4697] event 1485875324 R1 Feb 1 00:08:44 10.10.1.1 66: *Feb 1 09:08:42.711: %OSPF-5-ADJCHG: Process 1, Nbr 10.10.1.3 on FastEthernet1/0 from FULL to DOWN, Neighbor Down: Interface down or detached is older than opevents_max_event_age, skipping!
As of opEvents 2.2.1 we can provision opEvents to allow these wildly out of date syslog messages and replace the origin timestamp with its own.
### /usr/local/omk/config/opCommon.nmis 'opevents_max_action_queue_age' => 3600, 'opevents_max_event_age' => 7200, 'opevents_max_time_delta' => 1800, 'opevents_monthly_report_recipients' => [], 'opevents_monthly_report_title' => 'Monthly Summary Report',
Setting the 'opevents_max_time_delta' to a value of 1800 seconds will instruct opEvents to do the following:
- If the syslog message has a timestamp that is more than 1800 seconds off from the current server time:
- Accept the syslog message
- Remove and replace the timestamp with its own time stamp.
Appendix A: Upgrading rsyslog of RHEL and CentOS
The version of rsyslog which is included in older CentOS and RedHat systems is problematic and should be upgraded, the steps to do so are included below.
Create/Edit rsyslog.repo
cat /etc/yum.repos.d/rsyslog.repo
Result:
[rsyslog_v7] name=Adiscon CentOS-$releasever - local packages for $basearch baseurl=http://rpms.adiscon.com/v7-stable/epel-$releasever/$basearch enabled=1 gpgcheck=0 gpgkey=http://rpms.adiscon.com/RPM-GPG-KEY-Adiscon protect=1
Test Yum
yum search rsyslog
Install rsyslog
yum install rsyslog