Generally speaking the overall goal is to provide centralised logging services for the purposes of operations, compliance and audit. All systems should send log messages to the central server, where the logs will be kept in original form for the required period of time and the logs will be availble for event, incident and problem management purposes.
Table of Contents |
---|
Centralised Logging Architecture
Architectural Considerations
There are situations when it's desirable to forward syslog to a central location. are several considerations for creating a centralised logging solution.
- What devices, and operating systems will be sending logs.
- What applications will be sending logs.
- What protocol will be used to send log messages.
- What software will be used to send the log messages.
- What timezone are each of the devices sending logs in.
- What criticality of logs is required.
Devices Sending Logs
You should complete a table as below to catalogue all the devices which will be sending logs.
Name | Purpose | Logging System |
---|---|---|
Windows Event Manager | Event and Audit | Windows Event Logging |
CentOS Linux 5.x | Event and Audit | syslog |
CentOS Linux 6.x | Event and Audit | syslog |
Cisco IOS Switches | Event and Audit | Cisco IOS syslog |
Cisco IOS Routers | Event and Audit | Cisco IOS syslog |
Applications Sending Logs
The following applications logs need to be send centrally.
Application Name | Purpose | File | Device |
---|---|---|---|
Monkey Auth System | Audit | C:\Program Files\MAS\logs\monkeyauth.log | Windows 2008 Servers |
Elephant Financial | Audit | /data/elefin/log/app.log | CentOS Linux 6.8 |
Logging Protocol
syslog has proven to be a very robust protocol for large scale log management.
TCP should where reliable logging is required, UDP works very well, 99.99% of the time.
Centralised Logging and Archiving Solution
The following diagram shows how a system can be implemented using syslog and Opmantek opEvents.
Centralised Logging Design
Syslog Method and Transport
The following table summarises how the logs will get from the source into the opEvents server. Three basic techniques exist, native syslog transport supported by the operating system, a logging agent which monitors for data and sends the resulting data/events/log as a syslog.
There are several good choices for Windows, but NXLOG has proven to meet all the requirements, almost all other systems include embedded syslog systems
Source | Method and Transport |
---|---|
Windows 2003 Servers | nxlog monitoring Windows Event log, transport over syslog |
Windows 2008 Servers | nxlog monitoring Windows Event log, transport over syslog |
Windows 2012 Servers | nxlog monitoring Windows Event log, transport over syslog |
CentOS Linux 5.x | rsyslog 3.x |
CentOS Linux 6.x | rsyslog 7.6 |
Cisco IOS Switches | Native IOS syslog |
Cisco IOS Routers | Native IOS syslog |
Monkey Auth System | nxlog running on Windows. |
Elephant Financials | rsyslog running on Linux |
Logging Severity Levels
The requirement is to send level 6 and above.
https://en.wikipedia.org/wiki/Syslog#Severity_level
Value | Severity |
---|---|
0 | Emergency |
1 | Alert |
2 | Critical |
3 | Error |
4 | Warning |
5 | Notice |
6 | Informational |
7 | Debug |
syslog Facility
The best reference is: https://en.wikipedia.org/wiki/Syslog#Facility
We are primarily concerned with the facilities localX facilities. Logs will also grow at different rates and having them in separate files will allow for more granular control. The following table summarises which log files will end up in which files.
Device Type | syslog facility | Log file |
---|---|---|
| local0 | /data/log/local0.log |
Log server to log server (future) | local1 | /data/log/local1.log |
Application logging e.g. MonkeyAuth | local2 | /data/log/local2.log |
Windows servers (nxlog default) | local3 | /data/log/local3.log |
Cisco ASA default (VMware ESXi default) | local4 | /data/log/local4.log |
| local5 | /data/log/local5.log |
Linux syslog | local6 | /data/log/local6.log |
Cisco Routers and Switches | local7 | /data/log/local7.log |
Alternate file naming can be supported if required, e.g. cisco.log instead of local7.log.
Centralised Logging Implementation
Sample Configuration for rsyslog 7.6
The following is a config which sends all syslog over severity 6
Example Topology
In the example above all syslog messages received with a facility of local7 will be forwarded to the master server at 10.215.1.5. When this message is forwarded from the poller to the master, the poller will insert its own timestamp into the message.
...
- If the syslog message has a timestamp that is more than 1800 seconds off from the current server time:
- Accept the syslog message
- Remove and replace the timestamp with its own time stamp.
...
Appendix A: Upgrading rsyslog of RHEL and CentOS
The version of rsyslog which is included in older CentOS and RedHat systems is problematic and should be upgraded, the steps to do so are included below.
Create/Edit rsyslog.repo
Code Block |
---|
cat /etc/yum.repos.d/rsyslog.repo |
Result:
Code Block |
---|
[rsyslog_v7]
name=Adiscon CentOS-$releasever - local packages for $basearch
baseurl=http://rpms.adiscon.com/v7-stable/epel-$releasever/$basearch
enabled=1
gpgcheck=0
gpgkey=http://rpms.adiscon.com/RPM-GPG-KEY-Adiscon
protect=1 |
Test Yum
Code Block |
---|
yum search rsyslog |
Install rsyslog
Code Block |
---|
yum install rsyslog |