| Table of Contents |
|---|
Purpose
Purging may not have been set up at installation time. If the OMK database has grown large it's a good idea to purge it in 90 day increments until the desired retention period is realizedselect an event retention period and manually purge the OMK database.
Related Documents
Purging of old data in opEvents
Determine Retention Interval
Select a retention period that provides infrastructure engineers with necessary event information and also complies with your organizations security and compliance policies.
Run the opEvents Purge
...
In order to purge from the database in 90 day increments the date of the oldest record is required.
| Code Block |
|---|
[root@opmantek ~]# mongo -u opUserRW -p op42flow42 nmis --eval 'db.rawlogs.findOne()'
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017/nmis
MongoDB server version: 3.4.9
{
"_id" : ObjectId("5543df5db5d690352f1eb2cb"),
"entry" : "2015-05-01T16:17:19\t10.73.225.166\tUDP: [10.73.225.166]:14445->[10.72.0.31]\tSNMPv2-MIB::sysUpTime.0=23:5:15:52.26\tSNMPv2-MIB::snmpTrapOID.0=SNMPv2-SMI::enterprises.8072.4.0.3\tSNMPv2-MIB::snmpTrapEnterprise.0=SNMPv2-SMI::enterprises.8072.4",
"time" : NumberLong(1430511439),
"eventid" : ObjectId("5543df5db5d690352f1eb2ca"),
"type" : "nmis_traplog"
}
|
Notice the 'time' property above; 1430511439. The date command may be used to reveal when this was.
| Code Block |
|---|
[root@opmantek ~]# date --date @1430511439
Fri May 1 20:17:19 UTC 2015 |
...
Command
The purge action is run with the opevents-cli.exe command line utility. When using this utility the retention period is relative to the current time and is referred to as 'purge_older_than'. The value passed with the command may be expressed in minutes, hours, or days (combinations are also possible). Using days as an example, if the retention period were 1 year the command line option for the retention period would like like this:
| Code Block | ||
|---|---|---|
| ||
### CurrentRetention unix epoch time [root@opmantek ~]# date +%s 1513316674 ### Let X be The difference of current time and oldest record time converted to minutes >>> (1513316674-1430511439)/60 1380087.25 ### There are 129600 minutes in 90 days. Subtract 129600 minutes from X. >>> 1380087 - 129600 1250487 ### Based on this the period events_purge_older_than_=365d ### Entire command /usr/local/omk/bin/opevents-cli.exe act=purge events_purge_older_than should be 1250487 |
Run the opEvents Purge Command
...
=365d |
We recommend running the purge in simulate mode first to ensure the math is correctdetermine if the desired outcome will be achieved.
| Info |
|---|
For large databases this operation may take some time. |
| Code Block |
|---|
[root@opmantekroot@poller-office ~]# /usr/local/omk/bin/opevents-cli.exe act=purge events_purge_older_than=52560365d debug=1 simulate=true [Fri Dec 15 0522:5416:3128 2017] [debug] cli[3137711499] Purging unwanted database indices [Fri Dec 15 0522:5416:3128 2017] [debug] cli[3137711499] DbPurging purgedisabled offor rawlogs: 0Purging entries (of 7161464) in rawLogs are older than 1y270d50m Db purge of rawlogs: 0 entries (of 7161464) in rawLogs are older than 1y270d50mdisabled for rawlogs [Fri Dec 15 0522:5416:3128 2017] [debug] cli[31377] Simulated purging rawlogs: would remove 0 (of 7161464) entries from rawLogs Simulated purging rawlogs: would remove 0 (of 7161464) entries from rawLogs [Fri Dec 15 05:54:31 2017] [debug] cli[31377] 11499] Purging disabled for archivelogs Purging disabled for archivelogs [Fri Dec 15 0522:5416:3328 2017] [debug] cli[3137711499] Db purge of events: 46920440 entries (of 506091317947) in events are older than 36d12h1y Db purge of events: 46920440 entries (of 506091317947) in events are older than 36d12h1y [Fri Dec 15 0522:5416:3328 2017] [debug] cli[3137711499] Simulated purging events: would remove 46920440 (of 506091317947) entries from events Simulated purging events: would remove 46920440 (of 506091317947) entries from events [Fri Dec 15 0522:5416:3728 2017] [debug] cli[3137711499] Db purge of events: 128313340 entries (of 1380315613105) in actionlog are older than 36d12h1y Db purge of events: 128313340 entries (of 1380315613105) in actionlog are older than 36d12h1y [Fri Dec 15 0522:5416:3728 2017] [debug] cli[3137711499] Simulated purging events: would remove 128313340 (of 1380315613105) entries from actionlog Simulated purging events: would remove 128313340 (of 1380315613105) entries from actionlog [Fri Dec 15 0522:5416:3728 2017] [debug] cli[3137711499] Purging disabled for reports Purging disabled for reports |
Once satisfied that the desired action will occur; remove the debug simulate option and run the command again.Continue this 90 day cycle until the desired retention period is attained.
Recover Disk Space - Compact MongoDB
The old records have been removed, but the data base database is still taking the same amount of storage. In order to recover the disk space we will need to issue that the MongoDB 'compact' command that . This will rewrite and defrag defragment the database. The safe approach is to shut down all daemons that are writing the the database prior to issues issuing the 'compact' command.
Shutdown OMK Services
| Code Block |
|---|
service opeventsd stop service opflow stop service opconfigd stop service nmisd stop service omkd stop |
Enter the mongo Shell and Compact opEvents Collections
| Code Block |
|---|
[root@opmantek ~]# mongo -u opUserRW -p op42flow42 --authenticationDatabase=admin nmisng MongoDB shell version v3.4.9 connecting to: mongodb://127.0.0.1:27017/adminnmis MongoDB server version: 3.4.9 > db.runCommand({compact: 'events'}) { "ok" : 1 } > db.runCommand({compact: 'rawlogs'}) { "ok" : 1 > db.runCommand({compact: 'actionlog'}) { "ok" : 1 } > |
Start OMK Services
| Code Block |
|---|
service omkd start service nmisd start service opeventsd start service opconfigd start service opflowd start |
Set Purging Policy
Now that the database has been cleaned up, set the purging policy in opCommon. That procedure is found here: Purging of old data in opEvents