Troubleshooting NMIS

TABLE OF CONTENTS

RELATED ARTICLES

Devices Not Collecting / Device Information Not Displayed

Is the node reachable?

Ping it with a big echo request.

[root@opmantek conf]# ping -c 5 -s 1472 192.168.88.254
PING 192.168.88.254 (192.168.88.254) 1472(1500) bytes of data.
1480 bytes from 192.168.88.254: icmp_seq=1 ttl=63 time=319 ms
1480 bytes from 192.168.88.254: icmp_seq=2 ttl=63 time=323 ms
1480 bytes from 192.168.88.254: icmp_seq=3 ttl=63 time=321 ms
1480 bytes from 192.168.88.254: icmp_seq=4 ttl=63 time=320 ms
1480 bytes from 192.168.88.254: icmp_seq=5 ttl=63 time=322 ms
--- 192.168.88.254 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4330ms
rtt min/avg/max/mdev = 319.542/321.519/323.551/1.450 ms

What does nmap think about it?

[root@opmantek conf]# nmap 10.10.1.1

Starting Nmap 5.51 ( http://nmap.org ) at 2017-04-04 15:05 KST
Nmap scan report for 10.10.1.1
Host is up (0.011s latency).
Not shown: 998 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
23/tcp open  telnet

Nmap done: 1 IP address (1 host up) scanned in 13.53 seconds
[root@opmantek conf]#

Node Not Present in GUI

Example Case:

Suddenly the node cannot be found in the GUI. When attempting to re-add the node to NMIS via the GUI we receive a 'node already exists' error.

Issue:

Something has become very corrupt, we need to purge NMIS of all relevant node configuration.

Actions:

Open /usr/local/nmis8/conf/Nodes.nmis with an editor and delete the section for the problem node.
Remove the following files:
- /usr/local/nmis8/var/<node-name>-node.josn
- /usr/local/nmis8/var/<node-name>-view.json
Re-add the problem node via the NMIS GUI
Run the following commands:
- /usr/local/nmis8/bin/nmis.pl type=update node=<node-name> force=true
- /usr/local/nmis8/bin/nmis.pl type=collect node=<node-name> force=true

Verify

The problem node should now be functioning properly in the NMIS GUI.

Manual Update & Collect Actions

If a node isn't providing the data we think it should sometimes looking at manual update & collect debugs is helpful. Redirect or tee the output to a file in order to review latter.

[root@opmantek ~]# /usr/local/nmis8/bin/nmis.pl node=asgard debug=9 type=update > nodeUpdate.txt

-or-

[root@opmantek ~]# /usr/local/nmis8/bin/nmis.pl node=asgard debug=9 type=update | tee nodeUpdate.txt

###################

[root@opmantek ~]# /usr/local/nmis8/bin/nmis.pl node=asgard debug=9 type=collect > nodeCollect.txt

-or-

[root@opmantek ~]# /usr/local/nmis8/bin/nmis.pl node=asgard debug=9 type=collect | tee nodeCollect.txt

NMIS Tools and scripts

nmis.pl provides 2 methods for checking the directory structure of nmis and ensuring that the structure is complete and has the correct permissions (based on your Config.nmis). Running type=audit will report discrepancies between your structure and what is required, type=config will fix those errors

# optionally audit your config
/usr/local/nmis8/bin/nmis.pl type=audit
# fix your config
/usr/local/nmis8/bin/nmis.pl type=config

Additonally, the script fixperms.pl will go through and set the permissions on each file to ensure that NMIS can access the files it requires to operate normally

# fix permissions 
 /usr/local/nmis8/admin/fixperms.pl

SELinux Troubleshooting Tip

Sometimes there are things happening on Linux systems which don't make sense, many times it is because SELinux is preventing things. You can spend a lot of time getting SELinux to behave, or you can put it in permissive mode, or disable it, in the NMIS VM it has been disabled.

Much information to be found with Google, the following describes either option.

http://www.crypt.gen.nz/selinux/disable_selinux.html

Email alerts

Contacts.nmis must have the correct DutyTime format.

External Authentication

conf/Config.nmis must have the proper auth_method order as well as that method being provisioned.

If LDAP isnt working tcpdump can be used to see the response code from the LDAP server.

Long collect times

Are we collecting many interfaces that are not necessary?

Check the view.json file for number of interfaces and interface type. Look for common things such as interface type and description. Use models or Config.nmis to disable collection.

Syslog

When troubleshooting syslog issues the following script will gather more rsyslog daemon information then the nmis support tool.

getSyslogData.sh

snmptrapd

When troubleshooting snmptrapd issues the following script will gather more snmptrad daemon information then then nmis support tool.

getSnmpTrapdInfo.sh

Models

When troubleshooting models it's important to know if all the OID's that have a 'friendly name' are referenced within Model files have been defined in /usr/local/nmis8/mibs/nmis_mibs.oid. Some Model files import or call other Model, Graph or Common files. If an OID 'friendly name' has not been defined in nmis_mibs.oid it may not be obvious which model file is causing the problem. In order to validate friendly names more easily the script below has been provided. It will parse all the OID friendly names out of the model files and look for them in nmis_mibs.oid. If they are not found the operator will be notified. At some point this script should be converted to perl; this would make it much faster.

checkOid.sh