Upgrading Models

Owned by alexz (Unlicensed)
Last updated: Oct 20, 2016
7 min read

Overview

A substantial proportion of NMIS' functionality is provided by the device model and graph files, and using the newest model files is vital to ensure that your NMIS installation performs perfectly and benefits from the continual improvements that we make to NMIS. It is therefore very important to keep your model and graph files uptodate and in sync when you upgrade to a new NMIS version, but this can be a bit of a challenge as these are user-customisable and -extensible.

This document describes the new capabilities and limitations of the new automatic model upgrade tool (which ships with NMIS from 8.5.12G onwards, but can be downloaded from this page as well). For situations that exceed the automatic upgrader's capabilities, the recommended procedure for manual  model upgrades is also described.

The Automatic Model Upgrader

From version 8.5.12G onwards NMIS ships with a model upgrade tool and the installer offers to run it for you. If you are using NMIS 8.5.10G you can download the upgrade tool here; it is recommended that you save it as /usr/local/nmis8/admin/upgrade_models.pl.

Please note that the model upgrade tool is version-specific and will not work with NMIS releases other than the one it was shipped with. The current version for 8.5.10G handles model upgrades from any 8.5.X and 8.4.X versions to 8.5.10G.

Test Mode

Simply starting the tool without arguments will present a brief help text:

$ ./admin/upgrade_models.pl 
Usage: upgrade_models.pl [-u] [-o|-p] [-n regex] <new model dir> <live model dir>
-u: do perform the upgrade instead of just reporting model file states
-o: report only upgradeable files
-p: report only problematic files
-n: NEVER upgrade the matching files
...

The tool must be given the paths to the new models, i.e. /usr/local/nmis8/models-install and the "live" models, i.e. /usr/local/nmis8/models. It will not work if you give the directory paths in the wrong order.
If run without further arguments it will analyze the live models and determine which can be upgraded. It will NOT make any changes, but only print a line of status information for every model and graph file it operates on. Here is an example, upgrading from 8.5.2G (with custom extras) to 8.5.10G:

$ ./admin/upgrade_models.pl models-install models
Common-calls.nmis is uptodate.
...
Common-database.nmis is upgradeable: not modified since installation.
...
Common-Cisco-asset.nmis is upgradeable: new file.
Model-customhw.nmis is NOT UPGRADEABLE: locally created custom file.
Model-net-snmp.nmis is NOT UPGRADEABLE: has been modified since installation.

If you want to be informed of upgradable files only, you can add the option -o. If you want to see only files that can not be upgraded, use option -p.

Upgrade Mode

The upgrade tool will upgrade your model and graph files if and only if you run it with option -u (which can be combined with -o or -p). The output is the same as in test mode, except that as final step the actual changes will be made and the tool will report "Upgrading all upgradeable model files...Completed."

If desired you can also exclude certain model or graph files from the upgrade, using the -n option (which expects a regular expression argument, like "-n Model-custom").

It is safe to run the upgrade tool multiple times.

Limitations

  • As of version 8.6.0G, the installer instructs the upgrade tool to not handle Common-database.nmis, because of the possibility that an rrd location migration might be required.
    This would be the case if your current NMIS version is below 8.5.6G. In this situation, the adding of missing entries is left to the rrd migration tool (which is described further in the upgrade notes).
  • The upgrade tool compares the model file states against an embedded list of older releases and cannot upgrade models that are older than these known releases.
    For version 8.5.10G the upgrade tool was primed with states for releases 8.4.X and 8.5.X.
  • The upgrade tool will identify a file as 'locally modified' if you make any changes to it, and will not overwrite it under any circumstances.
    Changing a text field like '42' to the number 42 is considered a change. Adding white space outside of a string is not detected as change.
    Changing comments is not a change, nor is reordering items in a hash structure.
    All "locally modified" files will be left as-is by the upgrade tool.
  • The upgrade tool requires a complete and uptodate directory of "new" models.
    If you modify files in models-install the tool will refuse to work with these because of the state that is inconsistent with its embedded knowledge.
  • The upgrade tool is version specific and cannot  upgrade your models to any version but the one it was primed for and shipped with.
    I.e. the current tool targetting version 8.5.10G will be of no use to you if you want to upgrade to an older version of NMIS.

The Manual Upgrade Process

NMIS ships with new models in models-install/, and the installer does not make any changes to your live models/ directory unless directed to by you during the upgrade. This is because you may have created complex custom models for your installation or adjusted the standard models, both of which the installer mustn't overwrite or damage. The installer (and the upgrade_models tool) err on the side of caution, therefore some manual checks and merging are required if you  have local model customisations.

  1. Use upgrade_models.pl where possible.
  2. Compare the new models-install and the active models.
    Since NMIS 8.5.6G the installer performs that step for you, and leaves a comprehensive listing of the differences in a file in /tmp. See the section on Tools and Helpers below for details.
  3. Copy over any new files from models-install.
    A simple "cd /usr/local/nmis; cp -nav models-install/* models/" will copy any new files over (and print a line for each file that was copied). The "-n" is a crucial argument, meaning "don't  overwrite existing files".
  4. Replace existing files with new ones if the differences are not caused by your local modifications.
    After consulting the list of differences, you conclude that the differences in question are not caused by local modifications; In that case you'd simply use "cp -fav .... " to copy the specific file from models-install/ to models/.
    If you haven't made any model changes at all, then you can do a blanket cd /usr/local/nmis8; cp -fav models-install/* models/ and you're done.
  5. Merge any remaining local changes with new model files.
    If differences related to deliberate local changes remain, then you will have to use an editor to manually modify the model files in question to bring in new material.
    This can be a laborious process, and in all likelihood you will have to consult the documentation on NMIS modelling and the NMIS training material.
    The diffconfigs.pl tool will be (repeatedly) useful  to see if any todo's are left, and it is advisable that you run "perl -c <modelfile.nmis>" to syntax-check your final merged file.

Types of files in models and models-install

The models directory contains four types of files, which differ in the likelihood of local changes:

  • Common-X.nmis
    Common-database commonly has local adjustments if you want different RRD data retaining periods, or if you use a non-standard RRD directory schema, or if you have custom models with extra RRD databases.
    Common-thresholds is another likely candidate for local changes. Most of the other Common-X files rarely require local adjustments, unless you are customizing your models.
  • Model.nmis
    This file controls the model autodetection. If you haven't created custom models and added them to the model autodetection, then this file will not contain local changes.
  • Model-Y.nmis
    These files constitute the modelling configuration for all device types that NMIS knows about. Local changes  would be present if you extended or adjusted existing models in-place.
  • Graph-Z.nmis
    These files contain the graph definitions for NMIS's graphs. Local modifications are possible but not especially likely.

Tools and Helpers

compare_models

This tool performs a bulk comparison of all files in two models directories (usually models/ and models-install/). NMIS 8.5.6G and above ship with this tool as admin/compare_models.pl, and the installer will automatically invoke it for you.

The output will be similar to this example:

./admin/compare_models.pl models models-install 
Performing model difference check, please wait...
......................................................................
The comparison tool has detected some differences
between the old models (in models) and 
the new models (in models-install). 
The affected files are:
Common-database.nmis Common-heading.nmis Common-stats.nmis
Common-threshold.nmis Graph-QualityOfServiceStat.nmis Graph-SignalErrors.nmis
Graph-SignalLevel.nmis Graph-abits.nmis Graph-fan-status.nmis

A detailed listing of these differences has been 
saved in /tmp/model-diffs-2015-02-12.
You should review those differences (using less or an editor like 
nano, vi or emacs) and merge the model files where applicable.

The detailed listing is left in /tmp/model-diffs-<currentdate>, and contains a block of text for each file that was checked.  The actual comparison is performed by diffconfigs.pl, which is described below.

 

diffconfigs

The tool admin/diffconfigs.pl has been part of NMIS for quite some time. Its name implies that it compares configuration files, but it's flexible enough to also take care of models files.

It is invoked like this: ./admin/diffconfigs.pl somedir/oneconfig.nmis somedir/otherconfig.nmis and produces a list of differences similar to the following example:

./admin/diffconfigs.pl models/Model-CiscoNXOS.nmis  models-install/
Comparing models/Model-CiscoNXOS.nmis to models-install//Model-CiscoNXOS.nmis
Output format:
"Config Key Path:
-       Status in models/Model-CiscoNXOS.nmis
+       Status in models-install//Model-CiscoNXOS.nmis"
/systemHealth/sys/cdp:
-       <NOT PRESENT>
+       <DEEP STRUCTURE>
/systemHealth/sections:
-       env_temp
+       cdp,env_temp
Difference Summary:
        /systemHealth/sections
        /systemHealth/sys/cdp

A quick explanation of the format of the output:

  • The individual differences are shown with the path-like "location" of the  item in question, followed by two lines of difference info.
  • The line prefixed "-" describes the status in the first file, and the line with "+" describes the other.
  • If the difference is textual (like for systemHealth/sections in the example) then you will see the texts in question.
  • If the difference is structural (like for systemHealth/sys/cdp), then you will likely see descriptions like "NOT PRESENT".  If the structural types don't match you'll see info like "DEEP STRUCTURE, sometype".
  • If a structure is totally missing in one file, then no differences will be reported for its sub-elements - only the missing structure is indicated. This means that it's advisable to re-run diffconfigs after adding any structural items.
  • At the very end there's a summary of the differences, only listing the logical "locations" where differences were encountered.
  • Finally, the exit code of the command  is 0 if there were no differences, 1 if there were differences, and usually 255 if the operation failed altogether.

 

Note that if the file names of the files to compare are identical and only the directories differ, then you can call diffconfigs like this: ./admin/diffconfigs.pl somedir/oneconfig.nmis otherdir/ and it'll find the appropriate second file to compare.