...
Controlling how the command output is stored
Anchor | ||||
---|---|---|---|---|
|
In opConfig 3.1.1 an option for shadowing command output on disk was added: if you set the property shadow_file
to 1 or true (in the command's block, or in the command set's scheduling_policy
section), then opConfig will store the data both in the database and also on disk, in the same location and fashion as documented in the Tracking Files section below.
Tracking Files
opConfig version 3.0.3 introduces a new capability: Arbitrary files can now be downloaded from a node (with SCP), stored and tracked by opConfig.
Here is an snippet from the example command set named file_store.nmis
that ships with opConfig:
Code Block |
---|
# ...other command set structure
scheduling_info =>
{
# indicates work to be performed by and on the opConfig host
run_local => 'true',
},
commands => [
{
command => '_download_file_ scp:///var/log/secure',
store_internal => 'false',
tags => [ 'detect-change', 'other', 'custom', 'tags' ],
},
{
command => '_download_file_ scp://file_in_user_homedir',
store_internal => 'true', # is the default,
tags => [ 'detect-change' ],
},
],... |
To instruct opConfig to track one or more files, you have to
- set up a command set with the scheduling property
run_local
set to true, - and add a separate special
_download_file_
command for every file you want to track. - If you want the file data to be treated as binary, set
store_internal
to false.
The run_local
option indicates that all commands in this command set are to be run on the opConfig server, instead of on the node in question.
The special/pseudo-command _download_file_
requires an scp URI to be given. Note that the first two "/" characters in scp:///some/abs/path.txt
belong to the URI, and the path in question is /some/abs/path.txt
. In the second example above, the path is a plain relative file_in_user_homedir
which means scp will look for this file in the user's home directory.
If you leave the store_internal
option set to true (or omit it altogether), then the normal storage behaviour is selected: opConfig assumes your file contents are text, treats them as the 'command output' for this special command, and stores the output in the database. Hence, you'll see the whole file contents in the GUI, and change detection will be performed line-by-line. This cannot work for binary files, and cannot work for large files (above 16 megabytes) either (16 megabytes is MongoDB's size limit for individual documents).
On the other hand with this option set to false, opConfig stores a separate copy of the the file for each revision (under the directory configured with config option opconfig_external_store
, usually /usr/local/omk/var/opconfig
/external). The 'command output' is made up from the size and the SHA256 checksum of the file contents, and change detection (and the GUI) uses this data instead of the (binary or huge) file contents. This produces much more coarse change detection, but works with binary files. In the GUI you'll see the made up 'command output', and a button to download the actual file data.
All other opConfig capabilities work normally for file tracking commands; e.g. scheduling, tags, purging of old revisions, revision creation itself and so on.
The resulting files are stored in the directory /usr/local/omk/var/opconfig/external/<node name>/<command name>/<revision>
. opConfig 3.1.1 and newer also maintain a symbolic link latest
that points to the most recent revision.
Furthermore, in opConfig 3.1.1 and newer, any command output that is larger than 16 megabytes is automatically reconfigured for storage on disk. In these versions the configuration option store_internal
is also supported for any command whatsoever, not just _download_file_<xyz>
, but using the shadow_file
facility is more likely to be useful for 'normal' commandsBy default opConfig stores all command outputs in its MongoDB database, and only there.
In recent versions two new options let you fine-tune the behaviour and make use of on-disk storage where desired:
- Shadowing outputs on disk is controlled by the option
shadow_file
, - and the choice between database storage and disk storage is made via the option
store_internal
.
Disk or Database?
If you set the store_internal
option to "true" or omit it altogether, then all command outputs are stored in the database.
With this option set to false, opConfig creates a separate file on disk for each revision of the command.
The storage location for such files is configured with the config option opconfig_external_store
(default /usr/local/omk/var/opconfig/external
).
The actual files are then stored in the directory <opconfig_external_store>/<node name>/<command name>/<revision>
.
In version 3.1.1 and newer, an additional symbolic link latest
that points to the most recent revision is maintained in that directory.
For externally stored data the visible 'command output' in the GUI is made up from the size and the SHA256 checksum of the contents, and change detection (and the GUI) uses this data instead of the (possibly binary or huge) file contents. This produces much more coarse change detection, but works with binary files. In the GUI you'll see the made up 'command output', and a button to download the actua contents.
All other opConfig capabilities work normally for externally stored data; e.g. scheduling, tags, purging of old revisions, revision creation itself and so on.
Please note that in versions before 3.1.1 store_internal
is only supported for Tracking Files.
Commands or Files that cannot fit into the Database?
MongoDB doesn't support documents greater than 16 megabytes. In opConfig 3.1.1 and newer, any command output that is larger than this limit is automatically reconfigured for storage on disk, i.e. store_internal
is overridden and set to "false" for just this one revision.
Both Disk and Database?
In opConfig 3.1.1 an option for shadowing command output on disk was added: if you set the property shadow_file
to true (in the command's block, or in the command set's scheduling_policy
section), then opConfig will store the data both in the database and also on disk (in the same place as externally stored command outputs).
Please note that shadow_file
has no effect for commands that have been configured for disk storage.
Tracking Files
opConfig version 3.0.3 introduces a new capability: Arbitrary files can now be downloaded from a node (with SCP), stored and tracked by opConfig.
Here is an snippet from the example command set named file_store.nmis
that ships with opConfig:
Code Block |
---|
# ...other command set structure
scheduling_info =>
{
# indicates work to be performed by and on the opConfig host
run_local => 'true',
},
commands => [
{
command => '_download_file_ scp:///var/log/secure',
store_internal => 'false',
tags => [ 'detect-change', 'other', 'custom', 'tags' ],
},
{
command => '_download_file_ scp://file_in_user_homedir',
store_internal => 'true', # is the default,
tags => [ 'detect-change' ],
},
],... |
To instruct opConfig to track one or more files, you have to
- set up a command set with the scheduling property
run_local
set to true, - and add a separate special
_download_file_
command for every file you want to track. - If the file data is binary or larger than 16 megabytes, set
store_internal
to false.
The run_local
option indicates that all commands in this command set are to be run on the opConfig server, instead of on the node in question.
The special/pseudo-command _download_file_
requires an scp URI to be given. Note that the first two "/" characters in scp:///some/abs/path.txt
belong to the URI, and the path in question is /some/abs/path.txt
. In the second example above, the path is a plain relative file_in_user_homedir
which means scp will look for this file in the user's home directory.
The store_internal
option works as described in the previous section.
Grouping commands into sessions
...