Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: opFlow fixed size database

Table of Contents
maxLevel3

...

 

Code Block
cp install/01opmantek.conf /etc/apache2/conf.d/
service apache2 restart

 

...

Data

...

Storage Size 
Anchor
Considerations_for_NetFlow_Data_Storage
Considerations_for_NetFlow_Data_Storage

NetFlow data can get away on you, there could be several Gigabytes of NetFlow data each hour, day or month, this is all very dependant on where you are generating netflow from, the number of active users and the types of applications they are using.  A longer discussion on this will be written up, but in summary, if you are expecting large amounts of NetFlow data, then you will want to consider storing the MongoDB database and the NetFlow data into a filesystem with 50 gigabytes or more.  During development we found 20 gigabytes was enough for our purposes, but caused problems with the flow files mainly. If you are using the Opmantek Opmantek NMIS8 Virtual Machine please  please check out our instructions on on Resizing NMIS VMs.

opFlow is highly configurable and customisable, so it is easy to just have the flowd data be a seperate filesystem and update the configuration accordingly. 

If you are going to use a different directory, Modify the opFlow.nmis file and edit the <opflow_dir>
'<opflow_dir>' => '/data/opflow',
 
Edit the file /etc/init.d/mongod and change the correct directory, look for this line
mongodbpath=/data/mongodb
 
Edit the file /usr/local/etc/flowd.conf, and modify the entry for logfile.
logfile "/data/opflow/flowd" 

Alternate Installation Directory

opFlow can be installed into another directory if required, e.g. /opt/opmantek, the same process applies, but a few files will need to be changed.

Edit opFlow.nmis and opCommon.nmis and change the <omk_base> to be the new, e.g.

...

Size

opFlow uses fixed sizes for raw flows and conversations, when the setup tool (opflow_setup.pl) is run the database files will immediately be pre-allocated to the configured size and will never grow after this.  It is important that the sizes you choose fit on the partition you choose, the setup tool will warn you if they will not fit.  The default settings are: (found in /usr/local/opmantek/conf/opFlow.nmis)

Code Block
'opflow_db_conversation_collection_size' => 16106127360, #15G
'opflow_db_flow_collection_size' => 5368709120 # 5G

Please adjust them appropriately before running the setup tool. (which is done later in the instructions)

The output from the setup tool may tell you to run it again after adjusting your config with force=1 (opflow_setup.pl setup=db_config force=1) # again, only if required

Location

opFlow is highly configurable and customisable, so it is easy to just have the flowd data be a seperate filesystem and update the configuration accordingly. 

If you are going to use a different directory, Modify the opFlow.nmis file and edit the <opflow_dir>
'<opflow_dir>' => '/data/opflow',
 
Edit the file /etc/init.d/mongod and change the correct directory, look for this line
mongodbpath=/data/mongodb
 
Edit the file /usr/local/etc/flowd.conf, and modify the entry for logfile.
logfile "/data/opflow/flowd" 

Alternate Installation Directory

opFlow can be installed into another directory if required, e.g. /opt/opmantek, the same process applies, but a few files will need to be changed.

Edit opFlow.nmis and opCommon.nmis and change the <omk_base> to be the new, e.g.

'<omk_base>' => '/opt/opmantek',

...

When trying to add the _flowd user, you will get the following error message:

adduser: Please enter a username matching the regular expression configured
via the NAME_REGEX[_SYSTEM] configuration variable. Use the `--force-badname'
option to relax this check or reconfigure NAME_REGEX.
Code Block
adduser --force _flowd

Debian/Ubuntu follow the LSB (Linux Standard Base) specification, init script functions be available at /lib/lsb/init-functions.

Edit /etc/init.d/flowd

Code Block
#change this line: 
. /etc/init.d/functions
#to this: 
. /lib/lsb/init-functions

The start flowd

Code Block
sysv-rc-conf flowd on
service flowd start

 

...

Code Block
themeEmacs
cp /usr/local/opmantek/install/mongod.init.d /etc/init.d/mongod
chkconfig mongod on
service mongod start  

Debian/Ubuntu

Edit /etc/init.d/mongod

Code Block
#change this line: 
. /etc/init.d/functions
#to this: 
. /lib/lsb/init-functions

The start mongod

Code Block
sysv-rc-conf mongod on
service mongod start

 

...

  1. Ensure mongod is not running with the --auth switch, if you are using the mongod.init.d script included in opFlow run  run:

    Code Block
    /etc/init.d/mongod stop;
    /etc/init.d/mongod start_no_auth;
  2. Create the user, currently the opFlow user requires access to both the admin database and it's own. Start up the mongo shell and type these commands:

    Code Block
    use admin;
    db.addUser('opUserRW', 'op42flow42'); // these are the defaults, change them as well as your opFlow.nmis file
    use nmis; // again this is a default, it doesn't need to be changed
    db.addUser('opUserRW', 'op42flow42'); // these are the defaults, change them as well as your opFlow.nmis file, it should match the above user command 
  3. Restart the mongo server with authentication, --auth 

    Code Block/

    auth (again, only if you are using the mongod.init.d script included in opFlow)

    Code Block
    /etc/init.d/mongod stop;
    /etc/init.d/mongod start;

...

Code Block
themeEmacs
cp /usr/local/opmantek/install/opflowd.init.d /etc/init.d/opflowd
chkconfig opflowd on
service opflowd start

Debian/Ubuntu

Edit /etc/init.d/opflowd

Code Block
#change this line: 
. /etc/init.d/functions
#to this: 
. /lib/lsb/init-functions

The start opflowd

Code Block
sysv-rc-conf opflowd on
service opflowd start

 

...

Code Block
cd ~
# get release and untar it
wget opFlow-release.tgz
tar -zxvf opFlow-release.tgz
# make a backup of existing installation
cp -r /usr/local/opmantek /usr/local/opmantek-DAY-MON-YEAR
# stop the existing daemon so update can happen
service opflowd stop
# update the files
cp -r ./opmantek/* / files
cp -r ./opmantek/* /usr/local/opmantek/*

opFlow database now uses capped sizes, please adjust the following config settings to make sure they fit on your partition and are the size you want: (the defaults are listed)

Code Block
'opflow_db_conversation_collection_size' => 16106127360, #15G
'opflow_db_flow_collection_size' => 5368709120 # 5G

Now run the setup tool:

Code Block
/usr/local/opmantek/bin/opflow_setup.pl setup=db_config

Inspect the output and be sure you have configured the sizes the way you want. If you are unsure of the current size running opflow_setup.pl setup=db_config will inform you.

When you are happy with your configuration, re-run the setup tool with force=1: (this may take some time to complete if your existing database is large)

Code Block
/usr/local/opmantek/*

...

/bin/opflow_setup.pl setup=db_config force=1

Now update the configuration files run the following commands to do that.

Code Block
themeEmacs
/usr/local/opmantek/bin/opupdateconfig.pl /usr/local/opmantek/install/opFlow.nmis /usr/local/opmantek/conf/opFlow.nmis
/usr/local/opmantek/bin/opupdateconfig.pl /usr/local/opmantek/install/opCommon.nmis /usr/local/opmantek/conf/opCommon.nmis

/usr/local/opmantek/bin/opflow_setup.pl setup=all  
service opflowd start

Debian/Ubuntu

Verify/Edit /etc/init.d/flowd, /etc/init.d/mongod and /etc/init.d/opflowd

Code Block
#change this line: 
. /etc/init.d/functions
#to this: 
. /lib/lsb/init-functions