How to create and install an NPE/STM VM on VMWare ESXi

Terminology and Acronyms

AR – Available Rate
BITW – Bump in the Wire
CAC – Call Admission Control
DPDK – Data Plane Development Kit
DPI – Deep Packet Inspection
EFC – Egress Flow Class
EMAP – Egress Policy Map
FACL – Flow Access Control List
FR – Fixed Rate
HE – Host Equalization
HMAP – Host Policy Map
IFD – Intelligent Flow Delivery
IFC – Ingress Flow Class
IMAP- Ingress Policy Map
NPE – Network Performance Enforcement
P2P – Peer to Peer
QoS – Quality of Service
STM – Saisei Traffic Manager
TR – Total Rate
VoIP – Voice over IP
VM – Virtual Machine a software based virtual Guest or virtual appliance
Hypervisor – a hardware abstraction layer enabling multiple VM guests share the resources of a hardware platform, generally an X86 COTS server.

Introduction

The Saisei Traffic Manager platform™ is a highly distributable advanced real time network controller and real time monitoring application. This provides application visibility, policy control and advanced traffic engineering for customers looking to control their business applications through advanced network intelligence to provide deterministic levels of QoS. The patented flow engine provides real time policy based feedback loop to application traffic to ensure business policies are enforced. The NPE is a high capacity traffic manager designed to manage traffic flows to ensure desired performance and quality for all key business applications.

The software package is released in several formats for bare metal and hypervisor installations.

This document will concentrate explicitly on the requirements and steps for building and installing the NPE software package onto a ESXI VM.

Hardware Requirements

Obviously the basic hardware for any deployment is critical whether it be a bare metal or an ESXi deployment – in both cases the rules for core counts, memory, processor type and hard drive sizes still apply, for the latest rules the reader is referred to the NPE Hardware and System Verification document. This document will list the expectations of any hardware platform on which the NPE software package can run, at the time of writing Saisei will only support software running on certified systems as defined in the previously mentioned Verification Document.

Note: - The supported systems list contains Bare Metal systems only, to run in an ESXi environment the user is required to provide a system that contains the required core count for the desired service model and associated memory and MUST run the hardware verification commands to ensure the chosen XEON can support the required CPU command set.

Once logged on verify that memory, cores and drive space is available, here we can select the ESXi host address, Configuration tab and one of the following Processors, Memory and Storage entries to determine the free data store space: -

We can also select the Memory and Processor entries to check the capabilities of the system and finally also verify and ensure that Hyper Threading is turned off. Hyper Threading can only be changed if ALL VM’s currently loaded to the ESXi host are powered down – to check however select the ESXi host address followed by the Summary tab which will display core counts and Hyper Threading state.

If Hyper Threading is enabled it will be necessary to power down all VM’s on the ESXi host and then select the Configuration tab followed by Processors entry and finally the Properties option as seen below: -

Which results in the following display –

Uncheck the “Enabled” box and hit OK – notice for this to take effect a full reboot of the ESXi host is required.

A check for the “Power Management” model should also be performed since the default ESXi installation uses the “Balanced setting” and this should be set to “High Performance”, to view these settings: -

The Properties window opens and make the High Performance selection: -

And the Advanced section becomes: -

ESXi Prerequisites

The NPE software is released as a TGZ file which needs to be loaded and installed onto an Ubuntu 64 bit 14.04 Server Operating system.

Before you can create a VM for the NPE there are a few things that need to be present.

Creating VSwitches

After logging onto the ESXi host select the ESXi host address in the navigation list in this document we use 10.1.10.219 as the ESXi host, then select the “Configuration” tab and finally in the Hardware list select the Networking option. This will result in a screen similar to the following which list all existing vSwitches. Key in the Networking pane are the options at the top of the display and to create a new vSwitch choose the “Add Networking” option.

Key in the Networking pane are the options at the top of the display and to create a new vSwitch choose the “Add Networking” option which results in the following window being displayed: -

Verify that “Virtual Machine” is selected and then hit “Next” which results in the following window being displayed: -

If the actual NIC to be used for each Ethernet interface is NOT known this step can be skipped for now however be aware that this window will open with the first vmnic in the list pre-selected and will need to be deselected. Ideally the actual vmnic’s to use for eth1 and eth2 should be known and should be configured here. Note that eth1 (npe-eth1) is expected to connect to the Internet side of the network while eth2 (npe-eth2) is expected to be connected to the local or internal users. In devices with a large number of real NIC ports some amount of investigation is usually required to determine which physical interface links to which internal vmnic and at worst this may mean cabling each unused port separately and using the vSphere windows to find the cabled interface that displays a speed and link state in the Speed field as shown above. Thus assuming the interfaces for eth1 and eth2 are known select the appropriate interface for each as each vSwitch is configured.

The next step is to give the vSwitch a Network Label – npe-eth1 or npe-eth2 – and additionally enable all VLAN id’s from the pull-down menu list if VLAN operation is required: -

Now we can move to the next step and review the summary: -

But before closing the summary page now would be the best time to highlight the vSwitch entry in the configuration list followed by the Edit button – this will then allow the vSwitch to be reconfigured into “Promiscuous” mode which is essential for successful operation of the NPE software package.

Editing the vSwitch opens the window above with the “General” tab selected – you must now select the “Security” tab which will list the Policy Exceptions and will display the “Promiscuous Mode” entry as “Reject”, change this to “Accept as shown above.

Finally hit the “OK” button and the edit is complete.

At this point you will have successfully created the first vSwitch called npe-eth1, repeat the whole process one more time to create the second vSwitch called npe-eth2.

Creating an Ubuntu 64bit VM

The user must first create a basic ESXi Ubuntu 64bit VM by logging onto the system running VMs using vSphere perhaps and selecting the systems address in the Navigation tree: -

Now select the “Create a new virtual machine” options which results in: -

Where you need to select the “Typical” configuration option followed by “Next >”.

Give the VM a name and Next >

Select the Datastore to use, keep in mind the STM should have a minimum disc size of 500GB, ESXi this allow you to increase in the future should more be required, then Next >

Now set the Linux radio button and select the Ubuntu Linux (64bit) version then Next >

For the Networking portion the STM requires a minimum of 3 NIC interface, the first should point to the Management vSwitch, the second to a vSwitch called npe-eth1 and the third to a vSwitch called npe-eth2 – the adaptors for ALL vSwitches should be E1000 – DO NOT USE any of the other types of adapter, then Next >

Here we are setting a minimum disc of 500GB, it MUST be Thick Provision then Next >

All that remains is to verify the base configuration for the VM and Finish.

Installing the Ubuntu Operating System

Before the Ubuntu 14.04 Server operating system can be loaded it is the installers’ responsibility to preload the Ubuntu ISO file into the Hypervisors datastore through vSphere or other tool. A copy of the correct ISO file can be downloaded from the Saisei Support website http://support.saisei.com.

With this available the VM some edits to the VM need to be made.

Selecting the new VM name in the tree then allows the user to Edit the virtual machine settings.

The first thing to do is set the memory, use 16GB for a system running the Small model and 256GB for the Large model. Small is for 1Gbps and below and Large is for 10Gbps interface speeds.

Then set the cores per socket, this needs to be 4 for a Small model and 12 for the Large Model, wherever possible it is best to have ALL cores on the same socket.

Finally select the CD/DVD option and set the Datastore ISO File, browse to where the 14.04 Sever ISO resides and select then set the Connect at power on option. Then hit the OK button.

Finally when the Edit window closes, select the VM name and the console tab before powering up. The screens are now the same as doing an Ubuntu install on a bare metal device. Prior to starting this process it is best to verify that the VMNetwork vSwitch is connected to the network and onward to the Internet as updates will be pulled from the Internet.

The process now continues in the next section.

Ubuntu Installation Procedure

The example that follows will go through installing an Ubuntu 14.04 Server version of the base operating system.

The installation continues and asks for the language, this first language request merely allows the installer to give the correct text for the actual installation request: -

Which is then followed by Install Request Options: -

Before proceeding to the Language type for the entire installation: -

After the language the install requires the geographic location: -

Now we must answer questions about the keyboard, here it is usual to say no so that the keyboard type can be set manually: -

So now we select the keyboard: -

Followed by the layout: -

And now several applications and components will be installed which may take a while: -

Which will be followed by a request to select the Management interface for the system: -

With the Management interface selected we must now give a Hostname – in this case FlowCommand was chosen: -

Which will be followed by the Full name of the user who will automatically be given root access rights through the sudo command (in this case “saisei” was used)

DO NOT CREATE A USER CALLED “admin” THIS USER IS RESERVED FOR STM USAGE – ADDITIONALLY USERS CREATED IN THE STM SPACE CANNOT AND MUST NOT BE CREATED IN UBUNTU SPACE AS THE STM WILL OVERWRITE PASSWORDS AND SHELLS: -

The User then requires a user name for the account (In house built systems will use “saisei”): -

Followed by a password request (In house systems use “saisei” byt default): -

And password verification: -

And a request to encrypt the users home directory to which the answer is NO: -

The next step is to select a time zone: -

Followed by disk portioning where we select the guided option as shown (Note if over writing an existing installation always allow the system to overwrite and remove any previous information and stored data): -

Next we select the disk to partition: -

And confirm writing changes to the disk: -

The volume size will also be displayed to be confirmed: -

Before the final confirmation request : -

Triggering the system installation: -

Continuing to the HTTP Proxy where none is required: -

Which allows the installation to continue again: -

Before requesting how to manage updates, it is ESSENTIAL at this point to DECLINE any updates as the NPE requires explicit versions of various applications which it will retrieve during its own installation process: -

Of the optional software it is highly beneficial to select the Openssh server be installed – this will NOT cause any updates to be performed but merely loads the Openssh application, DO NOT load any of the other options presented: -

Now request that the GRUB boot loader be added to the Master Record: -

Upon successful completion of the installation the system will need to reboot – it will attempt to eject the CD to prevent restarting the process – please check and remove if necessary: -

And finally after a reboot the Operating System is installed: -

Following a successful installation it will be necessary to log onto the console or through SSH to determine systems IP address using the “ifconfig” command – by default this is configured to use DHCP – if however the user requires a static IP address then it will be necessary to change the /etc/network/interfaces file to configure the management Ethernet interface to use a static IP address and gateway etc.

Keep in mind that once rebooted the Saisei Traffic Manager will be running and if this is a new system it will default to a “Small” model but without any other configured elements. Changing the model will require a configuration save and further reboot to load the correct memory footprint for the chosen NPE model.

Installing and Upgrading the NPE Software Package

Loading the NPE software for the very first time to a bare metal system performs almost the same functions as upgrading an existing system the main difference being that the first load needs to create the base system directories for both what is called the current partition along with the partition for upgrades known as the alternate partition. This is the one and only time that the current partition will be written.

Subsequent upgrades will only ever write to the alternate partition ensuring that a running system remains unaffected by the installation process should a failure occur.

The upgrade process will load the new version of software into the “alternate” partition for security, it is then the responsibility of the user to save the current configuration to “Both” partitions before using the appropriate CLI or GUI option to reload the NPE to the “alternate” or newest software release.

This first step of this process is to download the latest release file from the support site, this file will be the same file you would use on a bare metal system and comes as a tgz tar ball file.

Once this has been downloaded from the support site onto a local drive the user must now “SCP” the file to the Bare Metal system either for the first install or an upgrade install – openssh-server will have been loaded provided the OS installation instructions were followed correctly.

Usually the file will be loaded to the base user’s home directory from where it is necessary for the user to be logged in as super user using the “sudo su” command.

As can be seen above the user is logged on and has entered root level, a typical release file is seen which must then be extracted using the command “tar zxvf filename.tgz”.

The result of the extraction is the creation of a subdirectory whose name is the same as the base tgz file, on closer inspection you will see this contains the OS that it was built for in this example Ubuntu 14.04 which is then followed by the NPE release number – in this case 5.1 and finally the build number which in this example is 4712.

As you can see from the following screen the extraction process has created a new directory loaded with 2 shell files and the release rpm file: -

Before executing the shell installation file it is worth reminding the user that for the installation to be successful the system MUST have access to the Internet as there are several hundred files belonging to the base Operating system that must be upgraded to those versions required by the NPE software package. Assuming connectivity exists the installation/upgrade can continue by changing directory to the newly created release directory and executing the stm-install shell file as shown below: -

The installation process will take several minutes since there are many updates that must be retrieved from the Internet as shown: -

A successful installation will result in the following display: -

Note that if this is the very first installation of the NPE software package it will be necessary to reboot the entire system before proceeding.

From Release 6.0 GA we have added automatic creation of management interfaces to deal with the external attacks seen on STM servers in the field. All Linux interfaces that have IP addresses configured when STM is started will be automatically created as STM management interfaces.

The Ubuntu firewall is enabled at every reboot with the previously defined rules being programmed into the firewall. Should a system be on public IP address at a remote data centre it will be necessary to create a list of allowable subnets in a using a comma sseparated value formatted file which must be present at boot time.

Failure to create a list of protected subnets in a file called “firewall_allowed_subnets” may actually lead to the inability to contact the system post upgrade to 6.0 releases and later systems.

It is therefore necessary to create the a file called "firewall_allowed_subnets" in both /etc/stm and /etc/stm.alt That file needs to have a comma separated list of the subnets you want to have access to the system, 0.0.0.0/0 is explicitely NOT allowed.

Add/create the file /etc/stm/firewall_allowed_subnets with the comma separated list of subnets (i.e 50.33.20.0/24) and copy to /etc/stm.alt

If running an older version of the STM and loading Release 6.0 GA for the first time following the installation of the software into the alternate partiion create and copy the firewall_allowed_subnets file into bot /etc/stm and /etc/stm.alt then do the following: -

cd /opt/stm/target.alt

sudo ./firewall_helper.sh

This will actually pre read the file and load the subnets into the firewall prior to it’s activation on rebooting to the new software.

With the firewall now loaded the operator can verify the correct allowed subnets using: - sudo ufw status

This command will list these rules that are in the firewall. If subnets are missing at this point that may need to manage the device the user should edit the firewall_allowed_subnets file prior to switchover and rerun the commands above – failure to add the correct subnets for management purposes will cause connectivity to the STM device to be lost.

If this is an upgrade the user is free to use the CLI or GUI to save any configuration changes to both software partitions and then request a reload of the NPE software package to the newest release.

Initialization Script

New in the 7.0 release is an initialization script designed to offer a user the opportunity to configure all required parameters to make the STM fully finctional including: -

System name - this usually defaults to “stm” when the Ubuntu system name and SNMP sysName are different.
Management IP address – usually defaults to DHCP unless otherwise set during the installation of Ubuntu
Management IP netmask – usually picked up from DHCP unless otherwise configured
Default gateway – usually picked up from DHCP unless otherwise configured
Management DNS server – usually picked up from DHCP unless otherwise configured
Management port allowed subnets – this is an entry in the Saisei “user_listener.py” script found in the scripts directory, the default allows ALL subnets but the user can limit Internal Hosts by configuring the subnet or subnets that are expected to be on the Internal side of the network. This is to make sure that Internal Hosts have not been infected by viruses that might spoof IP addresses.
Perspective – this release has two modes discussed leter in this document, normal operational mode uses the “base” perspective while Enterprise users may decide to use the “Enterprise” perspective which has been developed to simplify the configuration of the STM.
Automatic user detection – the STM can auto create user names based on the Internal IP address being used, the name is of the form User-a.b.c.d. If selected the STM will configure the “user_listener” script to start at power on.
Use reverse DNS for user identification – user_listener has the ability to check for a real user name using Reverse DNS, if chosen to do so user_listener will be modified to do the look up.
User subnets – this is an entry in the Saisei “user_listener.py” script found in the scripts directory, the default allows ALL subnets but the user can limit Internal Hosts by configuring the subnet or subnets that are expected to be on the Internal side of the network. This is to make sure that Internal Hosts have not been infected by viruses that might spoof IP addresses.
Model – by default the STM starts by using the “small” model, the user could change this to “large” if desired.
Internal BITW interface – each Bump in the Wire uses pair of interfaces, this option defines the Internal interface which is selected form a list
External BITW interface – defines the External interface to be paired with the previous option.

So when the user logs in for the very firt time the script will run and ask various questions, the responses will be as simple as yes or no or other specific items as defined in the text.

The order is: -

Do you wish to preconfigure the system?

You will need to answer some initial questions about the system usage

For the System name

Enter the system name (Simple string entry)

For the Management IP information

Do you wish to configure a static IP address for the management port? (If using DHCP)
The mamangement port has a static ip address of a.b.c.d configured (If static is found)
Enter the management port IP address (a.b.c.d) (When you decide to change)
Xxxx is not a valid IP address. Please enter a valid IP address (Possible error message)
Enter the management port netmask (a.b.c.d) (For subnet mask entry)
Xxxx is not a valid IP netmask. Please enter a valid netmask (Possible error message)
Enter the default gateway (a.b.c.d) (To enter the gateway)
Xxxx is not a valid IP address. Please enter a valid IP address (Possible error message)
Enter the DNS server (a.b.c.d)
Xxxx is not a valid IP address. Please enter a valid IP address (Possible error message)
Do you wish to allow access to the management port outside the directly connected network? (Allows special subnets to be added to the Allowable list and therefore the Ubuntu firewall)
Enter a subnet you want to allow access from (a.b.c.d/xx) where xx is number of significant bits. xx needs to be at least 8:
Xxxx is not a valid IP subnet. (Possible error message)
Do you wish to add more allowed subnets? (Allows the script to loop to add more subnets)

For User Listener – I.e. Create User names and link to internal Host IP addresses

Do you want to automatically link ip hosts to users? (To verify you want usernames)
Do you want to use reverse DNS to identify users? (If you want usernames then should we check if there is a DNS name)
Do you wish to restrict user creation to specific host subnets? (Should we use a restricted set of subnets)
Enter a subnet you want to create users from (a.b.c.d/xx) where xx is number of significant bits:
Please enter a valid subnet mask (Possible error message)
Do you wish to add more user subnets? (Allow multiple subnets to be added)

For the GUI display mode

Which perspective should be used: enterprise or base?
Please answer enterprise or base. (Possible error message)

For the STM model to use

Which model should be used: small or large?
Please answer small or large. (Possible error message)

For setting up a Bump in the Wire Interface pair

Do you wish to preconfigure a BITW?
Which interface is to be the internal interface of this BITW? See list above and enter the stm name: (Internal is the interface that points to the users)
Please answer with an stm name from the list. (Possible error message)
See the xxx interface LED flash on the server (The NIC link LED is flashed to help determine which interface has been chosen)
Which interface is to be the external interface of this BITW? See list above and enter the stm name:
Please answer with an stm name from the list. (Possible error message)
See the xxx interface LED flash on the server (The NIC link LED is flashed to help determine which interface has been chosen)
The external interface (external) is the same as the internal interface (internal). Please choose again. (Possible error when the same interface is chosen for both)