Replication Setup and Configuration Guide

opHA 5 introduces the concept of a Primary & multiple Poller servers and a message bus: (with replication)

• The Primary is the server that keeps the information of all the pollers and it is where we can read all the information from. The Consumer (ophad running on Primary) reads data continuously off the message bus and stores it in the database

• The Producer (ophad running on Pollers) has a change stream setup for the collections and recieves update from the database if there is any. This change stream data is pushed onto the message bus.

• With Replication and Nats if the main-primary goes down the secondary primary will take over also if a poller goes down the mirror will take over.

Here are terms used in this project together with their meanings.

1. Software Installation

1.1 Download Software

Download the required software onto each server.

1.2 Install Software

IMPORTANT : All the servers should be setup with the following

a. Pollers and mirrors discovered by the Main-Primary

b. Server names set

c. Secondary Primary server setup as Primary

c. One server needs to be an Arbiter. A poller can be used for this

c. Mongo 6 installed

In this setup guide we are using example server/host names so you will have to use the names that you have set for your own configuration. For this example we have the required 6 servers for the setup.

IMPORTANT :

One server should be set as an Arbiter and in this example we are using Poller1 Server

Main-Primary Server : opha-dev1.opmantek.net

Poller1 Server opha-dev2.opmantek.net [Arbiter Server]

Mirror1 Server opha-dev3.opmantek.net

Poller2 Server opha-dev4.opmantek.net

Mirror2 Server opha-dev5.opmantek.net

Second-Primary opha-dev6.opmantek.net

Run the installation commands on each server as needed.

All the following setup commands in this setup should be run as the root user

sh ./<executable>

e.g

sh ./opHA-Linux-x86_64-5.0.0.run

VERY IMPORTANT : During the installation of the arbiter server you must answer 'N' at the following prompt

2. NATS Setup

This configuration should be applied to the Main-Primary, Arbiter, and Secondary Primary servers ONLY.

1. Edit Configuration: Update the /etc/nats-server.conf file with the following settings:

   server_name: "opha-dev1.opmantek.net"  #The local server  
   http_port: 8222 
   listen: 4222 
   jetstream: enabled 
   accounts { 
     $SYS { 
       users = [ 
         { user: "admin",
           pass: "password" 
         } 
       ] 
     } 
     ophad: { 
       users: [ 
           { user: "omkadmin", password: "op42opha42" } 
       ] 
       jetstream: enabled 
     } 
   } 
   
   jetstream { 
     store_dir: "/opt/nats/storage" 
     max_memory_store: 1028M 
     max_file_store: 1028M 
   } 
   cluster { 
     name: "C1" 
     host: "opha-dev1.opmantek.net"  #The current host 
     # the current server 
     listen: "0.0.0.0:6222" 
     routes: [ 
       # secondary primary
       "nats://opha-dev6.opmantek.net:6222" 
       # server with the arbiter 
       "nats://opha-dev2.opmantek.net:6222" 
       # other servers 
     ] 
   }

3. Common Configuration for All Servers

3.1 Update /usr/local/omk/conf/opCommon.json

1. Backup Configuration: Make a backup copy of the opCommon.json file before making any changes.

2. Add NATS Cluster Configuration: Ensure the following configuration is included in opCommon.json on all servers. In the below example we have added the main-primary, arbiter and second-primary in the nats_cluster array under the “database“ json object

   "nats_cluster": [ "opha-dev1.opmantek.net:4222", "opha-dev2.opmantek.net:4222", "opha-dev6.opmantek.net:4222" ]

4. Server-Specific Configuration

4.1 Update /usr/local/omk/conf/opCommon.json for Main-Primary and Secondary Primary only

and Secondary Primary servers:

"db_replica_set": "rs1", 
"db_mongo_cluster": [ "opha-dev1.opmantek.net", "opha-dev6.opmantek.net" ], 
"nats_num_replicas": 3, 
"nats_cluster": [ "opha-dev1.opmantek.net", "opha-dev2.opmantek.net", "opha-dev6.opmantek.net" ]

4.2 Update License

Currently there are 2 available licenses and messagebus will not run without a license.

1. opHA-MB for NMIS & opCharts (This license is required for messagebus to run)

2. opHA-MB for opEvents (This license is an add-on for opEvents). With this add-on opEvents will use MessageBus to sync opEvents data from each poller/mirror to the main-primary and in replication mode opEvents data is synced to the secondary-primary. In replication mode, If the main-primary were to go down the secondary primary will retain the opEvents data.

   

5. MongoDB Setup (Main-Primary Only)

The following steps should be performed only on the Main-Primary MongoDB instance.

5.1 Initialize MongoDB Replica Set

Connect to MongoDB using the mongosh command and configure the replica set:

Run the following command

mongosh --username opUserRW --password op42flow42 admin

Run the following command

rs.initiate({ _id: "rs1", version: 1, members: [ { _id: 0, host : "opha-dev1.opmantek.net:27017" }, { _id: 1, host : "opha-dev6.opmantek.net:27017" } ] })

5.2 Set Default Write Concern

Run the following command to set the default write concern:

db.adminCommand({ "setDefaultRWConcern": 1, "defaultWriteConcern": { "w": 1 } })

5.3 Add Arbiter

Run the following command to add an arbiter to the replica set:

rs.addArb("opha-dev2.opmantek.net:27018")

5.4 Restart mongod on the Main-Primary and Second-Primary

Run the following command on the Main-Primary, Secondary Primary, and Arbiter servers ONLY:

sudo systemctl restart mongod

6. Starting Services

6.1 Start NATS

Start the NATS service on the Main-Primary, Secondary Primary, and Arbiter servers ONLY:

systemctl start nats-server

6.2 Restart mongod on Pollers and Mirrors only

After configuration changes, restart mongod on all Mirrors and Pollers servers ONLY :

systemctl restart mongod

6.3 Restart services on all servers

After configuration changes, restart the services on all the servers :

systemctl restart nmis9d opchartsd opeventsd omkd ophad

6.4 Start the Poller and Mirrors

Execute the following command on the poller and mirror servers.

cd /usr/local/omk/bin
./ophad cmd producer start

7. Final Steps

1. Accept EULAs: Make sure when you login to each server to confirm that all End User License Agreements (EULAs) are accepted.

8. Switching Main and Secondary Primary Servers

To switch from Main-Primary to Secondary Primary, follow these steps:

1. Connect to MongoDB:

   mongosh --username opUserRW --password op42flow42 admin

2. Update member priorities:

   cfg = rs.conf()
   cfg.members[0].priority = 0.6
   cfg.members[1].priority = 0.5
   rs.reconfig(cfg)