Changing the Arbiter Poller in a replicated setup opHA 5

Arbiter Poller is name given to a Poller which hosts

• Mongodb which acts as a Arbiter to the Primary and Secondary Mongodbs

• Nats: Third node of the Nats distributed setup.

This involves changes to distributed Mongodb and Nats configuration and opCommon.json that needs to be updated accordingly.

Procudure:

1. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)

sudo systemctl stop ophad

2. Main Primary: Delete all streams 

nats stream delete --user omkadmin --password op42opha42

If jq is installed on Main Primary, this below snippet can be used to automate deletion of all nats streams:

for stream_name in $(nats stream list --user omkadmin --password op42opha42 --json | jq -r '.[]'); 
      do nats stream rm --force --user omkadmin --password op42opha42 "$stream_name";
done

3. Skip the optional step to cleanup the Nats storage /opt/data/nats if not needed.
   Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)

   sudo systemctl stop nats-server

   clear /opt/nats/storage/jetstream

    rm -rf /opt/nats/storage

   Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)

   sudo systemctl start nats-server

4. Mongodb arbiter config changes done on the mongosh
   Main Primary : mongosh (https://www.mongodb.com/docs/manual/tutorial/replace-replica-set-member/ )

   mongosh --username opUserRW --password op42flow42 admin --port 27017
   rs1 [direct: primary] admin> cfg = rs.conf()
   rs1 [direct: primary] admin> cfg.members[2].host = "opha-dev6.opmantek.net:27018"
   rs1 [direct: primary] admin> rs.reconfig(cfg)
   rs1 [direct: primary] admin> rs.conf()

   Verify the mongodb cluster config
   

   mongosh --username opUserRW --password op42flow42 admin --port 27017
   
   rs1 [direct: primary] admin> rs.conf()

    members: [
       {
         _id: 0,
         host: 'opha-dev4.opmantek.net:27017',
         priority: 2,
       },
       {
         _id: 1,
         host: 'opha-dev7.opmantek.net:27017',
         priority: 1,
       },
       {
         _id: 2,
         host: 'opha-dev6.opmantek.net:27018',
         priority: 0,
       }
     ],

5. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)

Edit /usr/local/omk/conf/opCommon.json

omkadmin@lab-ophamb-mp01:/usr/local/omk/conf$ grep -a4 nats_cluster /usr/local/omk/conf/opCommon.json
      "db_use_v26_features" : 1,
      "redis_port" : 6379,
      "redis_server" : "localhost",
      "db_port" : "27017",
      "nats_cluster" : [
         “Main Primary ,
         “Sec Primary”,
         “New Arbiter Poller”
      ],

Restart the services for the changes made to opCommon.json to get reflected.

sudo systemctl restart omkd opeventsd opchartsd

5. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)

sudo systemctl start ophad

6. Ophad verify

/usr/local/omk/bin/ophad verify

7. Check to see if nats jetstream has been created 

nats stream list --user omkadmin --password op42opha42