Content Comparison

Arbiter Poller is name given to a Poller which hosts

• Mongodb which An arbiter Mongodb instance apart from the regular Mongodb instance. This mongodArb instance acts as a Arbiter to the Main Primary and Secondary MongodbsPrimary replicated Mongodb.

• Nats: Third node of the Nats distributed setup .

...

• apart from ones running on Main Primary and Secondary Primary.

A change to the IP address of a running arbiter Mongodb would involves changes to distributed Mongodb and , Nats configuration and opCommon.json that needs to be updated accordingly.

ProcudureInstructions:

1. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)

sudo systemctl stop ophad

2. Main Primary: Delete all streams 

nats stream delete --user omkadmin --password op42opha42

If jq is installed on Main Primary, this below snippet can be used to automate deletion of all nats streams:

for stream_name in $(nats stream list --user omkadmin --password op42opha42 --json | jq -r '.[]'); 
      do nats stream rm --force --user omkadmin --password op42opha42 "$stream_name";
done

...

1. Nats changes in Main Primary and Secondary Primary in /etc/nats-server.conf
   In the cluster section, replace old Arbiter DNS/IP address with new Arbiter DNS/IP address of
   "nats://opha-dev5.opmantek.net:6222" "nats://opha-dev6.opmantek.net:6222"

   Code Block
   cluster { name: "C1" host: "opha-dev4.opmantek.net" listen: "0.0.0.0:6222" routes: [ "nats://opha-dev6.opmantek.net:6222" "nats://opha-dev7.opmantek.net:6222" ] }

2. Nats changes in New Arbiter Poller
   - Replace server name with DNS/IP address of the new Arbiter.
   - Comment out cluster section and edit the routes with the DNS/IP address of Main Primary and Secondary Primary
   

   Code Block
   server_name: "opha-dev6.opmantek.net" <Last section> cluster { name: "C1" host: "opha-dev4.opmantek.net" listen: "0.0.0.0:6222" routes: [ "nats://opha-dev5.opmantek.net:6222" "nats://opha-dev7.opmantek.net:6222" ] }

3. Disable the old Arbiter Nats config (Not required if its the same instance that has the IP address change)

   Code Block
   sudo systemctl stop nats-server.service sudo systemctl disable nats-server.service

4. Mongodb arbiter config changes done on the mongosh
   Main Primary : mongosh (https://www.mongodb.com/docs/manual/tutorial/replace-replica-set-member/ )

   Code Block
   mongosh --username opUserRW --password op42flow42 admin --port 27017 rs1 [direct: primary] admin> cfg = rs.conf() rs1 [direct: primary] admin> cfg.members[2].host = "opha-dev6.opmantek.net:27018" rs1 [direct: primary] admin> rs.reconfig(cfg) rs1 [direct: primary] admin> rs.conf()

   Verify the mongodb cluster config

   Code Block
   mongosh --username opUserRW --password op42flow42 admin --port 27017 rs1 [direct: primary] admin> rs.conf()

   Code Block
   members: [     {       _id: 0,       host: 'opha-dev4.opmantek.net:27017',       priority: 2,     },     {       _id: 1,       host: 'opha-dev7.opmantek.net:27017',       priority: 1,     },     {       _id: 2,       host: 'opha-dev6.opmantek.net:27018',       priority: 0,     }   ],

   Heartbeats from Arbiter

   Code Block
   mongosh --username opUserRW --password op42flow42 admin --port 27017 rs1 [direct: primary] admin> rs.status()

   Code Block
   { _id: 2, name: 'opha-dev6.opmantek.net:27018', health: 1, state: 7, stateStr: 'ARBITER', uptime: 35, lastHeartbeat: ISODate('2025-11-17T06:38:32.593Z'), lastHeartbeatRecv: ISODate('2025-11-17T06:38:32.638Z'), }

   
   Disable the old mongodArb service (Not required if its the same instance that has the IP address change)

   Code Block
   sudo systemctl stop mongodArb.service sudo systemctl disable mongodArb.service

5. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)
   Edit /usr/local/omk/conf/opCommon.json

   Code Block

...

...

1. omkadmin@lab-ophamb-mp01:/usr/local/omk/conf$ grep -a4 nats_cluster /usr/local/omk/conf/opCommon.json       "db_use_v26_features" : 1,       "redis_port" : 6379,       "redis_server" : "localhost",       "db_port" : "27017",       "nats_cluster" : [          “Main Primary ,          “Sec Primary”,          “New Arbiter Poller”       ],

...

Restart the services for the changes made to opCommon.json to get reflected.

...

...

sudo systemctl restart omkd opeventsd opchartsd

2. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)

   Code Block

...

1. sudo systemctl

...

1. stop ophad

...

7. Check to see if nats jetstream has been created 

...

...

1. Main Primary: Delete all streams 

   Code Block

...

...

/usr/local/omk/bin/ophad verify

1. nats stream delete --user omkadmin --password op42opha42

   If jq is installed on Main Primary, this below snippet can be used to automate deletion of all nats streams:

   Code Block
   for stream_name in $(nats stream list --user omkadmin --password op42opha42 --json | jq -r '.[]');        do nats stream rm --force --user omkadmin --password op42opha42 "$stream_name"; done

   Verify to check if all streams have been deleted

   Code Block
   nats stream info --user omkadmin --password op42opha42

...

1. nats: error: could not pick a Stream to operate on: no Streams are defined

2. Optiomal (but recommended if Nats is upgraded) cleanup the Nats storage /opt/data/nats

...

1. .
   Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)

   Code Block

...

...

1. sudo systemctl stop nats-server

...

1. clear /opt/nats/storage/jetstream

...

1. Code Block

...

1. sudo rm -rf /opt/nats/storage

...

1. Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)

   Code Block
   sudo systemctl start nats-server

2. All vms (Main Pri, Sec Pri, Pollers, Mirrors) - Restart the services for the changes made to opCommon.json to get reflected.

   Code Block
   sudo systemctl restart ophad omkd opeventsd opchartsd​

3. Check to see if nats jetstream has been created 

sudo systemctl start nats-servernats stream list --user omkadmin --password op42opha42

...

Verify ophad to check nats config

/usr/local/omk/bin/ophad verify

2025-11-17T16:25:56.215+1100 [INFO]  ophad.verify: Nats connect:
  result=
  | can connect to nats-server: opha-dev4.opmantek.net version: 2.11.9 ✅
  | can connect to nats-server: opha-dev6.opmantek.net version: 2.11.9 ✅
  | can connect to nats-server: opha-dev7.opmantek.net version: 2.11.9 ✅
  | we can connect to Nats-server ✅

2025-11-17T16:25:56.216+1100 [INFO]  ophad.verify: ready for liftoff 🚀