Changing the Arbiter Poller in opHA 5 replicated setup

Arbiter Poller is name given to a Poller which hosts

• An arbiter Mongodb instance apart from the regular Mongodb instance. This mongodArb instance acts as a Arbiter to the Main Primary and Secondary Primary replicated Mongodb.

• Nats: Third node of the Nats distributed setup apart from ones running on Main Primary and Secondary Primary.

A change to the IP address of a running arbiter Mongodb would involves changes to distributed Mongodb, Nats configuration and opCommon.json that needs to be updated accordingly.

Instructions:

1. Nats changes in Main Primary and Secondary Primary in /etc/nats-server.conf
   In the cluster section, replace old Arbiter DNS/IP address with new Arbiter DNS/IP address of
   "nats://opha-dev5.opmantek.net:6222" "nats://opha-dev6.opmantek.net:6222"

   cluster { name: "C1" host: "opha-dev4.opmantek.net" listen: "0.0.0.0:6222" routes: [ "nats://opha-dev6.opmantek.net:6222" "nats://opha-dev7.opmantek.net:6222" ] }

2. Nats changes in New Arbiter Poller
   - Replace server name with DNS/IP address of the new Arbiter.
   - Comment out cluster section and edit the routes with the DNS/IP address of Main Primary and Secondary Primary
   
   

   server_name: "opha-dev6.opmantek.net" <Last section> cluster { name: "C1" host: "opha-dev4.opmantek.net" listen: "0.0.0.0:6222" routes: [ "nats://opha-dev5.opmantek.net:6222" "nats://opha-dev7.opmantek.net:6222" ] }

3. Disable the old Arbiter Nats config (Not required if its the same instance that has the IP address change)

   sudo systemctl stop nats-server.service sudo systemctl disable nats-server.service

4. Mongodb arbiter config changes done on the mongosh
   Main Primary : mongosh (https://www.mongodb.com/docs/manual/tutorial/replace-replica-set-member/ )

   mongosh --username opUserRW --password op42flow42 admin --port 27017 rs1 [direct: primary] admin> cfg = rs.conf() rs1 [direct: primary] admin> cfg.members[2].host = "opha-dev6.opmantek.net:27018" rs1 [direct: primary] admin> rs.reconfig(cfg) rs1 [direct: primary] admin> rs.conf()

   Verify the mongodb cluster config

   mongosh --username opUserRW --password op42flow42 admin --port 27017 rs1 [direct: primary] admin> rs.conf()

    

    members: [     {       _id: 0,       host: 'opha-dev4.opmantek.net:27017',       priority: 2,     },     {       _id: 1,       host: 'opha-dev7.opmantek.net:27017',       priority: 1,     },     {       _id: 2,       host: 'opha-dev6.opmantek.net:27018',       priority: 0,     }   ],

   Heartbeats from Arbiter

   mongosh --username opUserRW --password op42flow42 admin --port 27017 rs1 [direct: primary] admin> rs.status()
   { _id: 2, name: 'opha-dev6.opmantek.net:27018', health: 1, state: 7, stateStr: 'ARBITER', uptime: 35, lastHeartbeat: ISODate('2025-11-17T06:38:32.593Z'), lastHeartbeatRecv: ISODate('2025-11-17T06:38:32.638Z'), }

   
   Disable the old mongodArb service (Not required if its the same instance that has the IP address change)

   sudo systemctl stop mongodArb.service sudo systemctl disable mongodArb.service

    

5. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)
   Edit /usr/local/omk/conf/opCommon.json

   omkadmin@lab-ophamb-mp01:/usr/local/omk/conf$ grep -a4 nats_cluster /usr/local/omk/conf/opCommon.json       "db_use_v26_features" : 1,       "redis_port" : 6379,       "redis_server" : "localhost",       "db_port" : "27017",       "nats_cluster" : [          “Main Primary ,          “Sec Primary”,          “New Arbiter Poller”       ],

    

6. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)

   sudo systemctl stop ophad

7. Main Primary: Delete all streams 

   nats stream delete --user omkadmin --password op42opha42

   If jq is installed on Main Primary, this below snippet can be used to automate deletion of all nats streams:

   for stream_name in $(nats stream list --user omkadmin --password op42opha42 --json | jq -r '.[]');        do nats stream rm --force --user omkadmin --password op42opha42 "$stream_name"; done

   Verify to check if all streams have been deleted

   nats stream info --user omkadmin --password op42opha42 nats: error: could not pick a Stream to operate on: no Streams are defined

8. Optiomal (but recommended if Nats is upgraded) cleanup the Nats storage /opt/data/nats.
   Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)

   sudo systemctl stop nats-server

   clear /opt/nats/storage/jetstream

   sudo rm -rf /opt/nats/storage

   Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)

   sudo systemctl start nats-server

9. All vms (Main Pri, Sec Pri, Pollers, Mirrors) - Restart the services for the changes made to opCommon.json to get reflected.

   sudo systemctl restart ophad omkd opeventsd opchartsd​

10. Check to see if nats jetstream has been created 

Open

Verify ophad to check nats config