Changing the Arbiter Poller in opHA 5 replicated setup

Arbiter Poller is name given to a Poller which hosts

An arbiter Mongodb instance apart from the regular Mongodb instance. This mongodArb instance acts as a Arbiter to the Main Primary and Secondary Primary replicated Mongodb.
Nats: Third node of the Nats distributed setup apart from ones running on Main Primary and Secondary Primary.

A change to the IP address of a running arbiter Mongodb would involves changes to distributed Mongodb, Nats configuration and opCommon.json that needs to be updated accordingly.

Instructions:

Nats changes in Main Primary and Secondary Primary in /etc/nats-server.conf
In the cluster section, replace old Arbiter DNS/IP address with new Arbiter DNS/IP address of
~~"nats://opha-dev5.opmantek.net:6222"~~ "nats://opha-dev6.opmantek.net:6222"
```
cluster {
  name: "C1"
  host: "opha-dev4.opmantek.net"
  listen: "0.0.0.0:6222"
  routes: [
    "nats://opha-dev6.opmantek.net:6222"
    "nats://opha-dev7.opmantek.net:6222"
  ]
}
```

Nats changes in New Arbiter Poller
- Replace server name with DNS/IP address of the new Arbiter.
- Comment out cluster section and edit the routes with the DNS/IP address of Main Primary and Secondary Primary

server_name: "opha-dev6.opmantek.net"

<Last section>
cluster {
  name: "C1"
  host: "opha-dev4.opmantek.net"
  listen: "0.0.0.0:6222"
  routes: [
    "nats://opha-dev5.opmantek.net:6222"
    "nats://opha-dev7.opmantek.net:6222"
  ]
}

Disable the old Arbiter Nats config

sudo systemctl stop nats-server.service
sudo systemctl disable nats-server.service

Mongodb arbiter config changes done on the mongosh
Main Primary : mongosh (https://www.mongodb.com/docs/manual/tutorial/replace-replica-set-member/ )

mongosh --username opUserRW --password op42flow42 admin --port 27017
rs1 [direct: primary] admin> cfg = rs.conf()
rs1 [direct: primary] admin> cfg.members[2].host = "opha-dev6.opmantek.net:27018"
rs1 [direct: primary] admin> rs.reconfig(cfg)
rs1 [direct: primary] admin> rs.conf()

Verify the mongodb cluster config

mongosh --username opUserRW --password op42flow42 admin --port 27017

rs1 [direct: primary] admin> rs.conf()

 members: [
    {
      _id: 0,
      host: 'opha-dev4.opmantek.net:27017',
      priority: 2,
    },
    {
      _id: 1,
      host: 'opha-dev7.opmantek.net:27017',
      priority: 1,
    },
    {
      _id: 2,
      host: 'opha-dev6.opmantek.net:27018',
      priority: 0,
    }
  ],

Heartbeats from Arbiter

mongosh --username opUserRW --password op42flow42 admin --port 27017
rs1 [direct: primary] admin> rs.status()

    {
      _id: 2,
      name: 'opha-dev6.opmantek.net:27018',
      health: 1,
      state: 7,
      stateStr: 'ARBITER',
      uptime: 35,
      lastHeartbeat: ISODate('2025-11-17T06:38:32.593Z'),
      lastHeartbeatRecv: ISODate('2025-11-17T06:38:32.638Z'),
    }

Disable the old mongodArb service

sudo systemctl stop mongodArb.service
sudo systemctl disable mongodArb.service

All vms: (Main Pri, Sec Pri, Pollers, Mirrors)
Edit /usr/local/omk/conf/opCommon.json

omkadmin@lab-ophamb-mp01:/usr/local/omk/conf$ grep -a4 nats_cluster /usr/local/omk/conf/opCommon.json
      "db_use_v26_features" : 1,
      "redis_port" : 6379,
      "redis_server" : "localhost",
      "db_port" : "27017",
      "nats_cluster" : [
         “Main Primary ,
         “Sec Primary”,
         “New Arbiter Poller”
      ],

All vms: (Main Pri, Sec Pri, Pollers, Mirrors)
```
sudo systemctl stop ophad
```

Main Primary: Delete all streams

nats stream delete --user omkadmin --password op42opha42

If jq is installed on Main Primary, this below snippet can be used to automate deletion of all nats streams:

for stream_name in $(nats stream list --user omkadmin --password op42opha42 --json | jq -r '.[]'); 
      do nats stream rm --force --user omkadmin --password op42opha42 "$stream_name";
done

Verify to check if all streams have been deleted

nats stream info --user omkadmin --password op42opha42
nats: error: could not pick a Stream to operate on: no Streams are defined

Skip the optional step to cleanup the Nats storage /opt/data/nats if not needed.
Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)
```
sudo systemctl stop nats-server
```
clear /opt/nats/storage/jetstream
```
sudo rm -rf /opt/nats/storage
```
Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)
```
sudo systemctl start nats-server
```
All vms (Main Pri, Sec Pri, Pollers, Mirrors) - Restart the services for the changes made to opCommon.json to get reflected.
```
sudo systemctl restart ophad omkd opeventsd opchartsd
```
Check to see if nats jetstream has been created

nats stream list --user omkadmin --password op42opha42

Verify ophad to check nats config

/usr/local/omk/bin/ophad verify

2025-11-17T16:25:56.215+1100 [INFO]  ophad.verify: Nats connect:
  result=
  | can connect to nats-server: opha-dev4.opmantek.net version: 2.11.9 ✅
  | can connect to nats-server: opha-dev6.opmantek.net version: 2.11.9 ✅
  | can connect to nats-server: opha-dev7.opmantek.net version: 2.11.9 ✅
  | we can connect to Nats-server ✅

2025-11-17T16:25:56.216+1100 [INFO]  ophad.verify: ready for liftoff 🚀