Changing the Arbiter Poller in opHA 5 replicated setup

Arbiter Poller is name given to a Poller which hosts

An arbiter Mongodb instance apart from the regular Mongodb instance. This mongodArb instance acts as a Arbiter to the Main Primary and Secondary Primary replicated Mongodb.
Nats: Third node of the Nats distributed setup apart from ones running on Main Primary and Secondary Primary.

This involves changes to distributed Mongodb and Nats configuration and opCommon.json that needs to be updated accordingly.

Procedure:

Nats changes in Main Primary and Secondary Primary in /etc/nats-server.conf
In the cluster section, replace old Arbiter DNS/IP address with new Arbiter DNS/IP address of
~~"nats://opha-dev5.opmantek.net:6222"~~ "nats://opha-dev6.opmantek.net:6222"
```
cluster {
  name: "C1"
  host: "opha-dev4.opmantek.net"
  listen: "0.0.0.0:6222"
  routes: [
    "nats://opha-dev6.opmantek.net:6222"
    "nats://opha-dev7.opmantek.net:6222"
  ]
}
```

Nats changes in New Arbiter Poller
- Replace server name with DNS/IP address of the new Arbiter.
- Comment out cluster section and edit the routes with the DNS/IP address of Main Primary and Secondary Primary

server_name: "opha-dev6.opmantek.net"

<Last section>
cluster {
  name: "C1"
  host: "opha-dev4.opmantek.net"
  listen: "0.0.0.0:6222"
  routes: [
    "nats://opha-dev5.opmantek.net:6222"
    "nats://opha-dev7.opmantek.net:6222"
  ]
}

Disable the old Arbiter Nats config

sudo systemctl stop nats-server.service
sudo systemctl disable nats-server.service

Mongodb arbiter config changes done on the mongosh
Main Primary : mongosh (https://www.mongodb.com/docs/manual/tutorial/replace-replica-set-member/ )

mongosh --username opUserRW --password op42flow42 admin --port 27017
rs1 [direct: primary] admin> cfg = rs.conf()
rs1 [direct: primary] admin> cfg.members[2].host = "opha-dev6.opmantek.net:27018"
rs1 [direct: primary] admin> rs.reconfig(cfg)
rs1 [direct: primary] admin> rs.conf()

Verify the mongodb cluster config

mongosh --username opUserRW --password op42flow42 admin --port 27017

rs1 [direct: primary] admin> rs.conf()

 members: [
    {
      _id: 0,
      host: 'opha-dev4.opmantek.net:27017',
      priority: 2,
    },
    {
      _id: 1,
      host: 'opha-dev7.opmantek.net:27017',
      priority: 1,
    },
    {
      _id: 2,
      host: 'opha-dev6.opmantek.net:27018',
      priority: 0,
    }
  ],

Heartbeats from Arbiter

mongosh --username opUserRW --password op42flow42 admin --port 27017
rs1 [direct: primary] admin> rs.status()

    {
      _id: 2,
      name: 'opha-dev6.opmantek.net:27018',
      health: 1,
      state: 7,
      stateStr: 'ARBITER',
      uptime: 35,
      lastHeartbeat: ISODate('2025-11-17T06:38:32.593Z'),
      lastHeartbeatRecv: ISODate('2025-11-17T06:38:32.638Z'),
    }

Disable the old mongodArb service

sudo systemctl stop mongodArb.service
sudo systemctl disable mongodArb.service

All vms: (Main Pri, Sec Pri, Pollers, Mirrors)
Edit /usr/local/omk/conf/opCommon.json

omkadmin@lab-ophamb-mp01:/usr/local/omk/conf$ grep -a4 nats_cluster /usr/local/omk/conf/opCommon.json
      "db_use_v26_features" : 1,
      "redis_port" : 6379,
      "redis_server" : "localhost",
      "db_port" : "27017",
      "nats_cluster" : [
         “Main Primary ,
         “Sec Primary”,
         “New Arbiter Poller”
      ],

All vms: (Main Pri, Sec Pri, Pollers, Mirrors)
```
sudo systemctl stop ophad
```

Main Primary: Delete all streams

nats stream delete --user omkadmin --password op42opha42

If jq is installed on Main Primary, this below snippet can be used to automate deletion of all nats streams:

for stream_name in $(nats stream list --user omkadmin --password op42opha42 --json | jq -r '.[]'); 
      do nats stream rm --force --user omkadmin --password op42opha42 "$stream_name";
done

Verify to check if all streams have been deleted

nats stream info --user omkadmin --password op42opha42
nats: error: could not pick a Stream to operate on: no Streams are defined

Skip the optional step to cleanup the Nats storage /opt/data/nats if not needed.
Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)
```
sudo systemctl stop nats-server
```
clear /opt/nats/storage/jetstream
```
sudo rm -rf /opt/nats/storage
```
Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)
```
sudo systemctl start nats-server
```
All vms (Main Pri, Sec Pri, Pollers, Mirrors) - Restart the services for the changes made to opCommon.json to get reflected.
```
sudo systemctl restart ophad omkd opeventsd opchartsd
```
Check to see if nats jetstream has been created

nats stream list --user omkadmin --password op42opha42

Verify ophad to check nats config

/usr/local/omk/bin/ophad verify

2025-11-17T16:25:56.215+1100 [INFO]  ophad.verify: Nats connect:
  result=
  | can connect to nats-server: opha-dev4.opmantek.net version: 2.11.9 ✅
  | can connect to nats-server: opha-dev6.opmantek.net version: 2.11.9 ✅
  | can connect to nats-server: opha-dev7.opmantek.net version: 2.11.9 ✅
  | we can connect to Nats-server ✅

2025-11-17T16:25:56.216+1100 [INFO]  ophad.verify: ready for liftoff 🚀