Arbiter Poller is name given to a Poller which hosts

This involves changes to distributed Mongodb and Nats configuration and opCommon.json that needs to be updated accordingly.

Procedure:

  1. Nats changes in Main Primary and Secondary Primary in /etc/nats-server.conf
    In the cluster section, replace old Arbiter DNS/IP address with new Arbiter DNS/IP address of
    "nats://opha-dev5.opmantek.net:6222" "nats://opha-dev6.opmantek.net:6222"

    cluster {
  name: "C1"
  host: "opha-dev4.opmantek.net"
  listen: "0.0.0.0:6222"
  routes: [
    "nats://opha-dev6.opmantek.net:6222"
    "nats://opha-dev7.opmantek.net:6222"
  ]
}

  2. Nats changes in New Arbiter Poller
    - Replace server name with DNS/IP address of the new Arbiter.
    - Comment out cluster section and edit the routes with the DNS/IP address of Main Primary and Secondary Primary

    server_name: "opha-dev6.opmantek.net"

<Last section>
cluster {
  name: "C1"
  host: "opha-dev4.opmantek.net"
  listen: "0.0.0.0:6222"
  routes: [
    "nats://opha-dev5.opmantek.net:6222"
    "nats://opha-dev7.opmantek.net:6222"
  ]
}

  3. Disable the old Arbiter Nats config

    sudo systemctl stop nats-server.service
sudo systemctl disable nats-server.service

  4. Mongodb arbiter config changes done on the mongosh
    Main Primary : mongosh (https://www.mongodb.com/docs/manual/tutorial/replace-replica-set-member/ )

    mongosh --username opUserRW --password op42flow42 admin --port 27017
rs1 [direct: primary] admin> cfg = rs.conf()
rs1 [direct: primary] admin> cfg.members[2].host = "opha-dev6.opmantek.net:27018"
rs1 [direct: primary] admin> rs.reconfig(cfg)
rs1 [direct: primary] admin> rs.conf()

    Verify the mongodb cluster config 

    mongosh --username opUserRW --password op42flow42 admin --port 27017

rs1 [direct: primary] admin> rs.conf()

     members: [
    {
      _id: 0,
      host: 'opha-dev4.opmantek.net:27017',
      priority: 2,
    },
    {
      _id: 1,
      host: 'opha-dev7.opmantek.net:27017',
      priority: 1,
    },
    {
      _id: 2,
      host: 'opha-dev6.opmantek.net:27018',
      priority: 0,
    }
  ],

    Heartbeats from Arbiter

    mongosh --username opUserRW --password op42flow42 admin --port 27017
rs1 [direct: primary] admin> rs.status()

        {
      _id: 2,
      name: 'opha-dev6.opmantek.net:27018',
      health: 1,
      state: 7,
      stateStr: 'ARBITER',
      uptime: 35,
      lastHeartbeat: ISODate('2025-11-17T06:38:32.593Z'),
      lastHeartbeatRecv: ISODate('2025-11-17T06:38:32.638Z'),
    }


    Disable the old mongodArb service

    sudo systemctl stop mongodArb.service
sudo systemctl disable mongodArb.service

  5. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)
    Edit /usr/local/omk/conf/opCommon.json

    omkadmin@lab-ophamb-mp01:/usr/local/omk/conf$ grep -a4 nats_cluster /usr/local/omk/conf/opCommon.json
      "db_use_v26_features" : 1,
      "redis_port" : 6379,
      "redis_server" : "localhost",
      "db_port" : "27017",
      "nats_cluster" : [
         “Main Primary ,
         “Sec Primary”,
         “New Arbiter Poller”
      ],

  6. All vms: (Main Pri, Sec Pri, Pollers, Mirrors)

    sudo systemctl stop ophad

  7. Main Primary: Delete all streams 

    nats stream delete --user omkadmin --password op42opha42

    If jq is installed on Main Primary, this below snippet can be used to automate deletion of all nats streams:

    for stream_name in $(nats stream list --user omkadmin --password op42opha42 --json | jq -r '.[]'); 
      do nats stream rm --force --user omkadmin --password op42opha42 "$stream_name";
done

    Verify to check if all streams have been deleted

    nats stream info --user omkadmin --password op42opha42
nats: error: could not pick a Stream to operate on: no Streams are defined

  8. Skip the optional step to cleanup the Nats storage /opt/data/nats if not needed.
    Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)

    sudo systemctl stop nats-server

    clear /opt/nats/storage/jetstream

    sudo rm -rf /opt/nats/storage

    Main Pri, Sec Pri and Poller (all 3 servers where nats has been running)

    sudo systemctl start nats-server

  9. All vms (Main Pri, Sec Pri, Pollers, Mirrors) - Restart the services for the changes made to opCommon.json to get reflected.

    sudo systemctl restart ophad omkd opeventsd opchartsd​

  10. Check to see if nats jetstream has been created 

nats stream list --user omkadmin --password op42opha42
Screenshot 2025-11-17 at 17.28.52.png

Verify ophad to check nats config

/usr/local/omk/bin/ophad verify
2025-11-17T16:25:56.215+1100 [INFO]  ophad.verify: Nats connect:
  result=
  | can connect to nats-server: opha-dev4.opmantek.net version: 2.11.9 ✅
  | can connect to nats-server: opha-dev6.opmantek.net version: 2.11.9 ✅
  | can connect to nats-server: opha-dev7.opmantek.net version: 2.11.9 ✅
  | we can connect to Nats-server ✅

2025-11-17T16:25:56.216+1100 [INFO]  ophad.verify: ready for liftoff 🚀