Content Comparison

...

nats stream info --user omkadmin --password op42opha42

             Description: Streaming Message ophamb
                Subjects: 12da408c-adcc-441b-8918-1dafe07b3f88.*
                Replicas: 3
                 Storage: File


Cluster Information:

                    Name: C1
           Cluster Group: S-R3F-dLBC9xEB
                  Leader: http://10.1.50.21 (11d22h59m24s)
                 Replica: http://10.1.50.22/, current, seen 127ms ago
                 Replica: http://10.1.60.22, current, seen 127ms ago

Short script using jq to output the cluster information

for stream_name in $(nats stream list --user omkadmin --password op42opha42 --json | jq -r '.[]');
  do echo $stream_name;
    nats stream info --user omkadmin --password op42opha42 "$stream_name" --json | jq -r '.cluster.leader';
    nats stream info --user omkadmin --password op42opha42 "$stream_name" --json | jq -r '[.cluster.replicas[] |  .name ]';
  done

12da408c-adcc-441b-8918-1dafe07b3f88
http://10.1.50.21
[
  "http://10.1.50.22/",
  "http://10.1.60.22"
]
5e06e63b-2cbf-4aac-a2e9-81ee3133119e
http://10.1.60.22
[
  "http://10.1.50.21",
  "http://10.1.50.22/"
]
5f1a2b3a-1132-48bd-a60f-cefec693d30c
http://10.1.50.21
[
  "http://10.1.50.22/",
  "http://10.1.60.22"
]
7c20c663-e444-44af-9678-1e0a7534a834
http://10.1.50.21
[
  "http://10.1.50.22/",
  "http://10.1.60.22"
]

4. Mongo cluster heartbeat check on Main Primary

   Code Block

   shankarn@opha-dev4:/usr/local/omk/conf$ mongosh --username opUserRW --password op42flow42 admin --port 27017 rs1 [direct: primary] admin> rs.status() { ... members: [ { _id: 0, name: 'opha-dev4.opmantek.net:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 17503, optime: { ts: Timestamp({ t: 1763526818, i: 9 }), t: Long('7') }, optimeDate: ISODate('2025-11-19T04:33:38.000Z'), lastAppliedWallTime: ISODate('2025-11-19T04:33:38.225Z'), lastDurableWallTime: ISODate('2025-11-19T04:33:38.190Z'), }, { _id: 1, name: 'opha-dev7.opmantek.net:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 17496, optime: { ts: Timestamp({ t: 1763526814, i: 1 }), t: Long('7') }, optimeDurable: { ts: Timestamp({ t: 1763526814, i: 1 }), t: Long('7') }, optimeDate: ISODate('2025-11-19T04:33:34.000Z'), optimeDurableDate: ISODate('2025-11-19T04:33:34.000Z'), lastAppliedWallTime: ISODate('2025-11-19T04:33:38.225Z'), lastDurableWallTime: ISODate('2025-11-19T04:33:38.225Z'), lastHeartbeat: ISODate('2025-11-19T04:33:36.300Z'), lastHeartbeatRecv: ISODate('2025-11-19T04:33:37.493Z'), }, { _id: 2, name: 'opha-dev6.opmantek.net:27018', health: 1, state: 7, stateStr: 'ARBITER', uptime: 17496, lastHeartbeat: ISODate('2025-11-19T04:33:36.301Z'), lastHeartbeatRecv: ISODate('2025-11-19T04:33:36.290Z'), } ],

5. Run the command sudo /usr/local/omk/bin/ophad verify on all the Peers/Primary.

   The last line “ophad.verify: ready for liftoff 🚀 “ to indicate the configuration is good.

    

   Code Block

   shankarn@opha-dev5:~$ sudo /usr/local/omk/bin/ophad verify [sudo] password for shankarn: ophad v0.0.52: agent Appending to file "/usr/local/omk/log/ophad.log" Settings ----------------------------------------- * ClusterId: 783d7b91-6c64-4db9-a28f-6364a54b8505 * OMKDatabase: * ConnectionTimeout: 5h33m20s * RetryTimeout: 3m0s * PingTimeout: 33m20s * QueryTimeout: 1h23m20s * Port: 27017 * Server: localhost * MongoCluster: [] * ReplicaSet: (blank) * Name: omk_shared * Username: opUserRW * Password: ****** * WriteConcern: 1 * Uri: (blank) * BatchSize: 0 * BatchTimeout: 0 * NMISDatabase: * ConnectionTimeout: 2m0s * RetryTimeout: 3m0s * PingTimeout: 20s * QueryTimeout: 1h23m20s * Port: 27017 * Server: localhost * MongoCluster: [] * ReplicaSet: (blank) * Name: nmisng * Username: opUserRW * Password: ****** * WriteConcern: 1 * Uri: (blank) * BatchSize: 50 * BatchTimeout: 500 * OpEventsDatabase: * ConnectionTimeout: 2m0s * RetryTimeout: 3m0s * PingTimeout: 20s * QueryTimeout: 5m0s * Port: 27017 * Server: localhost * MongoCluster: [] * ReplicaSet: (blank) * Name: opevents * Username: opUserRW * Password: ****** * WriteConcern: 1 * Uri: (blank) * BatchSize: 50 * BatchTimeout: 500 * OMK: * LogLevel: info * BindAddr: * * Directories: * Base: /usr/local/omk * Conf: /usr/local/omk/conf * Logs: /usr/local/omk/log * Var: /usr/local/omk/var * OPHA: * DBName: opha * StreamingApps: [nmis opevents] * Logfile: /usr/local/omk/log/ophad.log * MongoWatchFilters: [] * StreamType: nats * AgentPort: 6000 * NonActiveTimeout: 8m0s * ResumeTokenCollection: resume_token * OpHACliPath: /usr/local/omk/bin/opha-cli.pl * Compression: true * Role: Poller * Consumer: false * Producer: false * ConsumerPollerSet: (blank) * DebugEnabled: false * Redis: * RedisServer: localhost * RedisPort: 6379 * RedisPassword: ****** * RetryTimeout: 3m0s * RedisStreamLenCheckPeriod: 5 * RedisProducerMaxStreamLength: 10000 * MaxRetries: 180 * RedisTLSEnabled: false * RedisTLSSkipVerify: false * RedisProducerDegradeTimeout: 10 * RedisProducerFullDegradeTimeout: 10 * Kafka: * Seeds: localhost:63616,localhost:63627,localhost:63629 * RetryTimeout: 3m0s * MaxRetries: 180 * Nats: * NatsServer: opha-dev4.opmantek.net * NatsCluster: [] * NatsPort: 4222 * NatsNumReplicas: 1 * NatsUsername: omkadmin * NatsPassword: ****** * RetryTimeout: 3m0s * NatsStreamLenCheckPeriod: 5 * NatsProducerMaxMsgPerSubject: 1000000 * NatsMaxAge: 604800 * MaxRetries: 180 * NatsTLSEnabled: false * NatsTLSCert: <path> * NatsTLSKey: <path> * NatsTLSSkipVerify: false * NatsProducerDegradeTimeout: 10 * NatsProducerFullDegradeTimeout: 10 * Authentication: * AuthTokenKeys: ****** -------------------------------------------------- 2025-10-22T08:01:46.329+1100 [INFO] ophad.verify: verify nmis9 mongodb connection with database: name=nmisng 2025-10-22T08:01:46.451+1100 [INFO] ophad.verify: MongoDB NMIS connect: maybe="found nodes collection in nmis9 ✅" 2025-10-22T08:01:46.451+1100 [INFO] ophad.verify: verify omk mongodb connection with database: name=opha 2025-10-22T08:01:46.551+1100 [INFO] ophad.verify: MongoDB OMK connect: maybe="found opstatus collection in omk database ✅" 2025-10-22T08:01:46.575+1100 [INFO] ophad.verify: Nats connect: result= | can connect to nats-server: opha-dev4.opmantek.net version: 2.11.9 ✅ | we can connect to Nats-server ✅ 2025-10-22T08:01:46.575+1100 [INFO] ophad.verify: ready for liftoff 🚀

    

...