WIP:Fix/mongo init container stuck

[Plesoun]

MR — mongo-init.js: preserve-live priority in mergeLiveMemberWithDesired + full PSA test suite

Cluster under test: lmio-jaguar (dev/test PSA). Nodes:

• lmio-jaguar 10.17.170.21 — mongo-1
• lmio-gepard 10.17.174.110 — mongo-2
• lmio-ibis 10.17.169.63 — mongo-3 (arbiter, ZK role [arbiter])

Orchestrator: ASAB Maestro / asab-remote-control v26.33-alpha3. All changes applied the correct way: model.yaml → ZooKeeper → up (remote-control API). No manual file ops on nodes.

---

1. What this MR contains

The only logic change is a revert-to-correct of a previous over-correction in
mergeLiveMemberWithDesired (mongo-init.js):

// BEFORE (regression — force-reset on every reconfig)
if (dm.priority !== undefined && dm.priority !== null) {
    out.priority = dm.priority
} else {
    out.priority = isArbiter ? 0 : 1
}

// AFTER (preserve-live — matches votes semantics & newMemberDocFromDesired)
if (dm.priority !== undefined && dm.priority !== null) {
    out.priority = dm.priority
} else if (out.priority === undefined || out.priority === null) {
    out.priority = isArbiter ? 0 : 1
}

Also dropped the redundant standalone if (dm.priority !== undefined) out.priority = dm.priority
line (L523) — the block above already handles dm.priority.

Why preserve-live is correct: out = Object.assign({}, cm) starts as a copy of the live
member. Votes, tags, secondaryDelaySecs all preserve live values and only default when absent.
Priority must behave the same. The earlier else { out.priority = isArbiter ? 0 : 1 } clobbered a
legitimate live priority (e.g. priority: 2) to 1 on every phase-5 rs.reconfig whenever
replica-set.json omits the field — a silent election-preference regression. New members still get
the arbiter ? 0 : 1 default via newMemberDocFromDesired (unchanged).

SHA: script bd5b355878d6…, 24606 B, 748 logical lines. ZK copy verified logically
identical to the reviewed sandbox file (comments only stripped).

---

2. Environment / Durable deployment

• Script served from ZK: /library/Site/ASAB Maestro/Files/mongo/script/mongo-init.js
• Propagated to disk on all 3 nodes by the orchestrator on the next up (ZK is the source of
  truth; disk overwrite on reconcile).
• Verification after P0 propagate: on-disk sha bd5b355878d6… on all 3 nodes = ZK. ✅

---

3. Test suite— execution & results

Pass criteria after every test:

1. RS healthy — rs.status(): all remaining members health=1, exactly one PRIMARY, expected member count.
2. No stopped mongo-*-init — docker ps -a across all nodes shows zero mongo-*-init in Exited (a transiently-Up init during reconcile is fine; it must exit 0 and be auto-deleted by the governator's ExitCode==0 gate).

#	Test	Model action	Result
P0	Propagate corrected script — up all 3 with unchanged model	ZK script → disk all nodes	✅ bd5b3558 everywhere, RS healthy, no lingering
T1	Remove a SECONDARY (gepard) via model	drop instance 2 → up all 3	✅ RS converged to jaguar P + ibis ARB, gepard gone, no lingering
T2	Re-add a node (gepard) via model	restore instance 2 → up all 3	✅ full PSA healthy, gepard prio=1 (normalized as fresh add), no lingering
T3	Concurrent-init race — non-impactful change to ALL members + simultaneous up	probe noop-4 → noop-5 on all 3 + simultaneous up	✅ race settled, RS healthy, noop-5 visible in compose, zero lingering inits
T4	Remove the current PRIMARY (jaguar) on an RS-with-arbiter	drop instance 1 → up all 3	✅ gepard cleanly elected PRIMARY (jaguar+ibis quorum), jaguar container gone, gepard prio=1 / ibis arb prio=0 preserved, no lingering
T4b	Re-add the removed PRIMARY (jaguar) via model	restore instance 1 → up all 3	❌ DETERMINISTIC FAILURE — see §4 (blind spot)

T4b failure — captured live

Sherpa output from the PRIMARY-side (gepard) run, captured during the re-add:

[boot] all mongods reachable after 7/60 attempts
Connection attempt 1/20
Initialization of replicaset failed.
Recoverable failure; exiting 0 so the sherpa is cleaned up (retried on next reconcile).

Resulting RS: 2 members (gepard P + ibis ARB) — jaguar not re-added. Zero lingering inits
(all exited 0 as designed), but the intended member never joined.

Summary of priority/failover verification (the point of this MR)

Everything the preserve-live change touches passed:

• Fresh re-adds normalize to prio 1 (T2).
• Non-arbiter keeps prio 1 through a concurrent race (T3) and after being elected PRIMARY (T4).
• Arbiter stays prio 0 (correct by Mongo design).
• Existing live priority is now preserved (not clobbered to 1).

---

4. Current blind spot (not covered by the sherpa — the reason T4b fails)

The sherpa cannot autonomously re-add a removed data member that sits at position 1 in
MONGO_HOSTNAMES and whose local mongod has no valid replica-set config.

Mechanism

Every node's sherpa env: MONGO_HOSTNAMES= mongo-1,mongo-2,mongo-3 (iterated in order).

1. jaguar = mongo-1 = position 1.
2. After being removed as PRIMARY (T4), jaguar's local mongod reports
   hello.info === "Does not have a valid replica set config" (its local system.replset holds
   the stale post-removal 2-member config; it is "not a member").
3. The sherpa connects to mongo-1 first, the branch at L686
   (!isWritablePrimary && !secondary && isreplicaset && info === "Does not have a valid replica set config")
   fires, it calls initiateReplicaSet() → rs.initiate(desired) → fails (an RS already
   exists on gepard) → quit(0) at L695.
4. The sherpa exits before ever iterating to mongo-2 (gepard, the live PRIMARY) to run
   rs.add.

So the sherpa never reaches the node that could actually add jaguar.

Why it's positional

• T1/T2 (removed/readded gepard, mongo-2): on re-add, mongo-1 (jaguar) was still a
  healthy PRIMARY, so the sherpa connected to jaguar first → found a live RS → correctly
  rs.add'd gepard. ✅
• T4b (removed/readded jaguar, mongo-1): the stale non-member is first, so the initiate
  short-circuit pre-empts the add path every time. ❌

Independent of this MR

This is not caused by the priority change. It is a pre-existing sherpa design gap in the
initiate-vs-add decision: the initiate branch should only fire when no reachable member is
already a valid primary. It reproduces deterministically (re-ran it twice in this session; identical
failure).

Secondary layer of the same blind spot (why plain rs.add isn't enough even if reached)

Even if the sherpa did reach gepard and call its normal addMemberFromDesired → rs.add, adding
an electable secondary to a set with only one writable data node (gepard) + arbiter is
rejected by MongoDB:

Rejecting reconfig where the new config has a PSA topology and the secondary is electable,
but the old config contains only one writable node.

The correct path requires MongoDB's rs.reconfigForPSASet (two-step: grant votes 1 / priority 0,
sync, then priority 1). The sherpa's flow-3 PSA branch (usePsa → rs.reconfigForPSASet) exists
in code, but is unreachable here because the initiate branch pre-empts it — leaving the removal
of a PRIMARY from a PSA set effectively one-way under pure-orchestrator control.

---

5. Manual workaround (used twice this session, verified)

To bring a removed PRIMARY back into a PSA set when the orchestrator can't:

// step 1 — add as NON-electable (votes 0, priority 0) — passes the PSA guard
rs.add({ _id: 1, host: "lmio-jaguar:27017", priority: 0, votes: 0 })
// wait for SECONDARY + initial sync (health=1, caught up)

// step 2 — promote electable via MongoDB PSA transition method
var c = rs.conf(); c.version++
var j = c.members.findIndex(m => m.host === "lmio-jaguar:27017")
c.members[j].priority = 1; c.members[j].votes = 1
rs.reconfigForPSASet(j, c)

Then verify 3 healthy members, priorities correct, zero lingering inits.

---

6. Recommended fixes (for follow-up, not part of this MR)

1. Sherpa initiate gate (root fix): only run the initiate branch when no reachable member
   is already a valid primary / a live replica set exists. This fixes the pre-emption regardless of
   hostname ordering.
2. PRIMARY-removal from PSA: either (a) short-term ops exception — orchestrated
   rs.stepDown() + handover before removal, or (b) teach the decommission path to use the
   PSA-safe reconfig so removal is not one-way.
3. Ensure hostname ordering can't mask this: consider pinning the added member's position in
   MONGO_HOSTNAMES, or make the loop resilient to a stale first host.

---