PB-16568: harden MongoDB replica set for HA under load

[pallav-px]

Problem

Under cluster load the px-central MongoDB replica set intermittently collapses to ReplicaSetNoPrimary, so px-backup API calls fail with server selection timeout and in-flight backup/restore operations fail (surfacing as flakes in BasicBackupAndRestoreWithParallelBackupSchedule). See PB-16568.

Root cause: the MongoDB liveness probe execs mongosh (Node.js cold-start + connect + auth) with a 5s timeout and asserts replica-set role. Because the pods are BestEffort (resources: {}), under node CPU/IO/network contention the mongosh exec cannot finish within 5s; after 6 failures kubelet kills the container — and when the killed member is the primary, this forces an election and the ReplicaSetNoPrimary window that fails px-backup.

Reproduced and fix-validated on a live cluster

Metric	Old (mongosh exec liveness)	New (tcpSocket)
Probe latency under identical CPU starvation	14,432 ms → fails 5s budget → kill	6 ms → passes
Member killed within 95s of starvation?	killed in ~50s (RESTARTS 0→1)	not killed (RESTARTS=0)
Role during starvation	primary killed → election	kept PRIMARY
px-backup server selection timeout errors	reproduced (8×)	0

Full step-by-step reproduction results are in a comment below.

Changes (chart px-central)

1. Liveness probe → tcpSocket:27017 — process-liveness only, so it cannot false-fail under starvation and can never restart a healthy primary. The isWritablePrimary || secondary role check moves to the readiness probe (which only removes the pod from the Service, never restarts it); readiness timeoutSeconds widened 5 → 15.
2. Container resources (values.mongodbResources) — moves the pods off BestEffort. Memory limit (6Gi) exceeds the 4 GB WiredTiger cache (persistentStorage.mongoCacheSize) so it isn't OOMKilled at the cache ceiling. Stops the starvation and stops the scheduler co-locating heavy workload onto the mongo nodes.
3. terminationGracePeriodSeconds: 120 (values.mongodbTerminationGracePeriodSeconds) — lets a SIGTERM'd mongod checkpoint and step down cleanly instead of being SIGKILLed on grace expiry (which caused WiredTiger crash recovery + oplog rollback).
4. podAntiAffinity default false → true (already values-gated in the chart) + a new PodDisruptionBudget (minAvailable: 2, toggle values.mongodbPodDisruptionBudget) — members never co-locate, and voluntary disruption can never drop below quorum.

All new behavior is values-tunable; set mongodbPodDisruptionBudget: false / podAntiAffinity: false for single-/two-node dev clusters.

Testing

• helm lint and helm template pass; rendered output verified for all four changes.
• Live A/B on a px-backup cluster (chart px-central-3.1.0): the old exec probe killed the primary under CPU starvation and reproduced the exact px-backup ReplicaSetNoPrimary error; after switching liveness to tcpSocket, the primary survived identical starvation with 0 restarts and px-backup logged 0 errors. Cluster restored to as-found state afterward.

Notes / follow-ups (not in this PR)

• The pxc-backup-mongodb-scripts ConfigMap (holding setup.sh) is referenced but not created by this chart — the slow-rejoin fast-path fix belongs wherever that script is generated (px-central operator).
• Optional defense-in-depth: raise px-backup's mongo serverSelectionTimeout and add Torpedo retry on transient server selection timeout.

Acceptance criteria (PB-16568)

• ✅ A liveness failure cannot, by itself, kill the current primary — met structurally by (1).
• ✅ Members not restarted/evicted on transient load — (1) + (2) + (3).
• ✅ Replica set retains a primary under the load profile — (2) + (4).

[pallav-px]

Live-cluster reproduction & fix validation (full results)

Run on a live px-backup cluster (4 nodes, v1.34.0) running the exact unfixed chart (px-central-3.1.0): all 3 mongo pods BestEffort, mongosh liveness probe (5s timeout, ×6, asserts RS role), terminationGracePeriodSeconds: 30, no PDB.

1. The probe has no headroom (baseline, idle)

The exact liveness command, timed in-pod on an idle node:

run 1: 886 ms   run 2: 895 ms   run 3: 935 ms      (budget = 5000 ms)

~0.9s with mongod healthy and unloaded — already ~1/5 of the budget spent on mongosh startup + auth alone.

2. Liveness kill reproduced (root cause)

Pinned a CPU hog to the primary's node (BestEffort mongo → CFS-starved). Re-timed the same probe:

run 1: 18868 ms  [ok]
run 2: 16401 ms  [server is in quiesce mode and will shut down]   ← kubelet already killing it
run 3: 12683 ms  [exit 137]

Kubelet events on the primary:

Warning  Unhealthy  Liveness probe failed: command timed out ... "mongosh --eval '...'" timed out after 5s
Normal   Killing    Container mongodb failed liveness probe, will be restarted

Pod RESTARTS 0 → 1 in-place — the exact signature from the diag bundles. Failover then elected a new primary.

3. px-backup ReplicaSetNoPrimary symptom reproduced

Forced a >30s no-primary window (froze both secondaries + stepped down the primary). px-backup's own reconcilers logged the ticket error verbatim (8× across workers):

failed to retrieve organizations: server selection error: server selection timeout,
current topology: { Type: ReplicaSetNoPrimary, Servers: [
  { Addr: pxc-backup-mongodb-0...:27017, Type: RSSecondary, Average RTT: 1120330 },
  { Addr: pxc-backup-mongodb-2...:27017, Type: RSSecondary, Average RTT: 630727 },
  { Addr: pxc-backup-mongodb-1...:27017, Type: RSSecondary, Average RTT: 1736972 } ] }

4. Fix validated — same hardware, same starvation, only the probe changed

Patched liveness → tcpSocket:27017 (this PR's change #1) and re-ran the identical CPU-hog starvation on the current primary:

	Old (mongosh exec)	New (tcpSocket)
Probe latency under starvation	14,432 ms → fails → kill	6 ms → passes
Member killed in 95s?	killed in ~50s	RESTARTS = 0, no Killing event
Role during starvation	primary killed → election	kept PRIMARY (health=1)
px-backup errors during window	ReplicaSetNoPrimary	0

Under the fix the starved member correctly went ready=false (readiness pulled it from the Service) but was never restarted and retained its PRIMARY role, so no election fired and px-backup stayed connected throughout.

5. Cleanup

Cluster restored to as-found state after validation: liveness reverted to the shipped mongosh exec, repro namespace deleted, replica set healthy 3/3, RESTARTS back to 0.

[px-kesavan]

Why cant we just increase the timeout to 60s or more, to address the livenessprobe issue ?

here we are checking only the port is open or not , which will be true in most of the time. We are losing the main functionality of livenessprobe here . where we fail to restart the pod for actual errors.

Old liveness probe actually checks mongod is up, reachable, authenticating, answering commands.

[pallav-px]

few reasons a bigger timeout alone doesn't fix this, and why the "deep" check isn't actually lost:

1. The bigger problem isn't just the timeout — it's the role assertion. The old probe requires isWritablePrimary || secondary. During any election a healthy member is transiently neither (STARTUP2/RECOVERING/candidate), so it fails the probe regardless of the timeout value. A 60s timeout wouldn't have prevented the kills — during an election storm the probe returns "not primary/secondary" quickly and still trips failureThreshold. A liveness probe should never assert replica-set role, because restarting a member mid-election makes the election worse.

2. You can't size a timeout against node starvation. Measured on a live cluster: the mongosh exec is ~0.9s idle but 14–19s under a single CPU hog, and the ticket's real load (90 MiB/s throttle + mysqlslap co-located) is heavier — it could exceed 60s. Whatever number we pick, enough contention beats it. And a 60s × 6 = 6-minute window to detect a genuinely hung mongod is worse than today for real failures.

3. We don't lose the deep check — it moves to readiness. The readiness probe still runs the full mongosh check (mongod up, reachable, authenticating, answering, correct role). The difference is what happens on failure: readiness pulls the pod from the Service (safe); liveness restarts the container (disruptive). For a stateful DB, a slow/busy member should be taken out of rotation, not killed — killing a live-but-busy PRIMARY is exactly what triggered the election + crash recovery in this bug. Also, a truly dead mongod (crash/OOM/panic) exits the process → the container exits → kubelet restarts it via restartPolicy: Always, no liveness needed.

Where you're right: tcpSocket won't restart a mongod that is deadlocked but still holding the port open. That's the one gap. In practice readiness removes it from the Service and the replica set heartbeats route around it, but liveness wouldn't auto-restart that specific case.

If the team wants liveness to still catch a hung-but-listening mongod, the compromise is a lightweight command liveness — mongosh --eval 'db.adminCommand("ping")' (ping only, no role assertion) with timeoutSeconds: 15 and a higher failureThreshold. That keeps "is mongod answering" as a restart signal while dropping the election-killer and adding headroom. It's still a heavy Node.js exec though, so it's less starvation-proof than tcpSocket — that's the tradeoff. Happy to switch to that if you prefer it over the pure TCP check.

[px-kesavan]

I still have a concern on 3rd point , even the readiness probe will face the same issue isn't it ? I see you have changed the timeout from 5 to 15 , based on your 2nd point , this will still fail , so pods will be 0/1 forever as it might crosses the timeout same as the older livenessprobe

[pallav-px]

readiness runs the same mongosh exec, so under hard enough starvation it can also hit 0/1. But that doesn't reproduce the bug, because the consequence differs:

• Liveness fail → container restart → primary killed → election → ReplicaSetNoPrimary. Outage.
• Readiness fail → 0/1, dropped from Service endpoints only. No restart, no election; mongod stays alive and stays PRIMARY, keeps serving.

And 0/1 doesn't cut off px-backup: the headless Service sets publishNotReadyAddresses: true (NotReady pods stay in DNS), and RS role + client routing are driven by mongod heartbeats/topology, not k8s readiness. Live proof: under identical starvation the member went ready=false but kept PRIMARY and px-backup logged 0 errors — vs. the old probe killing the primary with 8× ReplicaSetNoPrimary.

"0/1 forever" only lasts as long as the starvation — a capacity issue the resources change fixes (off BestEffort) — and even then it's a slow-but-serving DB, not an outage.

Net: worst case now = transiently NotReady but serving; worst case today = killed primary + no-primary window.