Promote the most advanced replica, and fence the old primary

[HectorIFC]

Context

Priority: high. This is a durability gap, found while auditing what ordering guarantee the system provides.

When an active segment's primary dies, Malachi.Cluster.Failover.promote/2 (lib/malachi/cluster/failover.ex:42-51) picks a new one like this:

case Enum.filter(segment.replica_set, &MapSet.member?(live, &1)) do
  [] -> []
  [new_primary | _] -> ...
end

It takes the first live replica in replica-set order. It does not compare how far each replica's log has advanced, and there is no epoch, term, or log truncation anywhere in the data plane (no match for epoch, fence or truncate in replication_server.ex, failover.ex or catchup.ex).

Why that matters with rf=3 and a quorum of 2: a batch is acknowledged once the primary and one follower have it durably. The other follower may not. If the promoted replica is that other follower, its log ends below the offsets the old primary already assigned and acknowledged. The new primary then appends new records at those same offsets (do_replicate_durable appends at its own log.next_offset), so two replicas end up holding different records under the same offsets.

Nothing detects that divergence today:

• the expected_first chain in follow/4 rejects a gap, not a conflict: a follower whose end is already past expected_first treats the batch as one it has and acks it;
• the integrity scrub verifies each copy against itself (checksums, and counts against the sealed metadata), so two internally valid but mutually divergent copies both pass.

What has been verified, and what has not. The absence of epoch, log comparison and fencing is verified in the code. No data loss has been observed in a run: the chaos certifications (scripts/docker-chaos-test.sh, scripts/docker-storage-chaos.sh) have passed repeatedly, including power-pull events. Treat this as a design gap that the harness has not yet hit, not as a reproduced incident.

Plan

1. Reproduce it first. A test that pins the scenario is worth more than the fix: three replication servers, ack a batch on a quorum that excludes one follower, kill the primary, force promotion of the replica that lacks the batch, produce again, and assert what happens to the acknowledged record. If it survives, this issue is wrong and should be closed with that evidence.
2. Then choose the fix. Three options, in increasing order of cost:
   • Promote the most advanced replica. Failover.plan/2 would compare the live replicas' durable ends before choosing. ReplicationServer.durable_end/4 already exists and already recovers from disk. Cheapest, and it removes the common case, but it is a heuristic: a replica can advance between the probe and the promotion.
   • Epoch per segment. Record a monotonically increasing epoch in the segment's metadata on every promotion, carry it in the replication messages, and have a replica reject an append from an older epoch. This is the standard fix (it is what Raft's term does) and it closes the race the heuristic leaves open.
   • Fence through the control plane before promoting, so the old primary cannot commit anything after the decision.
3. Whichever is chosen, the promotion path must stay a pure plan plus applied commands, as it is today, so it remains testable without processes.
4. Extend the chaos harness with the interleaving that provokes it (kill the primary right after an ack whose quorum excluded one follower), so the invariant is certified and not just unit tested.

Risks and open questions

• The heuristic and the epoch differ in guarantee, not just in cost. The PR must say plainly which one it buys.
• Promotion happens on the healing tick, so there is already a window between the primary dying and the promotion. Adding a probe per replica lengthens it; the trade against a stuck range needs a number, not a feeling.
• If divergence has already happened in a deployment, no fix repairs it retroactively. Detecting it (comparing replicas of a sealed segment byte for byte, which the storage-chaos drill already does across nodes) could become a separate follow-up.

Verification

• The reproduction test above, first failing, then passing.
• Existing failover and healing tests unchanged.
• The chaos drill extended, passing end to end.
• Full suite, mix credo --strict, mix dialyzer, single-node and 3-node loadtests with 0 errors.