Context
Malachi.Cluster.SelfHealing.repair_lost_copies/4 (lib/malachi/cluster/self_healing.ex:123-132) takes every sealed segment in the cluster metadata on every pass, and probe_and_repair/3 (:135) then issues one ReplicationServer.stored_bytes/3 call per replica of each of them (:165-169).
That is a cross-node GenServer.call per segment per replica, driven by Malachi.Cluster.HealCoordinator every 5 seconds (lib/malachi/cluster/heal_coordinator.ex:37). A cluster holding 10k sealed segments at RF=3 issues 30k cross-node calls per pass, and each one waits up to @probe_timeout of 2s (self_healing.ex:42) when a replica is slow.
The work has no bound at all, which is the part worth fixing. The comparable worker already solved this: Malachi.Cluster.Scrubber walks the same population, but a fixed :segments_per_tick at a time, refilling from the metadata when the cycle empties, so a node holding 10k sealed segments revisits each one about weekly instead of scanning all of them every minute. The healing pass has no equivalent, so its cost grows with the size of the cluster while its period stays fixed.
Two consequences, in order of how much they hurt:
- The pass runs inside the coordinator's own loop, so a pass that takes longer than the 5s interval simply keeps the coordinator busy, and membership healing (the part that actually re-replicates a lost replica, which NorthGuard puts on a clock) queues behind an integrity probe of segments nobody has touched in weeks.
- The probes are pure overhead in the steady state. Every sealed segment answers with the same byte count it answered with five seconds ago.
Plan
- Give the integrity half of the pass the same cycle treatment the scrubber has: a
:segments_per_tick bound with a pending list refilled from the metadata when it empties, so coverage is a period rather than a per-pass scan. Reuse the shape rather than reinventing it, and consider extracting it if the two copies would drift.
- Decide the default period deliberately. It should not be tighter than the scrub's, since the scrub is the mechanism that actually catches damage at rest; the byte probe is a cheaper, weaker check that mostly catches a truncated copy after a crash.
- Keep the membership half (under-replication) unbounded and on every pass. That one is genuinely urgent and is bounded by the number of dead brokers, not by the number of segments.
- Emit what the pass skipped, so a bounded pass never reads as full coverage in the telemetry.
Risks and open questions
- Splitting the two halves onto different cadences makes the coordinator's result map less uniform; the
on_result contract and HealCoordinator's logging both need to keep making sense.
- A slower integrity cycle means a truncated copy after a crash is detected later. Worth measuring against what the scrub already covers: if the scrub catches the same shape, this probe may deserve a much longer period than its current 5s, or may be redundant enough to fold into the scrub entirely. That question should be answered before the knob is added, not after.
Verification
- A test that a pass with N segments and a per-tick bound of K probes exactly K, and that the following pass continues from where it stopped rather than restarting.
- A test that the under-replication half is still evaluated on every pass.
- Full suite,
mix credo --strict, mix dialyzer, mix docs --warnings-as-errors, single-node and 3-node loadtests with zero errors.
Found during the CodeRabbit review of PR #44.
Context
Malachi.Cluster.SelfHealing.repair_lost_copies/4(lib/malachi/cluster/self_healing.ex:123-132) takes every sealed segment in the cluster metadata on every pass, andprobe_and_repair/3(:135) then issues oneReplicationServer.stored_bytes/3call per replica of each of them (:165-169).That is a cross-node
GenServer.callper segment per replica, driven byMalachi.Cluster.HealCoordinatorevery 5 seconds (lib/malachi/cluster/heal_coordinator.ex:37). A cluster holding 10k sealed segments at RF=3 issues 30k cross-node calls per pass, and each one waits up to@probe_timeoutof 2s (self_healing.ex:42) when a replica is slow.The work has no bound at all, which is the part worth fixing. The comparable worker already solved this:
Malachi.Cluster.Scrubberwalks the same population, but a fixed:segments_per_tickat a time, refilling from the metadata when the cycle empties, so a node holding 10k sealed segments revisits each one about weekly instead of scanning all of them every minute. The healing pass has no equivalent, so its cost grows with the size of the cluster while its period stays fixed.Two consequences, in order of how much they hurt:
Plan
:segments_per_tickbound with apendinglist refilled from the metadata when it empties, so coverage is a period rather than a per-pass scan. Reuse the shape rather than reinventing it, and consider extracting it if the two copies would drift.Risks and open questions
on_resultcontract andHealCoordinator's logging both need to keep making sense.Verification
mix credo --strict,mix dialyzer,mix docs --warnings-as-errors, single-node and 3-node loadtests with zero errors.Found during the CodeRabbit review of PR #44.