Stop writing to a sealed parent range after a split

[HectorIFC]

Context

Found while auditing the per-key ordering guarantee. It is a narrow but real violation of what the docs promise.

Malachi.Broker.ensure_segment/3 (lib/malachi/broker.ex:651-653) produces straight into the range's active segment when the frontend already holds it in its local cache, without consulting the control plane. The metadata cache refreshes on a timer (@default_brokers_refresh_interval 1_000, lib/malachi/broker_server.ex:50).

So when node B splits a range, node A keeps appending to the now-sealed parent segment for up to one refresh interval. The control-plane fence ({:error, :sealed} from fetch_active_range/2) only blocks opening a new segment on a sealed range; it does not stop appends to a segment that is already open on another node.

That inverts read order. history_sources/2 (lib/malachi/broker.ex:873-876) makes a child read its ancestors first, then itself. A record written to the parent at T+0.5s is therefore read before a record written to the child at T+0.1s. For a client that writes m1, waits for the ack, then writes m2 (the case where ordering is guaranteed), the inversion is observable if m1 landed on the node that already saw the split and m2 on the node that had not.

docs/guides/log-model.md:116-123 promises exactly what this window breaks: "A key hashes to a fixed position, so every record for that key lands in the same range, and the range's single primary appends them in the order it receives them", explicitly contrasted with Kafka's partition-count caveat.

Plan

1. Decide where the fence belongs. Three candidates:
   • Invalidate on refresh: when reconcile_metadata observes that a cached range is no longer active, drop its cached active segment so the next produce re-routes. Cheapest, and it shortens the window rather than closing it.
   • Check on produce when the cache is older than the refresh interval, which trades a metadata read for a bounded staleness.
   • Make the seal effective in the data plane: the segment's primary refuses appends once the segment is sealed, so a stale frontend gets an error and re-routes. This closes the window rather than shrinking it, and matches how the seal already works for readers.
2. Whatever is chosen, a stale produce must fail in a way the client retries transparently, the way :sealed already does, not surface as a lost write.
3. Add the case to the ordering documentation regardless (see the limits section), because even with the fix there is a moment where a produce is rejected and retried.

Risks and open questions

• Closing the window fully (option 3) puts a check on the produce hot path; it must be a local state check, not a control-plane read.
• Splits are currently operator-driven and rare, so the exposure today is small. It grows the moment automatic splitting lands, which makes this a prerequisite for that work rather than an independent nice-to-have.

Verification

• A test that reproduces the inversion: two frontends over one metadata, split through one, keep producing through the other, and assert the read order for a single key.
• The same test passing after the fix.
• Full suite, mix credo --strict, mix dialyzer, plus a loadtest to confirm no measurable cost on the produce path.