RFC: shared schema gate and the write critical section (#643)

[ragnorc]

What this proposes

A draft RFC (docs/rfcs/2026-09-18-shared-schema-gate.md) resolving RFC 0067's open question #643 — the second step of its throughput path:

• The process-local schema gate (("__schema_apply__", None)) becomes shared/exclusive: contract-lifecycle passes (schema apply, system-column upgrade, open/refresh/settle/reload/sync) take it exclusive; ordinary writers, merges, maintenance, branch control and read-view captures take a shared permit. Classification rule, auditable in one sentence: readers of the accepted view share; publishers of the accepted view exclude.
• The branch gate stays for same-branch writers; per-table gates keep stopping two same-table stagers from wasting one.
• Promotion moves after guard release (already gate-free-correct) as an independently revertible sub-decision.
• Nothing cross-process changes: durable sentinel + mono-branch refusal + manifest CAS remain the authority; the gate stays an in-process contention structure per the deny-list.

Why now

The concurrent-writes instrument (this stack's benchmark) measured the exclusive gate's cost, sequentially, on both engines: 8 writers deliver less than 1 — −9% on local FS, −68% at 30 ms object-store RTT (sidecar-era main: −76%) — with service-time p95 exploding from ~76 ms to seconds and manifest reads/op doubling from gate-serialized revalidation. Reads also capture catalogs behind writers' publish windows today (the architecture guide's contrary claim is corrected by this RFC either way). And step 3 (group commit) is starved without this: under the current gates writers never arrive at the publisher concurrently, so every batch would have size one.

The exclusive gate's founding rationale — a mutation advancing a Lance HEAD before discovering a schema lock — was abolished by RFC 0067's detached staging; a misclassified site can now cost only a stale publish the CAS refuses.

Evidence plan (acceptance bar for the implementation PR)

Before/after concurrent-writes at --writers 8 --write-branches {1,8}, local + RustFS(+30 ms); success = w8×B8 scales while w8×B1 stays branch-gate-bound. Existing pins mapped (branch-gate cells survive; the mid-apply blocking pin re-derives as reader-behind-writer; write_cost counts byte-identical). Commissions the missing DST concurrent-universe arm racing writers against a schema apply. Doc moves: writes.md gate order, architecture.md read-path sentence.

Stacking

Rebased onto main after #744 merged (the RFC commit replayed alone; the registry row and the RFC 0067 cross-link re-resolved against main). Docs only: the RFC, its registry row, and the 0067 unresolved-question pointer. The implementation PR follows on acceptance.