You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Browse filesBrowse the repository at this point in the historyBrowse files
SqlRush
committed
feat(cluster): crash-rejoin re-declare barrier (Shape A) + cold-GRD fail-closed
Closes the P0 that a crash-rejoined node with cluster.online_join=off
cold-serves its home blocks from a wiped GRD: a silent stale READ and a
silently-diverging WRITE that, after flush ordering, durably LOSES a
peer's committed rows (diagnosed 2026-07-15, both reproduced minimally on
main including the committed-write loss).
Fix 1 — the off-path counterpart of the online joiner tick. A per-
incarnation shmem flag ClusterGrdShared.offpath_boot_decided (init 0)
makes the phase gate fence self-home blocks RECOVERING from process start
(before any LMON tick), so reads AND writes fail-closed with zero cold-
serve window. cluster_reconfig_offpath_rejoin_tick() then classifies the
incarnation:
- crash-rejoin -> arm the join fence + demote self_join_admitted to 0
(writes 53R60) and LEAVE offpath_boot_decided 0, so the boot barrier
persists as the read fence for the incarnation. The epoch-keyed join
fence is a no-op on a fast rejoin (this node is still at
CLUSTER_EPOCH_INITIAL, so the monotonic-max fence epoch cannot rise);
the epoch-independent boot barrier carries the read fence.
- clean cold-bootstrap / single node -> set the flag, serve normally.
The crash-rejoin signal is the prior-incarnation voting-disk self-slot's
ALIVE bit (守门裁决 07-15: ALIVE, not epoch — a fast rejoin leaves BOTH
sides at INITIAL, so an epoch test is blind; a clean shutdown clears
ALIVE via qvotec_clear_self_alive_on_clean_shutdown while a crash leaves
it set). qvotec's startup ghost-detect latches it into
QvotecShmem.prior_unclean_death (offset 72; the convert-queue lane owns
64..71 — StaticAssert-pinned, this lane rebases after queue merges).
Fix 2 (defense in depth) — cluster_gcs_cold_grd_watermark_verdict(): an
InvalidScn master watermark under an active self-fence FAILs CLOSED
instead of SKIPping (the wiped watermark could mask a stale block),
except for a genuine extension block (never cross-node written). Pure,
header-only, unit-tested truth table; wired into fallback_verify_refresh.
Approved semantic consequence: a full-outage crash co-boot fences every
node (53R60) — no silent auto-formation; the cluster waits for online
admission (spec-5.22 cold-formation follow-up) or a clean restart. A
CLEAN full shutdown clears ALIVE and co-boots normally.
Tests: t/404 (crash-rejoin read/write/boot-race/full-outage/clean-restart
legs, all GREEN), fold unit truth table, t/249 grd_recovery roster
31->32 (offpath_crash_rejoin_fenced dump key).
Gates: unit clean 178/178, t/017+401+402+404 PASS, cluster_regress 13/13,
PG 219/219, format/headers/scn-cmp/ges-mode/clog all 0. (t/249's
cooperative-rebind timeout legs L6/L3/L13 flake under mac load — confirmed
pre-existing on a binary without these changes; the roster ripple passes.)
Known-scoped-out (守门裁决): a CLEAN fast-restart of a cluster member
(without clean-leave) shares the hazard but is out of this fix's scope
(the ALIVE bit is cleared on clean shutdown); clean departures should run
clean-leave flush/handoff.
Spec: spec-5.16 (extended to online_join=off crash-rejoin); Rule 8.A.
0 commit comments