fix(cluster): GCS block-ship race trio — dedup swallow on retryable deny, invalidate-slot instant fail, duplicate-reply overwrite - #43
Merged
Conversation
…eny, invalidate-slot instant fail, duplicate-reply overwrite
The 4-node S3 bench surfaced three functional defects in the GCS block
plane, all reproducible at low concurrency:
1. A retryable master denial (DENIED_PENDING_X reader guard, the e2
S->X nudge deny, and the direct-land forward denies) replied after
dedup registration but never released the IN_FLIGHT entry. The
requester's convergence retry reuses the same (request_id, epoch)
dedup key, so the retry was silently swallowed as
IN_FLIGHT_DUPLICATE until the TTL sweep — each swallowed round burned
a full cluster.gcs_reply_timeout_ms (observed: the bench's dominant
53R90 retransmit-exhaustion class and its ~26s/52s latency walls).
Fix: cluster_gcs_block_dedup_remove before every such deny (6 sites).
2. The node-wide invalidate broadcast slot failed a busy claim
INSTANTLY inside gcs_block_broadcast_invalidate_and_wait_ext, so two
concurrent local S->X upgrades on one node — even on unrelated
blocks — surfaced the loser as a spurious "S->X upgrade invalidate
did not complete" ERROR. Every caller of the blocking variant is a
backend (dispatch paths use the nowait fan-out), so the claim now
waits bounded by the ACK budget on the broadcast CV; the slot
release broadcasts the CV to wake claim waiters.
3. Reply delivery overwrote a received-but-unconsumed outstanding slot
on duplicate replies (dedup CACHED_REPLY resend racing the original)
while the requester consumed the 8KB image lock-free -> torn CRC32C
-> false DENIED_CHECKSUM_FAIL ("wire-ABI drift" on loopback). Fix:
first-reply-wins — every delivery path (wire handler, direct-land
completion, direct fail-slot) drops the duplicate once
reply_received is set (stale_reply_drop_count++), making the slot
reply fields immutable from reply_received=true until the owner
rearms them.
TDD: new t/390 (3-node) reproduces all three pre-fix — 5.15s
swallowed-retry walls on the organic and injected PENDING_X legs, the
exact "invalidate did not complete" collision error, and the silent
duplicate overwrite — and goes green with the fixes. New injection
point cluster-gcs-block-duplicate-grant-reply (registry 169 -> 170,
baselines refreshed); dedup unit U10 locks the remove-reopens-entry
contract.
Spec: spec-2.34-gcs-block-reliability-hardening.md
Spec: spec-6.12-crossnode-cache-fusion-perf-optimization.md
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three functional defects in the GCS block plane surfaced by the 4-node S3 bench, each reproducible at low concurrency and each mapping to a distinct S3 error signature:
cluster.gcs_reply_timeout_ms(the bench's dominant 53R90 retransmit-exhaustion class and its ~26s/52s latency walls). Fixed withcluster_gcs_block_dedup_removebefore every such deny (6 sites).S->X upgrade invalidate did not completeERROR. The blocking variant's callers are all backends, so the claim now waits bounded by the ACK budget on the broadcast CV; the release broadcasts the CV.reply_receivedis set (first-reply-wins).Test plan
Spec: spec-2.34-gcs-block-reliability-hardening.md
Spec: spec-6.12-crossnode-cache-fusion-perf-optimization.md