Skip to content

fix(events): surface the SUBSCRIBE error and refuse callers instead of admitting a dead stream (BUG-2764) - #1215

Merged
xarmian merged 10 commits into
mainfrom
fix/BUG-2764-subscribe-error-visible
Aug 27, 2026
Merged

fix(events): surface the SUBSCRIBE error and refuse callers instead of admitting a dead stream (BUG-2764)#1215
xarmian merged 10 commits into
mainfrom
fix/BUG-2764-subscribe-error-visible

Conversation

@xarmian

@xarmian xarmian commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Fixes BUG-2764. Files BUG-2799 and BUG-2800 for the two adjacent gaps review found that are different mechanisms.

The defect

establishSubscription opened the workspace subscription with client.Subscribe(ctx, channel). In go-redis v9.22.0 that call discards the SUBSCRIBE write's error (redis.go: _ = pubsub.Subscribe(ctx, channels...)), so a SUBSCRIBE the client could not deliver — refused dial, write failure, a caller's context ending mid-dial — came back as a PubSub indistinguishable from a healthy one. It was installed, its acknowledgement never arrived, and its callers were admitted as an unconfirmed subscription promised a reconcile that could never come: a stream subscribed to nothing, for the life of the process on heartbeat phase 1, and re-cycled every idle interval on phase 2.

The fix

  • Issue the SUBSCRIBE where its error is visible (client.Subscribe(ctx) then pubsub.Subscribe(ctx, channel)), at all three sites: internal/events (the filed one) and both internal/watchevents sites, where the failure used to surface only as the 5s confirmation wait timing out with an error that named the wait.
  • A third abandon reason in establishSubscription's deciding section, retiring the record exactly as the other two do.
  • SubscribeFailed, a new outcome: a caller whose loop ends with no live subscription and nobody establishing — after the loop's one built-in retry — is refused, not handed a channel wired to nothing. The post-loop check reads the pending record before the live entry (section 1's order, for the same reason), waits on an in-flight record and re-evaluates, and refuses on a closed bus.
  • SSE handler maps it to 503 + Retry-After: 5 + subscription_failed; admission slot released.
  • Uncovered-workspace retry: the idle pass now also re-establishes any workspace with subscribers, no entry and no record — the state a failed replacement leaves — so phase-2 recovery is not weaker than before (pre-fix the dead PubSub got cycled again; post-fix nothing was installed to cycle). Not counted as a cycle; quiet on shutdown.
  • watchevents constructor closes a failed PubSub and leaves the slot nil so cycleIfIdle's retry gate (b.pubsub == nil) can open on phase 2; resubscribe returns the write error immediately.
  • Prose sweep (CONVE-23): the idle-cycle "two reasons to install nothing" comment/log, the idle_timeout metric Help, and three docs/deployment.md clauses now name the third reason; the BUG-2738 narrative is rewritten as fixed and scoped to the activity stream.

Evidence (all on the head SHA unless stated)

  • go build ./..., go vet ./..., make lint 0 issues; go test ./... 28 ok / 0 FAIL / 4 no-test (233aef2; the final commit changes one log string); go test -race ok on internal/events, internal/watchevents, internal/metrics.
  • Tests: a Dialer whose net.Conn fails the SUBSCRIBE write (a proxy close races the kernel buffer). Events: all-fail → refused with nothing left behind; first-fail → the retry serves establisher and a mid-attempt joiner, each once; all-fail with a joiner → both refused; after Close → refused; failed replacement → next pass re-establishes and the original channel receives. Watchevents: slot left nil, resubscribe returns the injected error itself. Server: 503 on both subscribe branches, through the router.
  • Mutation matrix, 11 mutants: 10 detected via real assertions (drop the error term; drop the post-loop check; drop the handler case; drop the uncovered dispatch; closed bus falls through; keep the dead watch PubSub; drop the resubscribe check; drop the constructor branch — the last two re-run after the timing bounds were removed, still caught by state). One survived honestly: retiring the record outside the deciding section on the failure path — a joiner that catches the stale record wakes on done and re-decides (the loop's documented defence in depth), so there is no observable strand; the same-section retire is kept as the file's stated invariant.
  • Codex: 7 rounds. R1–R5 each found real defects, all fixed or filed; R6 (on-call walkthrough) and R7 (cold full-diff read) returned log-wording nits only, both fixed.

Recorded, not fixed

  • BUG-2799 — a SUBSCRIBE Redis rejects at the command level (-NOPERM etc.) arrives as the first reply, which go-redis's channel loop swallows; different mechanism, lands on the ack-consumption design BUG-2739/2747 built.
  • BUG-2800 — the watch bus has no failure outcome, so a watch stream on an instance with no subscription is still admitted and deaf (pre-existing; now visible as pubsub == nil, retryable on phase 2).
  • Close does not wait for maintenance-pass workers — pre-existing for cycle workers; the uncovered workers inherit exactly that posture (record retired, PubSub closed, eventual).
  • Under a persistently refusing Redis the uncovered pass costs one failed SUBSCRIBE + one log line per uncovered workspace per pass — the rate the cycle path already had.

xarmian added 10 commits August 27, 2026 02:34
…or shutdown, replacement refusals do not count as cycles (BUG-2764 codex round 4)
… docs scope the refusal to the activity stream (BUG-2764 codex round 5, BUG-2800)
@xarmian
xarmian marked this pull request as ready for review August 27, 2026 04:13
@xarmian
xarmian merged commit 1933041 into main Aug 27, 2026
7 checks passed
@xarmian
xarmian deleted the fix/BUG-2764-subscribe-error-visible branch August 27, 2026 04:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant