Skip to content

fix: stabilize hosted destroyer scenarios - #1

Merged
smiggleworth merged 3 commits into
mainfrom
fix/destroyer-ci-stability
Sep 3, 2026
Merged

fix: stabilize hosted destroyer scenarios#1
smiggleworth merged 3 commits into
mainfrom
fix/destroyer-ci-stability

Conversation

@smiggleworth

@smiggleworth smiggleworth commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • restore the five family-actor scenarios omitted from the hosted matrix and forward their required CPU/failpoint controls
  • start storage-fault writers before cutting their storage path and keep tmpfs-backed Sqrzl mounted during Fitz restart
  • keep same-type recovery requests sequential on one client connection, as required by the wire contract
  • reconcile ambiguous Stream pressure outcomes on an isolated replaceable connection and retain strict failure handling when reconciliation fails
  • pace sustained pressure according to domain semantics, bound broker sampling overhead, and prevent wall-clock sampling overruns from inventing progress windows
  • retry only typed Queue 4005 not-accepted responses with bounded exponential backoff while retaining exact reconciliation for unknown durable outcomes
  • allow active clients a bounded three-request observation window after correlated actor failure
  • update the harness to @cntryl/fitz 0.0.22

Why

The original full run at https://github.com/cntryl/fitz-destroyer/actions/runs/33666942180 exposed seven scenario failures and five matrix omissions. The failures came from harness ordering, invalid same-type request concurrency, tmpfs loss during storage restart, Stream sessions stranded after ambiguous BEGIN responses, unbounded diagnostic work during sustained pressure, a hosted active-fault observation window that was shorter than the in-flight request lifecycle, and broker liveness gaps corrected by cntryl/fitz#231, cntryl/fitz#232, and cntryl/fitz#233.

Validation

  • npm run check: TypeScript compilation and 183/183 tests passed
  • branch validation: https://github.com/cntryl/fitz-destroyer/actions/runs/33775542364 passed 60/60 jobs, including all 58 scenarios and aggregate analysis
  • post-merge main validation: https://github.com/cntryl/fitz-destroyer/actions/runs/33777076674 passed 60/60 jobs at b8f6cc4172f5ffb6cd412bf8456fafc5dc816885
  • actor-supervision evidence: all 7 isolated and 7 correlated domain faults, all 4 active clients, 8 readiness withdrawals, 8 recoveries, and every recovery canary
  • Queue overload evidence: all 64 operation identities reconciled for each of 4 producers; the exact local scenario also passed 3 consecutive runs against the corrected broker
  • post-merge harness image: ghcr.io/cntryl/fitz-destroyer:b8f6cc4172f5ffb6cd412bf8456fafc5dc816885, digest sha256:51e3b32c2ddf0454f489c0f5db13f8c9d0046157cc49cce47ed34b70b70799f5
  • broker image used by the final run: ghcr.io/cntryl/fitz:latest and :main, digest sha256:1a864bf434378f7de36c44e0d9ceca60e091bb2c58a6b3a924417a9a42337bb7

AI assistance

Codex assisted with hosted-artifact analysis, implementation, regression tests, local Docker validation, and end-to-end CI verification. I reviewed the complete diff and retained strict correctness assertions rather than suppressing scenario failures.

@smiggleworth
smiggleworth force-pushed the fix/destroyer-ci-stability branch from a7e1706 to 1424d75 Compare September 2, 2026 19:52
@smiggleworth
smiggleworth force-pushed the fix/destroyer-ci-stability branch from 1424d75 to eefddfc Compare September 2, 2026 20:09
@smiggleworth
smiggleworth merged commit b8f6cc4 into main Sep 3, 2026
60 checks passed
@smiggleworth
smiggleworth deleted the fix/destroyer-ci-stability branch September 3, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant