Skip to content

fix: e2e CI flake — goal-0017 emit fanout raced canvas-live-sync - #33

Merged
alicoding merged 1 commit into
mainfrom
fix/e2e-flake-hardening
Aug 12, 2026
Merged

fix: e2e CI flake — goal-0017 emit fanout raced canvas-live-sync#33
alicoding merged 1 commit into
mainfrom
fix/e2e-flake-hardening

Conversation

@alicoding

Copy link
Copy Markdown
Owner

Summary

  • Root cause found for the recurring shard-1 e2e flake (docs/goals/BACKLOG.md Standing build(deps): Bump golang.org/x/oauth2 from 0.35.0 to 0.36.0 in the go-dependencies group #1): goal 0017's per-service dataevent.Emit fanout made a single MCP update_workflow write fire mill-data-changed twice (SnapshotDraft + UpdateWorkflow), plus a third self-echo from the test's own earlier CreateWorkflow — three near-simultaneous events, each independently re-fetching, whichever resolved last won unconditionally regardless of dispatch order. A stale response could beat an already-applied newer one and wrongly show the external-change banner on a clean canvas.
  • Fixed in frontend/src/composition/useCanvasLiveSync.ts with a monotonic request-sequence guard (standard out-of-order-async-response pattern): drop a fetch response once a newer event has arrived since it was dispatched.
  • canvas-live-sync.spec.ts cleanup hardened into an outer try/finally (both tests) so a future failure can't leave an undeleted workflow / unattended-MCP-writes settings for later tests in the same shard worker.
  • resizable-table.spec.ts's one occurrence (PR feat: inline hotkey hints (goal 0015 remainder) + real Close-tabs bindings #24) hardened with a condition-based expect.poll wait for the resize handle's bounding box, additive to the suite's existing retries: 1.
  • docs/goals/BACKLOG.md Standing build(deps): Bump golang.org/x/oauth2 from 0.35.0 to 0.36.0 in the go-dependencies group #1 checked off with the full root-cause writeup; docs/SPEC.md's realtime-lock section gets an Update note generalizing the lesson for future mill-data-changed consumers.

Evidence

  • Real CI history: canvas-live-sync.spec.ts:151 failed 6/6 times sampled across the last ~30 ci.yml runs, always shard 1, always after goal 0017 (PR goal 0017: real-time surfaces audit — direct-UI/service mutations now emit mill-data-changed #16) merged, zero occurrences before. Every occurrence co-occurred with a configure-lists.spec.ts flake in the same run (recovered on Playwright's own retry every time — a resource-contention side effect of the extra retry overhead, not a separate state-cascade bug).
  • Local reproduction: 9/20 "clean canvas" repeats failed with zero artificial load, identical assertion/line to all CI failures — confirmed via a temporary event-trace instrument (removed before commit) showing exactly 3 near-simultaneous mill-data-changed arrivals per run.
  • Post-fix: 88 consecutive clean local repeats (0 failures), same build that showed 9/20 before the fix.

Test plan

  • Full local suite green: go vet, golangci-lint, go test ./... -race -cover, go build (desktop + server), npm run boundaries, npm run lint, npm run test (vitest), tsc --noEmit
  • Full local e2e suite green (179 tests, 176 passed + 3 pre-existing/unrelated local-multi-worker flakes recovered on retry — none in the target specs)
  • canvas-live-sync.spec.ts + resizable-table.spec.ts run --repeat-each=5, 0 failures
  • Lefthook pre-commit hooks passed on the real commit

🤖 Generated with Claude Code

https://claude.ai/code/session_018pkViCNAuZp2vBv2K9AbUh

…nding #1)

Root cause, confirmed via real CI history (6/6 shard-1 failures, all
after goal 0017 merged, zero before) and local reproduction (9/20
clean-canvas repeats failed with zero artificial load): goal 0017 gave
every direct-mutation service its own dataevent.Emit call, so a single
MCP update_workflow write now fires mill-data-changed TWICE
(SnapshotDraft + UpdateWorkflow) plus a third echo from the test's own
prior UI-driven CreateWorkflow. None carry payload content, so each
handler independently refetches — three fetches racing meant whichever
resolved last won unconditionally, letting a stale response beat an
already-applied newer one and wrongly show the external-change banner
on a clean canvas.

Fixed in useCanvasLiveSync.ts with a monotonic request-sequence guard
(the standard out-of-order-async-response fix): drop a fetch response
once a newer mill-data-changed event has arrived since it was
dispatched. Verified: 88 consecutive clean local repeats post-fix vs.
9/20 before it, same build.

Also: canvas-live-sync.spec.ts's cleanup hardened into an outer
try/finally (both tests) so a future assertion failure can't leave an
undeleted workflow / unattended-MCP-writes settings for later tests in
the same worker — defense-in-depth for the observed cascade,
independent of the root-cause fix. resizable-table.spec.ts's one
occurrence (PR #24, drag-handle bounding box) hardened with a
condition-based expect.poll wait at the point of use, additive to the
suite's existing retries: 1 (goal 0024 precedent, untouched).

BACKLOG.md Standing #1 checked off with the full root-cause writeup;
SPEC.md's realtime-lock section gets an Update note generalizing the
lesson for future mill-data-changed consumers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018pkViCNAuZp2vBv2K9AbUh
@alicoding
alicoding enabled auto-merge (squash) August 12, 2026 21:28
@alicoding
alicoding merged commit 02ecae9 into main Aug 12, 2026
16 checks passed
alicoding added a commit that referenced this pull request Aug 15, 2026
…nding #1) (#33)

Root cause, confirmed via real CI history (6/6 shard-1 failures, all
after goal 0017 merged, zero before) and local reproduction (9/20
clean-canvas repeats failed with zero artificial load): goal 0017 gave
every direct-mutation service its own dataevent.Emit call, so a single
MCP update_workflow write now fires mill-data-changed TWICE
(SnapshotDraft + UpdateWorkflow) plus a third echo from the test's own
prior UI-driven CreateWorkflow. None carry payload content, so each
handler independently refetches — three fetches racing meant whichever
resolved last won unconditionally, letting a stale response beat an
already-applied newer one and wrongly show the external-change banner
on a clean canvas.

Fixed in useCanvasLiveSync.ts with a monotonic request-sequence guard
(the standard out-of-order-async-response fix): drop a fetch response
once a newer mill-data-changed event has arrived since it was
dispatched. Verified: 88 consecutive clean local repeats post-fix vs.
9/20 before it, same build.

Also: canvas-live-sync.spec.ts's cleanup hardened into an outer
try/finally (both tests) so a future assertion failure can't leave an
undeleted workflow / unattended-MCP-writes settings for later tests in
the same worker — defense-in-depth for the observed cascade,
independent of the root-cause fix. resizable-table.spec.ts's one
occurrence (PR #24, drag-handle bounding box) hardened with a
condition-based expect.poll wait at the point of use, additive to the
suite's existing retries: 1 (goal 0024 precedent, untouched).

BACKLOG.md Standing #1 checked off with the full root-cause writeup;
SPEC.md's realtime-lock section gets an Update note generalizing the
lesson for future mill-data-changed consumers.


Claude-Session: https://claude.ai/code/session_018pkViCNAuZp2vBv2K9AbUh

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant