fix(ci): investigate recurring E2E Shard 3/4 failure on dep PRs #242

Activity

1. julianken

   added

   bugSomething isn't workingSomething isn't working

   on May 4, 2026
2. Repository owner deleted a comment from julianken-bot on May 4, 2026
3. julianken-bot commented on May 4, 2026

   

   julianken-bot

   on May 4, 2026

   Collaborator

   More actions

   Reviewer: julianken-bot (opus, fresh context)
   Verdict: APPROVED.

   Verification ledger

   • Title 60 chars (≤70).
   • Body owns the investigation; no source file changes; produces a child PR or quarantine commit per DONE WHEN.
   • Pre-req chore(deps): group ai-sdk + gcp-actions in dependabot #235 verified OPEN (chore(deps): group ai-sdk + gcp-actions in dependabot); ordering is correct — regroup first, then debug a regrouped PR.
   • DONE WHEN gives both exits: passing regrouped PR OR documented + quarantined spec with tracking ref in tests/e2e/README.md.
   • Open Dependabot PR scan confirms the pattern: build(deps): bump actions/checkout from 4 to 6 #124, build(deps): bump google-github-actions/auth from 2 to 3 #125, build(deps): bump google-github-actions/setup-gcloud from 2 to 3 #127, build(deps): bump actions/setup-node from 4 to 6 #128 fail E2E Shard 3/4; build(deps): bump google-github-actions/deploy-cloudrun from 2 to 3 #126 fails E2E Shard 4/4. So the failure is on shards 3 and 4, not just 3/4.

   Findings

   SUGGESTION — title/body scope is narrower than the data
   Four GH-Actions PRs fail Shard 3/4, but #126 (google-github-actions/deploy-cloudrun 2→3) fails Shard 4/4. The body's WHY says "four of five fail the same E2E Shard 3/4," which is accurate, but the investigation should explicitly include #126's Shard 4/4 in step 3, otherwise the diagnosis may stop at "Shard 3/4-specific" when the real cause is "tests run during CI on dep PRs touching workflow files." Minor edit to step 3: "Reproduce shard 3/4 and 4/4" — or generalize to "the failing shard."

   SUGGESTION — make the spec-identification step less hand-wavy
   Step 4 "likely in tests/e2e/" is a guess. Tighten to: "Pull the failed shard's Playwright HTML report from the workflow run artifacts (actions/upload-artifact step) to identify the failing spec deterministically." This removes a manual hunt and works even if the failing test isn't under tests/e2e/.

   Bottom line

   Issue is well-scoped, prereq-correctly-ordered, and has a clean two-branch DONE WHEN. The Shard 3/4-vs-4/4 framing and the "likely in tests/e2e/" hand-wave are polish, not blockers. Approve as-is; the suggestions can be folded into the first investigation comment when work begins.

   Verdict: APPROVED.
4. julianken commented on May 5, 2026

   

   julianken

   on May 5, 2026

   OwnerAuthor

   More actions

   Investigation complete — closed PRs were the source, no active flake

   Runs analyzed: 500 most recent (back to 2026-04-27) + targeted date-range queries for April 27–May 1.

   ---

   Root cause: two distinct bugs, both now fixed on main

   Pattern 1 — mermaid.spec.ts strict-mode violation (Shard 3/4)

   All five original GH-Actions dependabot PRs (#124–#128, opened 2026-04-27) failed E2E Shard 3/4 with this error:

   strict mode violation: locator('svg[id^="mermaid-"][role*="graphics-document"]') resolved to 2 elements
   

   MermaidDiagram.tsx pre-renders the SVG into both the in-page .mermaid-figure and the lightbox <dialog>, both carrying the same id. The bare locator matched both, and Playwright's strict mode rejected the expect() call.

   Fixed by PR #135 (fix/mermaid-e2e-strict, merged 2026-04-27T23:34:04Z) — scopes selector to .mermaid-figure svg[id^="mermaid-"]....

   Pattern 2 — theme-aware-hero.spec.ts (Shard 4/4)

   A separate test failure appeared on April 28 runs (theme-aware-hero.spec.ts:60,87) affecting branches that ran CI after the hero work landed but before #148.

   Fixed by PR #148 (fix(ThemeAwareHero): propagate display utilities to inner img, merged 2026-04-29T00:29:59Z).

   ---

   Post-#235 state: all passing

   The six new grouped dependabot PRs opened after #235 closed (2026-05-05) all pass E2E Shard 3/4:

   • dependabot/npm_and_yarn/payload-b1c7eb99fe → success
   • dependabot/npm_and_yarn/ai-sdk-c4bdf36872 → success (Shard 3/4: 15 passed)
   • dependabot/npm_and_yarn/tailwind-b04d16b6de → success
   • dependabot/npm_and_yarn/testing-021df8d6f7 → success
   • dependabot/github_actions/actions-core-ac27bb405c → success
   • dependabot/github_actions/gcp-actions-07446c8b8e → success

   ---

   Conclusion

   The Shard 3/4 failures were not Dependabot-specific. They were pre-existing spec bugs that happened to land on main in the same window as the original five Dependabot PRs. Both underlying bugs were fixed on main before #235 merged, so no code change is needed here. No quarantine or test.fixme() required.
5. julianken

   closed this as completedon May 5, 2026