Skip to content

feat(ui): add a project Flake Lab tab and a verified flake fix to the demo - #691

Merged
PhenX merged 5 commits into
mainfrom
ccr-292ba8c1-e4600b
Sep 29, 2026
Merged

PhenX merged 5 commits into
mainfrom
ccr-292ba8c1-e4600b

Conversation

@PhenX

@PhenX PhenX commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

What & why

The demo had one Flake Lab experiment, on one test, and nothing pointed to it; the lab also had no page of its own, so nothing showed which flaky tests had been through it and what each still needed.

A Flake Lab tab on the project page (snowflake icon, next to Failures), following the flake-lab capability:

  • tiles counting the flaky tests, the ones never run through the lab, the ones reproduced with no fix verified yet, and the verified fixes;
  • every test on the flaky ranking or with an experiment: where it stands (not tested, not reproduced, reproduced, still fails after a fix, verified fixed, flaky again), the condition that reproduced it, and a copy button for the command it needs next (piwi flake or piwi flake verify) — a fix to verify first, a fix that held last;
  • the project's newest experiments across its tests, each linking to the test's Flakiness tab.

It reads GET /api/projects/:id/flake-lab, served by a shared handler (getProjectFlakeLab) that the demo router reuses. The experiment row and its wording move out of the Flakiness tab into FlakeExperimentRow so both lists render experiments the same way, and one flakeCommand helper builds the commands for the two tabs and the plan_flake_experiment MCP tool.

A second demo example. The UI Components project's pagination test (test case 35) now runs the whole lab cycle: its three flaky runs get their failed first attempts with a slow rows request (and a count request slow beside it), so the flake profile ranks two suspects; a reproduce experiment reproduces it with the rows delay and rules the count out, a first verify still fails, and a second holds after the test's last retry-pass. It reads verified fixed, the Flaky view lists it apart, and its quarantine is proposed for release. The checkout test's failed-attempt split moves into a shared seed helper; its rows are byte-identical.

Docs. The Flake Lab page gets a "Try it in the demo" section and the tab under "Where results show"; the UI overview lists the tab; the catalog's where names it.

A proposal for docs-to-demo links (proposals/demo-examples.md, listed under Exploring in ROADMAP.md): one registry of demo examples the generated seed is checked against, a <DemoExamples> docs component, drift and runtime checks, and the screenshot scenes and run-app skill reading the registry instead of ~80 hardcoded ids. It includes the coverage of today's feature pages, a rollout in four PRs and the decisions left open.

How was it tested?

  • npm run app:test:unit: 4,718 passed. New tests/unit/project-flake-lab.test.ts covers the lab states, the next step and the handler (ordering, counts, verified and flaky-again tests, the limit, an unknown project); demo-seed-consistency.test.ts gains checks that every seeded experiment has finished by load time and that the verified fix holds (a retry-pass before it, none after).
  • E2E against a production build (CI=1, as CI runs it): flake-lab, flake-lab-runs, flake-suspects, flake-verified, desktop-flake-lab, capabilities-opt-out, keyboard-nav, scenario-gaps-decline — 36 passed. flake-lab.spec.ts gains a test for the new tab.
  • npm run app:generate:demo && npm run app:check:demo:runtime: passes, with new checks that the pagination test reads verified fixed and that the project's Flake Lab tab lists it and its three experiments. npm run app:check:demo covers the new route.
  • npm run app:typecheck, oxlint and oxfmt clean; the docs drift and word-budget tests pass (the Flake Lab page is at 1,497 of 1,500 words).
  • New screenshot scenes flake-lab-tab and flake-lab-tab-mobile, checked at 1280 px and 390 px (no horizontal scroll).

Checklist

  • PR title follows Conventional Commits (type(scope): subject)
  • Tests added/updated for behavior changes
  • Docs updated if user-facing (apps/docs/, README, or reporter README)

🤖 Generated with Claude Code

https://claude.ai/code/session_01QFSKHRYcgKdcMDqfaGmW2M


Generated by Claude Code

… from the docs

The demo held a single Flake Lab experiment, on one test, and nothing
pointed to it. The UI project's pagination test now carries a whole
lab cycle: its three flaky runs get their failed first attempts, with
the rows request slow and the count request slow beside it, so the
flake profile ranks two suspects. A reproduce experiment reproduces it
with the rows delay and rules the count out, a first verify still
fails, and a second holds after the test's last retry-pass. The test
reads verified fixed, the Flaky view lists it apart, and its
quarantine is proposed for release.

The checkout test's failed-attempt split moves into a shared helper;
its rows are unchanged. A seed consistency test and the demo runtime
check keep the verified fix holding, and the Flake Lab page links both
demo tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QFSKHRYcgKdcMDqfaGmW2M
The lab had no page of its own: its experiments lived on each flaky
test's Flakiness tab, so nothing showed which flaky tests had been
through it and what each still needed. The project page gets a Flake
Lab tab, with a snowflake icon, that lists every test on the flaky
ranking or with an experiment: where it stands (not tested, not
reproduced, reproduced, still failing after a fix, verified fixed,
flaky again), the condition that reproduced it and the piwi flake or
piwi flake verify command it needs next, then the project's newest
experiments. Tiles count the flaky, untested, awaiting-a-fix and
verified tests. The tab follows the flake-lab capability.

GET /api/projects/:id/flake-lab serves it from a shared handler the
demo router reuses. The experiment row and its wording move out of the
Flakiness tab so both lists render experiments the same way, and one
helper builds the piwi flake commands for the tab, the list and MCP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QFSKHRYcgKdcMDqfaGmW2M
A design record for linking each feature page to a concrete screen in
the live demo: one registry of examples the generated seed is checked
against, a docs component that renders a page's examples, drift and
runtime checks, and the screenshot scenes and run-app skill reading
the same registry instead of hardcoded ids. It includes the coverage
of today's feature pages, a rollout in four pull requests and the
decisions left open. ROADMAP.md lists it under Exploring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QFSKHRYcgKdcMDqfaGmW2M
@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Report for Reporter (./packages/reporter)

Status Category Percentage Covered / Total
🔵 Lines 76.47% 3911 / 5114
🔵 Statements 74.91% 4480 / 5980
🔵 Functions 79.66% 752 / 944
🔵 Branches 68.9% 3215 / 4666
File CoverageNo changed files found.
Generated in workflow #2272 for commit ca7951a by the Vitest Coverage Report Action

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Report for Application (./apps/application)

Status Category Percentage Covered / Total
🔵 Lines 63.19% 18031 / 28534
🔵 Statements 61.55% 21062 / 34214
🔵 Functions 63.4% 4073 / 6424
🔵 Branches 53.86% 15769 / 29273
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
apps/application/app/utils/flake-lab.ts 0% 0% 0% 0% 3-50
apps/application/app/utils/help-content.ts 100% 100% 100% 100%
apps/application/server/utils/mcp/tools.ts 12.52% 6.34% 12.98% 13.56% 165, 172-203, 215, 228, 234, 236, 268-338, 344-1632, 1641-1642, 1644-2304, 2365-2406, 2414, 2418, 2432-2444, 2457-2490, 2498, 2504, 2510, 2511, 2522, 2541-2632, 2647-2687, 2767-2771, 2778-2792, 2805, 2809, 2834-2919
apps/application/shared/flake-lab.ts 96.66% 93.02% 100% 100% 134
apps/application/shared/piwi-features.ts 100% 100% 100% 100%
apps/application/shared/handlers/flake-lab.ts 90.64% 80.59% 88.73% 94.78% 162, 183-185, 194, 214-215, 341, 344, 348, 396, 633, 643, 660, 668, 727, 751, 814, 829, 873, 893, 903-912
Generated in workflow #2272 for commit ca7951a by the Vitest Coverage Report Action

# Conflicts:
#	apps/application/public/demo/seed.version.json
@PhenX
PhenX merged commit a04c33e into main Sep 29, 2026
29 checks passed
@PhenX
PhenX deleted the ccr-292ba8c1-e4600b branch September 29, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants