feat(ui): add a project Flake Lab tab and a verified flake fix to the demo - #691
Merged
Merged
Conversation
… from the docs The demo held a single Flake Lab experiment, on one test, and nothing pointed to it. The UI project's pagination test now carries a whole lab cycle: its three flaky runs get their failed first attempts, with the rows request slow and the count request slow beside it, so the flake profile ranks two suspects. A reproduce experiment reproduces it with the rows delay and rules the count out, a first verify still fails, and a second holds after the test's last retry-pass. The test reads verified fixed, the Flaky view lists it apart, and its quarantine is proposed for release. The checkout test's failed-attempt split moves into a shared helper; its rows are unchanged. A seed consistency test and the demo runtime check keep the verified fix holding, and the Flake Lab page links both demo tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QFSKHRYcgKdcMDqfaGmW2M
The lab had no page of its own: its experiments lived on each flaky test's Flakiness tab, so nothing showed which flaky tests had been through it and what each still needed. The project page gets a Flake Lab tab, with a snowflake icon, that lists every test on the flaky ranking or with an experiment: where it stands (not tested, not reproduced, reproduced, still failing after a fix, verified fixed, flaky again), the condition that reproduced it and the piwi flake or piwi flake verify command it needs next, then the project's newest experiments. Tiles count the flaky, untested, awaiting-a-fix and verified tests. The tab follows the flake-lab capability. GET /api/projects/:id/flake-lab serves it from a shared handler the demo router reuses. The experiment row and its wording move out of the Flakiness tab so both lists render experiments the same way, and one helper builds the piwi flake commands for the tab, the list and MCP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QFSKHRYcgKdcMDqfaGmW2M
A design record for linking each feature page to a concrete screen in the live demo: one registry of examples the generated seed is checked against, a docs component that renders a page's examples, drift and runtime checks, and the screenshot scenes and run-app skill reading the same registry instead of hardcoded ids. It includes the coverage of today's feature pages, a rollout in four pull requests and the decisions left open. ROADMAP.md lists it under Exploring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QFSKHRYcgKdcMDqfaGmW2M
Contributor
Coverage Report for Reporter (./packages/reporter)
File CoverageNo changed files found. |
Contributor
Coverage Report for Application (./apps/application)
File Coverage
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
# Conflicts: # apps/application/public/demo/seed.version.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The demo had one Flake Lab experiment, on one test, and nothing pointed to it; the lab also had no page of its own, so nothing showed which flaky tests had been through it and what each still needed.
A Flake Lab tab on the project page (snowflake icon, next to Failures), following the
flake-labcapability:piwi flakeorpiwi flake verify) — a fix to verify first, a fix that held last;It reads
GET /api/projects/:id/flake-lab, served by a shared handler (getProjectFlakeLab) that the demo router reuses. The experiment row and its wording move out of the Flakiness tab intoFlakeExperimentRowso both lists render experiments the same way, and oneflakeCommandhelper builds the commands for the two tabs and theplan_flake_experimentMCP tool.A second demo example. The UI Components project's pagination test (test case 35) now runs the whole lab cycle: its three flaky runs get their failed first attempts with a slow rows request (and a count request slow beside it), so the flake profile ranks two suspects; a reproduce experiment reproduces it with the rows delay and rules the count out, a first verify still fails, and a second holds after the test's last retry-pass. It reads verified fixed, the Flaky view lists it apart, and its quarantine is proposed for release. The checkout test's failed-attempt split moves into a shared seed helper; its rows are byte-identical.
Docs. The Flake Lab page gets a "Try it in the demo" section and the tab under "Where results show"; the UI overview lists the tab; the catalog's
wherenames it.A proposal for docs-to-demo links (
proposals/demo-examples.md, listed under Exploring inROADMAP.md): one registry of demo examples the generated seed is checked against, a<DemoExamples>docs component, drift and runtime checks, and the screenshot scenes andrun-appskill reading the registry instead of ~80 hardcoded ids. It includes the coverage of today's feature pages, a rollout in four PRs and the decisions left open.How was it tested?
npm run app:test:unit: 4,718 passed. Newtests/unit/project-flake-lab.test.tscovers the lab states, the next step and the handler (ordering, counts, verified and flaky-again tests, the limit, an unknown project);demo-seed-consistency.test.tsgains checks that every seeded experiment has finished by load time and that the verified fix holds (a retry-pass before it, none after).CI=1, as CI runs it):flake-lab,flake-lab-runs,flake-suspects,flake-verified,desktop-flake-lab,capabilities-opt-out,keyboard-nav,scenario-gaps-decline— 36 passed.flake-lab.spec.tsgains a test for the new tab.npm run app:generate:demo && npm run app:check:demo:runtime: passes, with new checks that the pagination test reads verified fixed and that the project's Flake Lab tab lists it and its three experiments.npm run app:check:democovers the new route.npm run app:typecheck, oxlint and oxfmt clean; the docs drift and word-budget tests pass (the Flake Lab page is at 1,497 of 1,500 words).flake-lab-tabandflake-lab-tab-mobile, checked at 1280 px and 390 px (no horizontal scroll).Checklist
type(scope): subject)apps/docs/, README, or reporter README)🤖 Generated with Claude Code
https://claude.ai/code/session_01QFSKHRYcgKdcMDqfaGmW2M
Generated by Claude Code