Give Claude or Codex a repo. Get an independently verified Playwright evaluation.
Run this in your agent after the one-time setup below:
/canary-lab /absolute/path/to/your-app "checkout flow"
The /canary-lab skill takes the repo from first scan to finished evaluation. Your agent understands the code, gathers requirements, writes tests, and repairs application failures. Canary Lab starts the services, runs Playwright, stores the evidence, and owns the verdict.
Canary Lab requires Node.js 22.12 or newer and npm 9 or newer.
npx canary-lab init my-labinit creates a Canary Lab workspace, installs its dependencies and Chromium, and registers the skills plus one compact MCP connection (exec) for supported Claude and Codex clients. Restart your agent once so it discovers them.
The connection starts the local Canary Lab service when the skill needs it. To open the interface yourself:
cd my-lab
npx canary-lab uiIf /canary-lab or the Canary Lab exec tool is missing, run npx canary-lab setup --force from the workspace and restart your agent.
/canary-lab starts one resumable Flight:
scan repo → create suite → collect requirements → author and map tests → prepare isolated ports → run and repair → export evaluation
The skill keeps the reasoning work in your current Claude or Codex session. Canary Lab performs the mechanical checks and calculates every stage result from saved evidence.
- Requirements stay reviewable. Add a product requirements document, link a local file, gather relevant repo documents, or infer requirements from the branch diff.
- Coverage is explicit. Tests map to requirements and paths instead of producing a guessed percentage.
- Repairs stay isolated. Each run receives its own ports and Git worktree, so repairs do not alter your working copy.
- Progress survives interruption. Run
/canary-labagain with the same repo to continue the existing Flight. - The evaluation is the deliverable. The final archive contains the real verdict, test evidence, browser media, and per-test reasoning.
An agent can start a server and run Playwright itself. The gap is trust: the same actor that writes a fix should not be able to declare that fix correct.
| The agent can | The agent cannot |
|---|---|
| Read logs, traces, screenshots, and videos | Run Canary Lab's tests itself |
| Fix the application, or correct a test only when it is provably wrong | Declare a run green |
Signal rerun or restart |
Change Canary Lab's saved evidence |
Canary Lab adds:
- Results the agent does not own. The harness runs the tests and holds the pass/fail result.
- Concurrency without conflicts. Runs receive isolated ports and Git worktrees; conflicting work waits in a queue.
- Safe environment switching. Environment files are backed up before changes and restored when the run ends.
Use /canary-lab for the complete journey. Use a focused skill when you need only one part:
| Goal | Skill |
|---|---|
| Take a bare repo through a complete evaluation | /canary-lab |
| Run an existing suite and repair application failures | /canary-lab-run |
| Verify a running or deployed environment | /canary-lab-verify |
| Create a suite and its Playwright tests | /canary-lab-author |
| Build the requirement-to-test coverage ledger | /canary-lab-coverage |
| Prepare a suite for concurrent runs | /canary-lab-portify |
| Export a completed run as an evaluation | /canary-lab-export |
The skills share the same workspace, runs, evidence, and UI. Work started in one surface appears in the others.
The agent skill is the normal interactive path. The same Flight can also start from the Flights view or from a terminal:
npx canary-lab flight /absolute/path/to/your-app "checkout flow"Use the CLI for shell automation or when you want Canary Lab to conduct the Flight outside an existing agent conversation. Running the command again resumes existing work instead of creating a duplicate.
The UI and MCP server share one configurable port, 7421 by default. Choose another during setup with npx canary-lab init my-lab --port 8200, or change it later in Project Settings. The ui --port option is not supported.
Custom MCP clients should connect to http://localhost:<port>/mcp?profile=compact.
Canary Lab suites use normal Playwright tests plus a service configuration that names your existing development commands. Canary Lab assigns free ports, starts each service, waits for health checks, and tags the output so failures map back to the correct test.
See Suite Folders for the file structure and examples, or Guide for the complete run and repair workflow.
New workspaces include demonstrations for a prepared repair loop, a bare repo Flight, and focused coverage and authoring workflows. Delete the demo files after exploring them.
Use Docker Compose for infrastructure such as Postgres or Redis, and let Canary Lab start application services with their normal development commands. This keeps hot reload available during repair.
| Plain Playwright | Docker Compose with watch | Hosted dashboard | Canary Lab | |
|---|---|---|---|---|
| Runs existing development commands with hot reload | One service | Needs a development image and watch rules | Varies | Yes |
| Boots several services together | You script it | Yes | Varies | Yes |
| Runs concurrently on one machine | Manual | Not built in | Hosted | Isolated ports and worktrees |
| Keeps evidence outside the repair agent's control | No | No | Yes | Yes, locally |
| Switches environment files with backup and restore | Manual | Manual | No | Yes |
| Keeps harness data on your machine | Yes | Yes | No | Yes |
Use Canary Lab when failures depend on service startup, environment files, backend logs, requirement coverage, or evidence the agent should not control. Plain npx playwright test is enough when you do not need orchestration or independent repair evidence. Harness data stays local; repair agents may still need network access.
With Node.js 22.12 or newer:
npm install --save-dev canary-lab@latest
npx canary-lab upgradeRestart Canary Lab and connected agent apps afterwards so they load the refreshed skills and connection path. Existing suite folders under features/ and personal agent files are preserved.
- Node.js 22.12 or newer and npm 9 or newer.
- Chromium for Playwright, installed automatically by
canary-lab initunless installation is skipped. - A local UI server, started automatically through the agent connection or manually with
npx canary-lab ui. - Optional repair agents: supported Claude or Codex command-line clients on
PATH.
node-pty gives each service a real terminal. Its prebuilt binaries require no normal installation-time compilation. A postinstall step restores the execute bit on its helper binary when needed; it does nothing on Windows or when node-pty is absent.
- Repairs are only as useful as your service logs.
- Environment runs overwrite target files while active. If a process is killed during backup or restore, reopen the UI and use the environment controls to recover.
- Environment values are not validated, so stale configuration can surface as unclear failures.
- Linux and Windows workflows are not polished yet.
| Document | What it covers |
|---|---|
| Changelog | Release history. |
| Guide | Environment switching, run output, repairs, and evaluation reports. |
| Commands | Full CLI and trigger-surface reference. |
| Suite Folders | Suite structure, configuration, and Playwright tests. |
| Architecture | Module map, run lifecycle, concurrency, healing, and MCP. |
| Product Requirements | Product intent, non-goals, and quality bars. |
| Design System | Web UI tokens, primitives, and layout patterns. |
| Contributing | Code orientation and development workflow. |