diff --git a/apps/docs/.vitepress/config.mts b/apps/docs/.vitepress/config.mts index 37487d4f..63ca7d79 100644 --- a/apps/docs/.vitepress/config.mts +++ b/apps/docs/.vitepress/config.mts @@ -151,6 +151,7 @@ export default defineConfig({ { text: 'Apps & integrations', items: [ + { text: 'Piwi CLI', link: '/cli' }, { text: 'Desktop app', link: '/desktop' }, { text: 'Browser extension', link: '/extension' }, { text: 'Open in IDE', link: '/ide-integration' }, diff --git a/apps/docs/cli.md b/apps/docs/cli.md new file mode 100644 index 00000000..879b6333 --- /dev/null +++ b/apps/docs/cli.md @@ -0,0 +1,166 @@ +--- +title: Piwi CLI +lang: en-US +--- + +# Piwi CLI + +The `@piwitests/reporter` package ships a command-line tool, `piwi`, for the things that happen *around* a test run: wiring a project up, gating a CI job on the dashboard's analysis, running a saved test selection, and managing AI-step artifacts and agent skills. This page is the reference for every command and flag; each command's own `--help` prints the same list. + +## Invoking it + +Run it through the package name so `npx` resolves this package: + +```bash +npx @piwitests/reporter [options] +``` + +> The package is `@piwitests/reporter`; its command is `piwi`. A plain `npx piwi …` would fetch an unrelated `piwi` from npm — invoke it through the package name. Once the reporter is a project dependency, `npx piwi ` resolves the local binary and works too. + +| Command | What it does | +|---|---| +| [`init`](#init) | Wire a Playwright project up to a Piwi Dashboard | +| [`skills`](#skills) | Install the Piwi agent skills into this project | +| [`gate`](#gate) | Fail a CI job on the dashboard's analysis of a run | +| [`select`](#select-run) | Print the Playwright args for a saved test selection | +| [`run`](#select-run) | Run a saved test selection with `playwright test` | +| [`ai`](#ai) | Manage committed natural-language AI-step artifacts | + +Several commands read connection settings from the environment as a fallback: `PIWI_DASHBOARD_URL` (dashboard URL), `PIWI_API_KEY` (API key), and `PIWI_PROJECT_NAME` (project). A flag always wins over its environment variable. + +## `init` + +Wire a Playwright project up to a dashboard. It installs the reporter as a dev dependency, wraps `export default defineConfig(...)` with [`wrapConfig(...)`](./reporter#installing-via-wrapconfig), creates the [capture-fixtures](./capture-fixtures) file, records `PIWI_*` connection settings in `.env` / `.env.example` (and `.gitignore`), and installs the [agent skills](./mcp#agent-skills). **Every step is idempotent** — safe to re-run — and a config shape it will not rewrite is reported as `manual` with the exact change to make. See the [one-command setup](./getting-started#fast-path-one-command) in Getting started. + +```bash +npx @piwitests/reporter init --server-url http://localhost:3000 --project my-project +``` + +| Flag | Description | +|---|---| +| `--server-url ` | Dashboard URL to write into the config (env `PIWI_DASHBOARD_URL`, default `http://localhost:3000`) | +| `--project ` | Project name to report under (default: your package/folder name) | +| `--api-key ` | API key to write into `.env` (env `PIWI_API_KEY`). Omit to leave a `.env.example` placeholder you fill in yourself | +| `--cwd ` | Project root to operate on (default: current directory) | +| `--skills ` | Comma-separated skills to install, or `all` / `none` (default: the four workflow skills) | +| `--skills-dir ` | Directory to install skills into (default: `.claude/skills`) | +| `--skills-only` | Only install skills; do not touch the config or env | +| `--no-skills` | Configure the project but install no skills | +| `--no-install` | Do not run the package manager; record the dependency only | +| `--force` | Overwrite skill files that already exist | +| `--dry-run` | Report every change without writing anything | +| `--json` | Print the plan/result as JSON (for agents) | +| `-h`, `--help` | Show help | + +## `skills` + +Install the Piwi [agent skills](./mcp#agent-skills) into a project — agent-agnostic Markdown that lets a coding agent investigate failures, heal locators, and stabilize flaky tests. `init` installs these for you; use `skills` to add them to a project that already has the reporter, or to a different skills directory. + +```bash +npx @piwitests/reporter skills list +npx @piwitests/reporter skills add [names...] [options] +``` + +The five skills are `setup-piwi`, `investigate-failure`, `apply-locator-healing`, `stabilize-flaky-tests` and `run-the-right-tests`. `add` with no names installs all of them. + +| Flag (for `add`) | Description | +|---|---| +| `--dir ` | Directory to install into (default: `.claude/skills`) | +| `--cwd ` | Project root to operate on (default: current directory) | +| `--force` | Overwrite a skill file that already exists | +| `--dry-run` | Report what would be written without writing | +| `--json` | Print the results as JSON | + +## `gate` + +Fail a CI job on the dashboard's analysis of a run — the [merge gate](./ci#blocking-a-merge). Point it at a run, give it at least one policy rule, and it exits non-zero when the rule is violated. The run defaults to `./piwi-run.json` (the reporter's [output file](./ci#getting-the-run-url-back-out-of-ci)) when present. + +```bash +npx @piwitests/reporter gate --max-new-regressions 0 --fail-on-flaky +``` + +**Exit codes:** `0` satisfied · `1` violated · `2` could not evaluate. + +| Flag | Description | +|---|---| +| `--run-id ` | Run id to evaluate | +| `--from-file ` | Read `runId` from the reporter's output JSON | +| `--server-url ` | Dashboard URL (env `PIWI_DASHBOARD_URL`) | +| `--api-key ` | API key (env `PIWI_API_KEY`) | +| `--require-tag ` | Comma-separated; every test carrying the tag must pass | +| `--max-failed ` | Fail when more than *n* tests failed | +| `--max-new-regressions ` | Fail when more than *n* tests newly started failing | +| `--max-new-flaky ` | Fail when more than *n* tests newly became flaky | +| `--max-quarantined ` | Fail when more than *n* tests are quarantined | +| `--fail-on-new-cluster` | Fail when this run introduced a new failure cluster | +| `--fail-on-flaky` | Fail when this run contains any flaky test | +| `--require-selection ` | Fail when a test the named selection matches did not run or failed | +| `--json` | Print the raw result as JSON instead of a summary | +| `-h`, `--help` | Show help | + +The run source is resolved first-match-wins: `--run-id`, then `--from-file`, then `PIWI_OUTPUT_FILE`, then `./piwi-run.json`. At least one policy rule is required. + +## `select` / `run` {#select-run} + +Resolve a saved [test selection](./test-selection) to the tests it matches. `select` prints the Playwright args (so you can compose them yourself); `run` executes `playwright test` with them. `run impact --base ` runs the tests your working-tree diff impacts. + +```bash +npx @piwitests/reporter select smoke +npx @piwitests/reporter run smoke -- --workers=4 +npx @piwitests/reporter run impact --base origin/main +``` + +**Exit codes:** `0` ok · `1` the test run failed (`run` only) · `2` could not resolve. + +| Flag | Description | +|---|---| +| `--server-url ` | Dashboard URL (env `PIWI_DASHBOARD_URL`) | +| `--api-key ` | API key (env `PIWI_API_KEY`) | +| `--project ` | Project (env `PIWI_PROJECT_NAME`) | +| `--format ` | `args` (`file:line`, default) · `grep` · `files` · `json` | +| `--budget ` | Cap total time, e.g. `5m`, `90s`, `300000` (ms) | +| `--shard ` | Keep only shard *i* of *n*, balanced by test duration | +| `--fail-fast` | Order the least-reliable tests first | +| `--base ` | For `impact`: the ref to diff the working tree against | +| `--strict` | Fail (exit 2) instead of falling back when unreachable | +| `--pkg-runner ` | Package runner for the printed command (default `npx`) | +| `--json` | Print the full resolution as JSON (`select` only) | +| `-h`, `--help` | Show help | + +Pass extra Playwright arguments after `--`: `piwi run smoke -- --headed --workers=1`. + +## `ai` + +Manage committed natural-language [AI-step](./ai-steps) artifacts (`page.piwiLocator(...)` / `page.piwiRun(...)`). The LLM authors each entry once; CI replays the committed JSON deterministically with no model calls, so these commands are how you keep the committed set healthy. + +```bash +npx @piwitests/reporter ai check +npx @piwitests/reporter ai resolve --grep "checkout" +npx @piwitests/reporter ai prune +``` + +| Subcommand | What it does | +|---|---| +| `check` | Scan committed entries for orphans, non-canonical files and duplicate templates. Read-only; exits `1` when issues are found | +| `resolve` | Author missing entries by running the suite in resolve mode against the configured authoring server (forces `--workers=1`) | +| `prune` | Delete orphaned/dormant entries | + +| Flag | Applies to | Description | +|---|---|---| +| `--dir ` | `check` | Entry directory name per spec (env `PIWI_AI_DIR`, default `__piwi__`) | +| `--cwd ` | `check` | Root to scan (default: current directory) | +| `--json` | `check` | Emit findings as JSON | +| `--grep ` | `resolve` | Only author entries for matching tests | +| `--project ` | `resolve` | Author under one Playwright project (a resolve profile) | +| `--env K=V` | `resolve` | Extra env for the run (repeatable — flags/viewport profiles) | +| `--update-ai` | `resolve` | Re-author entries that already exist (needs `PIWI_DASHBOARD_URL` / `PIWI_API_KEY`) | + +**Exit codes:** `0` clean (or `--help`) · `1` hygiene issues found · `2` bad arguments / command unavailable. + +## Related + +- [Getting started](./getting-started) — `init` in the setup flow +- [CI & sharding](./ci) — `gate` in a CI job, and the run output file +- [Test selections](./test-selection) — what `select` / `run` resolve +- [AI steps](./ai-steps) — the authoring/replay lifecycle `ai` manages +- [MCP server → Agent skills](./mcp#agent-skills) — what `skills` installs