From 87ac299e146543e578b72ee95a45b763952bc2d3 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Mon, 6 Jul 2026 16:49:54 +0200 Subject: [PATCH 01/11] docs: add design spec for AI-friendly output and error diagnostics Co-Authored-By: Claude Fable 5 --- .../2026-07-06-ai-friendly-output-design.md | 142 ++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md diff --git a/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md b/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md new file mode 100644 index 0000000..ff2532e --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md @@ -0,0 +1,142 @@ +# AI-Friendly Output & Error Diagnostics for twd-cli + +**Date:** 2026-07-06 +**Status:** Approved for implementation + +## Motivation + +twd-cli's output is consumed primarily by CI logs and AI agents running TDD loops +(run tests → read failures → fix → re-run). An analysis against twd-relay's `run` +subcommand (`twd-relay/src/cli/run.ts`) confirmed that twd-relay's output design is +significantly better for this audience: + +1. **Failure detail placement.** twd-relay's final block shows the full + `suite > test` path plus the error message together. twd-cli's final + `Failed tests:` block shows only the bare test name — the error message is + buried in the per-test tree, forcing the reader to correlate them. +2. **Signal-to-noise.** twd-relay is silent during the run and prints one summary + block ("every line printed costs context tokens"). twd-cli prints a full config + JSON dump, status chatter, and the entire test tree including passing tests — + 100+ lines of noise per TDD iteration on a moderate suite. +3. **Actionable infrastructure errors.** twd-relay explains cause + remediation for + its failure modes. twd-cli dumps raw Puppeteer errors for the two most common + failures an AI hits: dev server down and TWD sidebar never mounting. (The one + exception — the existing `protocolTimeout` special case — is exactly the right + pattern and becomes the template.) +4. **Silent exits.** `bin/twd-cli.js`'s catch exits 1 with no output. + +## Decisions + +- **AI-first output is the default.** No `--reporter` flag; one code path. A + `--verbose` flag can be added later if the tree output is missed. +- **Mirror twd-relay's format exactly** (same block shape, `×` marker, no ANSI + colors in the block) so agents parse one format across the ecosystem. +- **In-place rewrite of twd-cli's reporting layer** (Approach A). No changes to + twd-js or twd-relay; no shared package (revisit if the format grows); no + relay-event rearchitecture (conflicts with the future twd-runner direction). + +## Output Format + +### Passing run (entire output) + +``` +Navigating to http://localhost:5173 ... +Running 12 test(s)... +Code coverage data written to .nyc_output/out.json + +--- Run complete --- + Passed: 12 | Failed: 0 | Skipped: 0 + Duration: 4.2s +``` + +### Failing run (adds failure block) + +``` +--- Run complete --- + Passed: 10 | Failed: 2 | Skipped: 0 + Duration: 4.2s + + Failed tests (2): + × Login > shows error on wrong password + Expected element to be visible (at http://localhost:5173/login) + × Signup > validates email + Timeout waiting for selector ".error" (at http://localhost:5173/signup) +``` + +### Rules + +- **Removed:** config JSON dump, `Starting TWD test runner...`, + `Page loaded. Starting tests...`, `Tests to report: N`, `Browser closed.`, and + the per-test tree (`reportResults` from `twd-js/runner-ci` is no longer called). +- **Kept:** `Navigating to ...` (context for which URL was targeted), the + `--test` filter messages (`Filtering: running N test(s)...` and the no-match + error), coverage lines, and the contract report (out of scope; already has its + own format). +- Failure paths are built with the existing `buildTestPath()` + (`suite > subsuite > test`). Error messages keep the current + `(at )` suffix so agents know the route. +- Retried tests appear inside the block in the same style: + `Retried (1): ✓ Login > shows error (passed on attempt 2)`. +- Multi-line error messages are indented to align under their test line + (same `replace(/\n/g, ...)` treatment as twd-relay). +- No ANSI color codes in the summary block. +- The `--- Run complete ---` block is always the **last** output of a completed + run — coverage and contract lines print before it, so the tail of the output is + always the summary + failures (the part an agent reads first). +- Exit codes unchanged: 0 = all pass, 1 = any failure (test, contract, or + infrastructure). + +## Error Diagnostics + +New module `src/diagnostics.js` exporting `explainError(error, config)` → a +diagnostic string for known failure modes, or `null` for unknown ones. The +`runTests()` catch block prints the error message first, then the diagnostic. +Messages interpolate the *actual* config values (url, timeout), not defaults. + +| # | Failure mode | Detection | Message (cause + remediation) | +|---|---|---|---| +| 1 | Dev server unreachable | `net::ERR_CONNECTION_REFUSED`, `ERR_NAME_NOT_RESOLVED`, `ERR_ADDRESS_UNREACHABLE` from `page.goto` | `Could not reach — connection refused.` / `Is your dev server running? Start it (e.g. \`npm run dev\`) or fix "url" in twd.config.json.` | +| 2 | TWD sidebar never mounts | `TimeoutError` from `waitForSelector('#twd-sidebar-root')` | `Page loaded but the TWD sidebar (#twd-sidebar-root) did not appear within ms.` / `Ensure twd-js is initialized in your app and your tests are registered.` / `If the app is slow to start, raise "timeout" in twd.config.json.` | +| 3 | Protocol timeout | existing `isProtocolTimeout()` check (moves into this module) | existing message, unchanged | +| 4 | Browser launch failure | `puppeteer.launch` errors matching `Could not find Chrome` or `Failed to launch the browser process` | `Puppeteer could not launch Chrome.` / `Run \`npx puppeteer browsers install chrome\`, or adjust "puppeteerArgs" in twd.config.json.` | + +- **Unknown errors:** print `Error running tests: ` then the stack — + message first, stack after (replaces today's raw object dump). +- **Silent-exit fix:** `bin/twd-cli.js` prints `error.message` in its catch before + `process.exit(1)` (covers `parseRunArgs` throws, which today exit silently). + +## Code Structure + +| File | Change | +|---|---| +| `src/testSummary.js` | Rewritten: `formatTestSummary` + `formatFailedTestsBlock` replaced by one `formatRunComplete({ testStatus, handlers, durationMs, retriedTests })` returning the full block as a string. Imports `buildTestPath`. `formatDuration.js` stays. | +| `src/diagnostics.js` | New: `explainError(error, config)`; absorbs `isProtocolTimeout()` from `index.js`. | +| `src/index.js` | Slimmed: drops config dump/chatter and the `reportResults` call, computes test count for the `Running N test(s)...` line from handlers it already queries, calls `formatRunComplete`, uses `explainError` in the catch. | +| `bin/twd-cli.js` | Prints `error.message` before `process.exit(1)`. | + +**Dependency consequence:** dropping `reportResults` removes twd-cli's only import +from `twd-js`. If nothing else imports it, remove `twd-js` from `package.json` — +which requires `npm run lock:linux` afterwards (Docker). Verify during +implementation. + +## Testing + +TDD with the existing vitest + mocked-Puppeteer setup in `tests/`: + +- **`formatRunComplete` unit tests:** all-pass block, failures block (asserting + `suite > test` path + indented error), retries, skips, multi-line error + indentation, and no ANSI codes in output. +- **`explainError` unit tests:** one per known failure mode asserting the message + references config values; unknown error returns `null`. +- **Run-flow tests updated:** existing output assertions revised; assert the config + dump is gone and the summary block prints; exit-code behavior unchanged. +- **Manual verification** against `test-example-app/`: a green run, a + deliberately-broken test (failure block), and a stopped dev server + (diagnostic #1). + +## Out of Scope + +- Contract report format (unchanged). +- JSON/machine-readable reporter. +- `--verbose` flag restoring the tree (add later if requested). +- Any changes to twd-js or twd-relay. From f02f6791346429de522c11bf0ca93142abfc588d Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 7 Jul 2026 10:04:01 +0200 Subject: [PATCH 02/11] docs: add implementation plan for AI-friendly output; fix spec formatDuration note Co-Authored-By: Claude Fable 5 --- .../plans/2026-07-07-ai-friendly-output.md | 995 ++++++++++++++++++ .../2026-07-06-ai-friendly-output-design.md | 2 +- 2 files changed, 996 insertions(+), 1 deletion(-) create mode 100644 docs/superpowers/plans/2026-07-07-ai-friendly-output.md diff --git a/docs/superpowers/plans/2026-07-07-ai-friendly-output.md b/docs/superpowers/plans/2026-07-07-ai-friendly-output.md new file mode 100644 index 0000000..be29e33 --- /dev/null +++ b/docs/superpowers/plans/2026-07-07-ai-friendly-output.md @@ -0,0 +1,995 @@ +# AI-Friendly Output & Error Diagnostics Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace twd-cli's verbose output with twd-relay's AI-optimized summary-block format and add actionable diagnostics for known infrastructure failures. + +**Architecture:** In-place rewrite of the reporting layer. `src/testSummary.js` becomes a single `formatRunComplete()` returning the relay-style block; a new `src/diagnostics.js` maps known Puppeteer errors to cause+remediation messages; `src/index.js` drops the config dump, chatter, and the per-test tree (`reportResults` from twd-js); `bin/twd-cli.js` never exits silently. The `twd-js` dependency becomes removable. + +**Tech Stack:** Node.js ESM (no TypeScript), vitest with mocked Puppeteer/fs, Puppeteer. + +**Spec:** `docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md` (approved). + +## Global Constraints + +- Output must mirror twd-relay's `run` format exactly: `--- Run complete ---` header, ` Passed: N | Failed: N | Skipped: N`, ` Duration: X.Xs` (`(durationMs / 1000).toFixed(1)`), failure marker `×` (U+00D7), retried marker `✓`. +- **No ANSI color codes** anywhere in the summary block. +- The `--- Run complete ---` block is always the **last** output of a completed run (coverage/contract lines print before it). +- Exit codes unchanged: 0 = all pass, 1 = any failure. +- Filter messages (`Filtering: running N test(s) matching --test filter(s).`, `No tests matched filter(s): ...`), coverage lines, and the contract report format are **unchanged**. +- Error messages interpolate actual config values (`config.url`, `config.timeout`), never hardcoded defaults. +- Node >= 18, ESM only (`"type": "module"`). Optional chaining is fine. +- Run tests with `npx vitest run ` (non-watch). Full suite: `npx vitest run`. +- All work happens on the existing `feat/ai-friendly-output` branch. Commit after each task (this repo allows autonomous commits on feat/* branches). + +--- + +### Task 1: `formatRunComplete` formatter + +Rewrite `src/testSummary.js` as one function returning the relay-style block. Delete `formatDuration.js` (the relay format makes it unused). + +**Files:** +- Modify: `src/testSummary.js` (full rewrite) +- Test: `tests/testSummary.test.js` (full rewrite) +- Delete: `src/formatDuration.js`, `tests/formatDuration.test.js` + +**Interfaces:** +- Consumes: `buildTestPath(testId, handlers)` from `src/buildTestPath.js` (exists; returns `'Suite > child > test'` or `null` if the id is unknown). +- Produces: `formatRunComplete({ testStatus, handlers, durationMs })` → `string` (the complete block, no trailing newline). `testStatus` entries are `{ id, status: 'pass'|'fail'|'skip', error?, retryAttempt? }`; `handlers` are `{ id, name, parent?, type }`. Task 3 calls this. + +- [ ] **Step 1: Write the failing tests** + +Replace the entire contents of `tests/testSummary.test.js` with: + +```js +import { describe, it, expect } from 'vitest'; +import { formatRunComplete } from '../src/testSummary.js'; + +const handlers = [ + { id: 's1', name: 'Login', type: 'suite' }, + { id: 't1', name: 'shows error on wrong password', parent: 's1', type: 'test' }, + { id: 't2', name: 'redirects on success', parent: 's1', type: 'test' }, + { id: 't3', name: 'validates email', parent: 's1', type: 'test' }, +]; + +describe('formatRunComplete', () => { + it('formats an all-pass run as the three-line block', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'pass' }, + { id: 't2', status: 'pass' }, + ], + handlers, + durationMs: 4200, + }); + expect(block).toBe( + '--- Run complete ---\n' + + ' Passed: 2 | Failed: 0 | Skipped: 0\n' + + ' Duration: 4.2s' + ); + }); + + it('counts skipped tests', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'pass' }, + { id: 't2', status: 'skip' }, + ], + handlers, + durationMs: 1000, + }); + expect(block).toContain(' Passed: 1 | Failed: 0 | Skipped: 1'); + }); + + it('appends the failure block with suite path and indented error', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'fail', error: 'Expected element to be visible (at http://localhost:5173/login)' }, + { id: 't2', status: 'pass' }, + { id: 't3', status: 'fail', error: 'Timeout waiting for selector ".error"' }, + ], + handlers, + durationMs: 4200, + }); + expect(block).toBe( + '--- Run complete ---\n' + + ' Passed: 1 | Failed: 2 | Skipped: 0\n' + + ' Duration: 4.2s\n' + + '\n' + + ' Failed tests (2):\n' + + ' × Login > shows error on wrong password\n' + + ' Expected element to be visible (at http://localhost:5173/login)\n' + + ' × Login > validates email\n' + + ' Timeout waiting for selector ".error"' + ); + }); + + it('indents multi-line error messages to align under the test line', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'fail', error: 'line one\nline two' }], + handlers, + durationMs: 500, + }); + expect(block).toContain(' line one\n line two'); + }); + + it('falls back to the test id when no handler matches', () => { + const block = formatRunComplete({ + testStatus: [{ id: 'ghost-id', status: 'fail', error: 'boom' }], + handlers: [], + durationMs: 500, + }); + expect(block).toContain(' × ghost-id'); + }); + + it('omits the failure block when everything passes', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'pass' }], + handlers, + durationMs: 500, + }); + expect(block).not.toContain('Failed tests'); + }); + + it('appends the retried block for tests that passed on retry', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'pass', retryAttempt: 2 }, + { id: 't2', status: 'pass' }, + ], + handlers, + durationMs: 500, + }); + expect(block).toContain( + '\n' + + ' Retried (1):\n' + + ' ✓ Login > shows error on wrong password (passed on attempt 2)' + ); + }); + + it('omits the retried block when no test was retried', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'pass' }], + handlers, + durationMs: 500, + }); + expect(block).not.toContain('Retried'); + }); + + it('contains no ANSI escape codes', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'fail', error: 'boom' }, + { id: 't2', status: 'pass', retryAttempt: 2 }, + { id: 't3', status: 'skip' }, + ], + handlers, + durationMs: 500, + }); + expect(/\x1b\[[0-9;]*m/.test(block)).toBe(false); + }); +}); +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `npx vitest run tests/testSummary.test.js` +Expected: FAIL — `formatRunComplete` is not exported (`SyntaxError` or undefined import). + +- [ ] **Step 3: Write the implementation** + +Replace the entire contents of `src/testSummary.js` with: + +```js +import { buildTestPath } from './buildTestPath.js'; + +export function formatRunComplete({ testStatus, handlers, durationMs }) { + const passed = testStatus.filter((t) => t.status === 'pass').length; + const failed = testStatus.filter((t) => t.status === 'fail').length; + const skipped = testStatus.filter((t) => t.status === 'skip').length; + const duration = (durationMs / 1000).toFixed(1); + + const lines = [ + '--- Run complete ---', + ` Passed: ${passed} | Failed: ${failed} | Skipped: ${skipped}`, + ` Duration: ${duration}s`, + ]; + + const failures = testStatus.filter((t) => t.status === 'fail'); + if (failures.length > 0) { + lines.push('', ` Failed tests (${failures.length}):`); + for (const failure of failures) { + const testPath = buildTestPath(failure.id, handlers) ?? failure.id; + lines.push(` × ${testPath}`); + if (failure.error) { + lines.push(` ${String(failure.error).replace(/\n/g, '\n ')}`); + } + } + } + + const retried = testStatus.filter((t) => t.status === 'pass' && t.retryAttempt >= 2); + if (retried.length > 0) { + lines.push('', ` Retried (${retried.length}):`); + for (const t of retried) { + const testPath = buildTestPath(t.id, handlers) ?? t.id; + lines.push(` ✓ ${testPath} (passed on attempt ${t.retryAttempt})`); + } + } + + return lines.join('\n'); +} +``` + +- [ ] **Step 4: Delete the now-unused duration formatter** + +```bash +git rm src/formatDuration.js tests/formatDuration.test.js +``` + +(`formatDuration` was only imported by the old `testSummary.js` — verify with `grep -rn "formatDuration" src/ tests/ bin/`, which must return nothing.) + +- [ ] **Step 5: Run the new tests to verify they pass** + +Run: `npx vitest run tests/testSummary.test.js` +Expected: PASS (9 tests). + +Note: `npx vitest run` (full suite) still fails at this point — `src/index.js` imports `formatTestSummary`/`formatFailedTestsBlock`, which no longer exist. That is expected until Task 3; do not run the full suite as a gate here. + +- [ ] **Step 6: Commit** + +```bash +git add -A src/testSummary.js src/formatDuration.js tests/testSummary.test.js tests/formatDuration.test.js +git commit -m "feat: add relay-style formatRunComplete block formatter" +``` + +--- + +### Task 2: `explainError` diagnostics module + +New module mapping known Puppeteer failures to actionable cause+remediation messages. Absorbs `isProtocolTimeout` from `src/index.js` (leave `index.js` untouched in this task; Task 3 rewires it). + +**Files:** +- Create: `src/diagnostics.js` +- Test: `tests/diagnostics.test.js` + +**Interfaces:** +- Consumes: nothing from other tasks. +- Produces: `explainError(error, config = {})` → `string | null` (diagnostic text for known failures, `null` for unknown) and `isProtocolTimeout(error)` → `boolean`. Task 3 calls both. + +- [ ] **Step 1: Write the failing tests** + +Create `tests/diagnostics.test.js`: + +```js +import { describe, it, expect } from 'vitest'; +import { explainError, isProtocolTimeout } from '../src/diagnostics.js'; + +const config = { url: 'http://localhost:5173', timeout: 10000 }; + +describe('explainError', () => { + it('explains connection refused with the configured url', () => { + const err = new Error('net::ERR_CONNECTION_REFUSED at http://localhost:5173'); + const msg = explainError(err, config); + expect(msg).toContain('Could not reach http://localhost:5173 (ERR_CONNECTION_REFUSED)'); + expect(msg).toContain('Is your dev server running?'); + expect(msg).toContain('"url" in twd.config.json'); + }); + + it('explains DNS resolution failures', () => { + const err = new Error('net::ERR_NAME_NOT_RESOLVED at http://myapp.local:5173'); + const msg = explainError(err, { ...config, url: 'http://myapp.local:5173' }); + expect(msg).toContain('Could not reach http://myapp.local:5173 (ERR_NAME_NOT_RESOLVED)'); + }); + + it('explains unreachable-address failures', () => { + const err = new Error('net::ERR_ADDRESS_UNREACHABLE at http://10.0.0.9:5173'); + expect(explainError(err, config)).toContain('(ERR_ADDRESS_UNREACHABLE)'); + }); + + it('explains the sidebar selector timeout with the configured timeout', () => { + const err = new Error( + "Waiting for selector '#twd-sidebar-root' failed: Waiting failed: 10000ms exceeded" + ); + err.name = 'TimeoutError'; + const msg = explainError(err, config); + expect(msg).toContain('TWD sidebar (#twd-sidebar-root) did not appear within 10000ms'); + expect(msg).toContain('Ensure twd-js is initialized'); + expect(msg).toContain('raise "timeout" in twd.config.json'); + }); + + it('does not claim a sidebar problem for unrelated TimeoutErrors', () => { + const err = new Error('Waiting for selector ".other-thing" failed'); + err.name = 'TimeoutError'; + expect(explainError(err, config)).toBeNull(); + }); + + it('explains protocol timeouts', () => { + const err = new Error('Runtime.callFunctionOn timed out.'); + err.name = 'ProtocolError'; + const msg = explainError(err, config); + expect(msg).toContain('protocolTimeout'); + expect(msg).toContain('twd.config.json'); + }); + + it('explains a missing Chrome install', () => { + const err = new Error('Could not find Chrome (ver. 131.0.6778.204).'); + const msg = explainError(err, config); + expect(msg).toContain('Puppeteer could not launch Chrome'); + expect(msg).toContain('npx puppeteer browsers install chrome'); + }); + + it('explains a browser process launch failure', () => { + const err = new Error('Failed to launch the browser process!\nspawn ENOENT'); + const msg = explainError(err, config); + expect(msg).toContain('Puppeteer could not launch Chrome'); + expect(msg).toContain('"puppeteerArgs" in twd.config.json'); + }); + + it('returns null for unknown errors', () => { + expect(explainError(new Error('something else entirely'), config)).toBeNull(); + }); + + it('tolerates a missing config and non-Error values', () => { + const err = new Error('net::ERR_CONNECTION_REFUSED at http://localhost:5173'); + expect(explainError(err)).toContain('Could not reach undefined (ERR_CONNECTION_REFUSED)'); + expect(explainError(null, config)).toBeNull(); + expect(explainError('boom', config)).toBeNull(); + }); +}); + +describe('isProtocolTimeout', () => { + it('matches ProtocolError timeouts', () => { + const err = new Error('Runtime.callFunctionOn timed out.'); + err.name = 'ProtocolError'; + expect(isProtocolTimeout(err)).toBe(true); + }); + + it('matches messages that mention protocolTimeout', () => { + expect(isProtocolTimeout(new Error('Increase the protocolTimeout setting'))).toBe(true); + }); + + it('rejects unrelated errors', () => { + expect(isProtocolTimeout(new Error('boom'))).toBe(false); + expect(isProtocolTimeout(null)).toBe(false); + }); +}); +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `npx vitest run tests/diagnostics.test.js` +Expected: FAIL — cannot resolve `../src/diagnostics.js`. + +- [ ] **Step 3: Write the implementation** + +Create `src/diagnostics.js`: + +```js +const NET_ERRORS = ['ERR_CONNECTION_REFUSED', 'ERR_NAME_NOT_RESOLVED', 'ERR_ADDRESS_UNREACHABLE']; + +export function isProtocolTimeout(error) { + const message = error && error.message ? error.message : ''; + return ( + (error && error.name === 'ProtocolError' && /timed out/i.test(message)) || + /protocolTimeout/i.test(message) + ); +} + +export function explainError(error, config = {}) { + if (!error || typeof error.message !== 'string') return null; + const message = error.message; + + const netMatch = message.match(/net::(ERR_[A-Z_]+)/); + if (netMatch && NET_ERRORS.includes(netMatch[1])) { + return ( + `Could not reach ${config.url} (${netMatch[1]}).\n` + + 'Is your dev server running? Start it (e.g. `npm run dev`) or fix "url" in twd.config.json.' + ); + } + + if (error.name === 'TimeoutError' && message.includes('#twd-sidebar-root')) { + return ( + `Page loaded but the TWD sidebar (#twd-sidebar-root) did not appear within ${config.timeout}ms.\n` + + 'Ensure twd-js is initialized in your app and your tests are registered.\n' + + 'If the app is slow to start, raise "timeout" in twd.config.json.' + ); + } + + if (isProtocolTimeout(error)) { + return ( + 'This looks like a Puppeteer protocolTimeout. The whole test suite runs in a single\n' + + 'page.evaluate call, so the run aborts if it takes longer than "protocolTimeout" (ms).\n' + + 'Raise it in twd.config.json, e.g. { "protocolTimeout": 600000 } (0 = no timeout).' + ); + } + + if (/Could not find Chrome|Failed to launch the browser process/.test(message)) { + return ( + 'Puppeteer could not launch Chrome.\n' + + 'Run `npx puppeteer browsers install chrome`, or adjust "puppeteerArgs" in twd.config.json.' + ); + } + + return null; +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `npx vitest run tests/diagnostics.test.js` +Expected: PASS (13 tests). + +- [ ] **Step 5: Commit** + +```bash +git add src/diagnostics.js tests/diagnostics.test.js +git commit -m "feat: add explainError diagnostics for known infrastructure failures" +``` + +--- + +### Task 3: Rewire `src/index.js` and `bin/twd-cli.js` + +Slim the orchestrator: drop the config dump, chatter, and the `reportResults` tree; print `Running N test(s)...`; print the block last; use `explainError` in the catch; never exit silently from the bin. + +**Files:** +- Modify: `src/index.js` (full rewrite below) +- Modify: `bin/twd-cli.js:13-15` (catch block) +- Test: `tests/runTests.test.js` (update mocks + assertions) + +**Interfaces:** +- Consumes: `formatRunComplete({ testStatus, handlers, durationMs })` from Task 1; `explainError(error, config)` from Task 2. +- Produces: `runTests(options)` keeps its exact current signature and return (`Promise` — `true` = failures). On error it sets `error.reported = true` after printing, rethrows; `bin/twd-cli.js` only prints `error.message` when `reported` is not set. + +- [ ] **Step 1: Update the run-flow tests** + +Apply these changes to `tests/runTests.test.js`: + +**(a)** Delete the `twd-js/runner-ci` mock and import (lines 6-8 and 22): + +```js +// DELETE these lines: +vi.mock('twd-js/runner-ci', () => ({ + reportResults: vi.fn(), +})); +// ...and: +import { reportResults } from 'twd-js/runner-ci'; +``` + +**(b)** Replace the `createMockPage` helper — every run now starts with an enumeration `page.evaluate` (returns the registered handler list) before the run `page.evaluate`: + +```js +function createMockPage(evaluateResult) { + return { + goto: vi.fn(), + waitForSelector: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(evaluateResult.handlers ?? []) // enumeration pass + .mockResolvedValue(evaluateResult), // run pass (+ coverage) + exposeFunction: vi.fn(), + }; +} +``` + +**(c)** In the test `"should print a protocolTimeout hint when the run aborts on timeout"`, the rejected evaluate now hits the enumeration call first — the flow still reaches the same catch, and the assertion (`errors.some((e) => e.includes('protocolTimeout'))`) still holds. No change needed beyond the helper. Verify it still passes in Step 4. + +**(d)** In `"preserves responseHeaders through the __twdCollectMock spread"`, the custom `page.evaluate` mock must serve the enumeration call first. Change the `evaluate` implementation to: + +```js + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) // enumeration pass + // Drive the registered __twdCollectMock callback from inside page.evaluate, + // mirroring how a real browser test would trigger it. + .mockImplementation(async () => { + const exposed = page.exposeFunction.mock.calls.find( + (c) => c[0] === '__twdCollectMock' + ); + expect(exposed).toBeDefined(); + const collectMock = exposed[1]; + await collectMock({ + alias: 'getPhoto', + url: '/v1/photo', + method: 'GET', + status: 200, + response: 'bin', + testId: 't-1', + responseHeaders: { 'Content-Type': 'image/png' }, + }); + return { handlers, testStatus }; + }), +``` + +**(e)** The four tests that assert on the run-evaluate arguments keep working unchanged (`toHaveBeenCalledWith` matches any call), but the two filter tests and `"skips coverage collection when a filter is active"` assert `page.evaluate` **call counts and positions** — these stay valid because filtered runs already used an enumeration-then-run sequence, which is unchanged. Leave them as-is. + +**(f)** Replace the two retry-summary tests with block-format assertions: + +```js + it("should include retried tests in the run-complete block", async () => { + const testStatus = [ + { id: '1', status: 'pass', retryAttempt: 2 }, + { id: '2', status: 'pass' }, + ]; + const handlers = [ + { id: '1', name: 'flaky test', type: 'test' }, + { id: '2', name: 'stable test', type: 'test' }, + ]; + const page = createMockPage({ handlers, testStatus }); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runTests(); + + const logs = consoleSpy.mock.calls.map(c => String(c[0])); + const block = logs.find(l => l.startsWith('--- Run complete ---')); + expect(block).toBeDefined(); + expect(block).toContain('Retried (1):'); + expect(block).toContain('✓ flaky test (passed on attempt 2)'); + }); + + it("should not include a retried section when no tests were retried", async () => { + const testStatus = [{ id: '1', status: 'pass' }]; + const handlers = [{ id: '1', name: 'test1', type: 'test' }]; + const page = createMockPage({ handlers, testStatus }); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runTests(); + + const logs = consoleSpy.mock.calls.map(c => String(c[0])); + const block = logs.find(l => l.startsWith('--- Run complete ---')); + expect(block).toBeDefined(); + expect(block).not.toContain('Retried'); + }); +``` + +**(g)** Replace the final test (`"should print the Tests: summary line and Failed tests block"`) with: + +```js + it("prints the run-complete block last, with failure paths and errors", async () => { + const testStatus = [ + { id: '1', status: 'pass' }, + { id: '2', status: 'fail', error: 'boom (at http://localhost:5173/form)' }, + { id: '3', status: 'skip' }, + ]; + const handlers = [ + { id: 's1', name: 'Form', type: 'suite' }, + { id: '1', name: 'should render', parent: 's1', type: 'test' }, + { id: '2', name: 'should submit form', parent: 's1', type: 'test' }, + { id: '3', name: 'should show error', parent: 's1', type: 'test' }, + ]; + const page = createMockPage({ handlers, testStatus }); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runTests(); + + const logs = consoleSpy.mock.calls.map((c) => String(c[0])); + const block = logs[logs.length - 1]; + expect(block.startsWith('--- Run complete ---')).toBe(true); + expect(block).toContain('Passed: 1 | Failed: 1 | Skipped: 1'); + expect(block).toContain('× Form > should submit form'); + expect(block).toContain('boom (at http://localhost:5173/form)'); + }); + + it("prints no config dump and no per-test tree chatter", async () => { + const testStatus = [{ id: '1', status: 'pass' }]; + const handlers = [{ id: '1', name: 'test1', type: 'test' }]; + const page = createMockPage({ handlers, testStatus }); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runTests(); + + const logs = consoleSpy.mock.calls.map((c) => String(c[0])); + expect(logs.some((l) => l.startsWith('Configuration:'))).toBe(false); + expect(logs.some((l) => l.startsWith('Starting TWD test runner'))).toBe(false); + expect(logs.some((l) => l.startsWith('Tests to report'))).toBe(false); + expect(logs.some((l) => l.startsWith('Browser closed'))).toBe(false); + expect(logs.some((l) => l === 'Running 1 test(s)...')).toBe(true); + }); + + it("marks rethrown errors as reported", async () => { + const page = createMockPage({ handlers: [], testStatus: [] }); + const bootError = new Error('net::ERR_CONNECTION_REFUSED at http://localhost:5173'); + page.goto = vi.fn().mockRejectedValue(bootError); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + + await expect(runTests()).rejects.toThrow('ERR_CONNECTION_REFUSED'); + + expect(bootError.reported).toBe(true); + const errors = errorSpy.mock.calls.map((c) => String(c[0])); + expect(errors.some((e) => e.includes('Is your dev server running?'))).toBe(true); + errorSpy.mockRestore(); + }); +``` + +- [ ] **Step 2: Run the tests to verify the new/changed ones fail** + +Run: `npx vitest run tests/runTests.test.js` +Expected: FAIL — the whole file errors because `src/index.js` still imports `formatTestSummary`/`formatFailedTestsBlock`, which no longer exist after Task 1. (That import error IS the red state for this task.) + +- [ ] **Step 3: Rewrite `src/index.js`** + +Replace the entire contents of `src/index.js` with: + +```js +import fs from 'fs'; +import path from 'path'; +import puppeteer from 'puppeteer'; +import { loadConfig } from './config.js'; +import { loadContracts, validateMocks } from './contracts.js'; +import { printContractReport } from './contractReport.js'; +import { generateContractMarkdown } from './contractMarkdown.js'; +import { buildTestPath } from './buildTestPath.js'; +import { formatRunComplete } from './testSummary.js'; +import { selectTestIds } from './filterTests.js'; +import { explainError } from './diagnostics.js'; + +export async function runTests(options = {}) { + const { testFilters = [] } = options; + let browser; + let config; + try { + config = loadConfig(); + const workingDir = process.cwd(); + + // Load contract validators if configured + let contractValidators = []; + if (config.contracts && config.contracts.length > 0) { + contractValidators = await loadContracts(config.contracts, workingDir); + } + + browser = await puppeteer.launch({ + headless: config.headless, + args: config.puppeteerArgs, + protocolTimeout: config.protocolTimeout, + }); + + const page = await browser.newPage(); + + // Register mock collector for contract validation + const collectedMocks = new Map(); + const occurrenceCounters = new Map(); + if (config.contracts && config.contracts.length > 0) { + await page.exposeFunction('__twdCollectMock', (mock) => { + const occKey = `${mock.alias}:${mock.testId}`; + const count = (occurrenceCounters.get(occKey) || 0) + 1; + occurrenceCounters.set(occKey, count); + + const dedupKey = `${mock.method}:${mock.url}:${mock.status}:${mock.testId}:${count}`; + collectedMocks.set(dedupKey, { ...mock, occurrence: count }); + }); + } + + // Navigate to your development server + const startedAt = Date.now(); + console.log(`Navigating to ${config.url} ...`); + await page.goto(config.url); + + // Wait for the selector to be available + await page.waitForSelector('#twd-sidebar-root', { timeout: config.timeout }); + + // Enumerate registered handlers (for the count line and --test filtering) + const registeredHandlers = await page.evaluate(() => { + const state = window.__TWD_STATE__; + if (!state || !state.handlers) return []; + return Array.from(state.handlers.values()).map((h) => ({ + id: h.id, + name: h.name, + parent: h.parent, + type: h.type, + })); + }); + + // Resolve --test filters to a concrete set of test ids (null = run all) + let selectedIds = null; + if (testFilters.length > 0) { + const { ids, unmatchedFilters } = selectTestIds(registeredHandlers, testFilters); + + if (ids.length === 0) { + console.error( + `No tests matched filter(s): ${testFilters.map((f) => `"${f}"`).join(', ')}` + ); + await browser.close(); + return true; + } + + if (unmatchedFilters.length > 0) { + console.warn( + `Warning: these filter(s) matched no tests (others did): ${unmatchedFilters.map((f) => `"${f}"`).join(', ')}` + ); + } + + selectedIds = ids; + console.log(`Filtering: running ${ids.length} test(s) matching --test filter(s).`); + } else { + const testCount = registeredHandlers.filter((h) => h.type !== 'suite').length; + console.log(`Running ${testCount} test(s)...`); + } + + // Execute all tests + const { handlers, testStatus } = await page.evaluate(async (retryCount, selectedIds) => { + const TestRunner = window.__testRunner; + const testStatus = []; + const runner = new TestRunner({ + onStart: (test) => { + test.status = "running"; + }, + onPass: (test, retryAttempt) => { + test.status = "done"; + const entry = { id: test.id, status: "pass" }; + if (retryAttempt !== undefined) entry.retryAttempt = retryAttempt; + testStatus.push(entry); + }, + onFail: (test, err) => { + test.status = "done"; + testStatus.push({ id: test.id, status: "fail", error: `${err.message} (at ${window.location.href})` }); + }, + onSkip: (test) => { + test.status = "done"; + testStatus.push({ id: test.id, status: "skip" }); + }, + }, { retryCount }); + const handlers = selectedIds + ? await runner.runByIds(selectedIds) + : await runner.runAll(); + return { handlers: Array.from(handlers.values()), testStatus }; + }, config.retryCount, selectedIds); + + const durationMs = Date.now() - startedAt; + + // Exit with appropriate code + let hasFailures = testStatus.some(test => test.status === 'fail'); + + // Enrich collected mocks with full test path names + for (const [, mock] of collectedMocks) { + if (mock.testId) { + mock.testName = buildTestPath(mock.testId, handlers); + } + } + + // Contract validation + if (config.contracts && config.contracts.length > 0) { + if (collectedMocks.size === 0) { + console.log('\nNo mocks collected — ensure twd-js supports contract collection'); + } + const validationOutput = validateMocks(collectedMocks, contractValidators); + const hasContractErrors = printContractReport(validationOutput); + if (hasContractErrors) { + hasFailures = true; + } + + // Write markdown report for CI/PR integration + if (config.contractReportPath) { + const reportPath = path.resolve(workingDir, config.contractReportPath); + const reportDir = path.dirname(reportPath); + if (!fs.existsSync(reportDir)) { + fs.mkdirSync(reportDir, { recursive: true }); + } + const markdown = generateContractMarkdown(validationOutput); + fs.writeFileSync(reportPath, markdown); + console.log(`Contract report written to ${config.contractReportPath}`); + } + } + + // Handle code coverage if enabled (skipped when a --test filter is active) + if (selectedIds && config.coverage) { + console.log('Skipping coverage collection (test filter active).'); + } + if (config.coverage && !hasFailures && !selectedIds) { + const coverage = await page.evaluate(() => window.__coverage__); + if (coverage) { + const coverageDir = path.resolve(workingDir, config.coverageDir); + const nycDir = path.resolve(workingDir, config.nycOutputDir); + + if (!fs.existsSync(nycDir)) { + fs.mkdirSync(nycDir, { recursive: true }); + } + if (!fs.existsSync(coverageDir)) { + fs.mkdirSync(coverageDir, { recursive: true }); + } + + const coveragePath = path.join(nycDir, 'out.json'); + fs.writeFileSync(coveragePath, JSON.stringify(coverage)); + console.log(`Code coverage data written to ${coveragePath}`); + } else { + console.log('No code coverage data found.'); + } + } + + await browser.close(); + + // The run-complete block is always the last output of a completed run + console.log(''); + console.log(formatRunComplete({ testStatus, handlers, durationMs })); + + return hasFailures; + + } catch (error) { + const message = error && error.message ? error.message : String(error); + console.error(`Error running tests: ${message}`); + const diagnostic = explainError(error, config); + if (diagnostic) { + console.error(`\n${diagnostic}`); + } else if (error && error.stack) { + console.error(`\n${error.stack}`); + } + if (error && typeof error === 'object') { + error.reported = true; + } + if (browser) await browser.close(); + throw error; + } +} +``` + +Deliberate changes from the old version (everything else is verbatim): +- Removed: `Starting TWD test runner...`, the `Configuration:` JSON dump, `Page loaded. Starting tests...`, `Tests to report: N`, `Browser closed.`, `Collecting code coverage data...`, the `reportResults` import+call, the old `⟳ Retried tests:` block, the old summary/failed-block printing, and the local `isProtocolTimeout`. +- Added: always-on handler enumeration (the filter path previously did this conditionally — same evaluate body), the `Running N test(s)...` line for unfiltered runs, `formatRunComplete` as the final output, `explainError` + `error.reported` in the catch. +- `config` is declared outside `try` so the catch can pass it to `explainError`. + +- [ ] **Step 4: Run the run-flow tests to verify they pass** + +Run: `npx vitest run tests/runTests.test.js` +Expected: PASS (all tests, including the updated ones). + +- [ ] **Step 5: Fix the silent bin catch** + +In `bin/twd-cli.js`, replace: + +```js + } catch (error) { + process.exit(1); + } +``` + +with: + +```js + } catch (error) { + if (!error?.reported) { + console.error(error?.message ?? String(error)); + } + process.exit(1); + } +``` + +- [ ] **Step 6: Run the full suite** + +Run: `npx vitest run` +Expected: PASS — every test file green (the Task 1 note about the broken full suite is now resolved). + +- [ ] **Step 7: Commit** + +```bash +git add src/index.js bin/twd-cli.js tests/runTests.test.js +git commit -m "feat: relay-style run output, actionable error diagnostics, no silent exits" +``` + +--- + +### Task 4: Remove the `twd-js` dependency + +`src/index.js` no longer imports `reportResults` — the package's only `twd-js` import. The in-browser `__testRunner` comes from the *user's app* bundling twd-js, not from this dependency. + +**Files:** +- Modify: `package.json` (remove dependency) +- Modify: `package-lock.json` (regenerated) + +**Interfaces:** +- Consumes: Task 3 (the import must already be gone). +- Produces: nothing for later tasks. + +- [ ] **Step 1: Verify nothing imports twd-js anymore** + +Run: `grep -rn "from 'twd-js" src/ bin/ tests/` +Expected: no output. (If anything matches, stop — Task 3 was not completed.) + +- [ ] **Step 2: Remove the dependency** + +In `package.json`, delete this line from `dependencies`: + +```json + "twd-js": "^1.8.2" +``` + +(leaving `openapi-mock-validator` and `puppeteer`). + +- [ ] **Step 3: Regenerate the lockfile — macOS pass, then Linux pass** + +```bash +npm install +npm run lock:linux +``` + +The second command is **required** (repo rule): npm on macOS leaves `@emnapi/*` transitive deps stale in the lock, which breaks `npm ci` on Linux CI. Docker must be running; if it is not, start Docker Desktop first. Verify the lockfile no longer contains a top-level `node_modules/twd-js` entry: `grep -c '"node_modules/twd-js"' package-lock.json` → expected `0`. + +- [ ] **Step 4: Run the full suite** + +Run: `npx vitest run` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add package.json package-lock.json +git commit -m "chore: drop twd-js dependency (reportResults no longer used)" +``` + +--- + +### Task 5: Update docs and verify end-to-end + +**Files:** +- Modify: `CLAUDE.md:29-46` (architecture steps 5-6 + Key Dependencies) +- Manual verification against `test-example-app/` + +**Interfaces:** +- Consumes: Tasks 1-4 complete. +- Produces: nothing — final task. + +- [ ] **Step 1: Update CLAUDE.md** + +In `CLAUDE.md`, replace the `runTests()` step list items 5-6: + +``` +5. Calls `window.__testRunner` in the browser context to execute all tests +6. Reports results via `reportResults()` from `twd-js/runner-ci` +``` + +with: + +``` +5. Calls `window.__testRunner` in the browser context to execute all tests +6. Prints a relay-style summary block (`formatRunComplete` in `src/testSummary.js`) as the last output: passed/failed/skipped counts, duration, failed tests with `suite > test` paths and error messages, and retried tests. Known infrastructure errors (dev server down, sidebar missing, protocol timeout, Chrome launch failure) get actionable diagnostics from `src/diagnostics.js`. +``` + +And replace the Key Dependencies bullet: + +``` +- **twd-js** — The TWD testing framework; provides `reportResults` from `twd-js/runner-ci` and the in-browser `__testRunner` +``` + +with: + +``` +- **twd-js** — not a dependency of this package; the user's app bundles it, which provides the in-browser `__testRunner` and `#twd-sidebar-root` this CLI drives +``` + +- [ ] **Step 2: Manual verification — green run** + +```bash +cd test-example-app && npm install && npm run dev & +sleep 5 +cd test-example-app && node ../bin/twd-cli.js run +``` + +Expected: `Navigating to http://localhost:5173 ...`, `Running N test(s)...`, contract report lines, then the `--- Run complete ---` block **last** with `Failed: 0`; exit code 0 (`echo $?`). + +- [ ] **Step 3: Manual verification — failing test** + +Temporarily break an assertion in one of `test-example-app/src/**` TWD test files (e.g. change an expected text), re-run `node ../bin/twd-cli.js run` from `test-example-app/`. +Expected: block shows `Failed: 1` and a `Failed tests (1):` entry with `× > ` plus the indented error message ending in `(at http://localhost:5173/...)`; exit code 1. **Revert the break afterwards.** + +- [ ] **Step 4: Manual verification — dev server down** + +Stop the dev server (`kill %1` or Ctrl-C the background job), then from `test-example-app/` run `node ../bin/twd-cli.js run`. +Expected output includes: + +``` +Error running tests: net::ERR_CONNECTION_REFUSED at http://localhost:5173 + +Could not reach http://localhost:5173 (ERR_CONNECTION_REFUSED). +Is your dev server running? Start it (e.g. `npm run dev`) or fix "url" in twd.config.json. +``` + +with no stack trace, exit code 1, and no duplicate message from the bin catch. + +- [ ] **Step 5: Commit** + +```bash +git add CLAUDE.md +git commit -m "docs: describe relay-style output and diagnostics in CLAUDE.md" +``` diff --git a/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md b/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md index ff2532e..e8e52b0 100644 --- a/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md +++ b/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md @@ -109,7 +109,7 @@ Messages interpolate the *actual* config values (url, timeout), not defaults. | File | Change | |---|---| -| `src/testSummary.js` | Rewritten: `formatTestSummary` + `formatFailedTestsBlock` replaced by one `formatRunComplete({ testStatus, handlers, durationMs, retriedTests })` returning the full block as a string. Imports `buildTestPath`. `formatDuration.js` stays. | +| `src/testSummary.js` | Rewritten: `formatTestSummary` + `formatFailedTestsBlock` replaced by one `formatRunComplete({ testStatus, handlers, durationMs })` returning the full block as a string. Imports `buildTestPath`. Duration is formatted relay-style (`(durationMs / 1000).toFixed(1)`s), so `formatDuration.js` and its test are deleted (nothing else uses them). | | `src/diagnostics.js` | New: `explainError(error, config)`; absorbs `isProtocolTimeout()` from `index.js`. | | `src/index.js` | Slimmed: drops config dump/chatter and the `reportResults` call, computes test count for the `Running N test(s)...` line from handlers it already queries, calls `formatRunComplete`, uses `explainError` in the catch. | | `bin/twd-cli.js` | Prints `error.message` before `process.exit(1)`. | From 187a3a793279db1c41de331f772ce5950343d61b Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 7 Jul 2026 10:13:56 +0200 Subject: [PATCH 03/11] feat: add relay-style formatRunComplete block formatter --- src/formatDuration.js | 7 -- src/testSummary.js | 47 +++++---- tests/formatDuration.test.js | 28 ----- tests/testSummary.test.js | 191 ++++++++++++++++------------------- 4 files changed, 113 insertions(+), 160 deletions(-) delete mode 100644 src/formatDuration.js delete mode 100644 tests/formatDuration.test.js diff --git a/src/formatDuration.js b/src/formatDuration.js deleted file mode 100644 index 0f283f9..0000000 --- a/src/formatDuration.js +++ /dev/null @@ -1,7 +0,0 @@ -export function formatDuration(ms) { - const totalSeconds = Math.floor(ms / 1000); - const minutes = Math.floor(totalSeconds / 60); - const seconds = totalSeconds % 60; - const millis = ms % 1000; - return `${minutes}:${String(seconds).padStart(2, '0')}.${String(millis).padStart(3, '0')}`; -} diff --git a/src/testSummary.js b/src/testSummary.js index 83c81bb..89c41bd 100644 --- a/src/testSummary.js +++ b/src/testSummary.js @@ -1,32 +1,37 @@ -import { formatDuration } from './formatDuration.js'; +import { buildTestPath } from './buildTestPath.js'; -const green = (s) => `\x1b[32m${s}\x1b[0m`; -const red = (s) => `\x1b[31m${s}\x1b[0m`; -const yellow = (s) => `\x1b[33m${s}\x1b[0m`; - -export function formatTestSummary({ testStatus, durationMs }) { +export function formatRunComplete({ testStatus, handlers, durationMs }) { const passed = testStatus.filter((t) => t.status === 'pass').length; const failed = testStatus.filter((t) => t.status === 'fail').length; const skipped = testStatus.filter((t) => t.status === 'skip').length; - const total = testStatus.length; - - const passedStr = `${green(passed)} passed`; - const failedStr = `${failed > 0 ? red(failed) : '0'} failed`; - const skippedStr = `${skipped > 0 ? yellow(skipped) : '0'} skipped`; + const duration = (durationMs / 1000).toFixed(1); - return `Tests: ${passedStr}, ${failedStr}, ${skippedStr} (${total} total) in ${formatDuration(durationMs)}`; -} + const lines = [ + '--- Run complete ---', + ` Passed: ${passed} | Failed: ${failed} | Skipped: ${skipped}`, + ` Duration: ${duration}s`, + ]; -export function formatFailedTestsBlock({ testStatus, handlers }) { const failures = testStatus.filter((t) => t.status === 'fail'); - if (failures.length === 0) return null; + if (failures.length > 0) { + lines.push('', ` Failed tests (${failures.length}):`); + for (const failure of failures) { + const testPath = buildTestPath(failure.id, handlers) ?? failure.id; + lines.push(` × ${testPath}`); + if (failure.error) { + lines.push(` ${String(failure.error).replace(/\n/g, '\n ')}`); + } + } + } - const handlersById = new Map(handlers.map((h) => [h.id, h])); - const lines = ['Failed tests:']; - for (const failure of failures) { - const handler = handlersById.get(failure.id); - const name = handler ? handler.name : failure.id; - lines.push(` ${red('✗')} ${name}`); + const retried = testStatus.filter((t) => t.status === 'pass' && t.retryAttempt >= 2); + if (retried.length > 0) { + lines.push('', ` Retried (${retried.length}):`); + for (const t of retried) { + const testPath = buildTestPath(t.id, handlers) ?? t.id; + lines.push(` ✓ ${testPath} (passed on attempt ${t.retryAttempt})`); + } } + return lines.join('\n'); } diff --git a/tests/formatDuration.test.js b/tests/formatDuration.test.js deleted file mode 100644 index b0b74db..0000000 --- a/tests/formatDuration.test.js +++ /dev/null @@ -1,28 +0,0 @@ -import { describe, it, expect } from 'vitest'; -import { formatDuration } from '../src/formatDuration.js'; - -describe('formatDuration', () => { - it('formats zero as 0:00.000', () => { - expect(formatDuration(0)).toBe('0:00.000'); - }); - - it('formats sub-second durations with leading zero minutes/seconds', () => { - expect(formatDuration(123)).toBe('0:00.123'); - }); - - it('formats single-digit seconds with a leading zero', () => { - expect(formatDuration(5_678)).toBe('0:05.678'); - }); - - it('formats the spec example (83.193s) as 1:23.193', () => { - expect(formatDuration(83_193)).toBe('1:23.193'); - }); - - it('formats a long duration past 10 minutes', () => { - expect(formatDuration(754_567)).toBe('12:34.567'); - }); - - it('pads milliseconds to three digits', () => { - expect(formatDuration(60_007)).toBe('1:00.007'); - }); -}); diff --git a/tests/testSummary.test.js b/tests/testSummary.test.js index 6298dd0..e7d1690 100644 --- a/tests/testSummary.test.js +++ b/tests/testSummary.test.js @@ -1,144 +1,127 @@ import { describe, it, expect } from 'vitest'; -import { formatTestSummary } from '../src/testSummary.js'; -import { formatFailedTestsBlock } from '../src/testSummary.js'; +import { formatRunComplete } from '../src/testSummary.js'; -const stripAnsi = (s) => s.replace(/\x1b\[[0-9;]*m/g, ''); +const handlers = [ + { id: 's1', name: 'Login', type: 'suite' }, + { id: 't1', name: 'shows error on wrong password', parent: 's1', type: 'test' }, + { id: 't2', name: 'redirects on success', parent: 's1', type: 'test' }, + { id: 't3', name: 'validates email', parent: 's1', type: 'test' }, +]; -describe('formatTestSummary', () => { - it('formats an all-pass run', () => { - const line = formatTestSummary({ +describe('formatRunComplete', () => { + it('formats an all-pass run as the three-line block', () => { + const block = formatRunComplete({ testStatus: [ - { id: '1', status: 'pass' }, - { id: '2', status: 'pass' }, - { id: '3', status: 'pass' }, + { id: 't1', status: 'pass' }, + { id: 't2', status: 'pass' }, ], - durationMs: 1234, + handlers, + durationMs: 4200, }); - expect(stripAnsi(line)).toBe('Tests: 3 passed, 0 failed, 0 skipped (3 total) in 0:01.234'); + expect(block).toBe( + '--- Run complete ---\n' + + ' Passed: 2 | Failed: 0 | Skipped: 0\n' + + ' Duration: 4.2s' + ); }); - it('formats a mixed run', () => { - const line = formatTestSummary({ + it('counts skipped tests', () => { + const block = formatRunComplete({ testStatus: [ - { id: '1', status: 'pass' }, - { id: '2', status: 'fail' }, - { id: '3', status: 'skip' }, + { id: 't1', status: 'pass' }, + { id: 't2', status: 'skip' }, ], - durationMs: 83_193, - }); - expect(stripAnsi(line)).toBe('Tests: 1 passed, 1 failed, 1 skipped (3 total) in 1:23.193'); - }); - - it('formats an empty run', () => { - const line = formatTestSummary({ testStatus: [], durationMs: 0 }); - expect(stripAnsi(line)).toBe('Tests: 0 passed, 0 failed, 0 skipped (0 total) in 0:00.000'); - }); - - it('keeps the "Tests:" label uncolored so grep "^Tests:" matches', () => { - const line = formatTestSummary({ - testStatus: [{ id: '1', status: 'pass' }], + handlers, durationMs: 1000, }); - expect(line.startsWith('Tests:')).toBe(true); + expect(block).toContain(' Passed: 1 | Failed: 0 | Skipped: 1'); }); - it('keeps the words "passed", "failed", "skipped" uncolored', () => { - const line = formatTestSummary({ + it('appends the failure block with suite path and indented error', () => { + const block = formatRunComplete({ testStatus: [ - { id: '1', status: 'pass' }, - { id: '2', status: 'fail' }, + { id: 't1', status: 'fail', error: 'Expected element to be visible (at http://localhost:5173/login)' }, + { id: 't2', status: 'pass' }, + { id: 't3', status: 'fail', error: 'Timeout waiting for selector ".error"' }, ], - durationMs: 1000, + handlers, + durationMs: 4200, }); - expect(line).toContain('passed'); - expect(line).toContain('failed'); - expect(/\x1b\[[0-9;]*m(passed|failed|skipped)/.test(line)).toBe(false); - expect(/(passed|failed|skipped)\x1b\[[0-9;]*m/.test(line)).toBe(false); + expect(block).toBe( + '--- Run complete ---\n' + + ' Passed: 1 | Failed: 2 | Skipped: 0\n' + + ' Duration: 4.2s\n' + + '\n' + + ' Failed tests (2):\n' + + ' × Login > shows error on wrong password\n' + + ' Expected element to be visible (at http://localhost:5173/login)\n' + + ' × Login > validates email\n' + + ' Timeout waiting for selector ".error"' + ); }); - it('colors the passed count green', () => { - const line = formatTestSummary({ - testStatus: [{ id: '1', status: 'pass' }], - durationMs: 1000, + it('indents multi-line error messages to align under the test line', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'fail', error: 'line one\nline two' }], + handlers, + durationMs: 500, }); - expect(line).toMatch(/\x1b\[32m1\x1b\[0m passed/); + expect(block).toContain(' line one\n line two'); }); - it('colors the failed count red only when > 0', () => { - const lineWithFailures = formatTestSummary({ - testStatus: [ - { id: '1', status: 'pass' }, - { id: '2', status: 'fail' }, - ], - durationMs: 1000, + it('falls back to the test id when no handler matches', () => { + const block = formatRunComplete({ + testStatus: [{ id: 'ghost-id', status: 'fail', error: 'boom' }], + handlers: [], + durationMs: 500, }); - expect(lineWithFailures).toMatch(/\x1b\[31m1\x1b\[0m failed/); + expect(block).toContain(' × ghost-id'); + }); - const lineNoFailures = formatTestSummary({ - testStatus: [{ id: '1', status: 'pass' }], - durationMs: 1000, + it('omits the failure block when everything passes', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'pass' }], + handlers, + durationMs: 500, }); - expect(lineNoFailures).not.toMatch(/\x1b\[31m0\x1b\[0m failed/); - expect(lineNoFailures).toContain('0 failed'); + expect(block).not.toContain('Failed tests'); }); - it('colors the skipped count yellow only when > 0', () => { - const lineWithSkips = formatTestSummary({ + it('appends the retried block for tests that passed on retry', () => { + const block = formatRunComplete({ testStatus: [ - { id: '1', status: 'pass' }, - { id: '2', status: 'skip' }, + { id: 't1', status: 'pass', retryAttempt: 2 }, + { id: 't2', status: 'pass' }, ], - durationMs: 1000, - }); - expect(lineWithSkips).toMatch(/\x1b\[33m1\x1b\[0m skipped/); - - const lineNoSkips = formatTestSummary({ - testStatus: [{ id: '1', status: 'pass' }], - durationMs: 1000, + handlers, + durationMs: 500, }); - expect(lineNoSkips).not.toMatch(/\x1b\[33m0\x1b\[0m skipped/); - expect(lineNoSkips).toContain('0 skipped'); + expect(block).toContain( + '\n' + + ' Retried (1):\n' + + ' ✓ Login > shows error on wrong password (passed on attempt 2)' + ); }); -}); -describe('formatFailedTestsBlock', () => { - it('returns null when there are no failures', () => { - const block = formatFailedTestsBlock({ - testStatus: [{ id: '1', status: 'pass' }], - handlers: [{ id: '1', name: 'test1', type: 'test' }], + it('omits the retried block when no test was retried', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'pass' }], + handlers, + durationMs: 500, }); - expect(block).toBeNull(); - }); - - it('returns null on an empty run', () => { - expect(formatFailedTestsBlock({ testStatus: [], handlers: [] })).toBeNull(); + expect(block).not.toContain('Retried'); }); - it('lists each failed test by name', () => { - const block = formatFailedTestsBlock({ + it('contains no ANSI escape codes', () => { + const block = formatRunComplete({ testStatus: [ - { id: '1', status: 'pass' }, - { id: '2', status: 'fail', error: 'boom' }, - { id: '3', status: 'fail', error: 'kaboom' }, + { id: 't1', status: 'fail', error: 'boom' }, + { id: 't2', status: 'pass', retryAttempt: 2 }, + { id: 't3', status: 'skip' }, ], - handlers: [ - { id: '1', name: 'should render', type: 'test' }, - { id: '2', name: 'should submit form', type: 'test' }, - { id: '3', name: 'should show error', type: 'test' }, - ], - }); - const stripped = stripAnsi(block); - expect(stripped).toContain('Failed tests:'); - expect(stripped).toContain('should submit form'); - expect(stripped).toContain('should show error'); - expect(stripped.indexOf('should submit form')).toBeLessThan(stripped.indexOf('should show error')); - }); - - it('falls back to the test id when no matching handler is found', () => { - const block = formatFailedTestsBlock({ - testStatus: [{ id: 'ghost-id', status: 'fail', error: 'boom' }], - handlers: [], + handlers, + durationMs: 500, }); - expect(stripAnsi(block)).toContain('ghost-id'); + expect(/\x1b\[[0-9;]*m/.test(block)).toBe(false); }); }); From b95292b7b27b0f1b3ab9b039c04f3a37d3c527a8 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 7 Jul 2026 10:18:46 +0200 Subject: [PATCH 04/11] feat: add explainError diagnostics for known infrastructure failures --- src/diagnostics.js | 47 ++++++++++++++++++++ tests/diagnostics.test.js | 92 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 139 insertions(+) create mode 100644 src/diagnostics.js create mode 100644 tests/diagnostics.test.js diff --git a/src/diagnostics.js b/src/diagnostics.js new file mode 100644 index 0000000..7cdcd9c --- /dev/null +++ b/src/diagnostics.js @@ -0,0 +1,47 @@ +const NET_ERRORS = ['ERR_CONNECTION_REFUSED', 'ERR_NAME_NOT_RESOLVED', 'ERR_ADDRESS_UNREACHABLE']; + +export function isProtocolTimeout(error) { + const message = error && error.message ? error.message : ''; + return ( + (error && error.name === 'ProtocolError' && /timed out/i.test(message)) || + /protocolTimeout/i.test(message) + ); +} + +export function explainError(error, config = {}) { + if (!error || typeof error.message !== 'string') return null; + const message = error.message; + + const netMatch = message.match(/net::(ERR_[A-Z_]+)/); + if (netMatch && NET_ERRORS.includes(netMatch[1])) { + return ( + `Could not reach ${config.url} (${netMatch[1]}).\n` + + 'Is your dev server running? Start it (e.g. `npm run dev`) or fix "url" in twd.config.json.' + ); + } + + if (error.name === 'TimeoutError' && message.includes('#twd-sidebar-root')) { + return ( + `Page loaded but the TWD sidebar (#twd-sidebar-root) did not appear within ${config.timeout}ms.\n` + + 'Ensure twd-js is initialized in your app and your tests are registered.\n' + + 'If the app is slow to start, raise "timeout" in twd.config.json.' + ); + } + + if (isProtocolTimeout(error)) { + return ( + 'This looks like a Puppeteer protocolTimeout. The whole test suite runs in a single\n' + + 'page.evaluate call, so the run aborts if it takes longer than "protocolTimeout" (ms).\n' + + 'Raise it in twd.config.json, e.g. { "protocolTimeout": 600000 } (0 = no timeout).' + ); + } + + if (/Could not find Chrome|Failed to launch the browser process/.test(message)) { + return ( + 'Puppeteer could not launch Chrome.\n' + + 'Run `npx puppeteer browsers install chrome`, or adjust "puppeteerArgs" in twd.config.json.' + ); + } + + return null; +} diff --git a/tests/diagnostics.test.js b/tests/diagnostics.test.js new file mode 100644 index 0000000..02c507a --- /dev/null +++ b/tests/diagnostics.test.js @@ -0,0 +1,92 @@ +import { describe, it, expect } from 'vitest'; +import { explainError, isProtocolTimeout } from '../src/diagnostics.js'; + +const config = { url: 'http://localhost:5173', timeout: 10000 }; + +describe('explainError', () => { + it('explains connection refused with the configured url', () => { + const err = new Error('net::ERR_CONNECTION_REFUSED at http://localhost:5173'); + const msg = explainError(err, config); + expect(msg).toContain('Could not reach http://localhost:5173 (ERR_CONNECTION_REFUSED)'); + expect(msg).toContain('Is your dev server running?'); + expect(msg).toContain('"url" in twd.config.json'); + }); + + it('explains DNS resolution failures', () => { + const err = new Error('net::ERR_NAME_NOT_RESOLVED at http://myapp.local:5173'); + const msg = explainError(err, { ...config, url: 'http://myapp.local:5173' }); + expect(msg).toContain('Could not reach http://myapp.local:5173 (ERR_NAME_NOT_RESOLVED)'); + }); + + it('explains unreachable-address failures', () => { + const err = new Error('net::ERR_ADDRESS_UNREACHABLE at http://10.0.0.9:5173'); + expect(explainError(err, config)).toContain('(ERR_ADDRESS_UNREACHABLE)'); + }); + + it('explains the sidebar selector timeout with the configured timeout', () => { + const err = new Error( + "Waiting for selector '#twd-sidebar-root' failed: Waiting failed: 10000ms exceeded" + ); + err.name = 'TimeoutError'; + const msg = explainError(err, config); + expect(msg).toContain('TWD sidebar (#twd-sidebar-root) did not appear within 10000ms'); + expect(msg).toContain('Ensure twd-js is initialized'); + expect(msg).toContain('raise "timeout" in twd.config.json'); + }); + + it('does not claim a sidebar problem for unrelated TimeoutErrors', () => { + const err = new Error('Waiting for selector ".other-thing" failed'); + err.name = 'TimeoutError'; + expect(explainError(err, config)).toBeNull(); + }); + + it('explains protocol timeouts', () => { + const err = new Error('Runtime.callFunctionOn timed out.'); + err.name = 'ProtocolError'; + const msg = explainError(err, config); + expect(msg).toContain('protocolTimeout'); + expect(msg).toContain('twd.config.json'); + }); + + it('explains a missing Chrome install', () => { + const err = new Error('Could not find Chrome (ver. 131.0.6778.204).'); + const msg = explainError(err, config); + expect(msg).toContain('Puppeteer could not launch Chrome'); + expect(msg).toContain('npx puppeteer browsers install chrome'); + }); + + it('explains a browser process launch failure', () => { + const err = new Error('Failed to launch the browser process!\nspawn ENOENT'); + const msg = explainError(err, config); + expect(msg).toContain('Puppeteer could not launch Chrome'); + expect(msg).toContain('"puppeteerArgs" in twd.config.json'); + }); + + it('returns null for unknown errors', () => { + expect(explainError(new Error('something else entirely'), config)).toBeNull(); + }); + + it('tolerates a missing config and non-Error values', () => { + const err = new Error('net::ERR_CONNECTION_REFUSED at http://localhost:5173'); + expect(explainError(err)).toContain('Could not reach undefined (ERR_CONNECTION_REFUSED)'); + expect(explainError(null, config)).toBeNull(); + expect(explainError('boom', config)).toBeNull(); + }); +}); + +describe('isProtocolTimeout', () => { + it('matches ProtocolError timeouts', () => { + const err = new Error('Runtime.callFunctionOn timed out.'); + err.name = 'ProtocolError'; + expect(isProtocolTimeout(err)).toBe(true); + }); + + it('matches messages that mention protocolTimeout', () => { + expect(isProtocolTimeout(new Error('Increase the protocolTimeout setting'))).toBe(true); + }); + + it('rejects unrelated errors', () => { + expect(isProtocolTimeout(new Error('boom'))).toBe(false); + expect(isProtocolTimeout(null)).toBe(false); + }); +}); From 5680d966fe6434d92494c9656eb2e51366c8b14b Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 7 Jul 2026 10:24:28 +0200 Subject: [PATCH 05/11] feat: relay-style run output, actionable error diagnostics, no silent exits --- bin/twd-cli.js | 3 + src/index.js | 90 +++++++++++------------------- tests/runTests.test.js | 123 ++++++++++++++++++++++++++--------------- 3 files changed, 112 insertions(+), 104 deletions(-) diff --git a/bin/twd-cli.js b/bin/twd-cli.js index 914fbd8..9d6da54 100755 --- a/bin/twd-cli.js +++ b/bin/twd-cli.js @@ -11,6 +11,9 @@ if (command === 'run') { const hasFailures = await runTests({ testFilters }); process.exit(hasFailures ? 1 : 0); } catch (error) { + if (!error?.reported) { + console.error(error?.message ?? String(error)); + } process.exit(1); } } else { diff --git a/src/index.js b/src/index.js index da40a61..3f3428b 100644 --- a/src/index.js +++ b/src/index.js @@ -1,33 +1,23 @@ import fs from 'fs'; import path from 'path'; import puppeteer from 'puppeteer'; -import { reportResults } from 'twd-js/runner-ci'; import { loadConfig } from './config.js'; import { loadContracts, validateMocks } from './contracts.js'; import { printContractReport } from './contractReport.js'; import { generateContractMarkdown } from './contractMarkdown.js'; import { buildTestPath } from './buildTestPath.js'; -import { formatTestSummary, formatFailedTestsBlock } from './testSummary.js'; +import { formatRunComplete } from './testSummary.js'; import { selectTestIds } from './filterTests.js'; - -function isProtocolTimeout(error) { - const message = error && error.message ? error.message : ''; - return ( - (error && error.name === 'ProtocolError' && /timed out/i.test(message)) || - /protocolTimeout/i.test(message) - ); -} +import { explainError } from './diagnostics.js'; export async function runTests(options = {}) { const { testFilters = [] } = options; let browser; + let config; try { - const config = loadConfig(); + config = loadConfig(); const workingDir = process.cwd(); - console.log('Starting TWD test runner...'); - console.log('Configuration:', JSON.stringify(config, null, 2)); - // Load contract validators if configured let contractValidators = []; if (config.contracts && config.contracts.length > 0) { @@ -63,22 +53,22 @@ export async function runTests(options = {}) { // Wait for the selector to be available await page.waitForSelector('#twd-sidebar-root', { timeout: config.timeout }); - console.log('Page loaded. Starting tests...'); + + // Enumerate registered handlers (for the count line and --test filtering) + const registeredHandlers = await page.evaluate(() => { + const state = window.__TWD_STATE__; + if (!state || !state.handlers) return []; + return Array.from(state.handlers.values()).map((h) => ({ + id: h.id, + name: h.name, + parent: h.parent, + type: h.type, + })); + }); // Resolve --test filters to a concrete set of test ids (null = run all) let selectedIds = null; if (testFilters.length > 0) { - const registeredHandlers = await page.evaluate(() => { - const state = window.__TWD_STATE__; - if (!state || !state.handlers) return []; - return Array.from(state.handlers.values()).map((h) => ({ - id: h.id, - name: h.name, - parent: h.parent, - type: h.type, - })); - }); - const { ids, unmatchedFilters } = selectTestIds(registeredHandlers, testFilters); if (ids.length === 0) { @@ -97,6 +87,9 @@ export async function runTests(options = {}) { selectedIds = ids; console.log(`Filtering: running ${ids.length} test(s) matching --test filter(s).`); + } else { + const testCount = registeredHandlers.filter((h) => h.type !== 'suite').length; + console.log(`Running ${testCount} test(s)...`); } // Execute all tests @@ -130,23 +123,6 @@ export async function runTests(options = {}) { const durationMs = Date.now() - startedAt; - console.log(`Tests to report: ${testStatus.length}`); - - // Display results in console - reportResults(handlers, testStatus); - - // Display retry summary if any tests were retried - const retriedTests = testStatus.filter(t => t.retryAttempt >= 2); - if (retriedTests.length > 0) { - console.log('\n⟳ Retried tests:'); - for (const t of retriedTests) { - const handler = handlers.find(h => h.id === t.id); - const name = handler ? handler.name : t.id; - console.log(` ✓ ${name} (passed on attempt ${t.retryAttempt})`); - } - console.log(` ${retriedTests.length} test(s) required retries to pass.`); - } - // Exit with appropriate code let hasFailures = testStatus.some(test => test.status === 'fail'); @@ -188,7 +164,6 @@ export async function runTests(options = {}) { if (config.coverage && !hasFailures && !selectedIds) { const coverage = await page.evaluate(() => window.__coverage__); if (coverage) { - console.log('Collecting code coverage data...'); const coverageDir = path.resolve(workingDir, config.coverageDir); const nycDir = path.resolve(workingDir, config.nycOutputDir); @@ -208,27 +183,24 @@ export async function runTests(options = {}) { } await browser.close(); - console.log('Browser closed.'); + // The run-complete block is always the last output of a completed run console.log(''); - console.log(formatTestSummary({ testStatus, durationMs })); - const failedBlock = formatFailedTestsBlock({ testStatus, handlers }); - if (failedBlock) { - for (const line of failedBlock.split('\n')) { - console.log(line); - } - } + console.log(formatRunComplete({ testStatus, handlers, durationMs })); return hasFailures; } catch (error) { - console.error('Error running tests:', error); - if (isProtocolTimeout(error)) { - console.error( - '\nThis looks like a Puppeteer protocolTimeout. The whole test suite runs in a single\n' + - 'page.evaluate call, so the run aborts if it takes longer than "protocolTimeout" (ms).\n' + - 'Raise it in twd.config.json, e.g. { "protocolTimeout": 600000 } (0 = no timeout).' - ); + const message = error && error.message ? error.message : String(error); + console.error(`Error running tests: ${message}`); + const diagnostic = explainError(error, config); + if (diagnostic) { + console.error(`\n${diagnostic}`); + } else if (error && error.stack) { + console.error(`\n${error.stack}`); + } + if (error && typeof error === 'object') { + error.reported = true; } if (browser) await browser.close(); throw error; diff --git a/tests/runTests.test.js b/tests/runTests.test.js index d7f9c59..32f0f7d 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -3,9 +3,6 @@ import { runTests } from "../src/index.js"; vi.mock('fs'); vi.mock('puppeteer'); -vi.mock('twd-js/runner-ci', () => ({ - reportResults: vi.fn(), -})); vi.mock('../src/config.js', () => ({ loadConfig: vi.fn(), })); @@ -19,7 +16,6 @@ vi.mock('../src/contractReport.js', () => ({ import fs from 'fs'; import puppeteer from 'puppeteer'; -import { reportResults } from 'twd-js/runner-ci'; import { loadConfig } from '../src/config.js'; import { loadContracts, validateMocks } from '../src/contracts.js'; import { printContractReport } from '../src/contractReport.js'; @@ -28,7 +24,9 @@ function createMockPage(evaluateResult) { return { goto: vi.fn(), waitForSelector: vi.fn(), - evaluate: vi.fn().mockResolvedValue(evaluateResult), + evaluate: vi.fn() + .mockResolvedValueOnce(evaluateResult.handlers ?? []) // enumeration pass + .mockResolvedValue(evaluateResult), // run pass (+ coverage) exposeFunction: vi.fn(), }; } @@ -116,7 +114,7 @@ describe("runTests", () => { errorSpy.mockRestore(); }); - it("should log retry summary when tests were retried", async () => { + it("should include retried tests in the run-complete block", async () => { const testStatus = [ { id: '1', status: 'pass', retryAttempt: 2 }, { id: '2', status: 'pass' }, @@ -131,13 +129,14 @@ describe("runTests", () => { await runTests(); - const logs = consoleSpy.mock.calls.map(c => c[0]); - expect(logs).toContain('\n⟳ Retried tests:'); - expect(logs.some(l => l.includes('flaky test') && l.includes('attempt 2'))).toBe(true); - expect(logs.some(l => l.includes('1 test(s) required retries'))).toBe(true); + const logs = consoleSpy.mock.calls.map(c => String(c[0])); + const block = logs.find(l => l.startsWith('--- Run complete ---')); + expect(block).toBeDefined(); + expect(block).toContain('Retried (1):'); + expect(block).toContain('✓ flaky test (passed on attempt 2)'); }); - it("should not log retry summary when no tests were retried", async () => { + it("should not include a retried section when no tests were retried", async () => { const testStatus = [{ id: '1', status: 'pass' }]; const handlers = [{ id: '1', name: 'test1', type: 'test' }]; const page = createMockPage({ handlers, testStatus }); @@ -146,8 +145,10 @@ describe("runTests", () => { await runTests(); - const logs = consoleSpy.mock.calls.map(c => c[0]); - expect(logs).not.toContain('\n⟳ Retried tests:'); + const logs = consoleSpy.mock.calls.map(c => String(c[0])); + const block = logs.find(l => l.startsWith('--- Run complete ---')); + expect(block).toBeDefined(); + expect(block).not.toContain('Retried'); }); it("should return true when tests have failures", async () => { @@ -233,25 +234,27 @@ describe("runTests", () => { goto: vi.fn(), waitForSelector: vi.fn(), exposeFunction: vi.fn(), - // Drive the registered __twdCollectMock callback from inside page.evaluate, - // mirroring how a real browser test would trigger it. - evaluate: vi.fn().mockImplementation(async () => { - const exposed = page.exposeFunction.mock.calls.find( - (c) => c[0] === '__twdCollectMock' - ); - expect(exposed).toBeDefined(); - const collectMock = exposed[1]; - await collectMock({ - alias: 'getPhoto', - url: '/v1/photo', - method: 'GET', - status: 200, - response: 'bin', - testId: 't-1', - responseHeaders: { 'Content-Type': 'image/png' }, - }); - return { handlers, testStatus }; - }), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) // enumeration pass + // Drive the registered __twdCollectMock callback from inside page.evaluate, + // mirroring how a real browser test would trigger it. + .mockImplementation(async () => { + const exposed = page.exposeFunction.mock.calls.find( + (c) => c[0] === '__twdCollectMock' + ); + expect(exposed).toBeDefined(); + const collectMock = exposed[1]; + await collectMock({ + alias: 'getPhoto', + url: '/v1/photo', + method: 'GET', + status: 200, + response: 'bin', + testId: 't-1', + responseHeaders: { 'Content-Type': 'image/png' }, + }); + return { handlers, testStatus }; + }), }; const browser = createMockBrowser(page); vi.mocked(puppeteer.launch).mockResolvedValue(browser); @@ -394,16 +397,17 @@ describe("runTests", () => { expect(fs.writeFileSync).not.toHaveBeenCalled(); }); - it("should print the Tests: summary line and Failed tests block", async () => { + it("prints the run-complete block last, with failure paths and errors", async () => { const testStatus = [ { id: '1', status: 'pass' }, - { id: '2', status: 'fail', error: 'boom' }, + { id: '2', status: 'fail', error: 'boom (at http://localhost:5173/form)' }, { id: '3', status: 'skip' }, ]; const handlers = [ - { id: '1', name: 'should render', type: 'test' }, - { id: '2', name: 'should submit form', type: 'test' }, - { id: '3', name: 'should show error', type: 'test' }, + { id: 's1', name: 'Form', type: 'suite' }, + { id: '1', name: 'should render', parent: 's1', type: 'test' }, + { id: '2', name: 'should submit form', parent: 's1', type: 'test' }, + { id: '3', name: 'should show error', parent: 's1', type: 'test' }, ]; const page = createMockPage({ handlers, testStatus }); const browser = createMockBrowser(page); @@ -411,15 +415,44 @@ describe("runTests", () => { await runTests(); - const stripAnsi = (s) => s.replace(/\x1b\[[0-9;]*m/g, ''); - const logs = consoleSpy.mock.calls.map((c) => stripAnsi(String(c[0]))); + const logs = consoleSpy.mock.calls.map((c) => String(c[0])); + const block = logs[logs.length - 1]; + expect(block.startsWith('--- Run complete ---')).toBe(true); + expect(block).toContain('Passed: 1 | Failed: 1 | Skipped: 1'); + expect(block).toContain('× Form > should submit form'); + expect(block).toContain('boom (at http://localhost:5173/form)'); + }); + + it("prints no config dump and no per-test tree chatter", async () => { + const testStatus = [{ id: '1', status: 'pass' }]; + const handlers = [{ id: '1', name: 'test1', type: 'test' }]; + const page = createMockPage({ handlers, testStatus }); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); - const summaryLine = logs.find((l) => l.startsWith('Tests:')); - expect(summaryLine).toBeDefined(); - expect(summaryLine).toMatch(/^Tests: 1 passed, 1 failed, 1 skipped \(3 total\) in \d+:\d{2}\.\d{3}$/); + await runTests(); - const failedHeader = logs.find((l) => l === 'Failed tests:'); - expect(failedHeader).toBeDefined(); - expect(logs.some((l) => l.includes('should submit form'))).toBe(true); + const logs = consoleSpy.mock.calls.map((c) => String(c[0])); + expect(logs.some((l) => l.startsWith('Configuration:'))).toBe(false); + expect(logs.some((l) => l.startsWith('Starting TWD test runner'))).toBe(false); + expect(logs.some((l) => l.startsWith('Tests to report'))).toBe(false); + expect(logs.some((l) => l.startsWith('Browser closed'))).toBe(false); + expect(logs.some((l) => l === 'Running 1 test(s)...')).toBe(true); + }); + + it("marks rethrown errors as reported", async () => { + const page = createMockPage({ handlers: [], testStatus: [] }); + const bootError = new Error('net::ERR_CONNECTION_REFUSED at http://localhost:5173'); + page.goto = vi.fn().mockRejectedValue(bootError); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + + await expect(runTests()).rejects.toThrow('ERR_CONNECTION_REFUSED'); + + expect(bootError.reported).toBe(true); + const errors = errorSpy.mock.calls.map((c) => String(c[0])); + expect(errors.some((e) => e.includes('Is your dev server running?'))).toBe(true); + errorSpy.mockRestore(); }); }); From 2fee382401455f2b402353f26eeaab63ec0e0eda Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 7 Jul 2026 10:29:58 +0200 Subject: [PATCH 06/11] chore: drop twd-js dependency (reportResults no longer used) --- package-lock.json | 250 +++++++--------------------------------------- package.json | 3 +- 2 files changed, 39 insertions(+), 214 deletions(-) diff --git a/package-lock.json b/package-lock.json index 4cce1a4..915a14d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -10,8 +10,7 @@ "license": "ISC", "dependencies": { "openapi-mock-validator": "^0.3.0", - "puppeteer": "^25.3.0", - "twd-js": "^1.8.2" + "puppeteer": "^25.3.0" }, "bin": { "twd-cli": "bin/twd-cli.js" @@ -40,26 +39,6 @@ "@types/json-schema": "^7.0.15" } }, - "node_modules/@babel/code-frame": { - "version": "7.29.0", - "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.0.tgz", - "integrity": "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw==", - "license": "MIT", - "dependencies": { - "@babel/helper-validator-identifier": "^7.28.5", - "js-tokens": "^4.0.0", - "picocolors": "^1.1.1" - }, - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/code-frame/node_modules/js-tokens": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz", - "integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==", - "license": "MIT" - }, "node_modules/@babel/helper-string-parser": { "version": "7.27.1", "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.27.1.tgz", @@ -74,6 +53,7 @@ "version": "7.28.5", "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.28.5.tgz", "integrity": "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==", + "dev": true, "license": "MIT", "engines": { "node": ">=6.9.0" @@ -95,15 +75,6 @@ "node": ">=6.0.0" } }, - "node_modules/@babel/runtime": { - "version": "7.29.2", - "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.2.tgz", - "integrity": "sha512-JiDShH45zKHWyGe4ZNVRrCjBz8Nh9TMmZG1kh4QTK8hCBTWBi8Da+i7s1fJw7/lYpM4ccepSNfqzZ/QvABBi5g==", - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, "node_modules/@babel/types": { "version": "7.29.0", "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.0.tgz", @@ -166,24 +137,26 @@ } }, "node_modules/@emnapi/core": { - "version": "1.11.1", - "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz", - "integrity": "sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==", + "version": "1.11.2", + "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.2.tgz", + "integrity": "sha512-TC8MkTuZUtcTSiFeuC0ksCh9QIJ5+F21MvZ4Wn4ORfYaFJ/0dsiudv5tVkejgwZlwQ39jL9WWDe2lz8x0WglOA==", "dev": true, "license": "MIT", "optional": true, + "peer": true, "dependencies": { "@emnapi/wasi-threads": "1.2.2", "tslib": "^2.4.0" } }, "node_modules/@emnapi/runtime": { - "version": "1.11.1", - "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.1.tgz", - "integrity": "sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==", + "version": "1.11.2", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.2.tgz", + "integrity": "sha512-kyOl3X0DuTiT1h2ft8r2fYO8JYtU9a9Xis/zBSiGArNaagCOWx90N1k2wxp18czFDH+OgcWGb5ZP/XMt3dcyPA==", "dev": true, "license": "MIT", "optional": true, + "peer": true, "dependencies": { "tslib": "^2.4.0" } @@ -507,6 +480,29 @@ "node": "^20.19.0 || >=22.12.0" } }, + "node_modules/@rolldown/binding-wasm32-wasi/node_modules/@emnapi/core": { + "version": "1.11.1", + "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz", + "integrity": "sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/wasi-threads": "1.2.2", + "tslib": "^2.4.0" + } + }, + "node_modules/@rolldown/binding-wasm32-wasi/node_modules/@emnapi/runtime": { + "version": "1.11.1", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.1.tgz", + "integrity": "sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, "node_modules/@rolldown/binding-win32-arm64-msvc": { "version": "1.1.4", "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.1.4.tgz", @@ -614,38 +610,6 @@ "dev": true, "license": "MIT" }, - "node_modules/@testing-library/dom": { - "version": "10.4.1", - "resolved": "https://registry.npmjs.org/@testing-library/dom/-/dom-10.4.1.tgz", - "integrity": "sha512-o4PXJQidqJl82ckFaXUeoAW+XysPLauYI43Abki5hABd853iMhitooc6znOnczgbTYmEP6U6/y1ZyKAIsvMKGg==", - "license": "MIT", - "dependencies": { - "@babel/code-frame": "^7.10.4", - "@babel/runtime": "^7.12.5", - "@types/aria-query": "^5.0.1", - "aria-query": "5.3.0", - "dom-accessibility-api": "^0.5.9", - "lz-string": "^1.5.0", - "picocolors": "1.1.1", - "pretty-format": "^27.0.2" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/@testing-library/user-event": { - "version": "14.6.1", - "resolved": "https://registry.npmjs.org/@testing-library/user-event/-/user-event-14.6.1.tgz", - "integrity": "sha512-vq7fv0rnt+QTXgPxr5Hjc210p6YKq2kmdziLgnsZGgLJ9e6VAShx1pACLuRjd/AS/sr7phAR58OIIpf0LlmQNw==", - "license": "MIT", - "engines": { - "node": ">=12", - "npm": ">=6" - }, - "peerDependencies": { - "@testing-library/dom": ">=7.21.4" - } - }, "node_modules/@tybys/wasm-util": { "version": "0.10.3", "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.3.tgz", @@ -657,16 +621,11 @@ "tslib": "^2.4.0" } }, - "node_modules/@types/aria-query": { - "version": "5.0.4", - "resolved": "https://registry.npmjs.org/@types/aria-query/-/aria-query-5.0.4.tgz", - "integrity": "sha512-rfT93uj5s0PRL7EzccGMs3brplhcrghnDoV26NqKhCAS1hVo+WdNsPvE/yb6ilfr5hi2MEk6d5EWJTKdxg8jVw==", - "license": "MIT" - }, "node_modules/@types/chai": { "version": "5.2.3", "resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz", "integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==", + "dev": true, "license": "MIT", "dependencies": { "@types/deep-eql": "*", @@ -677,6 +636,7 @@ "version": "4.0.2", "resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz", "integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==", + "dev": true, "license": "MIT" }, "node_modules/@types/estree": { @@ -870,46 +830,17 @@ } } }, - "node_modules/ansi-regex": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", - "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", - "license": "MIT", - "engines": { - "node": ">=8" - } - }, - "node_modules/ansi-styles": { - "version": "5.2.0", - "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-5.2.0.tgz", - "integrity": "sha512-Cxwpt2SfTzTtXcfOlzGEee8O+c+MmUgGrNiBcXnuWxuFJHe6a5Hz7qwhwe5OgaSYI0IJvkLqWX1ASG+cJOkEiA==", - "license": "MIT", - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/chalk/ansi-styles?sponsor=1" - } - }, "node_modules/argparse": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz", "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==", "license": "Python-2.0" }, - "node_modules/aria-query": { - "version": "5.3.0", - "resolved": "https://registry.npmjs.org/aria-query/-/aria-query-5.3.0.tgz", - "integrity": "sha512-b0P0sZPKtyu8HkeRAfCq0IfURZK+SuwMjY1UXGBU27wpAiTwQAIlq56IbIO+ytk/JjS1fMR14ee5WBBfKi5J6A==", - "license": "Apache-2.0", - "dependencies": { - "dequal": "^2.0.3" - } - }, "node_modules/assertion-error": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==", + "dev": true, "license": "MIT", "engines": { "node": ">=12" @@ -931,6 +862,7 @@ "version": "6.2.2", "resolved": "https://registry.npmjs.org/chai/-/chai-6.2.2.tgz", "integrity": "sha512-NUPRluOfOiTKBKvWPtSD4PhFvWCqOi0BGStNWs57X9js7XGTprSmFoz5F0tWhR4WPjNeR9jXqdC7/UpSJTnlRg==", + "dev": true, "license": "MIT", "engines": { "node": ">=18" @@ -1053,15 +985,6 @@ "dev": true, "license": "MIT" }, - "node_modules/dequal": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz", - "integrity": "sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==", - "license": "MIT", - "engines": { - "node": ">=6" - } - }, "node_modules/detect-libc": { "version": "2.1.2", "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", @@ -1078,12 +1001,6 @@ "integrity": "sha512-mXwg4Fqnv0WR4iuAT/gYUmctNkjILwXFHyZ+m7Ty1dfr0ezZt2U3gnrrJTfRobJTHoXf+IbuFvFITzLrLFjwJA==", "license": "BSD-3-Clause" }, - "node_modules/dom-accessibility-api": { - "version": "0.5.16", - "resolved": "https://registry.npmjs.org/dom-accessibility-api/-/dom-accessibility-api-0.5.16.tgz", - "integrity": "sha512-X7BJ2yElsnOJ30pZF4uIIDfBEVgF4XEBxL9Bxhy6dnrm5hkzqmsWHGTiHqRiITNhMyFLyAiWndIJP7Z1NTteDg==", - "license": "MIT" - }, "node_modules/emoji-regex": { "version": "10.6.0", "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-10.6.0.tgz", @@ -1576,15 +1493,6 @@ "url": "https://github.com/sponsors/antonk52" } }, - "node_modules/lz-string": { - "version": "1.5.0", - "resolved": "https://registry.npmjs.org/lz-string/-/lz-string-1.5.0.tgz", - "integrity": "sha512-h5bgJWpxJNswbU7qCrV0tIKQCaS3blPDrqKWx+QxzuzL1zGUzij9XCWLrSLsJPu5t+eWA/ycetzYAO5IOMcWAQ==", - "license": "MIT", - "bin": { - "lz-string": "bin/bin.js" - } - }, "node_modules/magic-string": { "version": "0.30.21", "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", @@ -1720,6 +1628,7 @@ "version": "1.1.1", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, "license": "ISC" }, "node_modules/picomatch": { @@ -1764,30 +1673,6 @@ "node": "^10 || ^12 || >=14" } }, - "node_modules/preact": { - "version": "10.29.1", - "resolved": "https://registry.npmjs.org/preact/-/preact-10.29.1.tgz", - "integrity": "sha512-gQCLc/vWroE8lIpleXtdJhTFDogTdZG9AjMUpVkDf2iTCNwYNWA+u16dL41TqUDJO4gm2IgrcMv3uTpjd4Pwmg==", - "license": "MIT", - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/preact" - } - }, - "node_modules/pretty-format": { - "version": "27.5.1", - "resolved": "https://registry.npmjs.org/pretty-format/-/pretty-format-27.5.1.tgz", - "integrity": "sha512-Qb1gy5OrP5+zDf2Bvnzdl3jsTf1qXVMazbvCoKhtKqVs4/YK4ozX4gKQJJVyNe+cajNPn0KoC0MC3FUmaHWEmQ==", - "license": "MIT", - "dependencies": { - "ansi-regex": "^5.0.1", - "ansi-styles": "^5.0.0", - "react-is": "^17.0.1" - }, - "engines": { - "node": "^10.13.0 || ^12.13.0 || ^14.15.0 || >=15.0.0" - } - }, "node_modules/puppeteer": { "version": "25.3.0", "resolved": "https://registry.npmjs.org/puppeteer/-/puppeteer-25.3.0.tgz", @@ -1826,35 +1711,6 @@ "node": ">=22.12.0" } }, - "node_modules/react": { - "version": "19.2.5", - "resolved": "https://registry.npmjs.org/react/-/react-19.2.5.tgz", - "integrity": "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA==", - "license": "MIT", - "peer": true, - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/react-dom": { - "version": "19.2.5", - "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.5.tgz", - "integrity": "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag==", - "license": "MIT", - "peer": true, - "dependencies": { - "scheduler": "^0.27.0" - }, - "peerDependencies": { - "react": "^19.2.5" - } - }, - "node_modules/react-is": { - "version": "17.0.2", - "resolved": "https://registry.npmjs.org/react-is/-/react-is-17.0.2.tgz", - "integrity": "sha512-w2GsyukL62IJnlaff/nRegPQR94C/XXamvMWmSHRJ4y7Ts/4ocGRmTHvOs8PSE6pB3dWOrD/nueuU5sduBsQ4w==", - "license": "MIT" - }, "node_modules/require-from-string": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", @@ -1898,13 +1754,6 @@ "@rolldown/binding-win32-x64-msvc": "1.1.4" } }, - "node_modules/scheduler": { - "version": "0.27.0", - "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.27.0.tgz", - "integrity": "sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==", - "license": "MIT", - "peer": true - }, "node_modules/semver": { "version": "7.8.5", "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", @@ -2058,29 +1907,6 @@ "license": "0BSD", "optional": true }, - "node_modules/twd-js": { - "version": "1.8.2", - "resolved": "https://registry.npmjs.org/twd-js/-/twd-js-1.8.2.tgz", - "integrity": "sha512-EZn4e2dLQ4jDLdgIbsz6+2WN1N4Q7FO5WnjKBno/2ldmp36+buc5k6/jLHka3EaSZjMgc+QilQxddym1fgMB+Q==", - "license": "MIT", - "dependencies": { - "@testing-library/dom": "^10.4.1", - "@testing-library/user-event": "^14.6.1", - "@types/chai": "^5.2.3", - "chai": "^6.2.2", - "preact": "^10.29.1" - }, - "bin": { - "twd-js": "dist/cli.js" - }, - "engines": { - "node": ">=16" - }, - "peerDependencies": { - "react": ">=17.0.0", - "react-dom": ">=17.0.0" - } - }, "node_modules/typed-query-selector": { "version": "2.12.2", "resolved": "https://registry.npmjs.org/typed-query-selector/-/typed-query-selector-2.12.2.tgz", diff --git a/package.json b/package.json index 02637da..a19409a 100644 --- a/package.json +++ b/package.json @@ -25,8 +25,7 @@ "license": "ISC", "dependencies": { "openapi-mock-validator": "^0.3.0", - "puppeteer": "^25.3.0", - "twd-js": "^1.8.2" + "puppeteer": "^25.3.0" }, "engines": { "node": ">=18.0.0" From 8f1d536c85f884aa4dc9f356f915bb427464c962 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 7 Jul 2026 10:38:22 +0200 Subject: [PATCH 07/11] docs: describe relay-style output and diagnostics in CLAUDE.md --- CLAUDE.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b08fc21..a746589 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,7 +30,7 @@ The codebase is a small ESM-only Node.js CLI with two core source files: 3. Navigates to the configured URL (default: `http://localhost:5173`) 4. Waits for `#twd-sidebar-root` selector (indicates app + TWD are ready) 5. Calls `window.__testRunner` in the browser context to execute all tests -6. Reports results via `reportResults()` from `twd-js/runner-ci` +6. Prints a relay-style summary block (`formatRunComplete` in `src/testSummary.js`) as the last output: passed/failed/skipped counts, duration, failed tests with `suite > test` paths and error messages, and retried tests. Known infrastructure errors (dev server down, sidebar missing, protocol timeout, Chrome launch failure) get actionable diagnostics from `src/diagnostics.js`. 7. Optionally collects `window.__coverage__` and writes to `.nyc_output/out.json` 8. Returns boolean `hasFailures` @@ -43,4 +43,4 @@ Tests are in `tests/` and use vitest. The test suite mocks `fs` to test config l ## Key Dependencies - **puppeteer** — Browser automation (launches Chrome/Chromium) -- **twd-js** — The TWD testing framework; provides `reportResults` from `twd-js/runner-ci` and the in-browser `__testRunner` +- **twd-js** — not a dependency of this package; the user's app bundles it, which provides the in-browser `__testRunner` and `#twd-sidebar-root` this CLI drives From 1dc4745d32e7c74be2bc20f2889ca73a0277e3f5 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 7 Jul 2026 10:49:48 +0200 Subject: [PATCH 08/11] polish: test stack-fallback branch, align count predicate, sync spec wording --- .../2026-07-06-ai-friendly-output-design.md | 2 +- src/index.js | 2 +- tests/runTests.test.js | 17 +++++++++++++++++ 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md b/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md index e8e52b0..247c00c 100644 --- a/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md +++ b/docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md @@ -95,7 +95,7 @@ Messages interpolate the *actual* config values (url, timeout), not defaults. | # | Failure mode | Detection | Message (cause + remediation) | |---|---|---|---| -| 1 | Dev server unreachable | `net::ERR_CONNECTION_REFUSED`, `ERR_NAME_NOT_RESOLVED`, `ERR_ADDRESS_UNREACHABLE` from `page.goto` | `Could not reach — connection refused.` / `Is your dev server running? Start it (e.g. \`npm run dev\`) or fix "url" in twd.config.json.` | +| 1 | Dev server unreachable | `net::ERR_CONNECTION_REFUSED`, `ERR_NAME_NOT_RESOLVED`, `ERR_ADDRESS_UNREACHABLE` from `page.goto` | `Could not reach ().` / `Is your dev server running? Start it (e.g. \`npm run dev\`) or fix "url" in twd.config.json.` | | 2 | TWD sidebar never mounts | `TimeoutError` from `waitForSelector('#twd-sidebar-root')` | `Page loaded but the TWD sidebar (#twd-sidebar-root) did not appear within ms.` / `Ensure twd-js is initialized in your app and your tests are registered.` / `If the app is slow to start, raise "timeout" in twd.config.json.` | | 3 | Protocol timeout | existing `isProtocolTimeout()` check (moves into this module) | existing message, unchanged | | 4 | Browser launch failure | `puppeteer.launch` errors matching `Could not find Chrome` or `Failed to launch the browser process` | `Puppeteer could not launch Chrome.` / `Run \`npx puppeteer browsers install chrome\`, or adjust "puppeteerArgs" in twd.config.json.` | diff --git a/src/index.js b/src/index.js index 3f3428b..81ac06f 100644 --- a/src/index.js +++ b/src/index.js @@ -88,7 +88,7 @@ export async function runTests(options = {}) { selectedIds = ids; console.log(`Filtering: running ${ids.length} test(s) matching --test filter(s).`); } else { - const testCount = registeredHandlers.filter((h) => h.type !== 'suite').length; + const testCount = registeredHandlers.filter((h) => h.type === 'test').length; console.log(`Running ${testCount} test(s)...`); } diff --git a/tests/runTests.test.js b/tests/runTests.test.js index 32f0f7d..50be79e 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -455,4 +455,21 @@ describe("runTests", () => { expect(errors.some((e) => e.includes('Is your dev server running?'))).toBe(true); errorSpy.mockRestore(); }); + + it("falls back to printing the stack for unrecognized errors", async () => { + const page = createMockPage({ handlers: [], testStatus: [] }); + const unknownError = new Error('weird boom'); + page.goto = vi.fn().mockRejectedValue(unknownError); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + + await expect(runTests()).rejects.toThrow('weird boom'); + + const errors = errorSpy.mock.calls.map((c) => String(c[0])); + expect(errors.some((e) => e.includes('Error running tests: weird boom'))).toBe(true); + expect(errors.some((e) => e.includes('at '))).toBe(true); + expect(errors.some((e) => e.includes('Is your dev server running?'))).toBe(false); + errorSpy.mockRestore(); + }); }); From 5851a71ab5c72e29a1f851f8f29878ace08094ee Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 7 Jul 2026 18:03:17 +0200 Subject: [PATCH 09/11] chore: bump version to 1.3.0-beta.1 for pre-merge beta testing Co-Authored-By: Claude Fable 5 --- package-lock.json | 4 ++-- package.json | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/package-lock.json b/package-lock.json index 915a14d..582f9dc 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "twd-cli", - "version": "1.2.0", + "version": "1.3.0-beta.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "twd-cli", - "version": "1.2.0", + "version": "1.3.0-beta.1", "license": "ISC", "dependencies": { "openapi-mock-validator": "^0.3.0", diff --git a/package.json b/package.json index a19409a..991f8bf 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "twd-cli", - "version": "1.2.0", + "version": "1.3.0-beta.1", "description": "CLI tool for running TWD tests with Puppeteer", "type": "module", "main": "src/index.js", From 595dc4b2e83974ece35f7bd88e638c11bea7bfd9 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 8 Jul 2026 13:39:37 +0200 Subject: [PATCH 10/11] ci: publish GitHub prereleases under the npm beta dist-tag Co-Authored-By: Claude Fable 5 --- .github/workflows/publish.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 28a51b8..b99756f 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -23,6 +23,7 @@ jobs: run: npm ci - name: Publish to npm - run: npm publish --access public + # Prereleases go to the "beta" dist-tag so `npm install twd-cli` keeps resolving to the stable release + run: npm publish --access public ${{ github.event.release.prerelease && '--tag beta' || '' }} env: NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} From 38999213aff83d5e26cc882a026a0c6a4f0de182 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 8 Jul 2026 17:07:54 +0200 Subject: [PATCH 11/11] chore: bump version to 1.3.0 Co-Authored-By: Claude Fable 5 --- package-lock.json | 4 ++-- package.json | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/package-lock.json b/package-lock.json index 582f9dc..f2ee5b5 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "twd-cli", - "version": "1.3.0-beta.1", + "version": "1.3.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "twd-cli", - "version": "1.3.0-beta.1", + "version": "1.3.0", "license": "ISC", "dependencies": { "openapi-mock-validator": "^0.3.0", diff --git a/package.json b/package.json index 991f8bf..6e1a9d1 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "twd-cli", - "version": "1.3.0-beta.1", + "version": "1.3.0", "description": "CLI tool for running TWD tests with Puppeteer", "type": "module", "main": "src/index.js",