From d0473ced86aac0f2f78fe4318da1c39b8cd78163 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 10:09:02 +0200 Subject: [PATCH 01/12] docs: design spec for fail-fast early bail + durable partial results Chunked, Node-driven execution so the CLI can stop after N failures (maxFailures, default 10) and never lose results on a timeout/crash. twd-cli-only; no twd-js changes. Co-Authored-By: Claude Opus 4.8 --- ...26-07-21-max-failures-early-bail-design.md | 156 ++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md diff --git a/docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md b/docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md new file mode 100644 index 0000000..3c0f1af --- /dev/null +++ b/docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md @@ -0,0 +1,156 @@ +# Design: Fail-fast early bail + timeout-durable partial results + +**Date:** 2026-07-21 +**Status:** Approved (pending spec review) +**Repo:** `twd-cli` + +## Problem + +On large suites the CLI loses everything on a timeout and wastes minutes on retries. + +Concrete case: `holafly/webapp-platform-admin` has 320+ tests. When ~40 fail, each +failing test is retried (`retryCount` default `2`), so the slowest tests run ~80 times. +The whole suite executes inside a **single `page.evaluate`** (`src/index.js:96`), and the +in-browser `testStatus` array is only returned to Node when `runAll()` fully resolves +(`twd/src/runner.ts:266`). When Puppeteer's `protocolTimeout` (default 300000ms) fires +mid-run, that one CDP call rejects, control jumps to the `catch`, and **all accumulated +results are lost** — the CI output is a bare timeout error with no test detail. + +Two distinct pains: +- **Wasted time / timeout:** running (and retrying) dozens of failing tests blows the + protocol timeout. +- **Total result loss:** a timeout (or any crash) mid-run discards every result gathered + so far. + +## Goals + +- Stop the run early once too many tests have failed, before the timeout is reached. +- Never lose already-gathered results: print partial results on both an early bail and a + genuine timeout/crash. +- Keep the change entirely within `twd-cli` — consuming apps upgrade only the CLI, not the + `twd-js` bundled in their app. + +## Non-goals + +- No changes to `twd-js` / the `TestRunner` class. +- No "consecutive failures" trigger (explicitly decided against — scattered failures would + never trip it and could still time out). Trigger is **cumulative total failures**. +- No change to retry semantics, coverage collection mechanics, or the relay-style summary + format beyond the additions below. + +## Decisions (locked) + +| Decision | Choice | +|---|---| +| Where the fix lives | `twd-cli` only (chunked execution driven from Node) | +| Bail trigger | Total failures ≥ `maxFailures` | +| `maxFailures` default | `10`, **on by default**; `0` disables (runs everything = today's behavior) | +| `chunkSize` default | `10` (advanced knob; bounds overshoot and per-timeout loss) | +| Execution order | Pre-order traversal identical to `runAll`; chunks are contiguous slices | + +## Approach + +Replace the single `page.evaluate(runAll)` with a Node-driven loop: + +1. Enumerate registered handlers (already done at `index.js:58`), **extended to include + `children`** so Node can compute execution order. +2. Compute the pre-order test-id list (new pure helper `src/testOrder.js`) mirroring + `runAll`'s traversal, so chunk boundaries never reorder tests. Matters for stateful apps + — there is no re-navigation between chunks; the page/app state persists exactly as in a + single run. +3. For each contiguous chunk of `chunkSize` ids: `page.evaluate` → `runByIds(chunk)` with + the same `onStart/onPass/onFail/onSkip` callbacks used today; return that chunk's + `testStatus`. Append to a Node-side `results` array. +4. After each chunk, if `maxFailures > 0` and `failed >= maxFailures`, stop. + +Retries are unchanged: a test only counts as failed after `retryCount` attempts are +exhausted (`onFail` fires once, post-retries). + +### `.only` handling + +If any `.only` is present, `runByIds` reports non-only tests as `skip` (existing runner +behavior in `runSuiteByIds`/`runTest`). Order and failure counts stay correct across +chunks; no special-casing needed. + +## Components + +### `src/config.js` +Add to `DEFAULT_CONFIG`: +- `maxFailures: 10` +- `chunkSize: 10` + +### `src/testOrder.js` (new) +Pure functions, no Puppeteer: +- `orderedTestIds(handlers)` → test ids in `runAll` pre-order (roots in order, DFS children + in order, `type === 'test'` only). +- `chunk(ids, size)` → array of contiguous slices. + +### `src/index.js` +- Extend handler enumeration to include `children`. +- Replace the single evaluate with the chunk loop. Accumulate `results` in a scope visible + to the `catch`. +- Track `executedCount`, `stoppedEarly`, and `totalTests` (count of `type === 'test'`). +- `hasFailures = stoppedEarly || results.some(fail)`. +- On `stoppedEarly`: **skip contract validation** (partial data → misleading) and say so. +- `catch`: if `results` is non-empty, print the partial summary **before** the diagnostic, + so a real timeout shows what ran. + +### `src/testSummary.js` +`formatRunComplete` gains optional `totalTests`, `stoppedEarly`, `maxFailures` +(backward compatible; existing callers/tests unaffected): +- `Not run: K` line when `totalTests > executed`. +- Early-stop banner: + ``` + ⚠ Stopped early: reached the failure limit (maxFailures=10). + 18 test(s) were not run. Fix the failures above, or set "maxFailures": 0 to run all. + ``` +- Failed-tests and retried lists unchanged. + +### `src/diagnostics.js` +Update the `protocolTimeout` message — it currently claims "the whole test suite runs in a +single page.evaluate call", which is no longer true. New text: a single chunk exceeded +`protocolTimeout` (likely one slow/hanging test); results up to that point are shown above; +raise `protocolTimeout` or lower `chunkSize`. + +## Coverage & contracts + +- **Coverage:** already skipped whenever there are failures; a bail implies failures, so no + change is required. +- **Contracts:** skipped on `stoppedEarly` (and on an interrupted/timeout run), with a + printed note that validation was skipped because the run was incomplete. + +## Error handling / data flow + +``` +loadConfig + → launch browser, goto url, wait for #twd-sidebar-root + → enumerate handlers (+children) + → orderedTestIds → chunk(size) + → for each chunk: + results += evaluate(runByIds(chunk)) // survives in Node + if maxFailures>0 and failed>=maxFailures: stoppedEarly=true; break + → (not stoppedEarly) contract validation + coverage as today + → print formatRunComplete(partial-aware) +catch (timeout/crash): + → if results non-empty: print formatRunComplete(partial) then diagnostic + → else: diagnostic only (as today) +``` + +Exit code: `hasFailures` (early bail ⇒ exit 1). + +## Testing (vitest, mocked Puppeteer + fs) + +- `orderedTestIds` preserves `runAll` order for nested suites; `chunk` slices correctly. +- Bail: failing chunks trip the threshold, loop stops, `Not run` computed, returns + `true` (failure). Overshoot bounded by `chunkSize`. +- Partial durability: `page.evaluate` throws on the 2nd chunk → partial summary printed + + diagnostic; results from chunk 1 present. +- `maxFailures: 0` disables bail (all chunks run). +- Clean pass: output and behavior unchanged from today (regression guard). +- Contracts skipped when `stoppedEarly`. + +## Rollout + +- Minor version bump (new default-on behavior, clearly messaged). +- Regenerate lockfile via `npm run lock:linux` if deps change (they should not here). +- Document `maxFailures` / `chunkSize` in README and `CLAUDE.md`. From d033dbabd1d54427f52c0b2b68a1dd286c864c47 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 10:20:24 +0200 Subject: [PATCH 02/12] docs: implementation plan for fail-fast early bail Co-Authored-By: Claude Opus 4.8 --- .../2026-07-21-max-failures-early-bail.md | 992 ++++++++++++++++++ 1 file changed, 992 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-21-max-failures-early-bail.md diff --git a/docs/superpowers/plans/2026-07-21-max-failures-early-bail.md b/docs/superpowers/plans/2026-07-21-max-failures-early-bail.md new file mode 100644 index 0000000..31ce76e --- /dev/null +++ b/docs/superpowers/plans/2026-07-21-max-failures-early-bail.md @@ -0,0 +1,992 @@ +# Fail-fast Early Bail + Durable Partial Results — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `twd-cli` stop a run once too many tests have failed, and never lose already-gathered results when a run times out or crashes. + +**Architecture:** Replace the single whole-suite `page.evaluate(runAll)` with a Node-driven loop that runs tests in ordered chunks via `runByIds`, accumulating results in Node after each chunk. After each chunk, stop if total failures ≥ `maxFailures`. Results live in Node, so a chunk timeout/crash still prints what completed. `twd-cli`-only; `twd-js` is untouched. + +**Tech Stack:** Node.js ESM, Puppeteer, Vitest (mocked `fs` + `puppeteer`). + +## Global Constraints + +- **Scope:** `twd-cli` only. Do **not** modify `twd-js` / the `TestRunner` class. Rely only on `runByIds`, the handler enumeration, and existing config. +- **Bail trigger:** cumulative **total** failures (`results.filter(status==='fail').length >= maxFailures`). No "consecutive" logic. +- **`maxFailures` default:** `10`, on by default. `0` disables bail (runs everything = today's behavior). +- **`chunkSize` default:** `10`. `<= 0` is treated as a single chunk (run everything at once). +- **Ordering contract:** the handler enumeration (`Array.from(window.__TWD_STATE__.handlers.values())`) is in insertion order, and twd inserts a suite before its children, so `filter(h => h.type === 'test')` is already pre-order execution order. `runByIds` runs a subset in that same tree order. Chunks are contiguous slices of that ordered id list. +- **Runtime:** ESM only, Node `>=18`. No new dependencies (so `npm run lock:linux` is not required for this work). +- **Commits:** Conventional Commits; this repo (`brikev/**`) keeps the `Co-Authored-By: Claude Opus 4.8 ` trailer. + +## File Structure + +- `src/config.js` (modify) — add `maxFailures`, `chunkSize` defaults. +- `src/testOrder.js` (create) — pure helpers `orderedTestIds(handlers)` and `chunk(items, size)`. +- `src/testSummary.js` (modify) — `formatRunComplete` gains optional `notRun`, `stoppedEarly`, `maxFailures`. +- `src/diagnostics.js` (modify) — reword the `protocolTimeout` explanation for the chunked model. +- `src/index.js` (modify) — chunked run loop, Node-side accumulation, bail, skip contracts on early stop, partial print on crash. +- `README.md`, `CLAUDE.md` (modify) — document the two config keys. +- Tests: `tests/config.test.js`, `tests/testOrder.test.js` (new), `tests/testSummary.test.js`, `tests/diagnostics.test.js`, `tests/runTests.test.js`. + +--- + +### Task 1: Config defaults (`maxFailures`, `chunkSize`) + +**Files:** +- Modify: `src/config.js:4-14` (`DEFAULT_CONFIG`) +- Test: `tests/config.test.js` + +**Interfaces:** +- Produces: `loadConfig()` returned object now includes `maxFailures: number` (default `10`) and `chunkSize: number` (default `10`), overridable from `twd.config.json`. + +- [ ] **Step 1: Update the default-config test expectations** + +In `tests/config.test.js`, add the two keys to every full-object `toEqual`. There are three: the "load default config" test (~line 26), the "merge user config" test (~line 51), and the "invalid JSON" test (~line 97). Add to each expected object: + +```js + retryCount: 2, + protocolTimeout: 300000, + maxFailures: 10, + chunkSize: 10, + }); +``` + +In the "override all default values" test (~line 68), add the two keys to the `userConfig` object so the full-override assertion still holds: + +```js + retryCount: 3, + protocolTimeout: 600000, + maxFailures: 5, + chunkSize: 20, + }; +``` + +- [ ] **Step 2: Add a focused test for the new defaults and overrides** + +Append inside the `describe('loadConfig', ...)` block in `tests/config.test.js`: + +```js + it('defaults maxFailures to 10 and chunkSize to 10', () => { + vi.mocked(fs.existsSync).mockReturnValue(false); + const config = loadConfig(); + expect(config.maxFailures).toBe(10); + expect(config.chunkSize).toBe(10); + }); + + it('allows user to override maxFailures and chunkSize (0 disables bail)', () => { + vi.mocked(fs.existsSync).mockReturnValue(true); + vi.mocked(fs.readFileSync).mockReturnValue( + JSON.stringify({ maxFailures: 0, chunkSize: 25 }) + ); + const config = loadConfig(); + expect(config.maxFailures).toBe(0); + expect(config.chunkSize).toBe(25); + }); +``` + +- [ ] **Step 3: Run the config tests to verify they fail** + +Run: `npx vitest run tests/config.test.js` +Expected: FAIL — defaults object does not yet include `maxFailures`/`chunkSize`. + +- [ ] **Step 4: Add the defaults to `DEFAULT_CONFIG`** + +In `src/config.js`, extend `DEFAULT_CONFIG`: + +```js +const DEFAULT_CONFIG = { + url: 'http://localhost:5173', + timeout: 10000, + coverage: true, + coverageDir: './coverage', + nycOutputDir: './.nyc_output', + headless: true, + puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], + retryCount: 2, + protocolTimeout: 300000, + maxFailures: 10, + chunkSize: 10, +}; +``` + +- [ ] **Step 5: Run the config tests to verify they pass** + +Run: `npx vitest run tests/config.test.js` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add src/config.js tests/config.test.js +git commit -m "feat: add maxFailures and chunkSize config defaults + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +### Task 2: Ordering + chunking helpers (`src/testOrder.js`) + +**Files:** +- Create: `src/testOrder.js` +- Test: `tests/testOrder.test.js` (create) + +**Interfaces:** +- Produces: + - `orderedTestIds(handlers: Array<{id, type, parent?}>) => string[]` — ids of `type === 'test'` handlers, in enumeration (pre-order) order. + - `chunk(items: T[], size: number) => T[][]` — contiguous slices of `size`; `size <= 0` returns a single chunk `[items]` (empty input returns `[]`). + +- [ ] **Step 1: Write the failing tests** + +Create `tests/testOrder.test.js`: + +```js +import { describe, it, expect } from 'vitest'; +import { orderedTestIds, chunk } from '../src/testOrder.js'; + +describe('orderedTestIds', () => { + it('returns test ids in enumeration order, skipping suites', () => { + const handlers = [ + { id: 's1', type: 'suite' }, + { id: 't1', type: 'test', parent: 's1' }, + { id: 's2', type: 'suite', parent: 's1' }, + { id: 't2', type: 'test', parent: 's2' }, + { id: 't3', type: 'test', parent: 's1' }, + ]; + expect(orderedTestIds(handlers)).toEqual(['t1', 't2', 't3']); + }); + + it('returns an empty array when there are no tests', () => { + expect(orderedTestIds([{ id: 's1', type: 'suite' }])).toEqual([]); + expect(orderedTestIds([])).toEqual([]); + }); +}); + +describe('chunk', () => { + it('splits into contiguous slices of the given size', () => { + expect(chunk([1, 2, 3, 4, 5], 2)).toEqual([[1, 2], [3, 4], [5]]); + }); + + it('returns a single chunk when size <= 0', () => { + expect(chunk([1, 2, 3], 0)).toEqual([[1, 2, 3]]); + expect(chunk([1, 2, 3], -4)).toEqual([[1, 2, 3]]); + }); + + it('returns an empty array for empty input', () => { + expect(chunk([], 10)).toEqual([]); + }); +}); +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `npx vitest run tests/testOrder.test.js` +Expected: FAIL with "Failed to resolve import '../src/testOrder.js'". + +- [ ] **Step 3: Implement the helpers** + +Create `src/testOrder.js`: + +```js +// The handler enumeration preserves insertion order, and twd registers each +// suite before its children, so filtering to tests yields pre-order execution +// order — the same order runByIds/runAll walk the tree in. +export function orderedTestIds(handlers) { + return handlers.filter((h) => h.type === 'test').map((h) => h.id); +} + +// Split items into contiguous slices of `size`. size <= 0 means "one chunk". +export function chunk(items, size) { + if (size <= 0) return items.length ? [items.slice()] : []; + const out = []; + for (let i = 0; i < items.length; i += size) { + out.push(items.slice(i, i + size)); + } + return out; +} +``` + +- [ ] **Step 4: Run the tests to verify they pass** + +Run: `npx vitest run tests/testOrder.test.js` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/testOrder.js tests/testOrder.test.js +git commit -m "feat: add test ordering and chunking helpers + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +### Task 3: Summary output — `Not run` line + early-stop banner + +**Files:** +- Modify: `src/testSummary.js` +- Test: `tests/testSummary.test.js` + +**Interfaces:** +- Consumes: nothing new. +- Produces: `formatRunComplete({ testStatus, handlers, durationMs, notRun?, stoppedEarly?, maxFailures? })`. New params are optional and default to no-ops (`notRun = 0`, `stoppedEarly = false`), so existing callers are unaffected. When `notRun > 0` a ` Not run: K` line is added; when `stoppedEarly` is true an early-stop banner is appended last. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/testSummary.test.js` inside the `describe('formatRunComplete', ...)` block: + +```js + it('adds a "Not run" line when notRun > 0', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'pass' }], + handlers, + durationMs: 1000, + notRun: 3, + }); + expect(block).toContain(' Not run: 3'); + }); + + it('omits the "Not run" line when notRun is 0', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'pass' }], + handlers, + durationMs: 1000, + }); + expect(block).not.toContain('Not run'); + }); + + it('appends an early-stop banner when stoppedEarly is true', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'fail', error: 'boom' }, + { id: 't2', status: 'fail', error: 'boom' }, + ], + handlers, + durationMs: 1000, + notRun: 5, + stoppedEarly: true, + maxFailures: 2, + }); + expect(block).toContain('Stopped early'); + expect(block).toContain('maxFailures=2'); + expect(block).toContain('5 test(s) were not run'); + expect(block).toContain('"maxFailures": 0'); + }); +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `npx vitest run tests/testSummary.test.js` +Expected: FAIL — no `Not run` line / banner yet. + +- [ ] **Step 3: Implement the additions** + +Replace the body of `formatRunComplete` in `src/testSummary.js` with: + +```js +export function formatRunComplete({ + testStatus, + handlers, + durationMs, + notRun = 0, + stoppedEarly = false, + maxFailures, +}) { + const passed = testStatus.filter((t) => t.status === 'pass').length; + const failed = testStatus.filter((t) => t.status === 'fail').length; + const skipped = testStatus.filter((t) => t.status === 'skip').length; + const duration = (durationMs / 1000).toFixed(1); + + const lines = [ + '--- Run complete ---', + ` Passed: ${passed} | Failed: ${failed} | Skipped: ${skipped}`, + ]; + if (notRun > 0) lines.push(` Not run: ${notRun}`); + lines.push(` Duration: ${duration}s`); + + const failures = testStatus.filter((t) => t.status === 'fail'); + if (failures.length > 0) { + lines.push('', ` Failed tests (${failures.length}):`); + for (const failure of failures) { + const testPath = buildTestPath(failure.id, handlers) ?? failure.id; + lines.push(` × ${testPath}`); + if (failure.error) { + lines.push(` ${String(failure.error).replace(/\n/g, '\n ')}`); + } + } + } + + const retried = testStatus.filter((t) => t.status === 'pass' && t.retryAttempt >= 2); + if (retried.length > 0) { + lines.push('', ` Retried (${retried.length}):`); + for (const t of retried) { + const testPath = buildTestPath(t.id, handlers) ?? t.id; + lines.push(` ✓ ${testPath} (passed on attempt ${t.retryAttempt})`); + } + } + + if (stoppedEarly) { + lines.push( + '', + `⚠ Stopped early: reached the failure limit (maxFailures=${maxFailures}).`, + ` ${notRun} test(s) were not run. Fix the failures above, or set "maxFailures": 0 to run all.` + ); + } + + return lines.join('\n'); +} +``` + +Leave the `import { buildTestPath }` line at the top of the file unchanged. + +- [ ] **Step 4: Run the full summary suite to verify pass + no regression** + +Run: `npx vitest run tests/testSummary.test.js` +Expected: PASS — new tests pass and the existing exact-block assertions (all-pass, failures, retried) are unchanged because `notRun`/`stoppedEarly` default off. + +- [ ] **Step 5: Commit** + +```bash +git add src/testSummary.js tests/testSummary.test.js +git commit -m "feat: show Not-run count and early-stop banner in run summary + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +### Task 4: Reword the `protocolTimeout` diagnostic + +**Files:** +- Modify: `src/diagnostics.js:31-37` +- Test: `tests/diagnostics.test.js:43-49` + +**Interfaces:** no signature change. The returned string must still contain the substrings `protocolTimeout` and `twd.config.json` (asserted by both `tests/diagnostics.test.js` and the `runTests` timeout test). + +- [ ] **Step 1: Update the existing diagnostic test to assert the new wording** + +Replace the "explains protocol timeouts" test in `tests/diagnostics.test.js` (~line 43) with: + +```js + it('explains protocol timeouts', () => { + const err = new Error('Runtime.callFunctionOn timed out.'); + err.name = 'ProtocolError'; + const msg = explainError(err, config); + expect(msg).toContain('protocolTimeout'); + expect(msg).toContain('twd.config.json'); + expect(msg).toContain('chunkSize'); + }); +``` + +- [ ] **Step 2: Run the diagnostics test to verify it fails** + +Run: `npx vitest run tests/diagnostics.test.js` +Expected: FAIL — current message does not mention `chunkSize`. + +- [ ] **Step 3: Reword the message** + +In `src/diagnostics.js`, replace the `isProtocolTimeout(error)` branch return in `explainError`: + +```js + if (isProtocolTimeout(error)) { + return ( + 'A single chunk of tests exceeded Puppeteer\'s protocolTimeout — usually one very\n' + + 'slow or hanging test. Any results printed above are partial (from chunks that\n' + + 'finished). Raise "protocolTimeout" in twd.config.json (0 = no timeout), or lower\n' + + '"chunkSize" so less work rides on each call.' + ); + } +``` + +- [ ] **Step 4: Run the diagnostics test to verify it passes** + +Run: `npx vitest run tests/diagnostics.test.js` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/diagnostics.js tests/diagnostics.test.js +git commit -m "docs: reword protocolTimeout diagnostic for chunked runs + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +### Task 5: Chunked execution refactor (behavior-preserving) + +Convert the single whole-suite `page.evaluate` into a chunked loop that runs `runByIds` per chunk and accumulates results in Node. No bail yet — every chunk runs, so behavior matches today. The chunk `evaluate` now returns a **test-status array** (not `{handlers, testStatus}`); handlers for the summary come from the enumeration. + +**Files:** +- Modify: `src/index.js` (imports + the run section, roughly current lines 95-122, 130-134, 187-189) +- Test: `tests/runTests.test.js` + +**Interfaces:** +- Consumes: `orderedTestIds`, `chunk` from `src/testOrder.js`; `loadConfig()` fields `chunkSize`, `retryCount`. +- Produces: unchanged public surface — `runTests(options?) => Promise`. Internally, each run `page.evaluate` is called as `(fn, retryCount, ids)` where `ids` is a concrete id array (never `null`). + +- [ ] **Step 1: Update the run-flow test harness for chunked returns** + +In `tests/runTests.test.js`: + +(a) Add the two new keys to `defaultMockConfig` (use a large `chunkSize` so every existing fixture is a single chunk, and keep bail effectively off for these tests): + +```js +const defaultMockConfig = { + url: 'http://localhost:5173', + timeout: 10000, + coverage: false, + coverageDir: './coverage', + nycOutputDir: './.nyc_output', + headless: true, + puppeteerArgs: [], + retryCount: 2, + maxFailures: 10, + chunkSize: 50, +}; +``` + +(b) Replace `createMockPage` so a chunk call resolves to a **testStatus array**: + +```js +function createMockPage({ handlers = [], testStatus = [] } = {}) { + return { + goto: vi.fn(), + waitForSelector: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) // enumeration pass returns handler metadata + .mockResolvedValue(testStatus), // each chunk run returns its testStatus array + exposeFunction: vi.fn(), + }; +} +``` + +(c) Update the two signature assertions that expected `null`: +- In "should pass retryCount to page.evaluate": change `toHaveBeenCalledWith(expect.any(Function), 3, null)` to `toHaveBeenCalledWith(expect.any(Function), 3, ['1'])`. +- In "passes selectedIds=null to the run evaluate when no filter": change `toHaveBeenCalledWith(expect.any(Function), 2, null)` to `toHaveBeenCalledWith(expect.any(Function), 2, ['1'])`, and update the test title to `"passes all test ids to the run evaluate when no filter"`. + +(d) In the "preserves responseHeaders" test, the mocked `page.evaluate` implementation currently returns `{ handlers, testStatus }`. Change its final line to return the array: + +```js + return testStatus; +``` + +(e) In the three filter tests ("runs only matching tests…", "warns about filters…", "skips coverage collection when a filter is active"), each builds a `runResult = { handlers: registry, testStatus: [...] }` and mocks the 2nd evaluate with it. Change each 2nd-evaluate mock to resolve the array directly, e.g.: + +```js + evaluate: vi.fn() + .mockResolvedValueOnce(registry) // enumeration pass + .mockResolvedValueOnce([{ id: 't1', status: 'pass' }]), // chunk run pass +``` + +Delete the now-unused `runResult` locals in those three tests. Leave every `toHaveBeenNthCalledWith(2, expect.any(Function), 2, ['t1'])` assertion as-is (a single filtered id is one chunk). + +- [ ] **Step 2: Run the run-flow tests to verify they fail** + +Run: `npx vitest run tests/runTests.test.js` +Expected: FAIL — `src/index.js` still returns/handles `{handlers, testStatus}` from a single evaluate. + +- [ ] **Step 3: Add the imports** + +In `src/index.js`, add after the existing imports: + +```js +import { orderedTestIds, chunk } from './testOrder.js'; +``` + +- [ ] **Step 4: Replace the single run evaluate with a chunked loop** + +In `src/index.js`, replace the block that currently runs the whole suite (the `const { handlers, testStatus } = await page.evaluate(...)` call through the `const durationMs = ...` line) with: + +```js + // Resolve the ordered id list to run: the filter result, or all tests. + const baseIds = selectedIds ?? orderedTestIds(registeredHandlers); + const chunks = chunk(baseIds, config.chunkSize); + + // Handlers for path-building/summary come from the enumeration so partial + // results are always printable even if a chunk never returns. + const handlers = registeredHandlers; + const testStatus = []; + let executed = 0; + + for (const ids of chunks) { + const chunkStatus = await page.evaluate(async (retryCount, chunkIds) => { + const TestRunner = window.__testRunner; + const testStatus = []; + const runner = new TestRunner({ + onStart: (test) => { + test.status = "running"; + }, + onPass: (test, retryAttempt) => { + test.status = "done"; + const entry = { id: test.id, status: "pass" }; + if (retryAttempt !== undefined) entry.retryAttempt = retryAttempt; + testStatus.push(entry); + }, + onFail: (test, err) => { + test.status = "done"; + testStatus.push({ id: test.id, status: "fail", error: `${err.message} (at ${window.location.href})` }); + }, + onSkip: (test) => { + test.status = "done"; + testStatus.push({ id: test.id, status: "skip" }); + }, + }, { retryCount }); + await runner.runByIds(chunkIds); + return testStatus; + }, config.retryCount, ids); + + testStatus.push(...chunkStatus); + executed += ids.length; + } + + const durationMs = Date.now() - startedAt; + const notRun = baseIds.length - executed; +``` + +- [ ] **Step 5: Point the summary + contract enrichment at the enumeration handlers** + +The replacement in Step 4 already defines `const handlers = registeredHandlers;`, so the existing `buildTestPath(mock.testId, handlers)` (contract enrichment loop) and the final `formatRunComplete({ testStatus, handlers, durationMs })` call keep working unchanged. Leave the final summary call as-is for this task: + +```js + console.log(''); + console.log(formatRunComplete({ testStatus, handlers, durationMs })); +``` + +- [ ] **Step 6: Run the run-flow tests to verify they pass** + +Run: `npx vitest run tests/runTests.test.js` +Expected: PASS. + +- [ ] **Step 7: Run the whole suite to confirm no cross-file regression** + +Run: `npm run test:ci` +Expected: PASS (all files). + +- [ ] **Step 8: Commit** + +```bash +git add src/index.js tests/runTests.test.js +git commit -m "refactor: run tests in Node-driven chunks via runByIds + +Accumulate results in Node per chunk instead of one whole-suite +page.evaluate. Behavior-preserving; sets up early-bail and durable +partial results. + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +### Task 6: Early bail on `maxFailures` + +Stop the chunk loop once total failures reach `maxFailures`, report `Not run` and the banner, and skip contract validation on an incomplete run. + +**Files:** +- Modify: `src/index.js` (the chunk loop from Task 5, the contracts block, the summary call) +- Test: `tests/runTests.test.js` + +**Interfaces:** +- Consumes: `config.maxFailures`. +- Produces: `runTests` returns `true` when it stops early; prints the early-stop summary; does not run contract validation when `stoppedEarly`. + +- [ ] **Step 1: Write the failing tests** + +Append to the `describe('runTests', ...)` block in `tests/runTests.test.js`: + +```js + it("stops early once maxFailures is reached and reports Not run", async () => { + const handlers = [ + { id: 's1', name: 'Suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 's1', type: 'test' }, + { id: 't2', name: 't2', parent: 's1', type: 'test' }, + { id: 't3', name: 't3', parent: 's1', type: 'test' }, + { id: 't4', name: 't4', parent: 's1', type: 'test' }, + { id: 't5', name: 't5', parent: 's1', type: 'test' }, + ]; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) // enumeration + .mockResolvedValueOnce([{ id: 't1', status: 'fail', error: 'a' }]) // chunk 1 + .mockResolvedValueOnce([{ id: 't2', status: 'fail', error: 'b' }]), // chunk 2 + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + maxFailures: 2, + chunkSize: 1, + }); + + const result = await runTests(); + + expect(result).toBe(true); + // enumeration + exactly 2 chunks (stopped; did NOT run t3..t5) + expect(page.evaluate).toHaveBeenCalledTimes(3); + const block = consoleSpy.mock.calls.map((c) => String(c[0])).at(-1); + expect(block).toContain('Not run: 3'); + expect(block).toContain('Stopped early'); + expect(block).toContain('maxFailures=2'); + }); + + it("runs every chunk when maxFailures is 0 (bail disabled)", async () => { + const handlers = [ + { id: 's1', name: 'Suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 's1', type: 'test' }, + { id: 't2', name: 't2', parent: 's1', type: 'test' }, + { id: 't3', name: 't3', parent: 's1', type: 'test' }, + ]; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) + .mockResolvedValue([{ id: 'x', status: 'fail', error: 'boom' }]), + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + maxFailures: 0, + chunkSize: 1, + }); + + await runTests(); + + // enumeration + 3 chunks; never bailed + expect(page.evaluate).toHaveBeenCalledTimes(4); + }); + + it("skips contract validation when the run stops early", async () => { + const handlers = [ + { id: 's1', name: 'Suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 's1', type: 'test' }, + { id: 't2', name: 't2', parent: 's1', type: 'test' }, + ]; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) + .mockResolvedValueOnce([{ id: 't1', status: 'fail', error: 'a' }]) + .mockResolvedValueOnce([{ id: 't2', status: 'fail', error: 'b' }]), + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + maxFailures: 2, + chunkSize: 1, + contracts: [{ source: './openapi.json' }], + }); + vi.mocked(loadContracts).mockResolvedValue([]); + + const result = await runTests(); + + expect(result).toBe(true); + expect(validateMocks).not.toHaveBeenCalled(); + }); +``` + +- [ ] **Step 2: Run the new tests to verify they fail** + +Run: `npx vitest run tests/runTests.test.js -t "stops early|maxFailures is 0|stops early"` +Expected: FAIL — no bail logic yet (loop runs all chunks; contracts still validated). + +- [ ] **Step 3: Add the bail check to the loop** + +In `src/index.js`, add a `stoppedEarly` flag and the threshold check inside the chunk loop (from Task 5). The loop becomes: + +```js + const handlers = registeredHandlers; + const testStatus = []; + let executed = 0; + let stoppedEarly = false; + + for (const ids of chunks) { + const chunkStatus = await page.evaluate(async (retryCount, chunkIds) => { + // ... unchanged evaluate body from Task 5 ... + }, config.retryCount, ids); + + testStatus.push(...chunkStatus); + executed += ids.length; + + if (config.maxFailures > 0) { + const failed = testStatus.filter((t) => t.status === 'fail').length; + if (failed >= config.maxFailures) { + stoppedEarly = true; + break; + } + } + } + + const durationMs = Date.now() - startedAt; + const notRun = baseIds.length - executed; + let hasFailures = stoppedEarly || testStatus.some((test) => test.status === 'fail'); +``` + +Remove the old standalone `let hasFailures = testStatus.some(test => test.status === 'fail');` line so `hasFailures` is only declared once (now including `stoppedEarly`). + +- [ ] **Step 4: Skip contract validation on an early stop** + +In `src/index.js`, guard the contracts block so it does not run on an incomplete run. Change the condition: + +```js + // Contract validation (skipped on an early stop — the data is partial) + if (!stoppedEarly && config.contracts && config.contracts.length > 0) { + // ... existing contract validation body unchanged ... + } else if (stoppedEarly && config.contracts && config.contracts.length > 0) { + console.log('\nSkipping contract validation — run stopped early (partial data).'); + } +``` + +- [ ] **Step 5: Pass the new fields to the summary** + +In `src/index.js`, update the final summary call: + +```js + console.log(''); + console.log(formatRunComplete({ + testStatus, + handlers, + durationMs, + notRun, + stoppedEarly, + maxFailures: config.maxFailures, + })); +``` + +- [ ] **Step 6: Run the run-flow tests to verify they pass** + +Run: `npx vitest run tests/runTests.test.js` +Expected: PASS (new bail tests and all prior tests). + +- [ ] **Step 7: Commit** + +```bash +git add src/index.js tests/runTests.test.js +git commit -m "feat: stop the run early after maxFailures failures + +Bail out of the chunk loop once total failures reach maxFailures, +report the Not-run count and banner, and skip contract validation on +the incomplete run. + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +### Task 7: Durable partial results on timeout/crash + +If a chunk `evaluate` throws (e.g. `protocolTimeout` on a hung test), print the results gathered from completed chunks before the diagnostic, instead of losing everything. + +**Files:** +- Modify: `src/index.js` (hoist run state above `try`; print partial results in `catch`) +- Test: `tests/runTests.test.js` + +**Interfaces:** no signature change. On a mid-run throw, `runTests` still rejects (as today) but first prints a partial run-complete block and an "interrupted" note. + +- [ ] **Step 1: Write the failing test** + +Append to the `describe('runTests', ...)` block in `tests/runTests.test.js`: + +```js + it("prints partial results when a chunk times out mid-run", async () => { + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + const handlers = [ + { id: 's1', name: 'Suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 's1', type: 'test' }, + { id: 't2', name: 't2', parent: 's1', type: 'test' }, + { id: 't3', name: 't3', parent: 's1', type: 'test' }, + ]; + const timeoutError = new Error('Runtime.callFunctionOn timed out.'); + timeoutError.name = 'ProtocolError'; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) // enumeration + .mockResolvedValueOnce([{ id: 't1', status: 'pass' }]) // chunk 1 ok + .mockRejectedValueOnce(timeoutError), // chunk 2 hangs + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + chunkSize: 1, + }); + + await expect(runTests()).rejects.toThrow('timed out'); + + const logs = consoleSpy.mock.calls.map((c) => String(c[0])); + const block = logs.find((l) => l.startsWith('--- Run complete ---')); + expect(block).toBeDefined(); + expect(block).toContain('Passed: 1'); + const errors = errorSpy.mock.calls.map((c) => String(c[0])); + expect(errors.some((e) => e.includes('protocolTimeout'))).toBe(true); + expect(browser.close).toHaveBeenCalled(); + errorSpy.mockRestore(); + }); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npx vitest run tests/runTests.test.js -t "prints partial results when a chunk times out"` +Expected: FAIL — no run-complete block is printed on the crash path (results are lost). + +- [ ] **Step 3: Hoist run state so `catch` can see partial results** + +In `src/index.js`, declare the accumulators alongside the existing `let browser; let config;` at the top of `runTests` (before the `try`): + +```js + let browser; + let config; + let startedAt = null; + let partialStatus = []; + let partialHandlers = []; +``` + +Inside the `try`, set `startedAt` where it is created today (`startedAt = Date.now();` instead of `const startedAt = ...`). After the enumeration, assign `partialHandlers = registeredHandlers;`. In the chunk loop, change the local accumulator to write through to the hoisted one: use `partialStatus` as the accumulation array (replace `const testStatus = [];` inside the try with `partialStatus = [];` and use `partialStatus` where the loop currently pushes and reads). Where the summary is built, use `const testStatus = partialStatus;` so the rest of the success path is unchanged. + +Concretely, the success-path lines become: + +```js + startedAt = Date.now(); + // ... goto / waitForSelector / enumeration ... + partialHandlers = registeredHandlers; + // ... filter resolution ... + + const baseIds = selectedIds ?? orderedTestIds(registeredHandlers); + const chunks = chunk(baseIds, config.chunkSize); + + const handlers = registeredHandlers; + partialStatus = []; + let executed = 0; + let stoppedEarly = false; + + for (const ids of chunks) { + const chunkStatus = await page.evaluate(/* ... */, config.retryCount, ids); + partialStatus.push(...chunkStatus); + executed += ids.length; + if (config.maxFailures > 0) { + const failed = partialStatus.filter((t) => t.status === 'fail').length; + if (failed >= config.maxFailures) { stoppedEarly = true; break; } + } + } + + const testStatus = partialStatus; + const durationMs = Date.now() - startedAt; + const notRun = baseIds.length - executed; +``` + +Everything below (`hasFailures`, contracts, coverage, summary) stays as written in Task 6. + +- [ ] **Step 4: Print partial results in `catch`** + +In `src/index.js`, at the top of the `catch (error) {` block (before the existing `console.error(...)` diagnostic lines), add: + +```js + if (partialStatus.length > 0) { + const durationMs = startedAt ? Date.now() - startedAt : 0; + console.log(''); + console.log(formatRunComplete({ + testStatus: partialStatus, + handlers: partialHandlers, + durationMs, + })); + console.log('\nRun interrupted before completion — results above are partial.'); + } +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `npx vitest run tests/runTests.test.js -t "prints partial results when a chunk times out"` +Expected: PASS. + +- [ ] **Step 6: Run the whole suite** + +Run: `npm run test:ci` +Expected: PASS (all files). In particular the existing "protocolTimeout hint when the run aborts" test still passes — its `page.evaluate` rejects on the first (enumeration) call, so `partialStatus` is empty and only the diagnostic prints. + +- [ ] **Step 7: Commit** + +```bash +git add src/index.js tests/runTests.test.js +git commit -m "feat: print partial results when a run is interrupted + +Accumulate results in Node-visible state so a mid-run protocolTimeout +or crash surfaces the tests that completed instead of discarding the +whole run. + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +### Task 8: Document the new config keys + +**Files:** +- Modify: `README.md` (config section) +- Modify: `CLAUDE.md` (the `src/config.js` description under Architecture) +- Modify: `docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md` (status line) + +**Interfaces:** none (docs only). + +- [ ] **Step 1: Document in README** + +Find the config/`twd.config.json` documentation section in `README.md` and add entries for the two keys (match the surrounding table/list style). Content to convey: +- `maxFailures` (default `10`): stop the run once this many tests have failed in total; the CLI prints the results gathered so far and exits non-zero. Set `0` to disable and always run every test. +- `chunkSize` (default `10`): how many tests run per browser call. Smaller values make the failure limit and timeouts more granular (less work lost if one chunk hangs); larger values reduce overhead. `0` runs everything in one call. + +Also add one line noting that on a `protocolTimeout` or crash mid-run, results from completed chunks are now printed instead of lost. + +- [ ] **Step 2: Update CLAUDE.md architecture note** + +In `CLAUDE.md`, update the `src/config.js` bullet to list `maxFailures` and `chunkSize` among the merged defaults, and update the `src/index.js` description: the suite no longer runs in a single `page.evaluate` — it runs in ordered chunks via `runByIds`, accumulating results in Node so the run can stop after `maxFailures` and partial results survive a timeout. + +- [ ] **Step 3: Flip the spec status to Implemented** + +In `docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md`, change `**Status:** Approved (pending spec review)` to `**Status:** Implemented`. + +- [ ] **Step 4: Verify the docs mention the keys** + +Run: `grep -n "maxFailures" README.md CLAUDE.md && grep -n "chunkSize" README.md CLAUDE.md` +Expected: at least one match in each file. + +- [ ] **Step 5: Commit** + +```bash +git add README.md CLAUDE.md docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md +git commit -m "docs: document maxFailures and chunkSize config + +Co-Authored-By: Claude Opus 4.8 " +``` + +--- + +## Self-Review + +**Spec coverage:** +- Chunked Node-driven execution → Task 5. +- `maxFailures` bail (total failures) → Task 6. +- `chunkSize` → Tasks 1, 2, 5. +- Durable partial results on timeout/crash → Task 7. +- `Not run` + early-stop banner → Task 3 (consumed in Task 6/7). +- Skip contracts on early stop → Task 6. +- `protocolTimeout` diagnostic reword → Task 4. +- Coverage unchanged (already skipped on failures) → no task needed; verified by existing suite in Task 5/6. +- Config defaults + docs → Tasks 1, 8. + +**Placeholder scan:** No TBD/TODO; every code step shows full code. + +**Type/name consistency:** `orderedTestIds`/`chunk` (Task 2) are consumed with those names in Task 5. `formatRunComplete` params `notRun`/`stoppedEarly`/`maxFailures` (Task 3) are passed with those names in Task 6/7. The chunk `evaluate` returns a testStatus array in Tasks 5–7 and the mock harness (Task 5 Step 1) matches. `stoppedEarly`/`notRun`/`executed`/`partialStatus`/`partialHandlers` are introduced and used consistently within `src/index.js`. + +**Note for the implementer:** Tasks 5–7 all edit the same run section of `src/index.js` in sequence; apply them in order. Task 5 leaves a plain chunk loop, Task 6 adds the bail + summary fields, Task 7 hoists the accumulators for the crash path. Re-read the current `src/index.js` before each of these tasks. From 234a92acdf27b6331b6a18d805f6928668dcc7ca Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 10:32:56 +0200 Subject: [PATCH 03/12] feat: add maxFailures and chunkSize config defaults Co-Authored-By: Claude Opus 4.8 --- src/config.js | 2 ++ tests/config.test.js | 25 +++++++++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/src/config.js b/src/config.js index 9d1ad5c..86a6899 100644 --- a/src/config.js +++ b/src/config.js @@ -11,6 +11,8 @@ const DEFAULT_CONFIG = { puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], retryCount: 2, protocolTimeout: 300000, + maxFailures: 10, + chunkSize: 10, }; export function loadConfig() { diff --git a/tests/config.test.js b/tests/config.test.js index 45bbdda..84f226f 100644 --- a/tests/config.test.js +++ b/tests/config.test.js @@ -33,6 +33,8 @@ describe('loadConfig', () => { puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], retryCount: 2, protocolTimeout: 300000, + maxFailures: 10, + chunkSize: 10, }); expect(fs.existsSync).toHaveBeenCalledWith(path.resolve(mockCwd, 'twd.config.json')); }); @@ -58,6 +60,8 @@ describe('loadConfig', () => { puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], retryCount: 2, protocolTimeout: 300000, + maxFailures: 10, + chunkSize: 10, }); expect(fs.readFileSync).toHaveBeenCalledWith( path.resolve(mockCwd, 'twd.config.json'), @@ -76,6 +80,8 @@ describe('loadConfig', () => { puppeteerArgs: ['--disable-dev-shm-usage'], retryCount: 3, protocolTimeout: 600000, + maxFailures: 5, + chunkSize: 20, }; vi.mocked(fs.existsSync).mockReturnValue(true); @@ -104,6 +110,8 @@ describe('loadConfig', () => { puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], retryCount: 2, protocolTimeout: 300000, + maxFailures: 10, + chunkSize: 10, }); expect(consoleWarnSpy).toHaveBeenCalledWith( expect.stringContaining('Warning: Could not parse twd.config.json'), @@ -150,4 +158,21 @@ describe('loadConfig', () => { expect(config.url).toBe('http://localhost:5173'); expect(config.timeout).toBe(10000); }); + + it('defaults maxFailures to 10 and chunkSize to 10', () => { + vi.mocked(fs.existsSync).mockReturnValue(false); + const config = loadConfig(); + expect(config.maxFailures).toBe(10); + expect(config.chunkSize).toBe(10); + }); + + it('allows user to override maxFailures and chunkSize (0 disables bail)', () => { + vi.mocked(fs.existsSync).mockReturnValue(true); + vi.mocked(fs.readFileSync).mockReturnValue( + JSON.stringify({ maxFailures: 0, chunkSize: 25 }) + ); + const config = loadConfig(); + expect(config.maxFailures).toBe(0); + expect(config.chunkSize).toBe(25); + }); }); \ No newline at end of file From 29f5cda66d2a0818379a937e86010c828ad35d49 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 10:36:09 +0200 Subject: [PATCH 04/12] feat: add test ordering and chunking helpers Co-Authored-By: Claude Opus 4.8 --- src/testOrder.js | 16 ++++++++++++++++ tests/testOrder.test.js | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) create mode 100644 src/testOrder.js create mode 100644 tests/testOrder.test.js diff --git a/src/testOrder.js b/src/testOrder.js new file mode 100644 index 0000000..e1d1db3 --- /dev/null +++ b/src/testOrder.js @@ -0,0 +1,16 @@ +// The handler enumeration preserves insertion order, and twd registers each +// suite before its children, so filtering to tests yields pre-order execution +// order — the same order runByIds/runAll walk the tree in. +export function orderedTestIds(handlers) { + return handlers.filter((h) => h.type === 'test').map((h) => h.id); +} + +// Split items into contiguous slices of `size`. size <= 0 means "one chunk". +export function chunk(items, size) { + if (size <= 0) return items.length ? [items.slice()] : []; + const out = []; + for (let i = 0; i < items.length; i += size) { + out.push(items.slice(i, i + size)); + } + return out; +} diff --git a/tests/testOrder.test.js b/tests/testOrder.test.js new file mode 100644 index 0000000..257b3c2 --- /dev/null +++ b/tests/testOrder.test.js @@ -0,0 +1,35 @@ +import { describe, it, expect } from 'vitest'; +import { orderedTestIds, chunk } from '../src/testOrder.js'; + +describe('orderedTestIds', () => { + it('returns test ids in enumeration order, skipping suites', () => { + const handlers = [ + { id: 's1', type: 'suite' }, + { id: 't1', type: 'test', parent: 's1' }, + { id: 's2', type: 'suite', parent: 's1' }, + { id: 't2', type: 'test', parent: 's2' }, + { id: 't3', type: 'test', parent: 's1' }, + ]; + expect(orderedTestIds(handlers)).toEqual(['t1', 't2', 't3']); + }); + + it('returns an empty array when there are no tests', () => { + expect(orderedTestIds([{ id: 's1', type: 'suite' }])).toEqual([]); + expect(orderedTestIds([])).toEqual([]); + }); +}); + +describe('chunk', () => { + it('splits into contiguous slices of the given size', () => { + expect(chunk([1, 2, 3, 4, 5], 2)).toEqual([[1, 2], [3, 4], [5]]); + }); + + it('returns a single chunk when size <= 0', () => { + expect(chunk([1, 2, 3], 0)).toEqual([[1, 2, 3]]); + expect(chunk([1, 2, 3], -4)).toEqual([[1, 2, 3]]); + }); + + it('returns an empty array for empty input', () => { + expect(chunk([], 10)).toEqual([]); + }); +}); From 1e6623386cb2b5f04a75c86b3d00c698284bfc45 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 10:39:16 +0200 Subject: [PATCH 05/12] feat: show Not-run count and early-stop banner in run summary Co-Authored-By: Claude Opus 4.8 --- src/testSummary.js | 20 ++++++++++++++++++-- tests/testSummary.test.js | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 2 deletions(-) diff --git a/src/testSummary.js b/src/testSummary.js index 89c41bd..b1e1cc2 100644 --- a/src/testSummary.js +++ b/src/testSummary.js @@ -1,6 +1,13 @@ import { buildTestPath } from './buildTestPath.js'; -export function formatRunComplete({ testStatus, handlers, durationMs }) { +export function formatRunComplete({ + testStatus, + handlers, + durationMs, + notRun = 0, + stoppedEarly = false, + maxFailures, +}) { const passed = testStatus.filter((t) => t.status === 'pass').length; const failed = testStatus.filter((t) => t.status === 'fail').length; const skipped = testStatus.filter((t) => t.status === 'skip').length; @@ -9,8 +16,9 @@ export function formatRunComplete({ testStatus, handlers, durationMs }) { const lines = [ '--- Run complete ---', ` Passed: ${passed} | Failed: ${failed} | Skipped: ${skipped}`, - ` Duration: ${duration}s`, ]; + if (notRun > 0) lines.push(` Not run: ${notRun}`); + lines.push(` Duration: ${duration}s`); const failures = testStatus.filter((t) => t.status === 'fail'); if (failures.length > 0) { @@ -33,5 +41,13 @@ export function formatRunComplete({ testStatus, handlers, durationMs }) { } } + if (stoppedEarly) { + lines.push( + '', + `⚠ Stopped early: reached the failure limit (maxFailures=${maxFailures}).`, + ` ${notRun} test(s) were not run. Fix the failures above, or set "maxFailures": 0 to run all.` + ); + } + return lines.join('\n'); } diff --git a/tests/testSummary.test.js b/tests/testSummary.test.js index e7d1690..1bb6950 100644 --- a/tests/testSummary.test.js +++ b/tests/testSummary.test.js @@ -124,4 +124,41 @@ describe('formatRunComplete', () => { }); expect(/\x1b\[[0-9;]*m/.test(block)).toBe(false); }); + + it('adds a "Not run" line when notRun > 0', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'pass' }], + handlers, + durationMs: 1000, + notRun: 3, + }); + expect(block).toContain(' Not run: 3'); + }); + + it('omits the "Not run" line when notRun is 0', () => { + const block = formatRunComplete({ + testStatus: [{ id: 't1', status: 'pass' }], + handlers, + durationMs: 1000, + }); + expect(block).not.toContain('Not run'); + }); + + it('appends an early-stop banner when stoppedEarly is true', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'fail', error: 'boom' }, + { id: 't2', status: 'fail', error: 'boom' }, + ], + handlers, + durationMs: 1000, + notRun: 5, + stoppedEarly: true, + maxFailures: 2, + }); + expect(block).toContain('Stopped early'); + expect(block).toContain('maxFailures=2'); + expect(block).toContain('5 test(s) were not run'); + expect(block).toContain('"maxFailures": 0'); + }); }); From e8087355bec60783f1013dcf7c344a9ca233e7da Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 10:43:28 +0200 Subject: [PATCH 06/12] docs: reword protocolTimeout diagnostic for chunked runs Co-Authored-By: Claude Opus 4.8 --- src/diagnostics.js | 7 ++++--- tests/diagnostics.test.js | 1 + 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/src/diagnostics.js b/src/diagnostics.js index 7cdcd9c..f5d8a78 100644 --- a/src/diagnostics.js +++ b/src/diagnostics.js @@ -30,9 +30,10 @@ export function explainError(error, config = {}) { if (isProtocolTimeout(error)) { return ( - 'This looks like a Puppeteer protocolTimeout. The whole test suite runs in a single\n' + - 'page.evaluate call, so the run aborts if it takes longer than "protocolTimeout" (ms).\n' + - 'Raise it in twd.config.json, e.g. { "protocolTimeout": 600000 } (0 = no timeout).' + 'A single chunk of tests exceeded Puppeteer\'s protocolTimeout — usually one very\n' + + 'slow or hanging test. Any results printed above are partial (from chunks that\n' + + 'finished). Raise "protocolTimeout" in twd.config.json (0 = no timeout), or lower\n' + + '"chunkSize" so less work rides on each call.' ); } diff --git a/tests/diagnostics.test.js b/tests/diagnostics.test.js index 02c507a..0dc95b2 100644 --- a/tests/diagnostics.test.js +++ b/tests/diagnostics.test.js @@ -46,6 +46,7 @@ describe('explainError', () => { const msg = explainError(err, config); expect(msg).toContain('protocolTimeout'); expect(msg).toContain('twd.config.json'); + expect(msg).toContain('chunkSize'); }); it('explains a missing Chrome install', () => { From ff37a245942aacb11b9694c78f74c6ba9a0052a8 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 10:48:18 +0200 Subject: [PATCH 07/12] refactor: run tests in Node-driven chunks via runByIds Accumulate results in Node per chunk instead of one whole-suite page.evaluate. Behavior-preserving; sets up early-bail and durable partial results. Co-Authored-By: Claude Opus 4.8 --- src/index.js | 70 +++++++++++++++++++++++++----------------- tests/runTests.test.js | 30 ++++++++---------- 2 files changed, 55 insertions(+), 45 deletions(-) diff --git a/src/index.js b/src/index.js index 81ac06f..2fb69cc 100644 --- a/src/index.js +++ b/src/index.js @@ -9,6 +9,7 @@ import { buildTestPath } from './buildTestPath.js'; import { formatRunComplete } from './testSummary.js'; import { selectTestIds } from './filterTests.js'; import { explainError } from './diagnostics.js'; +import { orderedTestIds, chunk } from './testOrder.js'; export async function runTests(options = {}) { const { testFilters = [] } = options; @@ -92,36 +93,49 @@ export async function runTests(options = {}) { console.log(`Running ${testCount} test(s)...`); } - // Execute all tests - const { handlers, testStatus } = await page.evaluate(async (retryCount, selectedIds) => { - const TestRunner = window.__testRunner; - const testStatus = []; - const runner = new TestRunner({ - onStart: (test) => { - test.status = "running"; - }, - onPass: (test, retryAttempt) => { - test.status = "done"; - const entry = { id: test.id, status: "pass" }; - if (retryAttempt !== undefined) entry.retryAttempt = retryAttempt; - testStatus.push(entry); - }, - onFail: (test, err) => { - test.status = "done"; - testStatus.push({ id: test.id, status: "fail", error: `${err.message} (at ${window.location.href})` }); - }, - onSkip: (test) => { - test.status = "done"; - testStatus.push({ id: test.id, status: "skip" }); - }, - }, { retryCount }); - const handlers = selectedIds - ? await runner.runByIds(selectedIds) - : await runner.runAll(); - return { handlers: Array.from(handlers.values()), testStatus }; - }, config.retryCount, selectedIds); + // Resolve the ordered id list to run: the filter result, or all tests. + const baseIds = selectedIds ?? orderedTestIds(registeredHandlers); + const chunks = chunk(baseIds, config.chunkSize); + + // Handlers for path-building/summary come from the enumeration so partial + // results are always printable even if a chunk never returns. + const handlers = registeredHandlers; + const testStatus = []; + let executed = 0; + + for (const ids of chunks) { + const chunkStatus = await page.evaluate(async (retryCount, chunkIds) => { + const TestRunner = window.__testRunner; + const testStatus = []; + const runner = new TestRunner({ + onStart: (test) => { + test.status = "running"; + }, + onPass: (test, retryAttempt) => { + test.status = "done"; + const entry = { id: test.id, status: "pass" }; + if (retryAttempt !== undefined) entry.retryAttempt = retryAttempt; + testStatus.push(entry); + }, + onFail: (test, err) => { + test.status = "done"; + testStatus.push({ id: test.id, status: "fail", error: `${err.message} (at ${window.location.href})` }); + }, + onSkip: (test) => { + test.status = "done"; + testStatus.push({ id: test.id, status: "skip" }); + }, + }, { retryCount }); + await runner.runByIds(chunkIds); + return testStatus; + }, config.retryCount, ids); + + testStatus.push(...chunkStatus); + executed += ids.length; + } const durationMs = Date.now() - startedAt; + const notRun = baseIds.length - executed; // Exit with appropriate code let hasFailures = testStatus.some(test => test.status === 'fail'); diff --git a/tests/runTests.test.js b/tests/runTests.test.js index 50be79e..676177c 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -20,13 +20,13 @@ import { loadConfig } from '../src/config.js'; import { loadContracts, validateMocks } from '../src/contracts.js'; import { printContractReport } from '../src/contractReport.js'; -function createMockPage(evaluateResult) { +function createMockPage({ handlers = [], testStatus = [] } = {}) { return { goto: vi.fn(), waitForSelector: vi.fn(), evaluate: vi.fn() - .mockResolvedValueOnce(evaluateResult.handlers ?? []) // enumeration pass - .mockResolvedValue(evaluateResult), // run pass (+ coverage) + .mockResolvedValueOnce(handlers) // enumeration pass returns handler metadata + .mockResolvedValue(testStatus), // each chunk run returns its testStatus array exposeFunction: vi.fn(), }; } @@ -47,6 +47,8 @@ const defaultMockConfig = { headless: true, puppeteerArgs: [], retryCount: 2, + maxFailures: 10, + chunkSize: 50, }; describe("runTests", () => { @@ -76,7 +78,7 @@ describe("runTests", () => { await runTests(); // page.evaluate is called with (fn, retryCount, selectedIds) - expect(page.evaluate).toHaveBeenCalledWith(expect.any(Function), 3, null); + expect(page.evaluate).toHaveBeenCalledWith(expect.any(Function), 3, ['1']); }); it("should pass protocolTimeout to puppeteer.launch", async () => { @@ -253,7 +255,7 @@ describe("runTests", () => { testId: 't-1', responseHeaders: { 'Content-Type': 'image/png' }, }); - return { handlers, testStatus }; + return testStatus; }), }; const browser = createMockBrowser(page); @@ -281,7 +283,7 @@ describe("runTests", () => { expect(entries[0].occurrence).toBe(1); }); - it("passes selectedIds=null to the run evaluate when no filter", async () => { + it("passes all test ids to the run evaluate when no filter", async () => { const testStatus = [{ id: '1', status: 'pass' }]; const handlers = [{ id: '1', name: 'test1', type: 'test' }]; const page = createMockPage({ handlers, testStatus }); @@ -290,7 +292,7 @@ describe("runTests", () => { await runTests(); - expect(page.evaluate).toHaveBeenCalledWith(expect.any(Function), 2, null); + expect(page.evaluate).toHaveBeenCalledWith(expect.any(Function), 2, ['1']); }); it("runs only matching tests when a --test filter is given", async () => { @@ -299,17 +301,13 @@ describe("runTests", () => { { id: 't1', name: 'shows error', parent: 's1', type: 'test' }, { id: 't2', name: 'redirects', parent: 's1', type: 'test' }, ]; - const runResult = { - handlers: registry, - testStatus: [{ id: 't1', status: 'pass' }], - }; const page = { goto: vi.fn(), waitForSelector: vi.fn(), exposeFunction: vi.fn(), evaluate: vi.fn() - .mockResolvedValueOnce(registry) // enumeration pass - .mockResolvedValueOnce(runResult), // run pass + .mockResolvedValueOnce(registry) // enumeration pass + .mockResolvedValueOnce([{ id: 't1', status: 'pass' }]), // chunk run pass }; const browser = createMockBrowser(page); vi.mocked(puppeteer.launch).mockResolvedValue(browser); @@ -350,14 +348,13 @@ describe("runTests", () => { { id: 's1', name: 'Login', parent: undefined, type: 'suite' }, { id: 't1', name: 'shows error', parent: 's1', type: 'test' }, ]; - const runResult = { handlers: registry, testStatus: [{ id: 't1', status: 'pass' }] }; const page = { goto: vi.fn(), waitForSelector: vi.fn(), exposeFunction: vi.fn(), evaluate: vi.fn() .mockResolvedValueOnce(registry) - .mockResolvedValueOnce(runResult), + .mockResolvedValueOnce([{ id: 't1', status: 'pass' }]), }; const browser = createMockBrowser(page); vi.mocked(puppeteer.launch).mockResolvedValue(browser); @@ -377,14 +374,13 @@ describe("runTests", () => { const registry = [ { id: 't1', name: 'shows error', parent: undefined, type: 'test' }, ]; - const runResult = { handlers: registry, testStatus: [{ id: 't1', status: 'pass' }] }; const page = { goto: vi.fn(), waitForSelector: vi.fn(), exposeFunction: vi.fn(), evaluate: vi.fn() .mockResolvedValueOnce(registry) - .mockResolvedValueOnce(runResult), + .mockResolvedValueOnce([{ id: 't1', status: 'pass' }]), }; const browser = createMockBrowser(page); vi.mocked(puppeteer.launch).mockResolvedValue(browser); From 1917a47bf3f6815daaa19f70734f536a5141d267 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 10:55:01 +0200 Subject: [PATCH 08/12] test: cover multi-chunk result accumulation Co-Authored-By: Claude Opus 4.8 --- tests/runTests.test.js | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/tests/runTests.test.js b/tests/runTests.test.js index 676177c..7f3cf65 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -177,6 +177,35 @@ describe("runTests", () => { expect(result).toBe(false); }); + it("accumulates results across multiple chunks", async () => { + const handlers = [ + { id: 's1', name: 'Suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 's1', type: 'test' }, + { id: 't2', name: 't2', parent: 's1', type: 'test' }, + { id: 't3', name: 't3', parent: 's1', type: 'test' }, + ]; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) // enumeration + .mockResolvedValueOnce([{ id: 't1', status: 'pass' }]) // chunk 1 + .mockResolvedValueOnce([{ id: 't2', status: 'fail', error: 'boom' }]) // chunk 2 + .mockResolvedValueOnce([{ id: 't3', status: 'pass' }]), // chunk 3 + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, chunkSize: 1 }); + + const result = await runTests(); + + expect(result).toBe(true); // one failure across chunks + expect(page.evaluate).toHaveBeenCalledTimes(4); // enumeration + 3 chunks + const block = consoleSpy.mock.calls.map((c) => String(c[0])).at(-1); + expect(block).toContain('Passed: 2 | Failed: 1 | Skipped: 0'); + }); + it("should skip contract validation when no contracts configured", async () => { const testStatus = [{ id: '1', status: 'pass' }]; const handlers = [{ id: '1', name: 'test1', type: 'test' }]; From c51989f56375a8c22f2cba8623cf7bbb4cd865f9 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 10:57:27 +0200 Subject: [PATCH 09/12] feat: stop the run early after maxFailures failures Bail out of the chunk loop once total failures reach maxFailures, report the Not-run count and banner, and skip contract validation on the incomplete run. Co-Authored-By: Claude Opus 4.8 --- src/index.js | 26 +++++++++-- tests/runTests.test.js | 97 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+), 4 deletions(-) diff --git a/src/index.js b/src/index.js index 2fb69cc..ac14c56 100644 --- a/src/index.js +++ b/src/index.js @@ -102,6 +102,7 @@ export async function runTests(options = {}) { const handlers = registeredHandlers; const testStatus = []; let executed = 0; + let stoppedEarly = false; for (const ids of chunks) { const chunkStatus = await page.evaluate(async (retryCount, chunkIds) => { @@ -132,13 +133,21 @@ export async function runTests(options = {}) { testStatus.push(...chunkStatus); executed += ids.length; + + if (config.maxFailures > 0) { + const failed = testStatus.filter((t) => t.status === 'fail').length; + if (failed >= config.maxFailures) { + stoppedEarly = true; + break; + } + } } const durationMs = Date.now() - startedAt; const notRun = baseIds.length - executed; // Exit with appropriate code - let hasFailures = testStatus.some(test => test.status === 'fail'); + let hasFailures = stoppedEarly || testStatus.some((test) => test.status === 'fail'); // Enrich collected mocks with full test path names for (const [, mock] of collectedMocks) { @@ -147,8 +156,8 @@ export async function runTests(options = {}) { } } - // Contract validation - if (config.contracts && config.contracts.length > 0) { + // Contract validation (skipped on an early stop — the data is partial) + if (!stoppedEarly && config.contracts && config.contracts.length > 0) { if (collectedMocks.size === 0) { console.log('\nNo mocks collected — ensure twd-js supports contract collection'); } @@ -169,6 +178,8 @@ export async function runTests(options = {}) { fs.writeFileSync(reportPath, markdown); console.log(`Contract report written to ${config.contractReportPath}`); } + } else if (stoppedEarly && config.contracts && config.contracts.length > 0) { + console.log('\nSkipping contract validation — run stopped early (partial data).'); } // Handle code coverage if enabled (skipped when a --test filter is active) @@ -200,7 +211,14 @@ export async function runTests(options = {}) { // The run-complete block is always the last output of a completed run console.log(''); - console.log(formatRunComplete({ testStatus, handlers, durationMs })); + console.log(formatRunComplete({ + testStatus, + handlers, + durationMs, + notRun, + stoppedEarly, + maxFailures: config.maxFailures, + })); return hasFailures; diff --git a/tests/runTests.test.js b/tests/runTests.test.js index 7f3cf65..7b6cc0b 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -497,4 +497,101 @@ describe("runTests", () => { expect(errors.some((e) => e.includes('Is your dev server running?'))).toBe(false); errorSpy.mockRestore(); }); + + it("stops early once maxFailures is reached and reports Not run", async () => { + const handlers = [ + { id: 's1', name: 'Suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 's1', type: 'test' }, + { id: 't2', name: 't2', parent: 's1', type: 'test' }, + { id: 't3', name: 't3', parent: 's1', type: 'test' }, + { id: 't4', name: 't4', parent: 's1', type: 'test' }, + { id: 't5', name: 't5', parent: 's1', type: 'test' }, + ]; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) // enumeration + .mockResolvedValueOnce([{ id: 't1', status: 'fail', error: 'a' }]) // chunk 1 + .mockResolvedValueOnce([{ id: 't2', status: 'fail', error: 'b' }]), // chunk 2 + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + maxFailures: 2, + chunkSize: 1, + }); + + const result = await runTests(); + + expect(result).toBe(true); + // enumeration + exactly 2 chunks (stopped; did NOT run t3..t5) + expect(page.evaluate).toHaveBeenCalledTimes(3); + const block = consoleSpy.mock.calls.map((c) => String(c[0])).at(-1); + expect(block).toContain('Not run: 3'); + expect(block).toContain('Stopped early'); + expect(block).toContain('maxFailures=2'); + }); + + it("runs every chunk when maxFailures is 0 (bail disabled)", async () => { + const handlers = [ + { id: 's1', name: 'Suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 's1', type: 'test' }, + { id: 't2', name: 't2', parent: 's1', type: 'test' }, + { id: 't3', name: 't3', parent: 's1', type: 'test' }, + ]; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) + .mockResolvedValue([{ id: 'x', status: 'fail', error: 'boom' }]), + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + maxFailures: 0, + chunkSize: 1, + }); + + await runTests(); + + // enumeration + 3 chunks; never bailed + expect(page.evaluate).toHaveBeenCalledTimes(4); + }); + + it("skips contract validation when the run stops early", async () => { + const handlers = [ + { id: 's1', name: 'Suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 's1', type: 'test' }, + { id: 't2', name: 't2', parent: 's1', type: 'test' }, + ]; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) + .mockResolvedValueOnce([{ id: 't1', status: 'fail', error: 'a' }]) + .mockResolvedValueOnce([{ id: 't2', status: 'fail', error: 'b' }]), + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + maxFailures: 2, + chunkSize: 1, + contracts: [{ source: './openapi.json' }], + }); + vi.mocked(loadContracts).mockResolvedValue([]); + + const result = await runTests(); + + expect(result).toBe(true); + expect(validateMocks).not.toHaveBeenCalled(); + }); }); From 9a21c2bea7a8184147aabdff4ffdffb3e3fd43de Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 11:02:01 +0200 Subject: [PATCH 10/12] feat: print partial results when a run is interrupted Accumulate results in Node-visible state so a mid-run protocolTimeout or crash surfaces the tests that completed instead of discarding the whole run. Co-Authored-By: Claude Opus 4.8 --- src/index.js | 23 +++++++++++++++++++---- tests/runTests.test.js | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 4 deletions(-) diff --git a/src/index.js b/src/index.js index ac14c56..e3aecd7 100644 --- a/src/index.js +++ b/src/index.js @@ -15,6 +15,9 @@ export async function runTests(options = {}) { const { testFilters = [] } = options; let browser; let config; + let startedAt = null; + let partialStatus = []; + let partialHandlers = []; try { config = loadConfig(); const workingDir = process.cwd(); @@ -48,7 +51,7 @@ export async function runTests(options = {}) { } // Navigate to your development server - const startedAt = Date.now(); + startedAt = Date.now(); console.log(`Navigating to ${config.url} ...`); await page.goto(config.url); @@ -66,6 +69,7 @@ export async function runTests(options = {}) { type: h.type, })); }); + partialHandlers = registeredHandlers; // Resolve --test filters to a concrete set of test ids (null = run all) let selectedIds = null; @@ -100,7 +104,7 @@ export async function runTests(options = {}) { // Handlers for path-building/summary come from the enumeration so partial // results are always printable even if a chunk never returns. const handlers = registeredHandlers; - const testStatus = []; + partialStatus = []; let executed = 0; let stoppedEarly = false; @@ -131,11 +135,11 @@ export async function runTests(options = {}) { return testStatus; }, config.retryCount, ids); - testStatus.push(...chunkStatus); + partialStatus.push(...chunkStatus); executed += ids.length; if (config.maxFailures > 0) { - const failed = testStatus.filter((t) => t.status === 'fail').length; + const failed = partialStatus.filter((t) => t.status === 'fail').length; if (failed >= config.maxFailures) { stoppedEarly = true; break; @@ -143,6 +147,7 @@ export async function runTests(options = {}) { } } + const testStatus = partialStatus; const durationMs = Date.now() - startedAt; const notRun = baseIds.length - executed; @@ -223,6 +228,16 @@ export async function runTests(options = {}) { return hasFailures; } catch (error) { + if (partialStatus.length > 0) { + const durationMs = startedAt ? Date.now() - startedAt : 0; + console.log(''); + console.log(formatRunComplete({ + testStatus: partialStatus, + handlers: partialHandlers, + durationMs, + })); + console.log('\nRun interrupted before completion — results above are partial.'); + } const message = error && error.message ? error.message : String(error); console.error(`Error running tests: ${message}`); const diagnostic = explainError(error, config); diff --git a/tests/runTests.test.js b/tests/runTests.test.js index 7b6cc0b..f557acb 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -594,4 +594,42 @@ describe("runTests", () => { expect(result).toBe(true); expect(validateMocks).not.toHaveBeenCalled(); }); + + it("prints partial results when a chunk times out mid-run", async () => { + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + const handlers = [ + { id: 's1', name: 'Suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 's1', type: 'test' }, + { id: 't2', name: 't2', parent: 's1', type: 'test' }, + { id: 't3', name: 't3', parent: 's1', type: 'test' }, + ]; + const timeoutError = new Error('Runtime.callFunctionOn timed out.'); + timeoutError.name = 'ProtocolError'; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) // enumeration + .mockResolvedValueOnce([{ id: 't1', status: 'pass' }]) // chunk 1 ok + .mockRejectedValueOnce(timeoutError), // chunk 2 hangs + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + chunkSize: 1, + }); + + await expect(runTests()).rejects.toThrow('timed out'); + + const logs = consoleSpy.mock.calls.map((c) => String(c[0])); + const block = logs.find((l) => l.startsWith('--- Run complete ---')); + expect(block).toBeDefined(); + expect(block).toContain('Passed: 1'); + const errors = errorSpy.mock.calls.map((c) => String(c[0])); + expect(errors.some((e) => e.includes('protocolTimeout'))).toBe(true); + expect(browser.close).toHaveBeenCalled(); + errorSpy.mockRestore(); + }); }); From d7d714f91ad09355e15f8273b293ea9852745600 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 11:07:32 +0200 Subject: [PATCH 11/12] docs: document maxFailures and chunkSize config Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 13 +++++++------ README.md | 10 ++++++++-- .../2026-07-21-max-failures-early-bail-design.md | 2 +- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a746589..fd64bfd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -20,19 +20,20 @@ The codebase is a small ESM-only Node.js CLI with two core source files: **`bin/twd-cli.js`** — CLI entry point. Parses `process.argv` for the `run` command, calls `runTests()`, and exits with code 0 (pass) or 1 (failure). -**`src/config.js`** — `loadConfig()` reads `twd.config.json` from `process.cwd()`, merges it with defaults (url, timeout, coverage, headless, puppeteerArgs, retryCount, protocolTimeout), and returns the merged config. Falls back to defaults if the file is missing or unparseable. +**`src/config.js`** — `loadConfig()` reads `twd.config.json` from `process.cwd()`, merges it with defaults (url, timeout, coverage, headless, puppeteerArgs, retryCount, protocolTimeout, maxFailures, chunkSize), and returns the merged config. Falls back to defaults if the file is missing or unparseable. -`protocolTimeout` (default `300000`, 5 min) is passed to `puppeteer.launch` and bounds Puppeteer's CDP commands. It matters because the entire suite runs inside a single `page.evaluate` (`Runtime.callFunctionOn`), so Puppeteer's implicit 180000ms ceiling would abort long-but-passing suites with no per-test output. Raise it for slow CI; `0` means no timeout. +`protocolTimeout` (default `300000`, 5 min) is passed to `puppeteer.launch` and bounds each chunk's CDP call. `maxFailures` (default `10`) stops the run after that many cumulative test failures; set to `0` to disable. `chunkSize` (default `10`) controls how many tests run per browser call. **`src/index.js`** — `runTests()` is the main orchestrator: 1. Loads config via `loadConfig()` 2. Launches Puppeteer with configured headless mode and args 3. Navigates to the configured URL (default: `http://localhost:5173`) 4. Waits for `#twd-sidebar-root` selector (indicates app + TWD are ready) -5. Calls `window.__testRunner` in the browser context to execute all tests -6. Prints a relay-style summary block (`formatRunComplete` in `src/testSummary.js`) as the last output: passed/failed/skipped counts, duration, failed tests with `suite > test` paths and error messages, and retried tests. Known infrastructure errors (dev server down, sidebar missing, protocol timeout, Chrome launch failure) get actionable diagnostics from `src/diagnostics.js`. -7. Optionally collects `window.__coverage__` and writes to `.nyc_output/out.json` -8. Returns boolean `hasFailures` +5. Enumerates all registered test handlers and computes pre-order execution order +6. Runs tests in ordered chunks via `runByIds(chunkIds)`, with chunk size controlled by config; accumulates results in Node so the run can stop after `maxFailures` failures and partial results survive a timeout or crash +7. Prints a relay-style summary block (`formatRunComplete` in `src/testSummary.js`) as the last output: passed/failed/skipped counts, duration, failed tests with `suite > test` paths and error messages, retried tests, and "Not run" count if stopped early. Known infrastructure errors (dev server down, sidebar missing, protocol timeout, Chrome launch failure) get actionable diagnostics from `src/diagnostics.js`. +8. Optionally collects `window.__coverage__` and writes to `.nyc_output/out.json` (skipped on early bail) +9. Returns boolean `hasFailures` **`test-example-app/`** — A React demo app with TWD tests integrated, used for manual testing/demonstration. Not part of the published package or test suite. diff --git a/README.md b/README.md index 5fee22e..eded0dc 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,9 @@ Create a `twd.config.json` file in your project root: "headless": true, "puppeteerArgs": ["--no-sandbox", "--disable-setuid-sandbox"], "retryCount": 2, - "protocolTimeout": 300000 + "protocolTimeout": 300000, + "maxFailures": 10, + "chunkSize": 10 } ``` @@ -79,10 +81,14 @@ Create a `twd.config.json` file in your project root: | `headless` | boolean | `true` | Run browser in headless mode | | `puppeteerArgs` | string[] | `["--no-sandbox", "--disable-setuid-sandbox"]` | Additional Puppeteer launch arguments | | `retryCount` | number | `2` | Number of attempts per test before reporting failure. Set to `1` to disable retries | -| `protocolTimeout` | number | `300000` | Puppeteer CDP `protocolTimeout` in ms (5 min). The whole suite runs inside one `page.evaluate`, so this bounds the **entire run** — raise it (e.g. `600000`) for slow CI; `0` means no timeout. Defaults above Puppeteer's implicit 180000ms ceiling | +| `protocolTimeout` | number | `300000` | Puppeteer CDP `protocolTimeout` in ms (5 min). Tests run in chunks via `runByIds`, so this bounds a **single chunk's browser call** (not the entire run) — raise it (e.g. `600000`) for slow CI or if individual chunks hang; `0` means no timeout. Defaults above Puppeteer's implicit 180000ms ceiling | +| `maxFailures` | number | `10` | Stop the run once this many tests have failed in total; the CLI prints the results gathered so far and exits non-zero. Set `0` to disable and always run every test | +| `chunkSize` | number | `10` | How many tests run per browser call. Smaller values make the failure limit and timeouts more granular (less work lost if one chunk hangs); larger values reduce overhead. `0` runs everything in one call | | `contracts` | array | — | OpenAPI contract validation specs (see [Contract Validation](#contract-validation)) | | `contractReportPath` | string | — | Path to write a markdown report for CI/PR integration | +**Partial Results on Timeout or Crash:** Tests run in chunks (controlled by `chunkSize`), so on a `protocolTimeout` or unexpected crash mid-run, results from completed chunks are printed instead of being lost entirely. + ## How It Works **Important**: Puppeteer is **not** used as a testing framework here. It simply provides a headless browser to load your application — the same way a user would open Chrome. Once the page loads, all test execution happens inside the real browser context through the [TWD runner](https://brikev.github.io/twd/). Your tests interact with real DOM, real components, and real browser APIs — Puppeteer just opens the door and gets out of the way. diff --git a/docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md b/docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md index 3c0f1af..316c5fb 100644 --- a/docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md +++ b/docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md @@ -1,7 +1,7 @@ # Design: Fail-fast early bail + timeout-durable partial results **Date:** 2026-07-21 -**Status:** Approved (pending spec review) +**Status:** Implemented **Repo:** `twd-cli` ## Problem From 3f68cc5e80148bb4d810491c98b973fa8419d602 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 21 Jul 2026 11:18:50 +0200 Subject: [PATCH 12/12] fix: dedupe skip counts across chunks and refine early-stop banner Address final-review findings: dedupe accumulated results by id so a describe.skip suite spanning multiple chunks is not counted skipped more than once; avoid a '0 test(s) were not run' early-stop banner; clarify the coverage-skip note in CLAUDE.md. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 2 +- src/index.js | 7 ++++++- src/testSummary.js | 11 ++++++----- tests/runTests.test.js | 27 +++++++++++++++++++++++++++ tests/testSummary.test.js | 16 ++++++++++++++++ 5 files changed, 56 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fd64bfd..e7e8200 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -32,7 +32,7 @@ The codebase is a small ESM-only Node.js CLI with two core source files: 5. Enumerates all registered test handlers and computes pre-order execution order 6. Runs tests in ordered chunks via `runByIds(chunkIds)`, with chunk size controlled by config; accumulates results in Node so the run can stop after `maxFailures` failures and partial results survive a timeout or crash 7. Prints a relay-style summary block (`formatRunComplete` in `src/testSummary.js`) as the last output: passed/failed/skipped counts, duration, failed tests with `suite > test` paths and error messages, retried tests, and "Not run" count if stopped early. Known infrastructure errors (dev server down, sidebar missing, protocol timeout, Chrome launch failure) get actionable diagnostics from `src/diagnostics.js`. -8. Optionally collects `window.__coverage__` and writes to `.nyc_output/out.json` (skipped on early bail) +8. Optionally collects `window.__coverage__` and writes to `.nyc_output/out.json` (skipped whenever the run has failures, including an early bail) 9. Returns boolean `hasFailures` **`test-example-app/`** — A React demo app with TWD tests integrated, used for manual testing/demonstration. Not part of the published package or test suite. diff --git a/src/index.js b/src/index.js index e3aecd7..4f083f9 100644 --- a/src/index.js +++ b/src/index.js @@ -107,6 +107,7 @@ export async function runTests(options = {}) { partialStatus = []; let executed = 0; let stoppedEarly = false; + const seenIds = new Set(); for (const ids of chunks) { const chunkStatus = await page.evaluate(async (retryCount, chunkIds) => { @@ -135,7 +136,11 @@ export async function runTests(options = {}) { return testStatus; }, config.retryCount, ids); - partialStatus.push(...chunkStatus); + for (const entry of chunkStatus) { + if (seenIds.has(entry.id)) continue; + seenIds.add(entry.id); + partialStatus.push(entry); + } executed += ids.length; if (config.maxFailures > 0) { diff --git a/src/testSummary.js b/src/testSummary.js index b1e1cc2..7497005 100644 --- a/src/testSummary.js +++ b/src/testSummary.js @@ -42,11 +42,12 @@ export function formatRunComplete({ } if (stoppedEarly) { - lines.push( - '', - `⚠ Stopped early: reached the failure limit (maxFailures=${maxFailures}).`, - ` ${notRun} test(s) were not run. Fix the failures above, or set "maxFailures": 0 to run all.` - ); + lines.push('', `⚠ Stopped early: reached the failure limit (maxFailures=${maxFailures}).`); + if (notRun > 0) { + lines.push(` ${notRun} test(s) were not run. Fix the failures above, or set "maxFailures": 0 to run all.`); + } else { + lines.push(' Fix the failures above, or set "maxFailures": 0 to run all.'); + } } return lines.join('\n'); diff --git a/tests/runTests.test.js b/tests/runTests.test.js index f557acb..0f99893 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -595,6 +595,33 @@ describe("runTests", () => { expect(validateMocks).not.toHaveBeenCalled(); }); + it("dedupes repeated suite-level skip entries across chunks", async () => { + const handlers = [ + { id: 'sk', name: 'Skipped suite', type: 'suite' }, + { id: 't1', name: 't1', parent: 'sk', type: 'test' }, + { id: 't2', name: 't2', parent: 'sk', type: 'test' }, + { id: 't3', name: 't3', parent: 'sk', type: 'test' }, + ]; + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce(handlers) // enumeration + .mockResolvedValueOnce([{ id: 'sk', status: 'skip' }]) // chunk 1 (t1) + .mockResolvedValueOnce([{ id: 'sk', status: 'skip' }]) // chunk 2 (t2) + .mockResolvedValueOnce([{ id: 'sk', status: 'skip' }]), // chunk 3 (t3) + }; + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, chunkSize: 1 }); + + await runTests(); + + const block = consoleSpy.mock.calls.map((c) => String(c[0])).at(-1); + expect(block).toContain('Skipped: 1'); + }); + it("prints partial results when a chunk times out mid-run", async () => { const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); const handlers = [ diff --git a/tests/testSummary.test.js b/tests/testSummary.test.js index 1bb6950..4950d75 100644 --- a/tests/testSummary.test.js +++ b/tests/testSummary.test.js @@ -161,4 +161,20 @@ describe('formatRunComplete', () => { expect(block).toContain('5 test(s) were not run'); expect(block).toContain('"maxFailures": 0'); }); + + it('does not claim "0 test(s) were not run" when nothing was skipped', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'fail', error: 'boom' }, + { id: 't2', status: 'fail', error: 'boom' }, + ], + handlers, + durationMs: 1000, + notRun: 0, + stoppedEarly: true, + maxFailures: 2, + }); + expect(block).toContain('Stopped early'); + expect(block).not.toContain('0 test(s) were not run'); + }); });