feat: fail-fast early bail + durable partial results

[kevinccbsg]

Problem

On large suites the CLI loses everything on a timeout and wastes minutes retrying.

Real case: holafly/webapp-platform-admin (320+ tests). When ~40 fail, each is retried (retryCount default 2) so the slowest tests run ~80×. The whole suite ran inside one page.evaluate, and testStatus was only returned to Node when runAll() fully resolved — so a protocolTimeout mid-run rejected that single CDP call and discarded every result, leaving a bare timeout error in CI.

What changed

The suite now runs in ordered chunks driven from Node (runByIds per chunk), accumulating results in Node. This gives two wins at once:

• Early bail — after each chunk, if total failures ≥ maxFailures, the run stops, prints the results gathered so far + a "Stopped early" banner with a Not run: K count, and exits non-zero.
• Durable results — if a chunk's page.evaluate times out/crashes, results from completed chunks are printed before the diagnostic instead of being lost.

twd-cli-only — no twd-js changes; consuming apps upgrade just the CLI.

Config (new)

key	default	behavior
maxFailures	10	Stop once this many tests have failed in total; 0 disables (run everything).
chunkSize	10	Tests per browser call; smaller = more granular bail/timeout bounding; 0 = one call.

Bail trigger is cumulative total failures (not consecutive). Execution order is preserved (chunks are contiguous slices of the runner's pre-order test list). Contract validation is skipped on an early stop (partial data). The protocolTimeout diagnostic was reworded for the chunked model.

Tests

242 passing. New coverage: multi-chunk accumulation, early-bail stops the loop at the threshold, maxFailures: 0 disables bail, partial results printed on a mid-run crash, contracts skipped on early stop, duplicate suite-skip dedupe across chunks, and the existing exact-block summary assertions unchanged (backward-compat on the normal path).

Design + plan: docs/superpowers/specs/2026-07-21-max-failures-early-bail-design.md, docs/superpowers/plans/2026-07-21-max-failures-early-bail.md.

🤖 Generated with Claude Code

[github-actions]

TWD Contract Validation

Spec	Passed	Failed	Warnings	Mode
./contracts/users-3.0.json	2	3	1	warn
./contracts/posts-3.1.json	2	2	0	warn
./contracts/products-3.0.json	13	23	2	warn
./contracts/events-3.1.json	6	13	0	warn

23 passed · 41 failed · 3 warnings · 1 skipped

Failed validations

./contracts/users-3.0.json

• GET /users/{userId} (200) — mock getUserNoAddress — in "Contract Validation - Mismatches > should fail: missing nested address field"
  • response.address: missing required property "address"
• GET /users/{userId} (200) — mock getUserBadAddress — in "Contract Validation - Mismatches > should fail: nested address missing required city"
  • response.address.city: missing required property "city"
  • response.address.country: missing required property "country"
• GET /users/{userId} (200) — mock getUserBadRole — in "Contract Validation - Mismatches > should fail: oneOf role with invalid variant"
  • response.role: oneOf best match (branch 2 of 2) failed: must be one of: "viewer"

./contracts/posts-3.1.json

• GET /posts/{postId} (200) — mock getPostNoAuthor — in "Contract Validation - Mismatches > should fail: post missing nested author object"
  • response.author: missing required property "author"
• GET /posts/{postId} (200) — mock getPostBadMeta — in "Contract Validation - Mismatches > should fail: post oneOf metadata matches neither variant"
  • response.metadata: oneOf best match (branch 1 of 2) failed: missing required property "category", unexpected property "duration", must be one of: "article"

./contracts/products-3.0.json

• GET /products (200) — mock getProductEmptyName — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: empty name violates minLength"
  • response[0].name: must NOT have fewer than 1 characters
• GET /products (200) — mock getProductBadSku — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid SKU pattern"
  • response[0].sku: must match pattern "^[A-Z]{2,4}-\d{4,8}$"
• GET /products (200) — mock getProductBadUuid — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid uuid format for id"
  • response[0].id: must match format "uuid"
• GET /products (200) — mock getProductBadDateTime — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid date-time format"
  • response[0].createdAt: must match format "date-time"
• GET /products (200) — mock getProductBadDate — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid date format"
  • response[0].releaseDate: must match format "date"
• GET /products (200) — mock getProductBadEmail — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid email format"
  • response[0].contactEmail: must match format "email"
• GET /products (200) — mock getProductBadUri — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid uri format"
  • response[0].website: must match format "uri"
• GET /products (200) — mock getProductBadIp — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid ipv4 format"
  • response[0].serverIp: must match format "ipv4"
• GET /products (200) — mock getProductBadIpV6 — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid ipv6 format"
  • response[0].serverIpV6: must match format "ipv6"
• GET /products (200) — mock getProductZeroPrice — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: price of 0 violates exclusiveMinimum"
  • response[0].price: must be > 0
• GET /products (200) — mock getProductNegQty — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: negative quantity violates minimum"
  • response[0].quantity: must be >= 0
• GET /products (200) — mock getProductOverQty — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: quantity exceeds maximum"
  • response[0].quantity: must be <= 999999
• GET /products (200) — mock getProductBadWeight — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: weight not multipleOf 0.01"
  • response[0].weight: must be multiple of 0.01
• GET /products (200) — mock getProductBadRating — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: rating above maximum (5)"
  • response[0].rating: must be <= 5
• GET /products (200) — mock getProductBadCurrency — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid enum value for currency"
  • response[0].currency: must be one of: "USD", "EUR", "GBP", "JPY"
• GET /products (200) — mock getProductBadCategory — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid enum value for category"
  • response[0].category: must be one of: "electronics", "clothing", "food", "books", "toys"
• GET /products (200) — mock getProductBadBool — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: string value for boolean inStock"
  • response[0].inStock: expected boolean, got string
• GET /products (200) — mock getProductDupTags — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: duplicate tags violates uniqueItems"
  • response[0].tags: must NOT have duplicate items (items ## 1 and 0 are identical)
• GET /products (200) — mock getProductTooManyTags — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: tags exceeds maxItems (10)"
  • response[0].tags: must NOT have more than 10 items
• GET /products (200) — mock getProductBadMeta — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: non-string value in metadata additionalProperties"
  • response[0].metadata.count: expected string, got number
• GET /settings (200) — mock getSettingsBadExtra — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: extra property on Settings (additionalProperties: false)"
  • response.extraField: unexpected property "extraField"
• GET /settings (200) — mock getSettingsBadLang — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: invalid language pattern in Settings"
  • response.language: must match pattern "^[a-z]{2}(-[A-Z]{2})?$"
• GET /products (200) — mock getProductBadNullable — in "Contract Validation - Products Mismatches (OpenAPI 3.0 — error mode) > should fail: wrong type for nullable description (number instead of string|null)"
  • response[0].description: expected string,null, got number

./contracts/events-3.1.json

• GET /events (200) — mock getEventsEmpty — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: empty events array violates minItems (1)"
  • response: must NOT have fewer than 1 items
• GET /events (200) — mock getEventShortName — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: event name too short (minLength: 3)"
  • response[0].name: must NOT have fewer than 3 characters
• GET /events (200) — mock getEventBadDate — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid date-time format for startDate"
  • response[0].startDate: must match format "date-time"
• GET /events (200) — mock getEventFloatId — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: float value for integer id"
  • response[0].id: expected integer, got number
  • response[0].id: must match format "int64"
• GET /events (200) — mock getEventBadBool — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: number value for boolean active"
  • response[0].active: expected boolean, got number
• GET /events (200) — mock getEventBadStatus — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid enum value for status"
  • response[0].status: must be one of: "draft", "published", "archived"
• GET /events (200) — mock getEventScoreMax — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: score at exclusiveMaximum boundary (100)"
  • response[0].score: must be < 100
• GET /events (200) — mock getEventLowPriority — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: priority below minimum (1)"
  • response[0].priority: must be >= 1
• GET /events (200) — mock getEventHighPriority — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: priority above maximum (5)"
  • response[0].priority: must be <= 5
• GET /events (200) — mock getEventDupAttendees — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: duplicate attendees violates uniqueItems"
  • response[0].attendees: must NOT have duplicate items (items ## 1 and 0 are identical)
• GET /events (200) — mock getEventNoAttendees — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: empty attendees array violates minItems (1)"
  • response[0].attendees: must NOT have fewer than 1 items
• GET /events (200) — mock getEventBadAttendee — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: invalid email format in attendees"
  • response[0].attendees[0]: must match format "email"
• GET /events/{eventId} (200) — mock getEventBadNullable — in "Contract Validation - Events Mismatches (OpenAPI 3.1 — error mode) > should fail: wrong type for nullable description (number instead of string|null)"
  • response.description: expected string,null, got number

View full report →