fix(tools): fail-loud native-crash retry for test:coverage:check (#1278)

[SisyphusZheng]

Closes #1278

Problem

test:coverage:check has died by SIGSEGV (exit 139) three times — Beta.1 CI run 33715486563 at 73a7900, a local run at 18778c9, and the first autoflow-ci run of PR #1291 (rerun green). Each time the crash was in the deno test --coverage subprocess (tools/check-coverage.ts reported "tests failed with code 139") while every other gate passed: a native-binding crash (rolldown/workerd class) under load, not a product signal. The gate is release-tier (tiers: ['ci', 'release'] in tools/autoflow/policy.ts), so a random crash could block a release.

Owner

v0.44 implementer (B2.13 early slice under stage issue #1288, umbrella #1155).

Before

Any non-zero exit from the coverage test run — assertion failure or native crash alike — threw tests failed with code N and failed the gate on the first occurrence. A random SIGSEGV blocked the gate exactly like a real regression, and a silent manual rerun hid the flake from the record.

After

tools/check-coverage.ts classifies the test-run exit code:

• Exit < 128 (incl. 1, the deno test assertion-failure code): real failure → fail immediately, zero retries. No blanket-retry masking.
• Exit ≥ 128 (signal termination, 128 + signo: 132 SIGILL / 134 SIGABRT / 139 SIGSEGV / …): native crash → bounded retry (default --crash-retries 2, i.e. max 3 attempts). Each retry starts from a clean .coverage-check dir so a crashed attempt can't poison deno coverage with partial profiles.
• Every crash is reported loudly on stderr ([check-coverage] NATIVE CRASH … attempt N/M (#1278)), and a recovered run prints a closing WARNING with the crash count — flakes stay visible and countable in CI logs.
• Crash exhaustion fails the gate explicitly ("crashed natively … on all 3 attempts; refusing to pass the gate on repeated native crashes").

Coverage semantics are untouched: thresholds, scopes, and the full-denominator logic are unchanged.

Why tool-level retry, not workflow-level

The gate reaches CI and release exclusively through the autoflow policy command deno task test:coverage:check (tools/autoflow/policy.ts, deno task autoflow:ci in .github/workflows/autoflow-ci.yml). Putting the retry inside the tool means:

1. It applies in the CI path by construction (the CI-gating rule) — no workflow change can accidentally bypass it, and local runs get identical semantics.
2. GitHub Actions has no native step retry; a workflow-level shell loop would duplicate the classification logic in YAML, fork CI/local semantics, and bury crash counts in a wrapper. Tool-level keeps one tested implementation with loud per-crash stderr lines in the gate log itself.

Why-not-second-owner

Bounded single-owner slice: one tool file plus its new test file; no frozen-semantics paths (freeze:semantics:check green), no workflow or governance edits.

Root-cause investigation

Attempted, not cheaply reachable:

• Serialization: Deno 2.9 deno test already runs test modules serially by default (verified locally: two 2 s test files take ~4 s with and without DENO_JOBS=1). The concurrency is inside tests — spawned subprocesses exercising npm/native bindings — so runner-level serialization is not a reachable fix.
• Pinning native bindings / core capture: no stack or core exists from the three crashes, and no local repro; pinning versions blind would risk destabilizing coverage semantics for an unproven cause.

The fail-loud retry is therefore the delivered fix; the loud crash counter keeps the flake measurable so a future root-cause attempt has data.

Evidence

• New tools/check-coverage.test.ts (7 tests, all green): exit classification; crash-without-assertion-failure retries then passes with per-crash reports; assertion failure fails immediately with the runner invoked exactly once; crash exhaustion after the bounded attempt count fails loudly; maxAttempts validation.
• Real-subprocess end-to-end proof (temp harness driving the exported retry logic against actual deno run children exiting 139/1/0):
  • crash, crash, pass → recovers, crashes=2, two loud reports, exit 0
  • assertion failure → tests failed with code 1, immediate, exit 1, no retry
  • crash ×3 → three loud reports, explicit exhaustion error, exit 1
• Full gate deno task test:coverage:check (runs the entire repo suite incl. the new tests): PASS, all thresholds met (packages: 81.94/85.12/90.13%; tools/lib: 77.56/85.92/79.22%).
• deno fmt --check, deno lint, deno check on touched files: clean.
• deno task arch:check, type-safety:check, text-integrity:check, freeze:semantics:check: PASS.
• Publishability: deno task pack:dry-run and deno task package-artifacts:check: PASS.

Scope

• tools/check-coverage.ts — exit classification + bounded fail-loud retry; --crash-retries flag (default 2).
• tools/check-coverage.test.ts — new, proves the retry contract.
• No workflow changes; no coverage-threshold or denominator changes; no production package changes.

Risk

Low. Worst case a native crash adds up to two extra full test runs before the gate fails (bounded, and each crash is loud in logs). Real failures are never retried. If the crash turns out to be deterministic rather than flaky, the gate still fails after 3 attempts — fail-closed, never silent PASS.

[cloudflare-workers-and-pages]

Deploying openelement with    Cloudflare Pages

Latest commit:	20edcaa
Status:	✅  Deploy successful!
Preview URL:	https://b65fca80.lessjs.pages.dev
Branch Preview URL:	https://fix-1278-coverage-sigsegv-re.lessjs.pages.dev

View logs

[github-actions]

APIError: Insufficient Balance

opencode session  |  github run