Baseline verification and stop autorun repair loops that make no progress

[marcellocurto]

Summary

Autorun cannot distinguish verification failures introduced by the current change from failures already present in the prepared repository. After Review A/B approve, almost every ordinary nonzero verification exit is treated as repairable, and Roark can consume every remaining fix pass even when neither the worktree nor verification output changes.

The same lifecycle writes ready-for-pr before running authoritative verification. On failure, it can publish that ready result immediately before posting a verification failure.

Add a durable pre-implementation verification baseline, structured verification snapshots, deterministic no-progress detection, and final readiness that includes verification. This refines the verification-repair behavior introduced by #82; it does not remove repair passes for failures plausibly introduced by the current change.

Goals

• Run the configured verification command in the prepared managed workspace before implementation can edit files.
• Run workspace copying and the beforeVerify hook at the same boundary for baseline and final verification so results are comparable.
• Make a failing baseline terminal by default, before implementation, with a distinct failed-baseline-verification outcome and actionable recovery information.
• Add an explicit --allow-failing-baseline CLI option and allowFailingBaseline repository configuration for issues intentionally authorized to repair a red baseline.
• Keep final verification mandatory even when a failing baseline is explicitly allowed; the override must never permit publication with a failed verifier.
• Persist structured snapshots for baseline, current verification, and archived pre-fix verification results.
• Calculate the verification fingerprint from normalized full command output, not the truncated Markdown tail.
• Include at least the command, exit code, timeout state, success state, output fingerprint, and worktree fingerprint in each structured snapshot.
• Normalize ANSI escapes, CRLF, the managed workspace’s absolute path, and trailing whitespace without removing diagnostic content.
• Calculate the worktree fingerprint from the binary diff against the captured pre-implementation Git baseline plus sorted untracked paths and their content hashes. Exclude .roark, ignored dependency state, and other ignored control/setup material.
• Stop automatic repair after a completed repair cycle when both the verification fingerprint and worktree fingerprint are unchanged from the preceding post-implementation verification.
• Continue to permit another repair when either the worktree or verifier output changed and fix budget remains.
• Preserve existing terminal treatment for timeouts, unavailable commands, permissions, and other failures already classified as non-repairable.
• Treat Review A/B approval as eligibility to verify, not final PR readiness.
• Run the successful lifecycle as reviews → final verification → final readiness → publication.
• Change readiness to a new structured version in which ready-for-pr requires triage proceed, an implementable plan, Review A approval, Review B approval, and passed verification.
• Produce not-ready when verification is failed, missing, or not run. Do not include a recommended PR title or body in a not-ready artifact.
• Never publish a ready-for-pr comment before verification succeeds.
• Make continue reuse valid structured baseline and verification snapshots, preserve no-progress terminal decisions, and safely recompute legacy version-2 readiness artifacts.
• Document baseline policy, the override, artifacts, readiness semantics, no-progress termination, and recovery behavior.
• Keep the behavior portable for global installs, CI, servers, and managed workspaces; do not introduce machine-local runtime dependencies.
• Limit this ticket to auto and continue. Do not broaden it into new baseline policy for review-pr or revise-pr, while preserving compatibility for their use of shared verification utilities.

Research and findings

Confirmed current behavior:

• lib/workflow/phases.ts captures only the Git head and reset information before implementation; it does not run verification.
• lib/autorun/verification.ts classifies nearly every ordinary nonzero exit as repairable.
• lib/autorun/publish-flow.ts schedules the next available fix pass without comparing output or worktree state.
• lib/autorun/attempt-lifecycle.ts repeats fix, refinement, Review A/B, readiness, and verification until the shared fix budget is exhausted.
• lib/workflow/readiness.ts derives ready-for-pr from triage, plan, and Review A/B only.
• lib/autorun/publish-flow.ts reads that readiness result before verification and publishes the ready artifact on the verification-failure path.
• lib/autorun/continue-plan.ts reconstructs verification from Markdown output and has no baseline or no-progress state.
• Existing final verification has Markdown tail and full-output artifacts but no structured JSON source of truth.

Issue marcellocurto/app.tenderintelligenceplatform.com#647 is the regression case:

• Review A and Review B approved passes 0 through 4.
• Initial verification reported 31 lint errors. Fix pass 1 correctly removed two errors introduced in the issue’s changed component.
• The next four complete verification outputs were byte-identical and reported 29 errors in unrelated unchanged files.
• Fix and refinement passes 2 through 4 reported no source edits.
• Roark posted ready-for-pr, then posted verification failure eight seconds later.
• The attempt used 22 agent phases, 3,153,686 tokens, approximately $7.78, and approximately 32.9 minutes.
• The managed workspace retained two modified files on a local unpushed branch, with no PR.
• The target repository required eslint-plugin-react-hooks 7.1.1 in package.json and bun.lock. The developer checkout had 7.0.1 installed, while Roark’s prepared workspace correctly installed 7.1.1 and exposed the baseline failures.

The failing-baseline override has deliberate semantics: it authorizes an issue to attempt repairing the baseline. Therefore, equality between the baseline and the first post-implementation verification must not by itself suppress the first repair. No-progress termination applies after a completed repair cycle leaves both code and verification unchanged.

Suggested artifacts:

• baseline-verification.json
• baseline-verification.md
• baseline-verification-full.md
• verification.json
• existing verification.md and verification-full.md
• verification-before-fix-N.json
• existing verification-before-fix-N.md and verification-before-fix-N-full.md

Current starting points include lib/workflow/artifact-catalog.ts, lib/workflow/artifacts.ts, lib/autorun/verification.ts, lib/autorun/discovery.ts, lib/autorun/attempt-lifecycle.ts, lib/autorun/publish-flow.ts, lib/autorun/continue-plan.ts, lib/autorun/attempts.ts, lib/workflow/progression.ts, lib/workflow/readiness.ts, lib/cli/args.ts, and lib/cli/hydrate.ts. These are starting points, not a required file inventory.

Validation

Behavior-level coverage must prove:

• A passing baseline permits implementation.
• A failing baseline prevents invocation of the implementation agent by default.
• The explicit override permits implementation but still requires a passing final verifier before publication.
• An introduced verifier failure can schedule a repair.
• Returning from 31 failures to the 29-failure baseline does not consume repeated no-op repair cycles.
• Under the failing-baseline override, the first unchanged final result can receive one repair opportunity, but an unchanged repair cycle terminates.
• Changed code or changed verifier output can continue within the remaining fix budget.
• Missing or failed verification cannot produce ready-for-pr.
• Successful final verification produces final ready artifacts and permits publication.
• No ready comment is published before verification passes.
• Continuation reuses valid snapshots and does not restart a stopped no-progress loop.
• Legacy version-2 attempts recover by deterministic recomputation rather than being silently misread.
• Existing targeted autorun, workflow, configuration, artifact, and readiness tests pass, followed by the repository verification command.