fix(stella-pipeline): verification follow-ups round 3 — flip rescues the fallback, budget stops degrade, arming symptoms recorded

[macanderson]

What this is

Round 3 of the verification-role work (#1798, #1813 were rounds 1–2): three decision-gated issues from the audit backlog, decided with the reasoning recorded in code, each with a witness test.

The decisions

1. A verifier outage is not a refutation (Closes #1788). heuristic_fallback now passes on an observed fail→pass flip, not only on green touched tests. The asymmetry it closes: Unverifiable abstains when the evidence is absent, but a missing checker drove a flip-verified candidate (diff over budget → ModelVerdict → provider down) to VerificationFailed — the checker's absence treated as the work's failure. With nothing deterministic positive the fallback still fails closed. This deliberately inverts the "even a flip doesn't rescue an unconfirmed suite" pin; the test now states why.

2. The scaffolding's budget must not discard the work (Refs #1789 — the reclassification + de-panic half). A budget stop during witness authoring/repair was rejected, aborting a candidate whose worker change was already complete. It degrades now: the budget guard still gates every later paid call (no overspend is possible), and what degrading buys is the deterministically-resolvable endings — a warranted waiver, an abstention — that need no further spend. The stage's two expects on workspaces became degradable aborts, per its own produce-vs-trust contract. The issue stays open for the end-to-end complete-without-further-spend scenario.

3. A build-failure baseline is recorded, never refused (Closes #1790). Resolved per the on-issue analysis: refusing SymptomClass::BuildFailure baselines (the issue's original framing) would reject the most common Rust witness shape — a missing-API test fails to compile on the old code by design. Instead the arming failure's class is recorded (witness_baseline=build_failure in the verifier's trusted evidence, plus a run warning), so a compile-armed flip is visible to the verdict and to anyone reading the evidence — the honest treatment for a shape that is legitimate for missing-API goals and identical to two-tree environment drift.

4. The cache-minimum measurement is a stated fact (Closes #1786 — the split landed in fix(stella-pipeline,stella-cli): verification follow-ups round 2 — role tuning parity, policy reach, cache-stable witness prompt, scoped goal verifier #1813). The management-prompt module doc records the measured prefix sizes: no fixed instruction block clears Anthropic's 1024-token minimum alone (verdict ~520, witness author ~620); the raw calls cache only with an agents.<role>.prompt override padding the prefix, while the witness author's engine turn crosses the minimum within its first tool round-trip — which is exactly where fix(stella-pipeline,stella-cli): verification follow-ups round 2 — role tuning parity, policy reach, cache-stable witness prompt, scoped goal verifier #1813's split pays.

Witness tests

heuristic_fallback_passes_only_on_confirmed_green_tests (inverted pin + new no-evidence case), a_build_failure_baseline_is_recorded_and_an_assertion_one_is_not. Both fail on main.

CI note

Until #1859 (witness_stage clippy, merged with #1813's premature auto-merge) and #1845 (file-size baseline skew: deck_render.rs +3) land, the required job is red on those pre-existing steps — none are in hunks this PR touches. I'll update-branch after they merge. Coordination: my #1795 implementation was dropped from this round in favor of the more complete open #1867; my duplicate unbreak #1874 was closed in favor of #1859/#1845.

Still tracked

#1787 (structured verdict output, per-candidate degradation records), #1789 (the e2e half), #1793 (FlipHalt — gated on a loop-bench measurement), #1794 (semantic witness review — needs its design pass; recommendation on the issue is extending the mutation audit to flip-corroborated ModelVerdict passes).

Closes #1788
Closes #1790
Closes #1786
Refs #1789

Summary by Sourcery

Adjust verification pipeline behavior to treat verifier outages, budget stops, and build-failure witnesses as recorded, degradable conditions rather than hard rejections, and document cache minimum measurements for management prompts.

Bug Fixes:

• Allow heuristic fallback verdicts to pass when a confirmed fail→pass flip exists even if the verifier is unavailable.
• Prevent witness authoring and repair stages from discarding completed work on budget stops or missing workspaces by degrading the run instead of rejecting or panicking.
• Record build-failure baselines for witnesses in pipeline state and evidence summaries instead of refusing them, ensuring missing-API compile errors are treated as legitimate but weaker evidence.

Enhancements:

• Expose the authored witness baseline symptom class in candidate state and evidence output so verifiers can distinguish build failures from assertion failures.
• Clarify management prompt documentation with measured token-prefix sizes and their implications for provider cache behavior.

Tests:

• Extend heuristic fallback tests to cover verifier outage behavior and no-positive-evidence failure cases.
• Add a witness-stage unit test to assert that build-failure baselines are recorded while assertion-based baselines are not.

[sourcery-ai]

Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
stella-cli-docs	Ignored		Aug 6, 2026 11:25am

[sourcery-ai]

Reviewer's Guide

Adjusts verification pipeline behavior to treat verifier outages and budget stops as degradable conditions rather than hard failures, records build-failure witness baselines as explicit evidence instead of rejecting them, and documents measured cache/prefix behavior in the management prompt module.

File-Level Changes

Change	Details	Files
Treat verifier outages as non-refuting when a fail→pass flip is already confirmed, while still failing closed when there is no deterministic positive evidence.	Updated heuristic_fallback to pass when either a fail→pass flip is achieved or touched tests are confirmed green, and to fail otherwise. Adjusted fallback reasoning strings to distinguish between flip-based passes, green-test passes, and cases with no deterministic evidence. Expanded heuristic_fallback_passes_only_on_confirmed_green_tests test to assert that a confirmed flip survives verifier outages and that lack of positive evidence still fails.	crates/stella-pipeline/src/verify.rs crates/stella-pipeline/src/verify/tests.rs
Make witness authoring/repair budget stops and missing workspaces degradable aborts so that completed work is preserved while still enforcing budget limits.	Replaced expect! panics on missing baseline and candidate workspaces with degradable WitnessAbort errors that preserve the run but drop witness authoring/grafting. Changed budget_abort handling in witness authoring and repair turns from rejected aborts to degradable aborts with clearer messages indicating the work stands unproven. Clarified stage-level documentation to treat mid-authoring budget stops as degradable conditions rather than hard failures.	crates/stella-pipeline/src/pipeline/witness_stage.rs
Record and surface build-failure baselines for witnesses as an explicit, weaker evidence signal instead of refusing them outright.	Added witness_baseline_symptom helper to classify baseline output as a build failure using SymptomClass and return a static tag. Extended candidate state to track an optional witness_baseline_symptom and initialized it in pipeline construction. Set witness_baseline_symptom when a witness’s baseline output is a build failure, and emit a warning explaining the implications for missing-API goals and environment drift. Included witness_baseline_symptom in the evidence summary string so verifiers and readers can see when the arming failure never ran a test. Added a unit test verifying that build-failure baselines are tagged and assertion-failure baselines are not.	crates/stella-pipeline/src/pipeline/witness_stage.rs crates/stella-pipeline/src/pipeline.rs crates/stella-pipeline/src/pipeline/evidence.rs
Document measured cache-minimum and prefix-size behavior for management prompts and witness authoring to clarify when cache hits are possible.	Expanded management_prompt module-level documentation to record measured token counts for various system prompts and to explain when the split yields real cache wins versus just prefix stability.	crates/stella-pipeline/src/management_prompt.rs

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#1786	Move the fixed hard-requirements block from the witness author user prompt into its system message, and add a prompt-shape test verifying this prompt structure.	❌	The diff does not modify crates/stella-pipeline/src/witness.rs or any witness author prompt construction. There is no change that moves the hard-requirements block into the system message, nor is there any new prompt-shape test asserting the system/user split for the witness author.
#1786	Measure and document management roles’ stable-prefix token counts against provider cache minimums, including the witness author, and state explicitly when the management prompt split yields cache benefits.	✅
#1788	Change heuristic_fallback so that a verifier outage is not treated as a refutation: deterministic positive evidence (especially a confirmed fail→pass flip) causes a pass, and the behavior is documented and tested.	✅
#1788	Align the HeuristicFallback rung’s scoring for VerifierUnavailable in reward.rs::outcome_term with the chosen behavior for verifier outages.	❌	The diff does not include any changes to reward.rs or outcome_term; there is no visible modification to the scoring for VerifierUnavailable to reflect the new heuristic_fallback behavior.
#1790	Use SymptomClass::BuildFailure to classify authored witness baseline failures (compile-error tails with no tests) in the witness baseline runs.	✅
#1790	Change behavior so that a baseline failure classified as BuildFailure is treated like an infrastructure outcome: the witness baseline run degrades and does not arm the flip oracle / failing baseline cannot be established.	❌	The PR introduces witness_baseline_symptom to classify and record build-failure baselines, adds a warning, and surfaces the symptom in the evidence summary, but it does not alter the arming or degradation logic. BuildFailure baselines are explicitly "recorded, never refused" and can still arm the flip; there is no change to make such baselines degrade or prevent establishment of a failing baseline.
#1790	Add or adjust tests (in the witness harness) so that a scripted baseline run returning a compile-error tail (no test names) is verified to degrade with a named reason instead of arming a failing baseline.	❌	The new test a_build_failure_baseline_is_recorded_and_an_assertion_one_is_not only checks that witness_baseline_symptom returns "build_failure" for compile-error output and None for an assertion failure. It does not exercise the end-to-end witness baseline behavior (arming vs degrading), nor does it live in the witness isolation harness described in the issue. No test verifies that compile-error baselines now degrade instead of arming.

Possibly linked issues

• Verifier outage fails closed (heuristic) while blind evidence abstains (Unverifiable) — decide the asymmetry or let a flip rescue the fallback #1788: The PR directly implements Verifier outage fails closed (heuristic) while blind evidence abstains (Unverifiable) — decide the asymmetry or let a flip rescue the fallback #1788’s decision: heuristic_fallback now passes on confirmed flips during verifier outages.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.