fix(verify): interactive probe stops false-negativing clean runs on natural-language expect - #11
Merged
Conversation
…rose `expect`
Live finding (cmax-building Plumb v1): the rollup verdict marked rollup-tests-green
and rollup-typecheck-clean as NOT met with failureCategory "interactive-failure",
while their own recorded evidence was `# pass 77 / # fail 0` and `tsc EXIT=0`. Root
cause: runInteractiveVerify required the model-authored natural-language `expect`
("EXIT=0 with no failing tests and at least 11 passing") to appear verbatim in
stdout (`combined.includes(expect)`). Prose never matches, so met = exit0 && false,
and the probe — authoritative in parseOneCondition — forced the condition to FAIL.
Fix: extract a tolerant evaluateExpect():
- mechanical success = echoed `EXIT=<n>` sentinel (the `; echo EXIT=$?` idiom
masks the real exit to 0) else the real process exit code;
- `expect` is corroboration, not a prose veto: literal substring, or all salient
tokens (TAP counters, ok/not-ok lines, quoted literals) must be present; a short
single-token sentinel still hard-fails if absent; pure prose is advisory.
Genuine failures (echoed EXIT!=0, real non-zero exit, "# fail 3" vs demanded
"# fail 0", absent literal sentinel) still yield met=false. 12 new regression tests;
full runtime suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live finding while cmax-building Plumb v1: the rollup verdict marked
rollup-tests-greenandrollup-typecheck-cleanNOT-met (interactive-failure) while their own evidence recorded# pass 77 / # fail 0andtsc EXIT=0.Root cause:
runInteractiveVerifyrequired the model's natural-languageexpectto be a literal substring of stdout. Prose never matches →met = exit0 && false→ the authoritative probe forces the condition to FAIL.Fix:
evaluateExpect()— mechanical signal from an echoedEXIT=<n>sentinel (or the real exit code);expectbecomes corroboration (literal substring / salient TAP tokens / advisory prose), never a prose veto. Short literal sentinels still hard-fail when absent; real failures still fail. +12 regression tests, full runtime suite green.🤖 Generated with Claude Code