fix(review): recover the model's JSON from reasoning-model output - #7
Closed
asdd-agent[bot] wants to merge 2 commits into
Closed
fix(review): recover the model's JSON from reasoning-model output#7asdd-agent[bot] wants to merge 2 commits into
asdd-agent[bot] wants to merge 2 commits into
Conversation
A reasoning model wraps its review object in analysis prose (with its own
braces), code fences, or trailing commentary, or leaves message.content empty
and emits the JSON in message.reasoning_content. The previous
first-brace-to-last-brace recovery captured an invalid span (or missed the
reasoning field entirely), so the runtime discarded a real review and fell
back to a 'human should review manually' placeholder even though the model
had reviewed.
Extraction now uses a real JSON parser (extract-json.py): it sweeps every
'{', lets the decoder consume the largest balanced object, and returns the
one that most looks like a review, while emitting nothing on genuine non-JSON
so the gate still fails closed. The adapter reads the answer across the
response shapes providers use (content string, content array of parts, legacy
text) AND message.reasoning_content, and logs a key-safe redacted diagnostic
on a persistent failure. Docs note added: reviewer and lens roles want a
JSON-reliable model; a reasoning model fits the free-form developer/council.
Spec: docs/specs/robust-review-json-extraction.md
Tests: .github/asdd/runtime/extract-json.test.sh, wired into validation/run-base.py
Signed-off-by: Christoph <awchristoph@gmail.com>
The de-slop gate hard-fails on a fixed fluff-word list; the spec title and its inbound links used one. Rename the spec to review-json-recovery.md and reword the links. No behaviour change. Signed-off-by: Christoph <awchristoph@gmail.com>
|
ASDD review - advisory (recommendation:
The review runtime returned invalid output; a human should review manually. Security scan (deterministic + SAST): no new issues. Impact scan: 1 finding(s), 0 block. SECURITY - okNo findings. IMPACT - concerns
Generated by the ASDD advisory review. Mode: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The review runtime discarded a valid review when the model returned its JSON object wrapped in reasoning prose, code fences, or a separate
reasoning_contentfield. The old first-brace-to-last-brace recovery captured an invalid span, so a real review degraded to a "human should review manually" placeholder.Extraction now uses a real JSON parser (
.github/asdd/runtime/extract-json.py) that recovers the review object wherever the model put it, and still emits nothing on genuine non-JSON so the gate keeps failing closed. The adapter reads the answer acrosscontent(string or array of parts), legacytext, andmessage.reasoning_content, and logs a key-safe redacted diagnostic on a persistent failure.The fix is at the shared adapter (
openai-compat.sh), so it also covers the operator-run fixed-prompt agents that call it throughrun-agent.sh(triage, support, review-contributor, review-merge). The Goose-run operate agents take a different path and are unaffected.Spec:
docs/specs/robust-review-json-extraction.md. Test:.github/asdd/runtime/extract-json.test.sh, wired intovalidation/run-base.py(base suite green).Authorship
Agent:trailer.Impact
generic.shstill validates the merged object and fails closed when extraction yields nothing).Checklist