Skip to content

fix(review): recover the model's JSON from reasoning-model output - #7

Closed
asdd-agent[bot] wants to merge 2 commits into
mainfrom
harden-review-runtime-json
Closed

fix(review): recover the model's JSON from reasoning-model output#7
asdd-agent[bot] wants to merge 2 commits into
mainfrom
harden-review-runtime-json

Conversation

@asdd-agent

@asdd-agent asdd-agent Bot commented Jul 23, 2026

Copy link
Copy Markdown

What this changes

The review runtime discarded a valid review when the model returned its JSON object wrapped in reasoning prose, code fences, or a separate reasoning_content field. The old first-brace-to-last-brace recovery captured an invalid span, so a real review degraded to a "human should review manually" placeholder.

Extraction now uses a real JSON parser (.github/asdd/runtime/extract-json.py) that recovers the review object wherever the model put it, and still emits nothing on genuine non-JSON so the gate keeps failing closed. The adapter reads the answer across content (string or array of parts), legacy text, and message.reasoning_content, and logs a key-safe redacted diagnostic on a persistent failure.

The fix is at the shared adapter (openai-compat.sh), so it also covers the operator-run fixed-prompt agents that call it through run-agent.sh (triage, support, review-contributor, review-merge). The Goose-run operate agents take a different path and are unaffected.

Spec: docs/specs/robust-review-json-extraction.md. Test: .github/asdd/runtime/extract-json.test.sh, wired into validation/run-base.py (base suite green).

Authorship

  • Authored by a human.
  • Authored or co-authored by an AI agent under human direction. The agent identity is named below and each agent commit carries the Agent: trailer.

Agent identity (if any): asdd-agent · Instructed by (human handle): welsbach

Impact

  • Non-normative: a fix that does not change the standard's text, the governance rules, or the review contract (generic.sh still validates the merged object and fails closed when extraction yields nothing).
  • Normative.

Checklist

  • Commits are signed off (DCO).
  • Conventional Commit message(s).

A reasoning model wraps its review object in analysis prose (with its own
braces), code fences, or trailing commentary, or leaves message.content empty
and emits the JSON in message.reasoning_content. The previous
first-brace-to-last-brace recovery captured an invalid span (or missed the
reasoning field entirely), so the runtime discarded a real review and fell
back to a 'human should review manually' placeholder even though the model
had reviewed.

Extraction now uses a real JSON parser (extract-json.py): it sweeps every
'{', lets the decoder consume the largest balanced object, and returns the
one that most looks like a review, while emitting nothing on genuine non-JSON
so the gate still fails closed. The adapter reads the answer across the
response shapes providers use (content string, content array of parts, legacy
text) AND message.reasoning_content, and logs a key-safe redacted diagnostic
on a persistent failure. Docs note added: reviewer and lens roles want a
JSON-reliable model; a reasoning model fits the free-form developer/council.

Spec: docs/specs/robust-review-json-extraction.md
Tests: .github/asdd/runtime/extract-json.test.sh, wired into validation/run-base.py
Signed-off-by: Christoph <awchristoph@gmail.com>
@asdd-agent
asdd-agent Bot requested a review from welsbach as a code owner July 23, 2026 22:11
@welsbach welsbach added the fix label Jul 23, 2026
The de-slop gate hard-fails on a fixed fluff-word list; the spec title and
its inbound links used one. Rename the spec to review-json-recovery.md and
reword the links. No behaviour change.

Signed-off-by: Christoph <awchristoph@gmail.com>
@welsbach welsbach closed this Jul 23, 2026
@welsbach
welsbach deleted the harden-review-runtime-json branch July 23, 2026 22:21
@github-actions

Copy link
Copy Markdown

ASDD review - advisory (recommendation: comment)

Produced by an automated agent under human direction. Advisory only: a human approves and merges.
Nothing here merges automatically.

The review runtime returned invalid output; a human should review manually. Security scan (deterministic + SAST): no new issues. Impact scan: 1 finding(s), 0 block.

SECURITY - ok

No findings.

IMPACT - concerns

  • warn: This change touches a behavioural surface (a gate, lens, agent prompt, or the runtime seam: .github/asdd/runtime/extract-json.py, .github/asdd/runtime/extract-json.test.sh, .github/asdd/runtime/openai-compat.sh, validation/run-base.py). If it changes behaviour adopters rely on for conformance, it is normative: declare it and add an impact analysis and target version. If it preserves behaviour, no action.

Generated by the ASDD advisory review. Mode: live.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant