fix: recover specs when a model drops a closing bracket

[RubenGlez]

The accuracy gate wasn't flaky — it was faithfully reporting a real bug that had been invisible for weeks.

What was actually happening

The gate's means were exact multiples of 1/14 and the worst fixture always scored 0.00 flat. With a weighted scorer (F1, partial credit per dimension) a slightly-wrong spec can't score zero, so every zero had to be the catch branch. It was:

threw: 4/14  |  scored-but-imperfect: 0

Every reply that parsed scored a perfect 1.00. The zeros weren't inaccuracy — they were exceptions. Capturing the raw HTTP response on the real buildOperation path:

[finish_reason=stop  completion_tokens=125]     ← not truncated by max_tokens
opening braces: 16 | closing braces: 15         ← one missing
chars after last }: ""                          ← no prose, no fences

deepseek-chat under-closes deeply nested response schemas and then stops, believing it finished — so all three attempts reproduced the identical mistake and the endpoint was documented as nothing at all.

This is not just a CI problem: every EasyDocs user on DeepSeek was losing ~30% of their endpoints. It's the cheapest supported provider and it's featured in BENCHMARK.md.

The fix, and why it isn't a one-liner

The obvious repair — append the missing } — parses, but produces the wrong document:

responses.security: Expected object, received array

The dropped bracket belonged before ,"security", not at the end. Closing at the end buries security inside responses, which then fails schema validation anyway.

So jsonCandidates yields each plausible reading (exact → closed at the end → closed at each member boundary) and the caller takes the first that satisfies OperationSchema. The schema is the only reliable arbiter of where the bracket belonged. A well-formed reply still takes the fast path and yields exactly one candidate.

Two supporting changes:

• Retry guidance names the likely cause instead of echoing Expected ',' or '}' ... at position 557, which the model demonstrably ignored — all three attempts repeated the same error.
• The gate prints why each imperfect fixture lost points. It previously reported a bare mean, which cannot distinguish "slightly wrong spec" from "generation threw and scored zero" — opposite fixes, and the second one hid this for weeks.

Measured effect

deepseek-chat on the eval suite, before:

mean=0.714   threw: 4/14      (samples across runs: 0.714, 0.786, 0.786, 0.857, 0.929)

After, three consecutive full gate runs:

deepseek/deepseek-chat   mean=0.951   worst post-orders.json (0.31)
deepseek/deepseek-chat   mean=0.951   worst post-orders.json (0.31)
deepseek/deepseek-chat   mean=0.964   worst post-orders.json (0.50)
openai/gpt-4o            mean=1.000
anthropic/claude-haiku   mean=1.000
✅ Accuracy gate passed (×3)

Zero generation failures, and the suite runs in half the time because it no longer burns three retries per failure.

post-orders.json now surfaces as a genuine partial miss (responses=0.00, responseSchema=0.00) rather than a hard zero — a real, separate accuracy issue that is now visible instead of masked.

Verification

• build, lint, typecheck green; 266 tests pass (was 256).
• 10 new tests in builder-json.test.ts, built on the verbatim malformed output captured from deepseek-chat. They assert the repair puts security at the top level and not inside responses — i.e. that it picked the correct insertion point, not merely a parseable one.
• Also asserted: repair happens on the first attempt without burning retries, braces inside string values don't unbalance the count, well-formed and fenced output are untouched, and output broken some other way (a missing comma) is not papered over.

Not included

JSON mode (response_format: json_object) would prevent the malformation at the source rather than repairing it, but it's provider-specific and worth evaluating separately.

https://claude.ai/code/session_01GrfumUQ4vFUAgsFYwzDyK8