RS-LoCoMo-Full-v3: strict-representable agent step - #145
Merged
Conversation
The first answer-stage execution against a real provider failed all eight
questions identically: Azure, serving gpt-4o-mini, rejected the request with
HTTP 400 — "Invalid schema for response_format 'AnswerAgentStep': in
('properties','arguments'), 'additionalProperties' is required to be supplied
and to be false." AnswerAgentStep.arguments was a free-form dict; pydantic
renders it as an open object; strict mode requires every object closed; and
closing an object with no properties would forbid all content. A free-form
object is therefore unrepresentable under strict structured output. The
providers that accepted v2's schema were the ones not enforcing strict mode at
all (design §2.4).
Tool arguments now travel as arguments_json, one JSON object encoded as a
string. Probing the shape live: Azure accepts the schema, and at temperature 0
gpt-4o-mini immediately demonstrated why the inner parse must be lenient — it
appended a sentence period inside the string, after the closing brace. The loop
takes the first complete JSON object and records any trailing text on the trace
row, so syntax noise neither reads as a retrieval failure nor vanishes.
The adapter now refuses to send a strict schema containing an open object,
failing at build time with a clear message instead of on the first compliant
provider in production. AnswerAgentStep was the only provider-bound model with
a free-form dict (DecisionRecord.edit_detail is persistence-only).
Protocol bumped v2 -> v3 per the design's own rule; no v2 score ever existed.
Design §2.4 gains the two new verified observations, §7 records the parse rule,
and D78 carries a dated amendment.
Validation: pyright 0 errors, 585 passed, ruff clean. New tests: trailing-junk
parse with recorded leftover, non-object rejection, answer steps ignoring the
field, and the guard proving no HTTP happens for an invalid schema.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKENhTLJg1HqhbdwCmmkbc
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
Grok review of the v3 change, all non-blocking, applied anyway: the open-object guard now raises a dedicated StrictSchemaError so a programmer schema regression is never mistaken for a flaky model reply by broad except blocks; parsed_arguments gains parametrized edge cases (empty, whitespace, nesting, unicode, fragments, non-objects); the WP-8.2 parenthetical now names both D78 amendments instead of only the judge story. Validation: pyright 0 errors, 586 passed, ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKENhTLJg1HqhbdwCmmkbc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The first answer-stage execution against a real provider failed all 8 questions identically:
AnswerAgentStep.argumentswas a free-formdict[str, object]. Strict mode requires every schema object closed — and closing an object with no properties would forbid all content. A free-form object is unrepresentable under strict structured output. The providers that accepted v2's schema were precisely the ones not enforcing strict mode (design §2.4).Fix, probed live before implementing
Tool arguments travel as
arguments_json— one JSON object encoded as a string. A live probe of the exact shape: Azure accepts it, and at temperature 0gpt-4o-miniimmediately showed why the inner parse must be lenient:The loop takes the first complete JSON object and records any trailing text on the trace row (
arguments_trailing) — syntax noise neither reads as a retrieval failure nor vanishes.Guard
_strict_json_schemanow refuses to send any strict schema containing an open object, failing at build time with a clear message instead of on the first compliant provider in production. A test proves no HTTP happens for an invalid schema.AnswerAgentStepwas the only provider-bound model with a free-form dict (DecisionRecord.edit_detailis persistence-only — checked all 15response_type=call sites).Protocol
v2 → v3 per the design's own rule (schema + prompt changed). No v2 score ever existed. Design §2.4 gains the two new verified observations (strict enforced only by some providers; string contamination), §7 records the parse rule, D78 carries a dated amendment.
Validation
pyright 0 errors; 585 passed, 310 skipped; ruff clean. New tests: trailing-junk parse with recorded leftover, non-object rejection, answer steps ignoring the field, build-time guard.
Contributor agreement