feat(openai): support structured output via response_format - #930
Merged
Conversation
rynfar
force-pushed
the
feat/openai-structured-output-merged
branch
from
September 4, 2026 09:15
b1e33f1 to
60eb91c
Compare
This was referenced Sep 4, 2026
The OpenAI endpoint dropped `response_format` entirely. Requests asking
for schema-valid JSON got prose back with no error
Translate `response_format` into the `output_config.format` that the
internal /v1/messages hop already understands. Enforcement is the SDK's
native structured output, including schema validation and retry, so this
is plumbing rather than a second implementation. Streaming works too:
the SDK buffers until the result validates, then emits it as one delta.
- `json_object` has no Anthropic equivalent — there is no schema-less JSON
mode — so it is rejected rather than widened into a permissive object
schema that would promise an enforcement the request never gets.
- `strict` and `name` are dropped: the SDK always validates, which is
never weaker than strict asked for, and `name` is a client-side label.
Tools remain incompatible with structured output, unchanged.
Errors name the field the caller actually sent. parseOutputFormat takes
the paths to report, and the OpenAI route validates before the hop —
otherwise a client sending `response_format` is told about
`output_config.format`, a field it never wrote.
Also strip a root-level `$schema` before handing the schema to the SDK.
Anything but the draft-07 URI made the model fail to submit its result,
returning HTTP 500 ("no structured_output result") after burning the
turn budget — and zod v4's `z.toJSONSchema()` emits the 2020-12 URI by
default, so every schema from that toolchain hit it. This affected the
native /v1/messages endpoint equally. The keyword only declares a
dialect and constrains nothing, so it is dropped rather than translated.
Many OpenAI-compatible clients serialize an unset optional as JSON null rather than dropping the key. translateResponseFormat read `.type` off the value before any validation, so `"response_format": null` threw a TypeError out of the /v1/chat/completions handler, which has no try/catch around the translation — turning a plain chat request that worked before structured output existed into a 500. Treat null as omission, and forward a non-object response_format untouched so parseOutputFormat rejects it with a 400 naming the client's own field instead of it being silently ignored. The boundary check in server.ts skips null for the same reason, which also keeps the error dialect honest: a null alongside an Anthropic-style output_config.format would otherwise report failures against `response_format.*`, a field the client did not meaningfully send.
The existing test asserts only what reaches the SDK, and its request actually ends in a 500: the mock yields no result carrying structured_output, so nothing downstream of the SDK boundary runs. The feature's whole point — schema-valid JSON in the response — had no coverage in either mode. Add a non-streaming case asserting the parsed JSON in choices[0].message.content, and a streaming case asserting the same value reassembled from the content deltas. Also restore translateOpenAiToAnthropic's JSDoc, which was left stranded above translateResponseFormat when that helper was inserted, so the "returns null if the request has no messages" contract that server.ts relies on documented the wrong function.
Structured-output mode buffers the SDK's wire events and replaces the response content, so it cannot coexist with a client-driven tool loop -- the tool_use turn gets swallowed. /v1/messages has always rejected the combination and continues to; nothing has depended on it working there. /v1/chat/completions is different. It dropped response_format entirely before structured output existed, so tool calling worked, and OpenAI permits both fields together -- LangChain agents and LiteLLM send them routinely. Routing those requests into the same rejection turned a working tool loop into a 400. Reject only when nothing the caller asked for can be honoured. With tools present the tools are honoured and the unsatisfiable schema is dropped and logged. json_object on its own stays a 400: there is no schema-less JSON mode to fall back to, and handing prose to a caller about to JSON.parse it is a worse failure than an actionable error.
rynfar
force-pushed
the
feat/openai-structured-output-merged
branch
from
September 4, 2026 18:45
60eb91c to
1b3f94a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #898 by @diwakar-s-maurya, cherry-picked so the original commit and authorship are preserved.
mainrequires signed commits, so fork PRs sit atmergeStateStatus: BLOCKEDregardless of CI — nothing the contributor could resolve from their side.What it does
/v1/chat/completionsdroppedresponse_formatentirely, so a request asking for schema-valid JSON got prose back. This translates it into theoutput_config.formatthat the internal/v1/messageshop already understands, so enforcement is the SDK's native structured output — schema validation and retry included — rather than a second implementation. The Anthropic spellingoutput_config.formatis accepted at this endpoint too.Errors are raised at the endpoint boundary in the client's own dialect, so an OpenAI caller sees
response_format.json_schema.schemarather than a field it never sent.Also drops a root-level
$schemabefore handing the schema to the SDK. Anything but the draft-07 URI makes the model fail to submit its result, which burns the turn budget and surfaces as a 500 — and that includes the 2020-12 URI that zod v4'sz.toJSONSchema()emits by default, so every schema from the zod/Vercel AI SDK toolchain hit it.One fix on top
translateResponseFormatread.typeoff the value before any validation, so"response_format": nullthrew aTypeErrorout of a handler with no try/catch around the translation:Plenty of OpenAI-compatible clients serialize an unset optional as
nullinstead of omitting the key, so this made a plain chat request that worked before structured output existed return a 500. Null is now omission, and a non-objectresponse_formatis forwarded untouched soparseOutputFormatrejects it with a 400 naming the client's field instead of silently ignoring it.Verified the regression test fails without the fix and passes with it.
Validation
npm test— 3036 pass, 1 skip, 0 failnpm run typecheck— clean