Skip to content

fix(openai): require nested strict schema fields - #441

Closed
Seth Juarez (sethjuarez) wants to merge 1 commit into
microsoft:mainfrom
sethjuarez:sejuare-microsoft/fix-strict-nested-schemas
Closed

Seth Juarez (sethjuarez) wants to merge 1 commit into
microsoft:mainfrom
sethjuarez:sejuare-microsoft/fix-strict-nested-schemas

Conversation

@sethjuarez

@sethjuarez Seth Juarez (sethjuarez) commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

Summary

  • make OpenAI strict JSON Schema conversion recursively include every nested object key in required
  • preserve Prompty optionality by rendering non-required fields as nullable unions
  • pin the behavior in the canonical shared wire vectors and specification
  • load generated property subtypes in the TypeScript vector harness so nested schemas are exercised faithfully
  • exercise the exact nested optional case in the ignored live Rust integration test

Runtime audit

Runtime Result
Rust Fixed recursive nested required generation
Python Already compliant; new shared vector passes
TypeScript Provider already compliant; vector harness now preserves nested generated properties
C# Already compliant; new shared vector passes
Go / Swift No OpenAI provider surface on main
Java Matching fix and live regression coverage are in PR #444 (2355dbe1, da5a3f6e)

Verification

  • cargo test -p prompty-openai
  • cargo fmt --all -- --check
  • uv run pytest tests/test_spec_vectors.py tests/test_executor.py tests/test_structured.py -q (219 passed, 17 skipped)
  • npm run test --workspace @prompty/core -- spec-vectors.test.ts (140 passed, 17 skipped)
  • npm run test --workspace @prompty/openai (91 passed, 18 skipped)
  • npm run lint --workspace @prompty/core
  • C# SpecVectorWireTests assembly run (25 passed)
  • live OpenAI test_structured_output E2E: accepted the nested strict schema and returned {"style":{"border":null,"color":"blue"}}
  • Java PR Add a Java runtime built on generated models #444: the canonical vector passes unmodified; reverting Java's one-line fix creates exactly one failure at response_format.json_schema.schema.properties.style.required (expected 2 keys, got 1), then the restored fix passes all 695 tests

The live Java negative check also reproduced OpenAI's nested-schema rejection verbatim: In context=('properties', 'address'), 'required' is required to be supplied and to be an array including every key in properties. Missing 'postcode'.

The contract matches OpenAI's Structured Outputs guidance: all fields must be required, optional fields are modeled with null, and the rule applies to nested object schemas.

Align the recursive strict JSON Schema contract with OpenAI by marking every nested object property required while representing Prompty optional fields as nullable. Add a shared wire vector, cross-runtime harness coverage, and a live nested structured-output check.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 4, 2026 09:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR tightens Prompty’s OpenAI Structured Outputs “strict” wire contract so that every object property is listed in required recursively, while Prompty-optional fields are modeled as nullable unions (e.g., type: ["string", "null"]). It updates both the normative spec/docs and the shared wire vectors, and adjusts harnesses/tests so nested schemas are exercised end-to-end.

Changes:

  • Updated the wire-format specification/docs to define strict-mode recursive required behavior and nullable modeling for optional fields.
  • Added a new canonical wire vector (structured_output_nested_optional) that asserts nested optional handling in strict mode.
  • Updated TypeScript vector harness loading to use generated Property.load(...) (preserving nested property shapes) and updated Rust wire/tests to require nested properties under strict mode.
Show a summary per file
File Description
web/src/content/docs/specification/wire-format.mdx Documents strict-mode recursive required + nullable optionals in schema conversion.
spec/vectors/wire/wire_vectors.json Adds canonical nested-optional structured output wire vector + expected strict schema.
spec/spec.md Updates normative spec text/pseudocode for strict schema generation rules.
runtime/typescript/packages/core/tests/spec-vectors.test.ts Uses Property.load so vectors preserve generated nested property subtypes.
runtime/rust/prompty-openai/tests/wire_vectors.rs Exercises the new vector and updates strict nested required assertions.
runtime/rust/prompty-openai/tests/integration.rs Updates ignored live test to validate nested optional strict output is nullable + required.
runtime/rust/prompty-openai/src/wire.rs Ensures strict mode includes nested properties in required arrays (recursive behavior).

Review details

  • Files reviewed: 7/7 changed files
  • Comments generated: 0
  • Review effort level: Lite

@sethjuarez

Copy link
Copy Markdown
Member Author

Superseded by #481, which re-authors this fix against current main (@typra/emitter 0.9.0 + the reorganized schema/model/ contracts/conformance/operations/wire layout). The core change — nested strict-mode requiring all object props and making optionals nullable — is included there, with the structured_output_nested_optional conformance vector authored in wire.tsp source and verified via cargo test -p prompty-openai. Closing in favor of #481.

Seth Juarez (sethjuarez) added a commit that referenced this pull request Aug 18, 2026
The #441 salvage added the structured_output_nested_optional vector to
wire.tsp and regenerated vectors.json + the Python conformance test, but
the generated TypeScript vector-conformance.test.ts was inadvertently
reverted along with unrelated prettier formatting churn. Regenerated with
the TS workspace installed (prettier present) so only the genuine vector
addition remains. Fixes the 'regeneration produces no diff' CI gate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seth Juarez (sethjuarez) added a commit that referenced this pull request Aug 18, 2026
…ing (#443) onto 0.9.0 (#481)

* fix(openai): require nested strict schema fields (rebase #441 onto emitter 0.9.0)

Salvages the still-valid correctness fix from PR #441, re-authored for main's
current state (schema reorg + @typra/emitter 0.9.0):

- prompty-openai wire.rs: in OpenAI strict mode, every object property must
  appear in 
equired (optional fields are represented as nullable).
- Vector structured_output_nested_optional authored at its NEW source of
  truth schema/model/conformance/vectors/wire.tsp; �ectors.json and the
  generated python conformance test are regenerated via
pm run generate
  (not hand-edited). PR #441's edit to the legacy spec/vectors/wire/
  wire_vectors.json is dropped since main removed that catalog.
- spec.md / wire-format.mdx: document the recursive strict-mode rule.

Verified: cargo test -p prompty-openai (lib + wire_vectors) 107 passed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(rust): harden Anthropic transport failures (rebase #443 onto main)

Salvages PR #443, re-merged onto main's evolved prompty-anthropic crate:

- Add endpoint.rs with shared build_api_url(): adds ANTHROPIC_BASE_URL env
  fallback and idempotent /v1 handling that main's inline build_url lacked.
- executor.rs and models.rs now delegate URL construction to the shared
  helper; classify transport/SSE failures as the PR intended.
- Conflict resolution: kept main's Agent type name (PR predated the
  Prompty->Agent rename) while adopting the PR's ANTHROPIC_BASE_URL test
  env guards.

Verified: cargo test -p prompty-anthropic — 55 lib + wire/discovery/vectors
tests pass; live integration tests remain env-gated (ignored).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore(schema): regenerate TS vector-conformance test for new vector

The #441 salvage added the structured_output_nested_optional vector to
wire.tsp and regenerated vectors.json + the Python conformance test, but
the generated TypeScript vector-conformance.test.ts was inadvertently
reverted along with unrelated prettier formatting churn. Regenerated with
the TS workspace installed (prettier present) so only the genuine vector
addition remains. Fixes the 'regeneration produces no diff' CI gate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants