Skip to content

Live integration vectors, reasoningEffort, turn-coverage gate, and Typra 2.1.8 - #522

Merged
Seth Juarez (sethjuarez) merged 10 commits into
mainfrom
sethjuarez/live-integration-vectors
Sep 9, 2026
Merged

Live integration vectors, reasoningEffort, turn-coverage gate, and Typra 2.1.8#522
Seth Juarez (sethjuarez) merged 10 commits into
mainfrom
sethjuarez/live-integration-vectors

Conversation

@sethjuarez

@sethjuarez Seth Juarez (sethjuarez) commented Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Hardens Prompty to run integration vectors against real LLM endpoints and closes several cross-runtime parity gaps surfaced along the way. Six commits:

Commit What
fc07e784 model: add reasoningEffort to ModelOptions (TypeSpec + regen across all runtimes)
09f5ce20 python: guard deprecated sampling params, translate reasoningEffort in the executor
c50c3cc0 python: live integration plumbing for real Entra-auth endpoints
c33ea820 docs: regenerate reference docs and vscode schema for reasoningEffort
f0862fb4 schema: add turn-engine vector coverage gate (forcing function)
deab0b69 schema: upgrade Typra emitter 2.1.6 → 2.1.8 and regenerate

Highlights

reasoningEffort support — new ModelOptions field flowing through TypeSpec to every runtime, with the Python executor translating it and guarding sampling params (temperature/top_p) that newer reasoning models deprecate.

Live integration harness — Python plumbing to exercise vectors against real OpenAI / Azure (Entra-auth) endpoints, falling back to mocks when env vars are absent.

Turn-engine vector coverage gate (schema/scripts/check-vector-coverage.mjs) — a forcing function that pins the canonical EngineEventKind TypeSpec alias (23 members) against what the shared @vector corpus asserts. Currently reports 15/23 and exits non-zero; the 8 uncovered kinds are a real cross-runtime engine-parity backlog tracked in #519. Exposed as the standalone npm run check:coverage; intentionally not wired into the blocking build chain until it's green.

Typra emitter 2.1.8 — fixes the generated conversion-test parity gap (sethjuarez/typra#328, now closed): wire-conversion tests (toWire/fromWire assertions) are now emitted for every backend, not just Go and Java. Regeneration adds them to Python, C#, Rust, Swift, and TypeScript, plus invalid-input rejection tests across all runtimes.

Verification

  • Python: full suite 1472 passed, 57 integration deselected
  • Rust: cargo test --no-run — all test binaries compile
  • C#: Prompty.Core.Tests builds, 0 errors
  • Go: model tests pass
  • Coverage gate behaves as designed (15/23, exit 1)

Known follow-ups (tracked separately)

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

@sethjuarez
feat(model): add reasoningEffort to ModelOptions
fc07e78 
Add a first-class reasoningEffort field to ModelOptions in the TypeSpec model with per-provider wire mappings (OpenAI chat/responses, Anthropic), and regenerate all runtimes (Python, C#, Go, Java, Rust, Swift, TypeScript). Frontier reasoning models deprecate scalar sampling params (temperature/top_p/top_k); reasoningEffort is the forward-compatible control surface.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sethjuarez
feat(python): guard deprecated sampling params, translate reasoningEf…
09f5ce2 
…fort

Keep the generated to_wire/_build_options mappers pure (deterministic knownAs renames so wire-conformance vectors still assert the rename), and apply capability guards only in the live execute path. OpenAI: drop temperature/top_p/top_k/frequency_penalty/presence_penalty for reasoning models (o-series, gpt-5) in the chat/responses arg builders. Anthropic: drop params the installed SDK no longer accepts and map reasoningEffort into output_config.effort. Adds TestReasoningEffortGuard coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sethjuarez
feat(python): live integration plumbing for real Entra-auth endpoints
c50c3cc 
Foundry: use azure.identity.aio DefaultAzureCredential for the async client. OpenAI: default base_url to OPENAI_BASE_URL or api.openai.com/v1 so key-less connections resolve. Add aiohttp to foundry/azure/all extras (async Entra token provider). Extend integration conftest and model-listing test for live runs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sethjuarez
docs: regenerate reference docs and vscode schema for reasoningEffort
c33ea82 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sethjuarez
schema: add turn-engine vector coverage gate (forcing function)
f0862fb 
Anchors cross-runtime turn parity to the canonical EngineEventKind TypeSpec
alias (23 members) and fails when any member is unasserted by the shared @vector
corpus (vectors.json). Today 15/23 are covered; the 8 uncovered kinds
(policy_applied, model_invocation_failed, model_reconciliation_required,
model_invocation_reconciled, tool_result_reconciled, turn_failed,
turn_reconciliation_required, post_commit_failed) are the vector-authoring
backlog -- currently implemented behaviorally only in the Rust engine.

Exposed as a strict standalone 'check:coverage' script. Intentionally NOT wired
into the blocking 'build' chain yet: it is red until the cross-runtime port +
covering vectors land, and blocking now would red unrelated in-flight work on
this branch. Flip it into 'build' once green.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sethjuarez
schema: upgrade Typra emitter 2.1.6 -> 2.1.8 and regenerate
deab0b6 
Typra 2.1.8 fixes the generated conversion-test parity gap reported in
sethjuarez/typra#328: wire-conversion tests (toWire/fromWire assertions) are now
emitted for every backend, not just Go and Java. Regenerating adds them to
Python, C#, Rust, Swift, and TypeScript, and additionally emits invalid-input
rejection tests (from_json_invalid / 'should reject malformed JSON') across all
runtimes.

Verified: Python model suite (766) and full Python suite (1472 passed, 57
integration deselected) green; Go model tests green. Turn-engine vector coverage
gate unchanged (15/23 by design; tracked separately in #519).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github-code-quality[bot]
github-code-quality Bot found potential problems Sep 9, 2026
View reviewed changes
Comment thread runtime/python/prompty/tests/integration/conftest.py
@sethjuarez Seth Juarez (sethjuarez) mentioned this pull request Sep 9, 2026
Open
@sethjuarez
schema: fix Swift reasoningEffort wire value and TS prettier regen drift
850106e 
Two CI regressions surfaced by the Typra 2.1.8 wire-conversion tests on PR #522:

1. Swift ModelOptions.toWire emitted the reasoningEffort RawRepresentable
   struct instead of its backing string (only field with a named string-union
   type), so a wire round-trip failed fromWire->load() with 'Expected string
   for field reasoningEffort.' Add a deterministic normalize-typra-output.mjs
   workaround appending .rawValue to that single assignment (mirrors the
   correct save() path). Runs in generate on local + CI, so regen stays
   reproducible. Verified: all 368 Swift tests pass.

2. TypeScript model sources were committed un-prettified because the TS
   workspace had no node_modules locally, so the emitter skipped prettier.
   Regenerated with TS deps installed; the 22 model .ts files now match CI's
   prettier output, fixing 'regeneration produces no diff.'

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sethjuarez Seth Juarez (sethjuarez) mentioned this pull request Sep 9, 2026
Open
@sethjuarez
schema: reference sethjuarez/typra#331 in Swift reasoningEffort worka…
0d5c3ec 
…round

Link the deterministic .rawValue toWire workaround to the upstream Typra
emitter bug it works around, so it can be removed once the Swift backend
unwraps RawRepresentable/named-union fields in the wire path.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sethjuarez
schema: link Swift/Go emitter workarounds to upstream typra issues
23a4031 
Add tracking references for the two previously-unnumbered 'tracked on
sethjuarez/typra' workarounds:
- Swift processor provider() shadowing -> sethjuarez/typra#304 (open, still
  reproduces on 2.1.8)
- Go value-backed coerce-union dead fmt import -> sethjuarez/typra#305 (appears
  fixed in 2.1.8; workaround now a no-op guarding against regression)

Verified against raw 2.1.8 emitter output. No generated-output change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sethjuarez
schema: reference sethjuarez/typra#332 for Swift main-target resource…
fbcbbf1 
…s gap

Link the restoreSwiftPackageResources workaround to the upstream feature
request for a main-target resources: emit option (the shipped test-resources
option from #260 only covers the test target and breaks Bundle.module for
production Discovery code).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sethjuarez
Seth Juarez (sethjuarez) merged commit 0d0c350 into main Sep 9, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@github-code-quality github-code-quality[bot] github-code-quality[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sethjuarez