Repository navigation
Service-quality spike: mock-fallback integration harness + drill YAGNI verdict (conformance rail unchanged) - #518
Merged
Conversation
Author the first prompty "service drill" -- a conformance vector that crosses the real Executor transport seam (live when creds exist, cassette-replay when absent) and asserts three planes: transport, wire, and semantic. The semantic plane IS a conformance vector over the Executor path. Spike finding: the existing @typra/emitter (2.1.7) carries a stage:"drill" vector plus its unknown `exchange` block through unchanged (unknown-key passthrough), into vectors.json and all 7 runtime harnesses -- zero emitter change needed to author or deliver a drill. - schema: add LiveChatConformance seam + drill.tsp vector; regen 7 runtimes. - python: drill adapter (base-URL redirect -> stdlib cassette-replay server, 3-plane assert + live/replay parity), cassette + recorder tooling. - typescript: mirror adapter + Node cassette-replay server + cassette copy. Drill agent built from @prompty/core (dist) so the executor's instanceof ApiKeyConnection matches across the dual-core test seam. - go/rust/swift/csharp/java: operation-level waivers so the drill vector skips cleanly where no adapter is registered (Python/TS-only for the spike). The requires guard is all-or-nothing, so the drill vector omits it and the adapter self-gates the live plane on credential presence while always running replay. Cassette bytes, SDK binding, and cred gating stay consumer-owned roster -- by design not modeled in TypeSpec. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 442651bd-34df-4b59-8e2f-d7093fa855a8
Service integration tests previously skipped entirely when API-key env vars were absent, leaving the executor->processor pipeline uncovered in CI. Add a mock-fallback harness so the same test bodies run against real services when creds are present and against deterministic mocks when they are not, driving coverage of the full provider pipeline in both modes with zero production-code changes. - Add tests/integration/mock_clients.py: request-aware sync+async mock clients returning real OpenAI SDK types (ChatCompletion, chunk streams, embeddings, images) and Anthropic SDK-shaped objects (attribute-access message + content blocks), covering chat, tool-call loops, structured output, streaming, embeddings and image generation. - conftest.py: autouse fixture patches only absent providers SDK constructors (openai.OpenAI/AsyncOpenAI/AzureOpenAI/AsyncAzureOpenAI, anthropic.Anthropic/AsyncAnthropic) and sets placeholder creds, so live providers are untouched (mixed real/mock mode works). Adds run_* markers. - Convert value-path test files (chat, streaming, structured, agent, embedding, image, anthropic) from skip_* to run_* decorators. Live-only tests (entra, model listing, tracing, direct) intentionally left skipped. Verified: 30 passed / 26 skipped in pure mock mode; 1371 unit tests green; ruff clean. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: faf27021-e24a-410d-bd40-fd03f22236e8 (cherry picked from commit 945f31a11f63a080401d85019b09089f90733ad1)
Reverts d8cb188. The service-drill spike answered its YAGNI-gate question, and the answer is "do not ship it in the emitted conformance rail." Why (the "no special flower" principle): every runtime's conformance surface must be identical and driven purely by Typra-emitted contracts -- no per-runtime extras that are not bound to a contract. The drill vector rides through the emitter fine (unknown stage:"drill" + exchange keys pass through unchanged), but everything that makes it *run* is irreducibly consumer-owned roster that the hard constraints forbid the emitter from ever owning: - the cassette-replay HTTP server - the SDK / base-URL transport binding - cassette bytes + exchange ref-resolution - the 3-plane (transport/wire/semantic) assertion - the var:live-enabled credential gate + live/replay parity Because the emitter can never generate that roster, authoring the drill as an emitted @vector forces EVERY runtime into one of two hand-authored, non-contract-bound choices: a drill adapter (Python, TS) or a waiver (Go, C#, Java, Rust, Swift). Both are "special flowers." There is no state in which the drill is simultaneously (a) an emitted uniform vector and (b) free of per-runtime roster. So it does not belong in the emitted conformance rail. This revert returns all 7 runtimes to the pre-drill #517 baseline (byte-identical for every schema, generated-model, runtime-model, and web file), so the conformance surface is once again uniform and contract-only. Verdict for the Typra follow-up: Typra does nothing. The drill is real x-N per-runtime duplication that the hard constraints keep out of the emitter, so there is no minimal stage:"drill"+exchange emitter rail worth scoping -- the emitter already passes unknown keys through, and the value-add would all be consumer roster. Service-quality probing, if wanted, lives in a separate consumer-owned integration layer (the mock-fallback harness in tests/integration/, kept on this branch), never the uniform conformance rail. Kept: the mock-fallback integration harness (b22ffaf) -- it is the consumer-owned, non-conformance integration layer where mixed real/mock service probing correctly belongs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 442651bd-34df-4b59-8e2f-d7093fa855a8
Seth Juarez (sethjuarez)
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR ships
Mock-fallback integration harness (
b22ffafc) —tests/integration/mock_clients.py+ autouse_mock_absent_servicesfixture that patches only absent providers' SDK constructors, withrun_*markers so value-path tests run in mock mode and real when creds exist. Python-only, consumer-owned, outside the conformance rail. Mock mode: 30 passed / 26 skipped.Service-drill spike verdict (
b1508712revertsd8cb188f) — the drill was authored on the callable seam, regenerated into all 7 runtimes, implemented end-to-end (Python + TS, 3-plane transport/wire/semantic, replay + live-parity), measured, and then deliberately backed out of the shared conformance rail.The drill verdict: excluded from the emitted rail (Typra does nothing)
Governing principle: no special-flower runtimes — every runtime's conformance surface must be identical and driven purely by Typra-emitted contracts, with no per-runtime extras that aren't bound to a contract.
stage:"drill"+exchangekeys ride through untouched — confirmed, zero emitter change).exchangeref-resolution, the 3-plane assertion, and thevar:live-enabledcred gate + live/replay parity.⇒ The drill is real ×N per-runtime duplication that the constraints keep out of the emitter. There is no minimal
stage:"drill"+exchangeemitter rail worth scoping — the value-add would all be consumer roster. Service-quality probing lives in the separate consumer-owned integration layer (the harness above), never the uniform conformance rail.Conformance rail: unchanged
The revert returns all 7 runtimes to the pre-drill #517 baseline — the schema, every
.typra-generatedartifact, every runtime's generated model + conformance harness, and the web reference docs are byte-identical toe769acce. The only delta from that baseline on this branch is the 9-file integration harness (+583/−30).Verification
ruff checkclean@prompty/coremodel conformance: 110 passed