Delete the python gateway data plane

[kfallah]

Previously stacked on #628, which has now merged to main; the base is back on main and the landed #628 series (including its post-review commits) is merged and reconciled here.

What this does

Release 0.6.0 removes the python gateway data plane. The compiled native engine (exp-gateway-native 0.2.0) is the only engine: every launch path (exp, exp run, exp --project, and the exp.load_router compatibility client) serves through it, and a missing compiled extension fails the launch with the exact build command instead of falling back.

Reconciliation with #628 (parameter gating)

#628's behavior survives the deletion in full, on the surviving native path:

• Parameter gating runs in native admission: route_generation_parameter_requests and the capability-aware preflight_gateway_request execute in native_bridge.admit before any payload is frozen; a ProviderParameterError finalizes the accepted request content-free and returns the public field error with no attempt row.
• The landed fix(gateway): gate generation parameters by route capabilities #628 series' retained provider-entry bounds (reserve_tool_entry/reserve_summary_entry) survive in the Rust dialects; only the python mirrors of that bounding died with the engine.
• The catalog capability intersection (_resolved_wire_profile) moved to exp/runtime/gateway/execution_resolution.py, now trimmed to the native branch only; the python_stream compatibility dialect died with the python engine. New execution_resolution_test.py covers the intersection and the identity fail-closed check.
• Provider clients keep fix(gateway): gate generation parameters by route capabilities #628's capability-gate constructors, reasoning-effort translation (reasoning_compat.py), and expanded wire profiles; only their executor-only stream() seams are deleted.
• fix(gateway): gate generation parameters by route capabilities #628's engine-parity served tests are ported to native-only assertions (native_messages_test.py: unsupported reasoning effort, unsupported sampling, catalog generation limits with zero upstream dispatch).

Reconciliation with #618 (Vertex AI)

Vertex serves natively: VertexClient keeps #628's capability-gated constructor and resolves through the shared gemini_generate_content dialect wire profile with the publisher-scoped SSE endpoint and a current OAuth bearer token (minted on the bridge's blocking-callback thread, never an event loop). Its executor-only stream() seam is deleted with the engine.

Regenerated goldens

Commit 1 froze the encoder parity contract as committed golden bytes; #628 then changed the encoders. The goldens in exp/runtime/gateway/testdata/parity_goldens.json are regenerated from the post-#628 python encoders, so the frozen contract now includes the Responses reasoning-summary lifecycle (rs_ items, reasoning_summary_part/text events, selected reasoning configuration in bodies). Two contracts are newly frozen: the Chat x-experiential-ignored-parameters disclosure (new golden plus a fixture argument on encode_chat_fixture) and the Messages surface dropping reasoning-summary deltas without changing its bytes. Every Rust encoder passes byte-for-byte against the new goldens, and against the live python encoders where those survive.

Differential loadtest findings (2.2M requests, report in comments)

• Fixed: a successful attempt terminating with zero semantic output (empty completion, or max_tokens truncating before any text) settled with the collected usage but dropped the usage event from the outward stream, so callers lost the token accounting on all six surfaces while billing stayed correct. The waterfall now returns the tracked usage ahead of the terminal; twelve served-engine repros (Chat/Responses/Messages, streaming and not, empty and truncated) were written failing first.
• Fixed: streaming responses again serve content-type: text/event-stream; charset=utf-8, matching the prior engine for strict clients.
• Named follow-up (bounded, non-failing): ledger calls run via spawn_blocking + Python::attach, and persistent_connection caches one SQLite connection per blocking thread in a threading.local. Tokio blocking threads retire without closing the cached connection, so gateway.db descriptors track ever-seen blocking threads (plateaued at 350 FDs / ~300 MB RSS in a 608k-request soak; bounded by the pool's thread ceiling, nothing fails). Candidate fixes: a dedicated small blocking-thread pool for bridge calls, a checkout/return connection pool instead of thread-locals, or thread-exit cleanup that closes the cached connection. Deferred because it changes the ledger connection strategy across every bridge call site, which deserves its own soak-validated change.

Deleted

• GatewayService request serving, create_gateway_app, and the uvicorn engine path.
• GatewayExecutor and the execution machinery (execution.py, aggregation.py, stream_attempts.py), plus the per-provider stream() seam and the python event mappers (providers/streaming.py, streaming_events.py, gemini_streaming.py, bedrock_streaming.py, streaming_usage.py) whose only consumer was that executor.
• The python Anthropic Messages encoders and error envelope (anthropic_protocol/encoding.py, errors.py); the native plane renders that surface.
• The public embeddable composition seam (create_gateway_runtime, GatewayRuntime, GatewayRuntimeConfig) and its docs section.
• The --engine flag (passing it is a usage error) and the deprecated fallback_app/fallback_port relay in the crate.
• The fastapi and uvicorn dependencies.
• Every test that existed solely for the deleted engine; the rest were converted to the component seam (load_gateway_components) or to the served native engine on real sockets.

Kept

Authority/store, ledger and group-commit, budgets, routing, DeploymentHealthRegistry (bridge-consumed), catalog lifecycle and hot reloader, all protocol decoders and payload builders, boundary.py error mapping, the whole native_* bridge stack, usage report and HTML via native callbacks, guardrails, replay/continuation stores, and the CLI native serving path.

Behavior notes

• Every encoder parity contract asserts against committed golden bytes (testdata/parity_goldens.json); no test imports a deleted python encoder or mapper.
• Readiness: the accounting latch lives on the components (accounting_healthy), so hosted compositions report their own store health and the bridge ANDs it with its settlement latch. Nothing references executor.require_healthy or mark_accounting_unhealthy.
• Startup keeps the credential/route proof: loading granted aliases builds each alias's credential-free route proof with actionable per-alias reasons, and native servability of every pool deployment is validated before binding.
• An {"escalate": reason} admission disposition is classified for content-free metrics and fails closed with the shared internal error; nothing hands a request to another engine. The accepted request is finalized content-free (no attempt row) before the disposition returns.
• Two regressions the installed-wheel release evidence caught during this work are fixed with unit repros: a continued Responses request now joins its first turn's selection episode (no second request-time embed), and ledger acceptance runs before route selection so a keyed replay that must fail closed never triggers learned-selection embedding.

Versions

Flagship experiential 0.6.0; exp-gateway-native 0.2.0 with the flagship floor >=0.2.0,<0.3; the lockstep test derives the ceiling from the crate version. cargo update -p exp-gateway-native and uv lock are current. (main carries 0.5.6/0.1.14 from #628; this PR supersedes both with the strictly greater pair.)

Accepted boundary

Platforms without a prebuilt wheel (musl, exotic architectures) must install from sdist and need a Rust toolchain to build exp-gateway-native. There is no interpreted fallback.

Release notes draft (0.6.0)

• The gateway now has exactly one data plane: the compiled native engine serves Chat Completions, Responses, Anthropic Messages, models, health, and usage on every launch path.
• Removed: the python serving engine, the --engine flag, and the embeddable create_gateway_runtime composition API. exp.load_router still returns an official OpenAI client, now backed by a private native gateway with a graceful stop handle.
• Removed dependencies: fastapi, uvicorn.
• Breaking: installs on platforms without a prebuilt exp-gateway-native wheel require a Rust toolchain (sdist build). A missing extension fails launch with the exact build command.

🤖 Generated with Claude Code

[kfallah]

@greptile-apps review

[greptile-apps]

Greptile Summary

The PR removes the Python gateway data plane and makes the compiled native engine the sole serving path.

• Deletes the Python service, executor, streaming adapters, and embeddable composition API.
• Moves all CLI launch paths to the native gateway and fails explicitly when the extension is unavailable.
• Defers readiness output until the native listener has successfully bound.
• Updates native protocol behavior, provider integration, tests, documentation, package versions, and generated parity goldens.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the native server now acquires the advertised listener before emitting readiness, and an occupied port exits without a ready receipt.

Important Files Changed

Filename	Overview
exp/cli/gateway/serve.py	Consolidates gateway startup onto the native engine and fixes the previously reported false-readiness race by emitting readiness from the post-bind callback.
exp/runtime/gateway/native/src/server.rs	Adds the listener callback after successful TCP binding and before request acceptance while preserving graceful shutdown and settlement draining.
exp/runtime/gateway/native_server.py	Exposes the native listener callback through the Python wrapper and maps native startup failures to the public server error.
exp/runtime/gateway/native_bridge.py	Retains the Python control-plane bridge for native admission, routing, accounting, guardrails, and provider callbacks.
exp/runtime/gateway/execution_resolution.py	Reduces execution resolution to the surviving native path while preserving capability-aware wire-profile selection.
pyproject.toml	Removes Python server dependencies and updates the flagship and native-extension package contract.

_{Reviews (6): Last reviewed commit: "Preserve client-visible usage on zero-ou..." | Re-trigger Greptile}

[kfallah]

@greptile-apps review

[kfallah]

@greptile-apps review

[kfallah]

@devin can you do in-depth integration testing with high volume traffic to ensure no regressions? spend up to $1k

[devin-ai-integration]

View Devin session

[kfallah]

@greptile-apps review

[devin-ai-integration]

PR #629 integration and regression report

Scope: does serving exclusively through the native data plane regress against the python data plane
that this PR deletes. Method: identical deterministic traffic sent to main with --engine python
and to the PR branch with the native plane, comparing statuses, normalized bodies, SSE frame
sequences, stream terminals, error envelopes and ledger accounting, plus bounded real-provider
traffic, lifecycle checks and long soaks.

Revisions measured: base a130bf20, PR head f868d010. Earlier findings from base 0389d28e /
PR 03e10f63 were re-verified against the new head and are labelled where they were not.

Traffic: roughly 2.2 million requests total across both planes. Real-provider spend: about $0.20.

Verdict

One client-visible regression, one bounded resource-retention issue worth a follow-up, and several
places where the native plane is strictly better. The regression is small and mechanical; nothing
found argues against deleting the python plane once it is fixed.

1. Regression: usage is dropped when a successful response has no output content

Reproducible with no provider involved. When an attempt terminates successfully with zero semantic
output (empty completion, or max_tokens truncating before any text), the native plane loses the
client-visible token usage the python plane returns. Billing is unaffected: the ledger is identical
on both planes.

Surface	python plane	native plane
Chat, non-stream	{"prompt_tokens":11,"completion_tokens":0,"total_tokens":11}	null
Chat, stream (include_usage)	final usage chunk emitted	usage chunk never emitted
Responses, non-stream	{"input_tokens":11,"output_tokens":0,...}	null
Responses, stream	usage on the completed event	null
Messages, non-stream	{"input_tokens":11,"output_tokens":0}	{"input_tokens":0,"output_tokens":0}
Messages, stream	final delta carries input 11	final delta carries input 0

Terminal states are correct on both planes (finish_reason=length, Responses incomplete with
max_output_tokens, Messages stop_reason=max_tokens), so only the accounting a caller reads back
is lost.

Cause: in waterfall.rs, the "successful terminal with no semantic output" branch settles with the
collected usage and then returns events: vec![event], dropping the usage event. The encoders take
usage from the settled event list, so they never see it. Every encoder gate (has_token_counts) is
identical between the two planes, which is why this only shows up on the no-output path. Attaching
the collected usage to the returned event list is enough.

Hit rate: 12 isolated cases in the local fault plan (Chat/Responses/Messages, streaming and not,
success and max_tokens), all 12 differing on every one of 5 repetitions, and confirmed against
real providers: Anthropic chat_json_tiny 40/40 (input 23, output 1 vs absent) and Gemini
chat_json_tiny 40/40 (input 16, output 0 vs absent).

Evidence: loadtest629/runs/repro-usage/transcript.txt, loadtest629/runs/vol25k-new/report_faults.json,
loadtest629/runs/real40-new/report_real.json.

2. Bounded resource retention on the native plane

The native plane retains SQLite descriptors and memory that the python plane does not. This is
bounded, not a runaway leak: over a 2700-second soak at concurrency 64 (608k requests) the
gateway.db descriptor count rose to 350 and then stopped moving at about 810 seconds, and total
FDs oscillated between roughly 500 and 729 for the remaining 30 minutes against a soft limit of
65536. Nothing failed: 608497/608497 HTTP 200, no accept failures, no CLOSE_WAIT accumulation, no
"too many open files".

Mechanism: ledger work runs through spawn_blocking + Python::attach on a tokio blocking pool
built with defaults, and persistent_connection caches one SQLite connection per thread in a
threading.local (2 FDs: db + wal). Blocking threads retire after their idle timeout without the
cached connection being closed, so the count tracks the number of distinct blocking threads that
ever touched the ledger rather than the number alive (live threads stayed at 29 while FDs reached
729). It plateaus because the pool's thread ceiling bounds it.

Memory follows the same shape: PR RSS 145 MB to about 300 MB and flattening, versus 151 MB to 177 MB
on the python plane. Latency degraded on both planes over 45 minutes (python 197 to 344 ms average,
native 239 to 303 ms), so the degradation is not native-specific, and native throughput was higher at
the end of the run.

Two lifecycle checks flag this and I left them failing rather than relaxing the tolerance: 100
client-aborted streams take FDs 18 to 130 with no release, and the 1200-second mixed soak measures
0.74 FD/s on native (max 986) against 0.037 FD/s on python (max 78).

Assessment: a follow-up, not a blocker. It is bounded at roughly 1 to 2 percent of the default FD
limit and caused no failures under sustained load, but the retained connections are never released
and would be worth closing on thread retirement (or by not caching per blocking thread).

Evidence: loadtest629/runs/fd-plateau-new/{pr,base}/plateau.json, loadtest629/runs/fd-probe-fd-probe-new/,
loadtest629/runs/lifecycle-full-new/summary.json.

3. Where the native plane is better

• At 300k requests/side (old revisions) the python plane dropped 193 client connections with read
  errors and raised two GatewayLedgerError: attempt is already settled with another terminal state
  exceptions from shielded futures. The native plane had zero of both.
• Gemini tool streaming: the python plane ended those streams with transport_error
  (peer closed connection without sending complete message body) 40/40 on both the old and new
  head; the native plane terminates them normally 40/40. This is a python-plane bug the PR removes.
• Throughput: 249 req/s versus 185 req/s on the 25k differential, about 35 percent faster, with
  identical ledgers.

4. Everything that matched

New-head 25k differential (26655 volume records plus 155 fault records per side): no status
transitions, no error-family transitions, no request present on only one side, and byte-identical
ledger totals (24130 requests, 24128 attempts, 1175925 input tokens, 50944 cached, 718352 output,
15007015 micro-USD known cost, 1985 unknown-cost attempts, terminal counts 22699 completed / 476
failed / 401 incomplete / 552 cancelled). Mock upstream counters agree exactly on both sides.

Fault parity across HTTP 429/500/400/401/503, streamed HTTP errors, malformed JSON, empty choices,
upstream connection close, delayed first byte, SSE truncation, SSE error frames, malformed SSE
frames, and the Messages and Responses equivalents: identical statuses and error envelopes
throughout.

Lifecycle: first-run setup, pool failover (100/100 with 2 dead-route attempts and consistent
unknown-cost accounting), restart persistence (identical usage, aliases and keys across restart; old
key still valid; port rebound in 1.85 s), authority enforcement (revoked key 401 invalid_key,
removed grant 403 model_not_granted, disabled identity 401, rejection envelopes exactly equal),
graceful shutdown (20 completed, 0 cut), and 50 load_router() create/use/close cycles with no bind
failures and no orphans.

Real providers, 2556 compared records: OpenAI, Anthropic, Gemini and Bedrock. Combined priced spend
$0.0929 on the new head. Ledger totals per plane are close but not equal, as expected with live
providers (base 58916 input / 26649 output / $0.05444 estimated; PR 61047 / 28230 / $0.059622).
Azure is not covered: the configured endpoint returns 404 for every probed deployment.

Known cosmetic difference, present everywhere: streaming content-type is
text/event-stream; charset=utf-8 on python and text/event-stream on native. Worth deciding
whether to restore the charset parameter, since strict clients do compare it.

Unresolved from the old revisions and not re-run on the new head: the 300k differential showed 18
PR-only records (12 Responses continuations, 6 idempotent replays) and small ledger differences, on
top of the python-plane read errors above. Worth a look, but the new-head 25k run reproduced neither.

5. Recommendation

Fix the no-output usage drop (item 1), then this is a safe deletion. Item 2 and the content-type
parameter are follow-ups. Bedrock chat_tools showed intermittent status differences in both
directions across runs and did not reproduce in dedicated bounded bursts, so I do not count it as a
finding.

Harness and all artifacts: /home/ubuntu/loadtest629. Nothing in the repository was modified.

Written by Devin

[kfallah]

@greptile-apps review

[devin-ai-integration]

Re-verified everything above against the new head 50347d8f (base cc446f77), since the merge brought in #628 parameter gating and #631 attribution:

• The no-output usage drop still reproduces on all six surfaces (Chat/Responses/Messages, streaming and not) and on the six max_tokens-truncated variants. Terminal states remain correct on both planes.
• A fresh 25k differential (26672 volume records plus 155 fault records per side) is otherwise clean: no status transitions, no error-family transitions, no record present on only one side, identical ledger totals (24101 requests, 1160433 input / 46848 cached / 715148 output tokens, 14903348 micro-USD known cost), and mock upstream counters agreeing exactly. Total latency 3394180 ms on native versus 6876928 ms on python.
• The parameter gating and attribution changes introduced no divergence: the 289 shared 400s are identical invalid_parameter / param=messages envelopes on both planes, and no request field is accepted on one plane and dropped on the other.

So the recommendation is unchanged: fix the no-output usage path, and the deletion is safe.

Written by Devin

[kfallah]

Thanks for the differential run — the no-output diagnosis was exact. Status on the three items, fixed on this branch:

1. Zero-output usage drop: fixed. The cause was where you pointed: the waterfall's "successful terminal with no semantic output" branch settled with the collected usage and then returned only the terminal event, so no encoder ever saw a usage event. The branch now returns the tracked usage event ahead of the terminal. Twelve served-native repros were written failing first and now pass: Chat, Responses, and Messages, streaming and non-streaming, for both the empty completion and the max_tokens truncation, against a mock upstream that returns zero content plus a real usage payload (exp/runtime/gateway/tests/native_messages_test.py::test_*_zero_output_*). Chat non-stream returns prompt_tokens with completion_tokens: 0, Chat streaming emits the include_usage final chunk, Responses carries usage on both the completed and incomplete terminals, and Messages reports the real input count.

2. Bounded FD/connection retention: deferred as a named follow-up, written up in the PR body. Your mechanism read matches the code: spawn_blocking + Python::attach with persistent_connection caching one SQLite connection per blocking thread in a threading.local, and retiring blocking threads never close it. The candidate fixes (dedicated small blocking pool, checkout/return connection pool, or thread-exit cleanup) all change the ledger connection strategy across every bridge call site, so it gets its own soak-validated change rather than riding this deletion. It is bounded by the pool ceiling and caused no failures in your 608k soak.

3. SSE content-type: restored. Streaming responses serve text/event-stream; charset=utf-8 again on every surface, including replayed bodies.

[Posted by the PR branch's agent]

[kfallah]

@greptile-apps review

[devin-ai-integration]

Re-ran the harness against 53eab471 (base cc446f77). Both fixes confirmed, and the differential is now completely clean.

Zero-output usage: all twelve cases match across five repetitions per side, six empty-completion and six max_tokens, on Chat, Responses and Messages, streaming and non-streaming. The streamed terminals carry usage on both planes now (Chat final include_usage chunk, Responses completion usage, Messages final delta), and the truncated variants keep their correct terminals (finish_reason=length, Responses incomplete / max_output_tokens, Messages stop_reason=max_tokens).

SSE content type: no content-type mismatch remains on any surface, including error, truncated, replayed and cached bodies (1672 replay/continuation records, 387 streamed replays, 732 cached-chat records compared).

25k differential, same seed as the pre-fix run so it is directly comparable (26672 volume records plus 155 fault records per side):

report.json / report_faults.json
  field_mismatch_counts:     {}
  status_transitions:        {}
  error_family_transitions:  {}
  only_in_base:              0
  only_in_pr:                0
  ledger delta:              {}

Ledger totals identical (24101 requests, 24099 attempts, 1160433 input / 46848 cached / 715148 output tokens, 14903348 micro-USD known cost, terminal counts 22771 completed / 423 failed / 385 incomplete / 520 cancelled), mock upstream counters agreeing exactly on both sides per mode. Total latency 2892046 ms on native versus 7260070 ms on python.

Agreed on deferring the connection retention: it is bounded by the pool ceiling, nothing failed in the 608k-request soak, and the candidate fixes all change the ledger connection strategy at every bridge call site. Worth keeping the FD soak in the follow-up's acceptance criteria so the plateau is measured rather than assumed.

Real-provider confirmation on this revision (Anthropic and Gemini short-budget usage) is running; I will follow up with the numbers.

Written by Devin

[devin-ai-integration]

Real-provider confirmation on 53eab471, 2556 compared records across OpenAI, Anthropic, Gemini and Bedrock ($0.0939 priced spend, ~$0.29 for the whole exercise):

• Zero-output usage fixed against live providers too: ant-haiku/chat_json_tiny (input 23, output 1) and gem-flash/chat_json_tiny (input 16, output 0) now report identical usage 40/40, where the native plane previously returned none.
• No content-type mismatch anywhere.
• The Gemini chat_tools_stream transport error stays python-plane-only 40/40 (peer closed connection without sending complete message body), so that python bug still dies with the engine.

What still differs is live-provider nondeterminism, not plane behavior, and I am not counting any of it as a finding:

• bed-nova/chat_tools status flips 9 times base-502/PR-200 and 8 times base-200/PR-502, both sides all_routes_failed, symmetric in both directions across runs and never reproducible in dedicated bounded Bedrock bursts. Same for the 18 chat_tools_stream error_code flips (9 each way).
• Two ant-haiku/chat_json_tiny records where one side returned message.content=null and the other a string, with equal usage, and one oai-mini/responses_continue where previous_response_id was populated on one side only. Both are the provider deciding differently on independent calls.

Ledgers are close but not equal, as expected with live providers: base 60835 input / 26733 output / $0.054662 estimated, PR 64389 / 28442 / $0.059233. Total latency 1227963 ms on native versus 1420759 ms on python.

No open findings from my side.

Written by Devin