Repository navigation
feat: routing table, per-key cost cap, and single-hop provider fallback (#4) - #40
Merged
Merged
Conversation
With PR #36's flare reactor-side TLS path shipped, the following files are no longer reachable from the production binary: - opengateway/mojo_bridge/tls_proxy.py -- vendored stdlib-ssl proxy. The reactor-side bind_tls/serve_tls in main.mojo replaces it; the wrapper was reverted from main.mojo when we discovered the proxy's daemon-thread recv stalls when the reactor also calls into the Python bridge (later diagnosed as a cert hostname issue, not the proxy). Kept around as a "vendored fallback" for the flare v0.9.0 pin, but nothing in main.mojo imports it and the diagnostic it was designed to enable is no longer relevant. - tests/test_tls_proxy.py -- the 3 tests for the above. Validated the proxy worked in pure Python (pytest passed), but with the proxy unwired they're pure overhead. - opengateway/mojo/repro_tls_thread.mojo + opengateway/mojo_bridge/_repro_tls.py -- the minimal reproducer for the stdlib-ssl + Mojo-runtime daemon-thread interaction that was filed as modular/modular#6865. That issue was closed (the only observable "stall" was cert hostname verification, not a runtime bug), and the reproducer no longer serves a purpose. Also add uv.lock to .gitignore -- opengateway uses pixi for env management, uv-generated lockfiles shouldn't be in the repo. Cleaned up opengateway/mojo_bridge/__init__.py to drop the start_tls_proxy import/export, and removed the corresponding "vendored fallback" paragraph from ADR-003 #4 since the fallback no longer exists. Verified: 39 pytest pass (down from 42 -- the 3 tls_proxy tests gone), ruff + mypy clean, Mojo router tests pass. Co-authored-by: opencode-agent <noreply@opencode.ai>
flare is managed by pixi (see pixi.toml) and checked out into a local `flare/` dir during install. It is not part of the repo.
…ck (#4) - New ``RouteRule`` config (``ROUTES_JSON`` env var) maps a model to a primary + optional fallback provider, with per-1k-token pricing and a max-output-tokens ceiling. Empty list keeps the legacy prefix-based routing for backward compatibility. - Per-virtual-key ``max_cost_usd`` cap (new column in ``virtual_keys``, additive migration). Enforced pre-call via worst-case estimate = ``max_tokens`` × output price + prompt estimate × input price. Over-estimation is the right bias for a cap. Skipped when no route is configured (no pricing data) or when the key has no cap. - Single-hop provider fallback on retryable failures (5xx, 429, connection / timeout). Non-retryable 4xx other than 429 surfaces immediately. Streaming falls back only before the first frame — mid-stream errors would corrupt the SSE shape, so they're logged and the stream ends with the canonical ``[DONE]`` terminator. - Routing lives in the Python bridge (``opengateway.mojo_bridge.routing``), keeping the Mojo router (prefix → module) unchanged. Drift is still guarded by ``tests/test_mojo_bridge.py::test_routing_rules_match_mojo_router``. - New ``AnthropicProvider`` stub reusing the OpenAI-compatible ``/v1/messages`` httpx plumbing — enough for the fallback story without adding the ``anthropic-python`` SDK as a second dependency. - Stream path pre-flights the primary's API key (and fallback's) so a missing key surfaces as 502 before the pump thread starts. Only enforced for known provider modules; custom adapters and test stubs are passed through. Tests: ``tests/test_cost_cap.py`` (6) and ``tests/test_provider_fallback.py`` (11) cover the new surface. 56 passed, 0 failed; ruff + mypy clean.
johnnyhuy
added a commit
that referenced
this pull request
Sep 19, 2026
Post-merge format cleanup. CI's pinned ruff version enforces a tighter format than the local install; collapse 4 long-form expressions into single lines and unwrap multi-line dict literals. Co-authored-by: Claude <noreply@anthropic.com>
johnnyhuy
added a commit
that referenced
this pull request
Sep 19, 2026
johnnyhuy
added a commit
that referenced
this pull request
Sep 19, 2026
Captures the user-facing surface of #4 (PR #40) that landed in the code without a README update. - New "Routing, Cost Caps, and Fallbacks" section: explains ROUTES_JSON shape, per-key max_cost_usd enforcement, single-hop fallback on retryable failures, and streaming fallback's pre-first-frame boundary. - Configuration table gains ROUTES_JSON. - Repository structure updated: providers/anthropic.py (stub), mojo_bridge/{stream,db,ratelimit,routing}.py, tests/test_cost_cap.py, tests/test_provider_fallback.py. - Roadmap corrected to match reality: Anthropic stub in-tree but not a real adapter; Postgres-backed virtual keys, Prometheus metrics, Redis rate limiting, in-binary TLS, and calver release-please are all shipped (was marked [ ]). - Philosophy test count bumped 23 → 56. Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4.
Summary
Three new routing capabilities for the Mojo bridge:
RouteRuleconfig (ROUTES_JSONenv var) maps a model to a primary + optional fallback provider with per-1k-token pricing and a max-output-tokens ceiling. Empty list preserves the legacy prefix-based routing.max_cost_usdcap (newvirtual_keys.max_cost_usdcolumn, additive migration). Pre-call worst-case estimate; over-estimation is the right bias.[DONE].Design boundaries
max_cost_usdis a separate USD cap. Pre-call only; no persistent USD ledger yet — overspend is logged not aborted.test_routing_rules_match_mojo_routertest.AnthropicProviderstub reuses the OpenAI-compatible/v1/messageshttpx plumbing. Fullanthropic-pythonSDK integration is a follow-up.Files changed
Also includes the prior
e2e5a44(orphan TLS cleanup) +f7d7cda(gitignoreflare/).Testing
pytest: 56 passed (39 prior + 17 new)ruff check: cleanmypy opengateway/: cleanMigration notes
Operators upgrading from
v0.1.1:virtual_keys.max_cost_usd DOUBLE PRECISION(nullable).ensure_schema()runs the ALTER idempotently.ROUTES_JSON. Unset = identical pre-Add routing guardrails for cost caps and provider fallback #4 behaviour.OPENAI_API_KEYandANTHROPIC_API_KEYsemantics unchanged. The new anthropic module is a stub until a real adapter lands.