Skip to content

concepts.md

Latest commit

 

History

History
60 lines (41 loc) · 2.32 KB

concepts.md

File metadata and controls

60 lines (41 loc) · 2.32 KB
Raw

Concepts

Switchyard vocabulary, mapped to this kit.

Runtime surfaces

Surface Role
switchyard-server Standalone HTTP proxy. TOML deployment. OpenAI Chat, OpenAI Responses, Anthropic Messages.
switchyard-libsy Embeddable Rust library. Picks targets; host makes HTTP calls.
switchyard launch Starts the native server via PyO3 and points Claude Code / Codex / OpenClaw at it.
This kit’s Python routers Educational offline stand-ins for demos and labs.

Request flow

  1. Client sends OpenAI or Anthropic shaped request.
  2. Switchyard decodes into provider-neutral protocol types.
  3. Route algorithm selects a target.
  4. Target’s LLM client encodes into the upstream format and calls the backend.
  5. Response is translated back to the client’s API.

Three-layer deployment

[llm_clients.openrouter]   # transport + credentials env
[targets.weak]             # model id + client
[routes.smart]             # client-visible id + algorithm
Layer Defines
LLM client base_url, format (openai_chat | openai_responses | anthropic_messages), api_key_env
Target Upstream model ID + which client to use
Route Model name clients put in model field + routing type

Table keys (routes.smart) are local names. Only id is exposed on GET /v1/models.

Roles vs models

Strong / weak / capable / efficient are roles inside an algorithm, not fixed properties of a vendor model. The same GPT-4o-mini can be weak in one route and the only target of a passthrough in another.

Protocol translation

switchyard-translation converts requests, buffered responses, and streams across:

  • OpenAI Chat Completions
  • OpenAI Responses
  • Anthropic Messages

That is why Claude Code can speak Anthropic while the backend is OpenAI-compatible vLLM.

Metrics

Production Switchyard exposes Prometheus metrics (requests, errors, latency, tokens, routing overhead). This kit simulates tokenomics offline via examples/lib/metrics.py.

Further reading