Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/lib/components/HomeMetrics.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

const metrics = [
{
title: 'TIFT REDUCTION',
title: 'TTFT REDUCTION',
prefix: '',
value: 70,
suffix: '%',
Expand All @@ -26,11 +26,11 @@
description: 'Metrics Available'
},
{
title: 'OPEN AI COMPATIBLE',
title: 'OPENAI-COMPATIBLE',
prefix: '',
value: 100,
suffix: '%',
description: 'Open AI Compatible'
description: 'OpenAI-Compatible'
}
] as const;
</script>
Expand Down
2 changes: 1 addition & 1 deletion src/lib/config/reference-nav.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ export const referenceNav: DocsNavSection[] = [
{
label: 'API',
items: [
{ label: 'OpenAI Compatible', href: `${base}/reference/api/openai` },
{ label: 'OpenAI-Compatible', href: `${base}/reference/api/openai` },
{ label: 'Responses API', href: `${base}/reference/api/responses` },
{ label: 'Anthropic Messages API', href: `${base}/reference/api/messages` },
{ label: 'Admin API', href: `${base}/reference/api/admin` },
Expand Down
6 changes: 3 additions & 3 deletions src/lib/content/concepts/architecture/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ The gRPC path provides maximum performance by handling all text processing at th

---

## HTTP Path (OpenAI Compatible)
## HTTP Path (OpenAI-Compatible)

The HTTP path supports two modes for OpenAI-compatible backends.

Expand All @@ -147,9 +147,9 @@ Disaggregated inference with separate prefill and decode workers:

---

## 3rd Party Path
## Third-Party Path

The 3rd party path routes to external LLM providers through a unified interface.
The third-party path routes to external LLM providers through a unified interface.

### Model Discovery

Expand Down
6 changes: 5 additions & 1 deletion src/lib/content/getting-started/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,8 +206,10 @@ curl http://localhost:30000/v1/responses \
### Advanced Features

- [Load Balancing](load-balancing.md)
- [KV Events Cache-Aware Routing](kv-events-cache-aware.md)
- [Tokenizer Caching](tokenizer-caching.md)
- [MCP in Responses API](mcp.md)
- [External Providers](external-providers.md)

---

Expand Down Expand Up @@ -521,14 +523,16 @@ curl http://localhost:30000/workers
### Reliability and Data

- [Reliability Controls](reliability-controls.md) — concurrency limits, retries, and circuit breakers
- [Data Connections](data-connections.md) — history backend setup for Postgres, Redis, and Oracle
- [Data Connections](data-connections.md) — history backend setup for PostgreSQL, Redis, and Oracle
- [Tokenization and Parsing APIs](tokenization-and-parsing.md) — tokenize, detokenize, and parser endpoints

### Advanced Features

- [Load Balancing](load-balancing.md) — policy selection and tuning
- [KV Events Cache-Aware Routing](kv-events-cache-aware.md) — prefix routing driven by engine KV cache events
- [Tokenizer Caching](tokenizer-caching.md) — L0/L1 cache setup for gRPC mode
- [MCP in Responses API](mcp.md) — configure and execute MCP tools through `/v1/responses`
- [External Providers](external-providers.md) — route to OpenAI, Anthropic, Gemini, and xAI backends

## Troubleshooting

Expand Down
6 changes: 3 additions & 3 deletions src/lib/content/getting-started/kv-events-cache-aware.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ This guide walks through wiring an **SGLang worker emitting KV cache events** to

## Why event-driven?

Cache-aware routing has three internal flavours. The one this guide configures is the most accurate of the three because it routes against the worker's **actual** KV cache state rather than an approximation.
Cache-aware routing has three internal flavors. The one this guide configures is the most accurate of the three because it routes against the worker's **actual** KV cache state rather than an approximation.

| Flavour | Tree | Input | Worker connection | Triggered when |
| Flavor | Tree | Input | Worker connection | Triggered when |
|---|---|---|---|---|
| **Event-driven** | `PositionalIndexer` (event-built) | Token IDs | gRPC | Worker emits KV events |
| Approximate token tree | `TokenTree` (prefix observed at routing time) | Token IDs | gRPC | Worker is gRPC but emits no events |
Expand Down Expand Up @@ -113,7 +113,7 @@ Event-driven routing needs the worker in **SMG gRPC mode** (`--grpc`) — KV eve
| `publisher: "zmq"` | Selects the ZMQ publisher the servicer bridges. |
| `endpoint` / `topic` | ZMQ `PUB` address and topic prefix. For data-parallel, the port is `endpoint_port + dp_rank`; SMG currently consumes rank 0. |

SMG learns the block size from the `BlockStored` events themselves, so you needn't set it; pass vLLM's `--block-size N` (the analogue of SGLang's `--page-size`) only to pin a non-default value.
SMG learns the block size from the `BlockStored` events themselves, so you needn't set it; pass vLLM's `--block-size N` (the analog of SGLang's `--page-size`) only to pin a non-default value.

Everything downstream — SMG flags, block-size learning, and the verification logs — is unchanged; `KvEventMonitor` consumes the events the same way for any gRPC worker.

Expand Down
2 changes: 1 addition & 1 deletion src/lib/content/getting-started/logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Configure structured logging with multiple output formats and integrate with log

#### Before you begin

- SMG [installed](index.md)
- SMG [installed](index.md#install)
- Completed the [Getting Started](index.md) guide
- Log aggregation system (optional): Elasticsearch, Loki, or similar

Expand Down
2 changes: 1 addition & 1 deletion src/lib/content/getting-started/pd-disaggregation.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ registered id until the worker is re-registered.
With vLLM data parallelism (`data_parallel_size > 1`), run SMG with
`--dp-aware`: SMG pins each request to a DP rank and mints the decode params
with the matching `{engine_id}_dp{rank}` engine-core id. Without `--dp-aware`,
DP>1 prefill workers are not minted for and SMG falls back to legacy host/port
decode params are not minted for DP>1 prefill workers, and SMG falls back to legacy host/port
injection (decode recomputes the prompt locally). External-LB DP
(`--data-parallel-external-lb`, one pod per rank) is unsupported for Mooncake
minting: every pod's engine core is `{engine_id}_dp0`, so register pods as
Expand Down
4 changes: 2 additions & 2 deletions src/lib/content/getting-started/tokenization-and-parsing.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,6 @@ curl http://localhost:30000/parse/reasoning \

## Next Steps

- [Gateway Extensions API](../reference/api/extensions.md#tokenize)
- [Gateway Extensions API](../reference/api/extensions.md#parse-function-calls)
- [Gateway Extensions API](../reference/api/extensions.md#tokenizer-management)
- [Gateway Extensions API](../reference/api/extensions.md#parser-utilities)
- [Admin API Reference](../reference/api/admin.md)
2 changes: 1 addition & 1 deletion src/lib/content/reference/api/extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ These run behind protected-route auth middleware:

For OpenAI-compatible endpoints (`/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`), see:

- [OpenAI Compatible API](openai.md)
- [OpenAI-Compatible API](openai.md)
- [Responses API](responses.md)

---
Expand Down
4 changes: 2 additions & 2 deletions src/lib/content/reference/api/responses.md
Original file line number Diff line number Diff line change
Expand Up @@ -544,7 +544,7 @@ curl http://localhost:30000/v1/conversations/conv_abc123def456

### Update Conversation

Update conversation metadata. Uses merge semantics - set a key to `null` to delete it.
Update conversation metadata. Uses merge semantics set a key to `null` to delete it.

```
POST /v1/conversations/{conversation_id}
Expand Down Expand Up @@ -744,7 +744,7 @@ Returns the item object.

### Delete Conversation Item

Remove an item from a conversation. This performs a soft delete - the item may still exist if referenced by other conversations.
Remove an item from a conversation. This performs a soft delete the item may still exist if referenced by other conversations.

```
DELETE /v1/conversations/{conversation_id}/items/{item_id}
Expand Down
2 changes: 1 addition & 1 deletion src/lib/content/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -538,7 +538,7 @@ headers can otherwise spoof storage hook request context values.

Controls the tokio async runtime that backs request handling.

By default the runtime is **container-aware**. tokio sizes its worker pool to
By default the runtime is **container-aware**. Tokio sizes its worker pool to
`std::thread::available_parallelism()`, which on Rust 1.95+ already reads the
cgroup CPU quota — so under a Kubernetes `limits.cpu` the worker count matches
the pod's quota, not the host's core count. No extra configuration is needed for
Expand Down
2 changes: 1 addition & 1 deletion src/lib/content/reference/priority-scheduler.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ The scheduler surfaces admission and preemption outcomes as HTTP status codes. E

| Status | Condition | `X-SMG-Error-Code` | Extra headers |
|--------|-----------|--------------------|---------------|
| **503** Service Unavailable | **Preempted** — admitted, then cancelled before its first byte to make room for a higher-priority request | `scheduler_preempted` | `X-SMG-Preempted: true`, `Retry-After: 1` |
| **503** Service Unavailable | **Preempted** — admitted, then canceled before its first byte to make room for a higher-priority request | `scheduler_preempted` | `X-SMG-Preempted: true`, `Retry-After: 1` |
| **429** Too Many Requests | **Queue full** — the request's per-class queue is at its configured depth | `scheduler_queue_full` | — |
| **408** Request Timeout | **Queue timeout** — the request waited longer than its class's `queue_timeout` | `scheduler_queue_timeout` | — |
| **499** Client Closed Request | **Client gone** — the client disconnected before admission completed (nginx convention; never actually read) | `scheduler_client_cancelled` | — |
Expand Down
Loading