Add per-model KV quantization and physical request-capacity accounting

[Gajesh2007]

Summary

Add optional paged full-attention KV quantization (int4, k8v4, int8) with exact-model overrides. Packed pages, resident reuse and complete checkpoints retain a versioned format identity. Native KV remains the default; weight quantization does not select KV precision. K4/V4 uses five effective bits per value including FP32 metadata, giving 3.2× BF16 payload compression before other costs.

Admission now prices physical storage, page/fixed costs and guaranteed GPU workspace while reporting literal prompt-plus-generation tokens. Coordinator reservations debit both routing and public capacity consistently. Execution-specific throughput and TTFT histories prevent native rates from being reused for quantized KV; unknown formats bootstrap at concurrency one, retain cold-load deadline bounds, and need the existing same-format sample threshold before expanding within the configured ceiling.

An explicit benchmark-only prefill option reserves additional FP32 scratch for forced fused SDPA on eligible shapes. Budget refusal keeps the funded direct path. Deadline arrivals retire any owned optional step before forecasting against their original absolute deadline. Production packed prefill stays direct.

Before

flowchart LR
  R[Request] --> C["TPSRegistry / ttftCalibrator: model and hardware key"]
  C --> A["EngineV2Bridge.capacity: native KV sizing"]
  A --> P["PagedLayerCache.updateAndAttend: native K/V"]
  P --> S["CompleteCheckpointCodec: native tensor layout"]
  P --> H["BackendSlotCapacity heartbeat"]
  H --> C

Loading

After

flowchart LR
  R[Request] --> C["snapshotProvider / remainingSlotTokenBudget: execution-scoped rates and pending debit"]
  C --> A["EngineV2Bridge.capacity / AdmissionV2: physical bytes and raw tokens"]
  A --> K{EngineV2KVQuantizationPolicy}
  K -->|native| N[Existing native attention]
  K -->|packed| P["PagedLayerCache.updateAndAttend: packed direct attention"]
  P --> S["CompleteCheckpointCodec: packed bytes and format identity"]
  P --> F["EngineLoopV2.finalize: GPU retirement before refund"]
  N --> H["BackendSlotCapacity: actual execution_identity"]
  F --> H
  H --> C
  U[Uncalibrated quantized slot] --> B["effectiveMaxConcurrencyForModel: cap 1; unknown TTFT plus cold-load bound"]
  B --> C

Loading

Validation and limits

Local numerical, stride, storage, checkpoint, ownership, cancellation and deadline tests passed, along with the full coordinator suite, focused race checks, provider capacity/identity/TTFT tests and TypeScript mirror lint. Raw logs and exact model/runtime hashes are retained in the implementation report.

The small authored generation probes found GPT-OSS K4/V4 regressions: native scored 8/8 graded cases, K4/V4 6/8, and K8/V4/K8/V8 8/8. Qwen and Gemma INT4 matched their native 6/8 scores. Eight graded cases do not establish broad quality or a one-percentage-point non-inferiority margin.

Measured Qwen 16K prefill averaged 11.95s native, 36.51s direct INT4 and 18.73s optional fused INT4 on the same local executable. This is not a demonstrated native-speed prefill path. The final quiet 4K repeat also remained slower: native/INT4 common decode was 95.67/76.04 TPS at B1 and 224.14/168.01 at B4. Strict mixed-arrival runs were inconclusive for both formats because the unchanged 5ms delivery tolerance could not be met. No compute-concurrency increase or serving default is promoted.

Coordinator-first rollout is required because older coordinators ignore the new identity field. Scalar rates remain context-sensitive and heartbeat samples are not independent requests. Native-only bandwidth anomaly expectations exclude quantized samples until separate expectations are qualified. M5/Qwen3.8, vision and broad long-context/MTP quality remain unqualified.

Dependency stack

Pinned draft dependencies: mlx #17, mlx-c #9, mlx-swift #22, and mlx-swift-lm #142. They provide the FP32 Metal tile correction, explicit fused API and packed runtime. No merge, production configuration change or deployment is included.

The threat-model CI job currently fails before review because its API credential is invalid. The separate E2E load benchmark awaits the repository's deliberate manual cost approval. These are distinct from local validation and the ordinary CI test results.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
d-inference	Ready	Preview	Sep 7, 2026 4:03pm UTC
d-inference-console-ui-dev	Ready	Preview	Sep 7, 2026 4:03pm UTC
d-inference-landing	Ready	Preview	Sep 7, 2026 4:03pm UTC