diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 22c8a8dd..bd2da29c 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -49,7 +49,10 @@ jobs: - name: Typecheck run: |- python -m pip install ty - pip install --prefer-binary -e ".[tests]" + # Install optional modules whose concrete types are part of the source + # tree. Otherwise unresolved-import=ignore turns them into Unknown and + # CI misses diagnostics that appear in a full local development env. + pip install --prefer-binary -e ".[tests,tui]" huggingface-hub ruamel.yaml ty check ./infer_stack build_and_test_sdist: ## diff --git a/.gitignore b/.gitignore index f7556690..943676a4 100644 --- a/.gitignore +++ b/.gitignore @@ -17,10 +17,6 @@ generated/* !generated/kubeai/ generated/kubeai/* !generated/kubeai/.gitignore -!examples/ -!examples/single-node/ -!examples/single-node/config.yaml -!examples/single-node/models.yaml # rsync target for e2e harness reports (developer artifacts, not source) e2e-report-*/ diff --git a/AGENTS.md b/AGENTS.md index 7d57786e..f03cef37 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,3 +47,7 @@ new entry; keep diffs small. If a lesson seems important, but a MWE is not possible, label the lesson as SPECULATIVE. + +## Commit attribution + +Every agent-authored commit must carry a `Co-authored-by:` trailer naming the exact model (and weights/config variant, when known) that produced it, per its own runtime identification — never copied from earlier commits; if undeterminable, say so. diff --git a/CHANGELOG.md b/CHANGELOG.md index ebbc081b..56802a42 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,311 @@ We [keep a changelog](https://keepachangelog.com/en/1.0.0/). We aim to adhere to [semantic versioning](https://semver.org/spec/v2.0.0.html). +### Allow slow K3s installs during bootstrap and join + +- `infer-stack kube k3s bootstrap` and `infer-stack kube k3s join` now give the + upstream K3s installer an explicit 10-minute timeout instead of inheriting + the generic 60-second command timeout. Other commands keep their existing + timeout behavior. Regression tests cover both server and worker installs. + +### Bump the default vLLM engine image to v0.30.0-cu129 + +- `PINNED_IMAGES['vllm']` is now `vllm/vllm-openai:v0.30.0-cu129` (previously + `v0.19.1`). Compose endpoints without a catalog `runtime.image` override now + render the CUDA 12.9 build; the image is part of service identity, so an + existing vLLM container is recreated once on the next converge. The KubeAI + backend is unaffected: its vLLM image comes from the KubeAI chart's default + profile, and `runtime.image` overrides are not supported there. +- The dev pre-pull instructions (`dev/e2e_tests/README.md`, + `dev/leasing-test-plan.md`) track the new tag. +- The dev-only patched image `dockerfiles/vllm-hack.dockerfile` still bases on + `v0.19.0`; bump it separately if that transformers patch needs the new + baseline. + +### Reversible Kubernetes / Compose workstation handoff + +- Added `infer-stack kube node status|detach|attach`. `detach` is a preview by + default; `--yes` marks an infer-stack-owned maintenance cordon, drains + controller-managed workloads, and leaves the Kubernetes agent/control plane + running so the host can temporarily use its GPU through the local Compose + backend. `attach --yes` waits for the same node identity to be Ready and + uncordons it after the operator has stopped local Compose GPU workloads. +- The lifecycle refuses bare/unmanaged pods rather than force-deleting them and + refuses to attach/detach through a cordon owned by unrelated cluster + maintenance. Interrupted detaches are marked so they can be retried or + reversed safely. +- `kube nodes` now exposes schedulability and reports an infer-stack-detached + node as `SCHED=compose`. +- Documented the practical two-workstation test workflow: preserve existing + Compose config/ledger, run KubeAI from a separate data root with + `INFER_STACK_BACKEND=kubeai`, and switch physical nodes between Kubernetes + scheduling and direct Compose use without uninstalling/rejoining Kubernetes. + +### Kubernetes integration setup moves into `infer-stack kube` + +- Kept the generic Kubernetes manager/backend distribution-neutral: generic + diagnostics now describe Kubernetes capabilities rather than K3s remediation, + while K3s provisioning remains isolated under `infer-stack kube k3s`. Added a + dedicated cluster setup runbook that documents the control-plane/worker model, + K3s as the first provisioning target, and the contract future distro-specific + integrations must preserve. +- Added `infer-stack kube nodes` and capability-driven `infer-stack kube setup`; + setup is read-only by default and `--apply` reconciles only infer-stack's + Kubernetes integration requirements. Existing GPU Operator / managed-cluster + NVIDIA resources and externally managed KubeAI installations are accepted + rather than replaced. +- `kube setup --apply` can reconcile the pinned NVIDIA device plugin with GPU + Feature Discovery, install/reconcile KubeAI, and merge discovered GPU + resource profiles without overwriting operator-owned profiles. Existing + KubeAI Helm versions and optional operator values are preserved unless an + explicit override is requested. Existing/operator `secrets.*` values are + carried only through a temporary mode-0600 Helm values file, never copied + into infer-stack's persistent generated YAML. +- Added explicit K3s convenience commands, `infer-stack kube k3s bootstrap` + and `infer-stack kube k3s join`; the old bootstrap/join scripts are now + compatibility wrappers so cluster lifecycle knowledge has one implementation. + Host NVIDIA driver/container-runtime installation remains outside infer-stack. +- KubeAI diagnostics and catalog guidance now point at the shared Kubernetes + setup surface instead of requiring users to compose README recipes and shell + scripts by hand. + +### External endpoints; gateway routes derived from the published catalogs + +- A catalog endpoint can name an OpenAI-compatible server infer-stack does + not run: `external: {api_base, model, api_key_env}` (or `catalog endpoint + add NAME --external-api-base ... --external-model ...`). It has no model, + lease or deployment; the gateway routes its alias to that server, sending + the key named by `api_key_env` (the name only is stored). Runtime options + beside `external:` are refused, and `api_key_env` may not name one of + infer-stack's own secrets. +- Published endpoint definitions outlive quiescence: an acquire merges its + catalogs into the published union instead of replacing it, so an unrelated + runbook no longer unpublishes another's endpoints. `routes prune` is the + explicit removal. +- Gateway routes are derived at every render from the published union and + the placed deployments. `litellm_registry.json` now keeps only the routes + of deployments no catalog defines (remembered past release, as before); + rows an older version wrote are still routed. `routes seed` publishes + catalogs, `routes prune` unpublishes, and `routes list` shows each route's + origin. A state dir from before the registry is no longer seeded from its + `litellm_config.yaml`. +- With dynamic routing, an external endpoint's route has a stable id from its + alias, so pointing it at another server replaces one route. +- An external key reaches the gateway by name: the host gateway's + environment lists it (a changed value recreates the gateway on `apply`), + the in-cluster gateway's Secret holds it (a change rolls the pod). `env + NAME=...` says when the gateway picks it up; `routes seed` refuses a key + with no value. +- A runbook whose catalog is already published no longer warns that its + catalogs differ from the recovery snapshot. +- An alias a resident deployment serves keeps its meaning even when no + catalog defines it (an ad-hoc acquire): `routes seed --replace`, `access` + and `acquire` refuse to publish a different definition until it is + released or evicted. +- A published external endpoint whose key has no value refuses every + publication (acquire, apply, config publish, ...) before anything changes, + with the `infer-stack env NAME=...` to fix it; no command recreates the + gateway without the key. +- Publishing endpoints is transactional: an acquire commits its catalogs + with its lease and publication marker, and a fresh ledger's first snapshot + is written with its first marker, never before. `gc`, `release` and + `evict` publish no endpoints on a fresh ledger; `acquire`, `access` and a + first `apply` / `stack up` do. +- `infer-stack access NAME...` makes endpoints reachable whoever runs them: + it leases the managed ones (as `acquire`), publishes external ones with no + lease, waits for the front door, and writes the same env-file (with no + `INFER_STACK_LEASE_ID` when nothing is leased). `run` uses it, so a mixed + bundle takes one lease for its managed members. `acquire` of an external + endpoint points at `access`; `release --env-file` of a lease-less env-file + has nothing to release. + +### Hardening from a review of the backend-parity work + +- `routes seed --replace` replaces exactly what it showed: an alias another + process redefined meanwhile refuses, and nothing is written. + +- A KubeAI Model with several replicas (`min_replicas`) is resident like one + pod: releasing its last keep-warm lease no longer prunes it, `leases` shows + it running, and it fails fast when every replica crash-loops. +- On KubeAI, a pod the scheduler cannot place evicts an idle Model only when + the scheduler reports a capacity shortage, and only one on a node the pod + could use; an impossible node selector or a taint evicts nothing. +- An apply that reached the runtime but not its routes keeps the publication + and its approved render pending, so a render that changes before the retry + (an upgrade) must be approved again. KubeAI's apply reports its host + gateway's result instead of dropping it. +- `secrets rotate` restores the old key when the publication fails before + its apply began. +- With dynamic routing, routes are removed and verified before their + upstreams are stopped. +- An acquire coalesces onto an adequate live deployment before reviving an + older idle one. +- `routes seed` refuses to redefine an existing alias unless `--replace`. +- A preview (admission) no longer writes the gateway's secrets. + +### `/dev/shm` for vLLM on Compose, and the old recipes as suggestions + +`runtime.shm_size` (e.g. `16g`) sets a vLLM container's `/dev/shm` on the +Compose backend; Docker's default is 64 MiB, and vLLM's workers share memory +when a model spans GPUs. It is opt-in, so no running engine is recreated; a +parallel engine without it logs a warning, and `catalog suggest` sets it on +multi-GPU entries. The pre-leasing `docs/demos/`, `recipies/` and `examples/` +are removed: their 4 x 96 GB vLLM tuning is now three suggestion variants +(`qwen3.5-122b-a10b-tp4-128k`, `qwen3.5-122b-a10b-fp8-tp4-262k`, +`qwen3.6-35b-a3b-tp2-262k`). + +### The gateway can run inside the cluster + +`infer-stack config set kubeai_gateway cluster` runs the LiteLLM gateway as a +Deployment and a NodePort Service in the KubeAI namespace instead of on the +host running infer-stack, so no single host is in every request's path and +no `kubectl port-forward` is needed: an env file's `OPENAI_BASE_URL` is a +node's address on the NodePort (or `kubeai_gateway_url`), and every node +answers. It is the same gateway, key and route registry; `secrets rotate` +updates its Secret and rolls it, `doctor` checks it, and `stack down` +removes it. It takes static routes only. `docs/kubeai-backend.md` gains +"Add a workstation", the runbook for a second GPU machine. + +### KubeAI picks a resource profile by GPU size + +On the kubeai backend, an endpoint with `placement.min_vram_gib` and no +`resource_profile` gets the smallest resource profile whose nodes' GPUs are +that large, read from GPU Feature Discovery's `nvidia.com/gpu.memory` label; +before, `min_vram_gib` was warned about and ignored. `infer-stack measure +--record` works there too, and a recorded measurement feeds the same choice. +`infer-stack catalog suggest` on kubeai sizes the catalog to the cluster's +largest GPU node and proposes a resource profile per GPU product for the +helm values. `dev/k3s_agent_container.sh` adds a second (simulated) node to +a development cluster. + +### A parity suite for the two backends + +`tests/test_parity.py` runs one scenario per row that `docs/backend-parity.md` +marks *same*, on the compose backend over a fake Docker and on the kubeai +backend over a fake kubectl, from the client contract to the TUI. The TUI no +longer swaps an injected Docker runner for the real one. + +### The gateway in front of a cluster is the compose gateway + +On the kubeai backend the gateway now takes the same settings as on compose: +Open WebUI (`ui`, on by default), the reverse proxy, and dynamic routing, +under which each deployment is its own Model (`-`), so the +same model acquired `--dedicated` twice runs as two Models behind one alias. +Catalog endpoints are routed before they run, as on compose, so a new Model +no longer recreates the gateway. The gateway's changes are shown and +approved with the acquire's, before its lease commits. `routes list / seed / +prune` work on kubeai and show a cluster route's upstream (they showed `?`), +and `gc --orphans` and `clean --orphans` run there (and find nothing: the +backend only reads its own labelled pods). + +### `ps`, `logs`, `status` and the TUI read the backend, on either backend + +`infer-stack ps` lists what the backend runs, containers or pods and the +gateway, in one table that says what each serves; `infer-stack logs` takes an +endpoint alias, an instance name, a container id prefix or a deployment id, +and `-f` keeps following instances that start or restart later. Both read the +same strict residency the controller decides with, so they work on KubeAI; +`status` does too, and says `starting` for an instance that is up but not yet +ready. `stack up` is now `apply` and `stack down` stops everything on either +backend; the raw Compose verbs (`stack compose -- …`, `restart`, `pull`, +`start`, `stop`) act on the Compose project on this host, which is the +gateway's on KubeAI. The TUI's runtime pane (formerly "docker") follows pod +logs, lists pods, and its Apply / Down buttons work on KubeAI. + +Fixed on the way: every kubectl call from the TUI failed, because the TUI +replaced the backend's command runner with Docker's, whose environment has no +`KUBECONFIG`; the TUI's "engines" log view included Open WebUI and Postgres; +and a flag before a positional ate it (`logs -f qwen` followed everything, +`acquire --yes qwen` named no endpoint), on every command. + +### One acquire path + +The controller's pre-admission branch is gone: every backend, including the +dry-run backend, acquires, renders, renews and publishes through admission. +A backend with only `realize` / `teardown` gets the admission surface from +`SimpleAdmission`. Two behaviours changed with it. While residency cannot +be read, no acquire is admitted; before, a request needing no new GPU was +admitted and then failed at its render, after the lease was committed. And +`routes prune` computes the desired set with the same view the render uses. + +### KubeAI acquires go through admission, as Compose's do + +An acquire on the KubeAI backend is now previewed in memory and commits its +lease only if every deployment renders, the same path Compose takes. A +refused acquire (no resource profile, an ollama endpoint, a served-name +collision) writes no lease and runs no `kubectl`; before, the lease was +committed, rendered, and then rolled back. A `--queue` acquire of such an +endpoint fails at once instead of waiting out its timeout. The cluster +schedules, so a KubeAI deployment commits an empty GPU allocation; `leases` +still shows no GPU for it, and a `--no-apply` acquire now reports it as +cluster-scheduled rather than unplaced. An idle +keep-warm Model is kept only while its pod has started, as a keep-warm +container is on Compose. The approval digest moved into the scaffold both +backends share, and one function now answers whether a backend allocates +GPUs. Removing the older acquire branch, which only test fakes still use, +is the next step (roadmap P1b). + +### Compose and KubeAI: a parity matrix and a roadmap + +`docs/backend-parity.md` states the relationship the two backends are meant +to have (KubeAI is Compose plus a scheduler: more information supplied, more +tools installed, the same catalog, verbs, env file and TUI) and records, row +by row, where that holds today, where it does not, and which differences are +deliberate. `docs/planning/backend-parity-roadmap.md` plans the rest in five +phases with exit criteria, from one acquire path to the gateway inside the +cluster for a multi-workstation run. The README's KubeAI section, which still +described the pre-leasing `setup` / `deploy` workflow and a live patch that +is no longer needed, now points at the current docs and keeps only the +cluster prerequisites and the `kubectl` debugging checks. + +### Endpoints|models and leases|deployments are tabs + +The sidebar shows Endpoints and Models as two tabs instead of two panes split +by a divider, so the endpoint list gets the whole height; the main area does +the same for Leases and Deployments. A tab's label carries its counts +(`Leases 1/3`), so the hidden one is still readable. Each pair is one switch +(`InferStackTUI.TABBED_CATALOG`, `TABBED_TABLES`); setting it to `False` +restores the draggable split, and both layouts are tested. + +### One Clean up, in the footer, with a CLI command behind it + +The leases and deployments panes each had a Clean up button, and both did the +same thing: forget released/expired leases and stopped deployments. Nothing +running changes. That is now one `x Clean up` in the footer, and the same +action on the CLI is `infer-stack gc --forget`. + +The TUI log now names the CLI command for the actions that were missing one: +`r` (`status`), Clean up (`gc --forget`), saving settings (`config set …`), +and the API tab's send, test-all (`test `) and model list (a `curl` +that reads the key through `infer-stack env` and never prints it). Poll +intervals are TUI-only, and the log says so. + +### The CLI is built on kwconf instead of scriptconfig + +kwconf is scriptconfig's successor, which aiq-magnet, cmd_queue and kwdagger +already use. `Value`, `ModalCLI` and `__command__` carry over; `DataConfig` +is now `Config`. One behavioural difference needed care: kwconf reads the +strings `null`, `true` and numbers on the command line as values, so +`--backend null` arrived as `None`. Options whose choices are strings now +take `type=str`. The program name stays `infer-stack` in usage and errors +(`__prog__`). + +`uv.lock` and `requirements/locks/tests.txt` were regenerated: kwconf 0.11.0 +added, scriptconfig removed. The uv `exclude-newer` cutoff (2026-06-04), which +predated every kwconf release, is removed; the locked versions are unchanged. + +### The TUI stops repainting what did not change + +Every refresh tick repainted the lease and deployment panes, because setting +a pane's border title repaints it even when the text is unchanged: about +7 KB/s of terminal output with nothing on screen changing, which is what an +SSH session feels. Titles are now set only on change (75 bytes/s idle). +Streamed engine logs are drawn in batches of up to 200 lines every 100 ms +instead of one UI message per line, and a flood keeps its newest lines. A +watchdog reports any UI-thread stall over 0.5 s in the TUI log, with the +stack it was stuck in written to the error log file. `dev/profile_tui.py` +measures event-loop lag for idle, docker-pane and log-flood cases. + ### Ledger reads are safe across threads The ledger shares one SQLite connection between threads, but only write @@ -23,6 +328,73 @@ scheduler with its own GPU accounting: a lease held outside it, manual or keep-warm, occupies a GPU the scheduler believes is free, and the job it places there cannot start. +### An idle keep-warm model gives way to a leased one on KubeAI too + +The rule: a keep-warm model that no lease holds is always a candidate for +eviction when a model with a lease needs a resource. Compose admission already +applied it when placing. On KubeAI the cluster schedules and never evicts, so +a leased Model could sit `Unschedulable` behind idle warm ones until its +timeout. A readiness probe can now report `needs_room`; the wait then evicts +the longest-idle deployment, one per 30 s, never a leased one, and never +after the wait's deadline. Verified on k3s: a leased Model that did not fit +beside an idle one was ready 107 s later, with the idle one evicted. + +### The gateway is its own module + +The front door (LiteLLM config and service, route registry, dynamic-route +reconciliation, the managed keys, Open WebUI, the reverse proxy) moved out of +`leasing/compose.py` into `leasing/gateway.py`, with a `Gateway` object that +`ComposeBackend` owns (`backend.gateway`). The service naming rules moved to +`leasing/naming.py`. Backends now hand the gateway their route rows rather +than the gateway reading backend state. Rendered output is unchanged +(verified byte for byte), and `compose.py` is about a third smaller. Code +that imported gateway names from `infer_stack.leasing.compose` should import +them from `infer_stack.leasing.gateway`; the public naming helpers are still +importable from `compose`. + +### The kubeai backend fails fast on an engine that cannot start + +The crash diagnosis (restart count, exit code, and the engine log classified +as fatal, transient or unknown) was Docker-only, so on a cluster a model that +could never start held its lease for the whole timeout. It now lives in +`infer_stack/leasing/diagnosis.py` and both backends use it. The kubeai +backend reads pods through a new strict `residency()` (a kubectl failure +raises instead of reading as "nothing running") and quotes the crashed run's +log (`kubectl logs --previous`). A not-ready wait names the pod's reason, such +as `Unschedulable` or `ImagePullBackOff`. vLLM rejecting a flag (`error: +unrecognized arguments`) is now recognised as fatal on both backends; before, +it waited for two restarts. + +`runtime.env` now works on the kubeai backend (the Model's `spec.env`, with +the same templates and reserved names). The served-name rule lives in one +place, `leasing.models.served_name`. Two gateway route builders fell back to +the deployment id where the engine used the first served alias, so a +deployment without `served_model_name` routed to a name its engine did not +serve. + +`gc --orphans` and `network migrate` now refuse a non-compose backend +explicitly. They had used "the backend has `residency`" to mean compose. + +### The kubeai backend puts the LiteLLM gateway in front of the cluster + +On the kubeai backend a client had to name a model by its KubeAI Model name +(a DNS slug of the served name), not the endpoint alias. Cards send the +alias, so a card that ran on the compose backend got HTTP 404 on a cluster +(verified on k3s). The kubeai backend now runs the same LiteLLM gateway, +routing each alias to its Model: one `OPENAI_BASE_URL`, the managed key, and +the alias as the model name on both backends. `secrets rotate` works on it. +`--no-litellm` keeps the old direct access. New setting: +`kubeai_gateway_upstream`, for a gateway that cannot reach the cluster +Service's IP. + +The profile-era KubeAI renderer (`infer_stack.backends.render_kubeai_artifacts`) +is removed. Nothing called it since the profile path was excised; the kubeai +backend is the one renderer for KubeAI. + +`dev/kubeai_e2e.sh` now sends the alias as a card does, and fails when the +request fails. Before, a failed generation fell through to PASS: the check +sat in a `&&` list, where `set -e` does not apply. + ### Custom container launches are catalog data, not recipes `runtime.serve_recipe` is gone. An endpoint whose image has its own launcher @@ -807,7 +1179,10 @@ States the part that is not automatic: `$SLURM_JOB_GPUS` must be passed as including cards allocated to another job, and the failure surfaces later as a CUDA OOM in whichever job loses. -## Version 0.7.1 - Unreleased +## Version 0.7.2 - Unreleased + + +## Version 0.7.1 - Released 2026-09-24 ## Version 0.7.0 - Released 2026-08-28 diff --git a/Makefile b/Makefile index 68465960..55eefcb8 100644 --- a/Makefile +++ b/Makefile @@ -1,34 +1,18 @@ PYTHON ?= python -EXAMPLE ?= single-node - -init: - $(PYTHON) manage.py init - -render: - $(PYTHON) manage.py render - -deploy: - $(PYTHON) manage.py deploy +# Optional extra Helm values; the common case infers GPU profiles. +VALUES ?= status: $(PYTHON) manage.py status -smoke-test: - $(PYTHON) manage.py smoke-test - -example-single-node: - cp examples/single-node/config.yaml ./config.yaml - cp examples/single-node/models.yaml ./models.yaml - @echo "Copied single-node example into repo root. Edit hostname/model ids as needed." +render: + $(PYTHON) manage.py render bootstrap-k3s: - bash scripts/bootstrap_k3s.sh + $(PYTHON) manage.py kube bootstrap --provider=k3s --apply install-kubeai: - bash scripts/install_kubeai.sh generated/kubeai/kubeai-values.yaml kubeai - -bootstrap-single-node: example-single-node bootstrap-k3s - @echo "Single-node example copied and K3s bootstrapped. Next: python manage.py render && python manage.py deploy" + $(PYTHON) manage.py kube install --apply $(if $(VALUES),--values "$(VALUES)",) port-forward-kubeai: kubectl -n kubeai port-forward svc/kubeai 8000:80 diff --git a/README.md b/README.md index 41a2877a..8d5caab3 100644 --- a/README.md +++ b/README.md @@ -4,14 +4,10 @@ [![Python versions](https://img.shields.io/pypi/pyversions/infer-stack.svg)](https://pypi.org/project/infer-stack/) [![License](https://img.shields.io/pypi/l/infer-stack.svg)](https://github.com/AIQ-Kitware/infer_stack/blob/main/LICENSE) -> **Heads up — the leasing model is now the primary workflow.** Declare models -> in a catalog (`infer-stack catalog …`) and `acquire`/`run` endpoints -> on demand; see `docs/source/manual/` (the Ollama + Open WebUI tutorial and the -> leasing demo) and `infer-stack help tree`. The -> **named stack profiles** documented below (`setup`/`render`/`up`/`switch`/…) -> are the pre-leasing model and now live under **`infer-stack legacy `** -> (e.g. `infer-stack legacy render`). This README still describes that legacy -> flow; a leasing-oriented rewrite is pending. +Declare models in a catalog (`infer-stack catalog …`) and `acquire` or `run` +endpoints on demand. `infer-stack help tree` prints the whole command surface; +[docs/source/manual/](docs/source/manual/) has the Ollama + Open WebUI +tutorial and the leasing demo. ## Primary leasing workflow @@ -23,7 +19,12 @@ infer-stack catalog suggest --apply infer-stack acquire ``` -`config.yaml` and `catalog.yaml` are the user configuration. Leasing keeps an +No GPU here? `infer-stack catalog suggest --simulator --apply` adds +`mock-smol`, a simulator that answers like vLLM with random text, so the same +three steps run on any host with Docker ([docs/mock-endpoints.md](docs/mock-endpoints.md)). + +`settings.yaml` (`infer-stack config …`) and `catalog.yaml` are the user +configuration. Leasing keeps an internal frozen recovery snapshot so a crash cannot re-render committed state with different settings, but ordinary `acquire` advances that snapshot automatically. Compatible catalog additions can be acquired while other models @@ -35,22 +36,21 @@ step. See [ADR 0001](docs/adr/0001-user-config-is-authoritative.md). ## Related work - [HyperQwen](https://github.com/syv-ai/HyperQwen) is a specialized model-preparation - and serving stack for running Qwen3.8-27B efficiently on 24-GiB consumer GPUs, - with its published tuning and measurements centered on the RTX 3090. - infer-stack's suggestion for it delegates the requantization, patched-vLLM - launcher, speculative decoding, and GPU-level tuning to HyperQwen's image, - described entirely in catalog data (see "Images with their own launcher" - below); infer-stack adds hardware discovery, catalog suggestions, - exact GPU affinity, lease lifecycle, and routing around that serving stack. - `catalog suggest` offers its fast endpoint on any Ampere-or-newer GPU with - 24 GiB; on a detected RTX 3090 it also offers explicit `-long` (150K) and - `-huge` (245,760) endpoint variants from HyperQwen's measured single-user - profiles. It has also been measured unchanged on an RTX PRO 6000 Blackwell. - The base suggestion is - named `qwen3.8-27b-dbirks-hyperqwen` because it starts from the + and serving stack for Qwen3.8-27B. infer-stack delegates requantization, + patched-vLLM launchers, speculative decoding, and GPU-level tuning to its + image, while hardware discovery and suggestion choose explicit catalog + profiles by GPU *class* (compute capability + VRAM) rather than product-name + strings. Memory-tight 24-32 GiB Ampere-or-newer cards get the reference + fast/long/huge choices. High-VRAM Turing cards (sm75-sm79, >=48 GiB) get the + measured full-262K FP16/Triton profiles, with the prepared `-fast` checkpoint + preferred. Roomy Ampere-or-newer cards (>=48 GiB) get the fast baseline plus + a conservative/provisional full-262K ordinary-KV prefab; use + `dev/profile_qwen38_hyperqwen.sh` to refine that class on new hardware such as + Blackwell. Exact GPU-name gates remain available for future exceptions backed + by model-specific measurements. The model identity is + `qwen3.8-27b-dbirks-hyperqwen` because it starts from the `dbirks/Qwen3.8-27B-W4A16-AutoRound` derivative; the unsuffixed - `qwen3.8-27b` identity is left available for the official - `Qwen/Qwen3.8-27B` checkpoint. + `qwen3.8-27b` identity is left available for the official checkpoint. ## Supported platform @@ -62,49 +62,51 @@ Operational and security constraints that are accepted during the current planning-stage release are tracked in [docs/planning/known-limitations.md](docs/planning/known-limitations.md). -`infer_stack` manages **named stack profiles** for local and Kubernetes-backed inference. - -A stack profile is a small graph made from: - -* **providers** — inference runtimes such as vLLM and Ollama -* **gateways** — optional API routers such as LiteLLM -* **frontends** — optional UIs such as Open WebUI -* **routes** — optional public model aliases exposed through a gateway - -This repo can render those profiles through two backends: +`infer-stack` serves the endpoints declared in a catalog. `acquire` takes a +lease on an endpoint; the controller places its engine on free GPUs and +reconciles the backend to run it: -* **Compose** for local single-host serving. Compose supports vLLM, Ollama, optional LiteLLM, and optional Open WebUI. -* **KubeAI** for Kubernetes-backed vLLM serving. KubeAI support is vLLM-only for now. +* **engines**: vLLM (one container per deployment) and Ollama (one daemon per + `runtime_hosts` entry, serving many tags); +* **LiteLLM gateway**: one OpenAI base URL, `http://127.0.0.1:14042/v1`, in + front of every endpoint alias. On by default; `config set litellm false` + drops it; +* **Open WebUI**: on by default at `http://127.0.0.1:13000`; + `config set ui false` or `acquire --no-ui` drops it; +* **reverse proxy**: an optional single-port nginx in front of both. -The direct Ollama path can run without LiteLLM and without predeclaring models. vLLM profiles still use explicit runtimes, placement, and runtime settings. +Two backends run this: **Compose** (single host; vLLM and Ollama) and +**KubeAI** (a Kubernetes cluster; vLLM only). ## Main commands ```bash -infer-stack setup --backend compose --profile ollama-direct -# or: infer-stack setup --backend compose --profile qwen2-5-7b-instruct-turbo-default -infer-stack list-profiles -infer-stack describe-profile -infer-stack validate -infer-stack render -infer-stack up -d -infer-stack deploy -infer-stack switch --apply # re-render and converge; no separate up needed -infer-stack status -infer-stack smoke-test -infer-stack version # print the installed version -infer-stack config paths # show where config / artifacts / caches live +infer-stack config init # data dir + default backend -> settings.yaml +infer-stack catalog suggest --apply # seed catalog.yaml from this host's GPUs +infer-stack catalog show # what can be acquired +infer-stack kube inventory # Kubernetes/GPU/KubeAI facts, before or after install +infer-stack kube doctor # detailed readiness + actionable fixes +infer-stack kube node test NODE --expected-gpus=1 # plan worker GPU + real generation acceptance +infer-stack kube node detach NODE # preview handing a cluster GPU host to Compose +infer-stack acquire # lease, render, bring up, wait for a real generation +infer-stack access --env-file e.env # reach it, managed or external; leases only what runs here +infer-stack test # one generation through the gateway +infer-stack leases # desired vs running, per deployment +infer-stack status # paths, backend and a lease summary +infer-stack release --all # drop every lease +infer-stack paths # where settings, catalog, ledger and caches live +infer-stack version +infer-stack help tree # the whole command surface ``` -The CLI is built on [`scriptconfig`](https://gitlab.kitware.com/utils/scriptconfig), -so every subcommand is also importable as a Python class — useful for -notebooks, tests, and other scripts: +The CLI is built on [`kwconf`](https://github.com/Erotemic/kwconf), +so every subcommand is also importable as a Python class: ```python -from infer_stack.cli import RenderCLI, SmokeTestCLI +from infer_stack.cli import AcquireCLI, TestCLI -RenderCLI.main(argv=False, profile="qwen2-5-7b-instruct-turbo-default", yes=True) -SmokeTestCLI.main(argv=False, model="qwen/qwen2.5-7b-instruct-turbo") +AcquireCLI.main(argv=False, names=['smol135-1'], yes=True) +TestCLI.main(argv=False, name='smol135-1') ``` `manage.py` and `infer-stack` are aliases for the same entry point; @@ -112,247 +114,235 @@ shell examples below use `infer-stack`. ## Operating the rendered Compose stack -Once the stack is up, common docker compose operations are available as -`infer-stack` subcommands so you don't have to `cd` into the rendered -output directory or repeat the `-f docker-compose.yml --env-file .env` -flags. They all resolve the rendered location via the same -`output.generated_dir` chain as the rest of the CLI. +`ps` and `logs` read the backend directly; `stack` wraps `docker compose` on +the rendered project, so you never `cd` into it or repeat `-f`/`--env-file`. ```bash -infer-stack ps # docker compose ps -infer-stack ps -a # include stopped -infer-stack logs -f open-webui # follow one service -infer-stack logs --tail=200 litellm vllm-* # tailored backlog -infer-stack logs -f --raw litellm # full LiteLLM tracebacks -infer-stack restart open-webui # restart specific services -infer-stack stop # stop everything (no remove) -infer-stack start # start back up -infer-stack pull # refresh images +infer-stack ps # engines, gateway, UI +infer-stack ps -a # include exited instances +infer-stack logs -f # follow whatever serves an endpoint +infer-stack logs --tail 200 litellm # the gateway's backlog +infer-stack logs -f --raw litellm # full LiteLLM tracebacks +infer-stack stack restart open-webui # docker compose restart +infer-stack stack stop # stop everything (no remove) +infer-stack stack start # start it back up +infer-stack stack pull # refresh images +infer-stack stack compose -- ps --format json # any other docker compose command ``` -Interactive ``infer-stack logs -f`` compacts only explicitly registered, known-noisy -LiteLLM traceback shapes; unknown tracebacks pass through unchanged. Redirected or -piped output stays raw, and ``--raw`` disables compaction in an interactive follow. -The compacted CLI path preserves Compose ANSI service colors when attached to a TTY; -``--no-color`` still disables them. The TUI uses the same conservative compactor when -LiteLLM logs are visible. +`logs` accepts a service or pod name, a container id prefix, a deployment id +or an endpoint alias. Interactive `infer-stack logs -f` compacts only +explicitly registered, known-noisy LiteLLM traceback shapes; unknown +tracebacks pass through unchanged. Redirected or piped output stays raw, and +`--raw` disables compaction in an interactive follow. `--no-color` drops the +name-prefix colors. The TUI uses the same compactor. -For Ollama model management inside the rendered Ollama service, prefer the -CLI wrappers: +Ollama tags are pulled into the daemon on the first `acquire` of an endpoint +that serves them. `stack compose` reaches the daemon for anything else, e.g. +`infer-stack stack compose -- exec ollama-local-ollama ollama list`. + +On the KubeAI backend `ps` and `logs` read pods, and the `stack` Compose verbs +act on the gateway's Compose project on this host. + +## Inspect an endpoint before running it ```bash -infer-stack ollama-pull smollm2:135m -infer-stack ollama-list -infer-stack ollama-ps +infer-stack catalog show +infer-stack acquire --no-apply # declare + write the compose project; start nothing +infer-stack paths leasing # where docker-compose.yml landed +infer-stack apply # start it (or `release --all` to discard) ``` -For other interactive one-shot commands inside a container, use -`infer-stack logs`, `infer-stack ps`, `infer-stack restart`, or fall back to raw -Compose only when no wrapper exists. +## Catalog model -On the KubeAI backend these wrappers raise ``NotImplementedError`` — -use the equivalent ``kubectl`` commands in the meantime. +`catalog.yaml` is the one user-edited description of what can run. Its +sections: -## Inspect a profile before running it +* `models`: weight sources (`hf://org/name`, with optional `revision`, + `quantization`, `dtype`); +* `endpoints`: served API names. Each picks an `engine` (`vllm` or `ollama`), + a `model` (a `models` key, or an Ollama tag), and optionally `runtime` + (vLLM settings), `protocol`, `placement`, `sharing` and `reclaim`; +* `runtime_hosts`: Ollama daemons, each with its GPUs and daemon settings; +* `bundles`: named lists of endpoints to acquire together. + +An endpoint can instead name a server that already runs elsewhere, with +`external: {api_base, model, api_key_env}` in place of `engine`/`model`. +It has no lease: `infer-stack access` (or `run`) publishes its route on the +LiteLLM front door, and the client asks for the alias through the same base +URL as any other endpoint. Moving an alias between a managed runtime and an +external server changes neither the alias nor the workflow: ```bash -infer-stack describe-profile qwen2-5-7b-instruct-turbo-default --format yaml +infer-stack env REMOTE_QWEN_KEY=sk-... # the upstream's key, by name +infer-stack access qwen-remote --env-file qwen.env +source qwen.env # OPENAI_BASE_URL, OPENAI_API_KEY, ... ``` -## Stack profile model +`api_base` is resolved from inside the gateway's container or pod: +`localhost` there is the gateway itself, so a server on the same host is +reached through the Docker bridge address (e.g. `http://172.17.0.1:8000/v1`). +See [docs/planning/external-endpoints.md](docs/planning/external-endpoints.md). -Profiles are written as stack graphs. The main sections are `providers`, `gateways`, `frontends`, and `routes`. For details and examples, see [docs/stack-graph-profiles.md](docs/stack-graph-profiles.md). +```yaml +models: + smol135: + source: hf://HuggingFaceTB/SmolLM2-135M-Instruct -Common shapes: +endpoints: + smol135-1: + engine: vllm + model: smol135 + runtime: {max_model_len: 8192} + chat: + engine: ollama + host: local-ollama + model: qwen3.5:4b -```text -Open WebUI -> Ollama # ollama-direct, no LiteLLM -Open WebUI -> LiteLLM -> vLLM # classic vLLM compose profiles -Open WebUI -> LiteLLM -> Ollama # Ollama with stable aliases -Open WebUI -> LiteLLM -> Ollama + vLLM # mixed migration / test stacks -Ollama API + vLLM API directly # raw backend profiles +runtime_hosts: + local-ollama: + engine: ollama + placement: {gpu_indices: [0]} + settings: {keep_alive: 30m} + +bundles: + both: [smol135-1, chat] ``` -Custom provider models and custom profiles live in the configured `catalog.user_models_file`, which defaults to `~/.config/infer_stack/models.yaml`. New files should prefer provider-specific top-level keys: +Edit it with `infer-stack catalog model|endpoint|host|bundle add …`, or by +hand with `infer-stack catalog edit`; `infer-stack catalog validate` checks it. +The schema reference is the docstring of `infer_stack/leasing/catalog.py`. -```yaml -vllm_models: - my-vllm-model: - hf_model_id: org/model - -ollama_models: - my-ollama-model: - tag: qwen3.5:4b - -profiles: - my-stack: - providers: {} - gateways: {} - frontends: {} - routes: {} +Shapes the stack renders: + +```text +Open WebUI -> LiteLLM -> vLLM / Ollama # the default + \-> an external OpenAI-compatible server (external:) +Open WebUI -> vLLM / Ollama # config set litellm false ``` -`models:` is still interpreted as a vLLM model catalog for convenience, but new docs and recipes use `vllm_models:` / `ollama_models:`. +The named stack profiles of earlier releases (`setup`, `switch`, +`--profile`) are gone; see +[docs/stack-graph-profiles.md](docs/stack-graph-profiles.md). ## Where config and rendered artifacts live -`infer-stack` follows XDG basedir conventions, so where you invoke it -from never changes which config it reads or where it writes rendered -artifacts: - -There are exactly two path roots: +`infer-stack` follows XDG basedir conventions, so the directory you invoke it +from never changes which config it reads or where it writes. There are two +roots: | What | Default location | How to relocate | | --- | --- | --- | -| `config.yaml`, `models.yaml`, `kubeai-values.local.yaml` | `~/.config/infer_stack/` (resp. `$XDG_CONFIG_HOME`) | `--config-dir` (or `INFER_STACK_CONFIG_DIR`) | -| **Everything generated** — `generated/` (docker-compose.yml, .env, plan.yaml, kubeai/*) **and** `state/` (hf-cache, postgres volumes, Ollama store, runtime bind mounts) | `~/.local/share/infer_stack/` (resp. `$XDG_DATA_HOME`) | `--data-dir` (or `INFER_STACK_DATA_DIR`) | +| `settings.yaml`, `catalog.yaml` | `~/.config/infer_stack/` (resp. `$XDG_CONFIG_HOME`) | `--config-dir` or `INFER_STACK_CONFIG_DIR` | +| **Everything generated**: `leasing/` (the ledger, the compose project, its `.env`) and the bind-mounted state (`hf-cache/`, `vllm-cache/`, `open-webui/`, `ollama/`, …) | `~/.local/share/infer_stack/` (resp. `$XDG_DATA_HOME`) | `config set data_dir `, `--data-dir` or `INFER_STACK_DATA_DIR` | + +`infer-stack paths` prints every resolved path and whether it exists. -`--data-dir` is the single knob for "put everything I generate in one -directory." It **relocates the one infer-stack installation controlling a +The data dir **relocates the one infer-stack installation controlling a host/backend; it does not create an isolated second installation**. Do not run controllers from multiple config/data roots against the same Docker host or Kubernetes namespace. See the [single-owner limitation](docs/planning/known-limitations.md#one-control-plane-per-host-or-backend-namespace). -Set it once at `setup`; it is baked into the absolute -`state.*` and `output.generated_dir` paths written to `config.yaml`, so -later commands don't need it again: - ```bash -# All rendered artifacts and bind-mount state land under one directory. -infer-stack setup \ - --backend compose \ - --profile ollama-direct \ - --data-dir /data/service/docker/vllm-stack - -infer-stack render --yes -``` +# Persist it once; later commands read it from settings.yaml. +infer-stack config set data_dir /data/service/docker/infer-stack -```bash -# Keep config.yaml in a checkout for ad-hoc experiments. -infer-stack setup --config-dir $PWD --backend compose --profile

--data-dir $PWD/stack +# Or keep both roots in a checkout for an ad-hoc experiment. +export INFER_STACK_CONFIG_DIR=$PWD/cfg INFER_STACK_DATA_DIR=$PWD/stack +infer-stack paths ``` -`--config-dir` / `--data-dir` live on every subcommand, so they appear -**after** the subcommand name. For "set once for the whole shell" use the -env vars instead. For a bespoke split layout (e.g. big `state/` on a data -disk, artifacts elsewhere), edit `state.*` / `output.generated_dir` in -`config.yaml` directly. +`--config-dir` / `--data-dir` are accepted by every subcommand, after the +subcommand name. ## Constraining placement to specific GPUs -If some of your GPUs are tied up by other work, restrict the planner -(and the rendered ``device_ids``) to the subset you want it to use: - ```bash # Only place onto GPU 1 (e.g. GPU 0 is running a display). -infer-stack render --yes --profile test-single-11gb --allowed-gpus 1 +infer-stack acquire --allowed-gpus 1 -# Or pin a TP=2 profile to physical GPUs 1 and 3. -infer-stack render --yes --profile test-multi-gpu --allowed-gpus 1,3 +# Or confine a TP=2 endpoint to physical GPUs 1 and 3. +infer-stack acquire --allowed-gpus 1,3 ``` -``--allowed-gpus`` (or ``INFER_STACK_ALLOWED_GPUS=1,3``) filters the -detected inventory before placement — real indices are preserved, so -the rendered compose stack pins ``device_ids: ["1", "3"]`` to those -exact physical GPUs. Useful for integration tests that need to share a -host with other jobs. - -## Demos / integration recipes +`--allowed-gpus` (or `INFER_STACK_ALLOWED_GPUS=1,3`) filters the detected +inventory before placement for that call only. Real indices are preserved, so +the rendered compose stack pins `device_ids` to those exact GPUs. The durable +forms live in data: -End-to-end examples under [docs/demos/](docs/demos/) are written as -markdown tutorials. The CI smoke test is runnable with pytest-codeblocks: +* `placement: {gpu_indices: [1]}` on a vLLM endpoint pins it exactly (the list + length must equal tp×pp×dp); Ollama daemons pin through their + `runtime_hosts` entry; +* `placement: {min_vram_gib: 24}` makes smaller GPUs ineligible; +* `config set skip_display_gpus true` (or `--skip-display-gpus`) leaves the + GPU driving a monitor free. -```bash -pytest --codeblocks docs/demos/ci_smoke_test.md -``` - -Each ``bash`` block is a self-contained shell snippet you can also -copy-paste into a terminal. See -[docs/demos/ci_smoke_test.md](docs/demos/ci_smoke_test.md) for the -``setup → describe → validate → render`` flow on the smallest test -profiles. - -For a real running vLLM stack on a workstation, see -[docs/demos/quickstart.md](docs/demos/quickstart.md). For direct Ollama on a dual GTX 1080 Ti style host, see -[docs/demos/ollama_direct_quickstart.md](docs/demos/ollama_direct_quickstart.md). For a focused GPU-1 backend switch test, see [docs/demos/smollm2_gpu1_backend_switch.md](docs/demos/smollm2_gpu1_backend_switch.md). +## Demos / integration recipes -User-supplied paths on the CLI (`--file`, `--from-file`, -`--resource-profiles-file`, `--output-dir`) still resolve against the -current working directory — they're meant to behave as typed. +The user manual under [docs/source/manual/](docs/source/manual/) has two +walkthroughs on the current CLI: +[the Ollama + Open WebUI tutorial](docs/source/manual/ollama-openwebui-tutorial.md) +and [the leasing demo](docs/source/manual/leasing-demo.md) (standing service, +Open WebUI, several models side by side). --- ## Backend 1: Compose -Use Compose for local single-host deployments. It can render direct Ollama stacks, vLLM stacks, mixed Ollama+vLLM stacks, and raw backend-only stacks. +Use Compose for single-host serving. It runs vLLM and Ollama engines, mixed +freely, behind the optional gateway and UI. ### Getting started -Prerequisite: Docker and the `docker compose` plugin must be installed. +Prerequisite: Docker and the `docker compose` plugin. ```bash -# Direct Ollama, no LiteLLM and no predeclared models. -infer-stack setup --backend compose --profile ollama-direct -infer-stack validate --simulate-hardware 2x11 -infer-stack render --yes --simulate-hardware 2x11 -infer-stack up -d - -# Classic vLLM through LiteLLM/Open WebUI. -infer-stack setup --backend compose --profile qwen2-5-7b-instruct-turbo-default -infer-stack validate -infer-stack render -infer-stack up -d -``` - -### Test that it is responding +infer-stack config init --backend compose +infer-stack doctor --gpu +infer-stack catalog init -When LiteLLM is enabled, the default Compose front door is: +# A vLLM endpoint. +infer-stack catalog model add smol135 --source hf://HuggingFaceTB/SmolLM2-135M-Instruct +infer-stack catalog endpoint add --model smol135 # -> smol135-1 +infer-stack acquire smol135-1 -```text -http://127.0.0.1:14042/v1 +# An Ollama endpoint, on a daemon pinned to GPU 0. +infer-stack catalog host add local-ollama --engine ollama --gpu 0 +infer-stack catalog endpoint add chat --engine ollama --host local-ollama --model qwen3.5:4b +infer-stack acquire chat ``` -When using `ollama-direct`, Open WebUI talks to Ollama directly and the Ollama API is available at: - -```text -http://127.0.0.1:11434 -http://127.0.0.1:11434/v1 -``` +`infer-stack catalog suggest --apply` fills the catalog with endpoints sized +for the detected GPUs instead. -unless you changed the relevant ports in config. +### Test that it is responding -Wait until the active profile can serve a real request through its resolved default endpoint: +With LiteLLM enabled, every endpoint is reachable by its alias at: -```bash -infer-stack wait-ready +```text +http://127.0.0.1:14042/v1 ``` -`wait-ready` is stronger than Docker Compose health: it probes the user-facing -LiteLLM, Ollama, or direct vLLM access surface and, by default, requires a tiny -generation/completion to succeed. The smoke test runs this readiness probe by -default before issuing its normal test request: +`acquire` already blocks until the endpoint returns a real generation through +that front door, which is stronger than Docker's container health. After +`acquire --no-wait`, block separately; to check again later, send one request: ```bash -infer-stack smoke-test +infer-stack wait smol135-1 +infer-stack test smol135-1 +infer-stack test chat --prompt "Name three colors." --max-tokens 32 ``` -For direct Ollama profiles, pull a model first and then smoke-test that model: +Clients read the front door and the managed key from the env file: ```bash -infer-stack ollama-pull qwen3.5:4b -infer-stack ollama-list -infer-stack smoke-test --model qwen3.5:4b +export OPENAI_BASE_URL=$(infer-stack env OPENAI_BASE_URL) +export OPENAI_API_KEY=$(infer-stack env LITELLM_MASTER_KEY) ``` -For LiteLLM profiles, `smoke-test` reads the rendered `.env` automatically and -uses the active profile's resolved OpenAI-compatible front door. You can inspect -individual secrets when needed: - -```bash -infer-stack env LITELLM_MASTER_KEY -infer-stack env VLLM_BACKEND_API_KEY -``` +or get both, plus per-endpoint names, from +`infer-stack acquire --env-file lease.env`. To replace the gateway's master key (refused while leases are active; the gateway restarts, and clients must fetch the key again): @@ -361,244 +351,111 @@ gateway restarts, and clients must fetch the key again): infer-stack secrets rotate ``` -When you intentionally want the old quick behavior, skip the readiness wait: - -```bash -infer-stack smoke-test --no-wait --model gpt2 -``` - ### Stop it ```bash -infer-stack down +infer-stack release --all # drop every lease +infer-stack release --all --evict # ...and stop the engines now +infer-stack clean -f # no leases, nothing on a GPU; the gateway stays +infer-stack stack down # docker compose down, bypassing the ledger ``` -`down` never removes named volumes. The Postgres data directory and the -Open WebUI volume are preserved across `down`, `up`, `switch`, and `render`. - -### Open WebUI authentication +After a plain `release`, `keep-warm` endpoints (the default `reclaim` policy) +stay loaded until another lease needs their GPUs or `infer-stack evict` stops +them. `stack down` releases no lease, so the next `apply` or `acquire` brings +leased models back. -By default Open WebUI runs with `WEBUI_AUTH=False` — no login screen, -anyone who can reach the port gets straight into the UI. This is the -expected behavior for a local dev box. To re-enable login/signup, set -in `config.yaml`: +All state is bind-mounted from the data dir, so none of these delete it, +including `stack down --volumes`. For a destructive reset, remove the +directories `infer-stack paths` lists. -```yaml -open_webui: - auth: true -``` +### Open WebUI authentication -and re-render. Existing accounts stored in the `postgres-open-webui` -volume are preserved across the toggle. +Open WebUI runs with `WEBUI_AUTH=False`: no login screen, and anyone who can +reach port 13000 gets the UI. No setting changes that. Keep the host on a +trusted network, or run without the UI (`config set ui false`). -### Reverse proxy (TLS) and LDAP +### Reverse proxy -Open WebUI can be fronted by an opt-in nginx TLS reverse proxy, and its -login can be backed by an LDAP directory. Both are off by default and -configured as ordinary config fields. The built-in `openwebui-tls-ldap` -profile wires them together as a worked example (Ollama + Open WebUI -behind nginx, no public Open WebUI/Ollama ports); see -[`examples/openwebui-tls-ldap/`](examples/openwebui-tls-ldap/). +An optional nginx service publishes one HTTP port with the UI at `/` and the +API at `/v1`. It needs the LiteLLM gateway. ```bash -infer-stack setup --backend compose --profile openwebui-tls-ldap -``` - -**Reverse proxy.** Enable it under `frontends.reverse_proxy`. It renders -an nginx service plus a generated `state.runtime/nginx.conf`: - -```yaml -frontends: - reverse_proxy: - enabled: true - target: open_webui # or litellm / ollama / a custom upstream - server_name: host.example.com - ssl: - enabled: true - certificate: ./certs/site.crt - certificate_key: ./certs/site.key - dhparam: ./dhparam.pem # optional +infer-stack config set reverse_proxy true # port 80 +infer-stack config set reverse_proxy '{enabled: true, port: 8080}' ``` -When `ssl.enabled` is true, port 80 redirects to HTTPS (`force_https`) -and the cert/key/dhparam host paths are bind-mounted read-only. When -`ssl.enabled` is false, only HTTP is published (HTTPS publishing is -gated on TLS so you never get a `:443` mapping with nothing listening). - -> **Path caveat.** Relative `certificate`/`certificate_key`/`dhparam`/ -> `config_path` values are written verbatim into the generated -> `docker-compose.yml`, so Docker Compose resolves them **relative to the -> generated directory** (where the compose file lives), not your CWD. -> `infer-stack render` warns when a referenced cert or config file is not -> found. Use absolute paths if you want to avoid the ambiguity. - -**LDAP.** Enable it under `frontends.open_webui.ldap`. The directory -settings render as Open WebUI `LDAP_*` environment variables, and -secrets/site-specific values are emitted as `.env` placeholders -(`LDAP_HOST`, `LDAP_PASSWD`, `LDAP_SEARCH_BASE`, …) so you can fill them -in after the first render without re-touching the compose YAML: - -```yaml -frontends: - open_webui: - ldap: - enabled: true - env_defaults: - LDAP_PORT: '636' - LDAP_USE_TLS: 'true' - LDAP_ATTRIBUTE_FOR_USERNAME: uid -``` +`acquire --reverse-proxy` turns it on for one call. Add `config_path: +/path/to/nginx.conf` to the block to mount your own config at +`/etc/nginx/conf.d/default.conf` instead of the generated one. -**Manual escape hatches.** When the typed renderer is not enough, drop -to manual control without leaving infer-stack: - -* `frontends.reverse_proxy.config_path` — mount an existing nginx config - file instead of rendering one. -* `frontends.reverse_proxy.extra_config` — inject extra directives into - the rendered HTTPS `server` block. -* Every rendered service (`ollama`, vLLM runtimes, `litellm`, - `open_webui`, `reverse_proxy`) accepts generic overrides: - `extra_env`, `env_file`, `extra_volumes`, `extra_hosts`, `labels`, - `additional_ports`, and `gpus` (scalar `all`/count or a structured - device-request list). - -Field precedence (lowest to highest) is: top-level config section -(`reverse_proxy:` / `open_webui:` / `ollama:`) → the matching -`frontends.*` / `providers.*` / `gateways.*` section → the active -profile. Newer configs should prefer the `frontends.*` / `providers.*` -form shown above. +It does no TLS and no authentication. Only the one port is published and no +certificate is mounted, so terminate TLS in a proxy in front of it. The TLS +and LDAP settings of the pre-leasing profiles no longer exist. ### Persistent state and database layout -Compose renders stateful services only when their components are enabled: - -* `postgres-open-webui` — rendered only when Open WebUI is enabled. It stores chats, accounts, and settings in `state.postgres_open_webui`. -* `postgres-litellm` — rendered only when LiteLLM is enabled. It stores router state in `state.postgres_litellm`. -* `ollama` — rendered only when the Ollama provider is enabled. Its model store is `state.ollama`, mounted at `/root/.ollama`. -* vLLM runtimes mount `state.hf_cache` for Hugging Face weights and `state.vllm_cache` for compiled artifacts. - -Each Postgres container has its own `POSTGRES_DB`, `POSTGRES_USER`, and -`POSTGRES_PASSWORD`, sourced from component-specific `.env` keys. There is no shared Postgres instance and no `postgres-init` bootstrap service. - -Open WebUI chat history is **not** tied to the model currently being served, so after a profile switch old chats may reference model IDs the current gateway no longer advertises — that is expected. +Everything lives under the data dir (`infer-stack paths`): -### Operational tips +* `open-webui/`: Open WebUI's data directory (accounts, chats, settings), + mounted at `/app/backend/data`; +* `postgres-litellm/`: LiteLLM's route store, rendered only with + `config set dynamic_routing true`; +* `ollama/`: the Ollama model store, mounted at `/root/.ollama`; +* `hf-cache/`, `vllm-cache/`, `torch-cache/`, `triton-cache/`, `cuda-cache/`: + vLLM weights and compile caches (see + [docs/persistent-caches-and-warm-restarts.md](docs/persistent-caches-and-warm-restarts.md)); +* `runtime/`: directories an endpoint's `runtime.mounts` asks for; +* `leasing/`: the ledger and the rendered compose project. -Prefer scoping commands to specific services rather than relying on -container names. Use only the services rendered by the active profile: - -```bash -# LiteLLM gateway profile -infer-stack logs -f litellm - -# Direct Ollama profile -infer-stack logs -f ollama - -# Ollama model store helpers -infer-stack ollama-list -infer-stack ollama-ps -``` - -You do not need to delete any volume during normal operation. If you -ever want a destructive reset, do it explicitly with -`docker compose down -v` against `generated/docker-compose.yml` — the -toolchain itself never does this. +Open WebUI chat history is not tied to the models currently served, so old +chats may name aliases the gateway no longer advertises. That is expected. ### Custom .env values are preserved -`generated/.env` is rewritten non-destructively. Any `KEY=value` pair -you add manually (for example `VERBOSE=1`, `HF_HOME=/data/hf`, or any -key this program does not yet know about) is preserved across -`render`, `setup`, `switch`, `up`, and `deploy`. Comments and the order -of existing lines are preserved where practical. +The compose project's `.env` holds the managed secrets (`LITELLM_MASTER_KEY`, +`HF_TOKEN`, …). `infer-stack env KEY=VALUE` merges a value into it, and keys +you add are kept across renders. Compose uses the file for interpolation, so a +key reaches a container only when the rendered service references it (as +`HF_TOKEN` does for vLLM). Set `HF_TOKEN` before the first `acquire` of a +gated model. -### Switching profiles +### Switching models + +There is no single active model to switch. `acquire` another endpoint and it +runs beside the first; release the first when you are done: ```bash -infer-stack switch --apply +infer-stack acquire smol135-1 +infer-stack acquire chat # now both are served +infer-stack leases # find smol135-1's lease id +infer-stack release ``` -`switch --apply` re-renders from the updated `config.yaml`, then brings the -stack up convergently with `--remove-orphans` so a separate `infer-stack up` is -not needed. Components/runtimes that are no longer in the rendered compose file -are dropped. Compose preserves existing containers whose service definitions did -not change. For vLLM-to-vLLM profile switches, unchanged Open WebUI stays up; -LiteLLM is refreshed through its admin API when possible. The live refresh -path treats LiteLLM's "model not found in db" response for config-backed -models as non-fatal, so switching aliases can add the new route without -tearing LiteLLM down. That can temporarily leave stale config-backed aliases -in `/v1/models`; restart LiteLLM manually only when you want to clean those -up. If Compose already created or recreated LiteLLM while converging the new -stack, no extra router refresh is attempted because the new container has -already loaded the freshly rendered YAML. Profiles that do not render -LiteLLM, such as direct Ollama profiles, skip the router refresh path even if -an old `runtime/litellm_config.yaml` file remains from a previous profile. -Switches that change Open WebUI's provider wiring, such as -`Open WebUI -> LiteLLM` to `Open WebUI -> Ollama`, necessarily recreate -Open WebUI because its environment changes. Postgres volumes and provider -caches are left untouched. vLLM runtime containers are named after their -Compose service, for example `vllm-chat`, so `docker ps` and -`infer-stack logs vllm-chat` clearly identify them as vLLM containers. +The gateway carries a route for every catalog endpoint, so adding or removing +a model does not recreate LiteLLM, and Open WebUI stays up. When GPUs are +short, `acquire` fails fast; `--queue` waits for a GPU instead, and idle +`keep-warm` deployments are evicted to make room. vLLM containers are named +after the served alias (`vllm-`), so `docker ps` and +`infer-stack logs vllm-` identify them. ### Protocol modes for base vs. instruct models -Profiles declare a `protocol_mode` (`chat` or `completions`) that the -served model must support. Models also declare which protocols they -support via `supported_protocols`. Validation runs before render and -fails with an actionable message if a profile asks for `chat` on a -completions-only model. - -Practical guidance: - -* Instruct/chat models (with a chat template) can use either, but - default to `chat`. -* Base models like Pythia, Llama-2 base, Mistral-v0.1 base, and Falcon - base do not define a chat template. Their HELM profiles use - `protocol_mode: completions` and the `smoke-test` command will - exercise `/v1/completions` for them. -* The rendered LiteLLM config uses `text-completion-openai/` - as the upstream provider for completions-only services. That means - even chat-shaped requests sent through Open WebUI to a Pythia model - get translated by LiteLLM into upstream `/v1/completions` calls — no - second vLLM container is needed to support Open WebUI for a - completions-only model. -* Open WebUI is still a chat UI, so prompt formatting matters. - HELM/eval clients should call `/v1/completions` directly for exact - prompt control rather than going through the chat frontend. - -### Chat-shaped clients on top of completions models - -Some clients (e.g. InspectAI / Inspect Evals stock MMLU tasks) only -speak `/v1/chat/completions` and cannot be reconfigured. For those -cases, profiles can opt into a LiteLLM-only adapter: - -```yaml -chat_compat: - enabled: true - strategy: flat_messages -``` - -When set on a `protocol_mode: completions` service, the rendered -LiteLLM config keeps the `text-completion-openai/` upstream -and adds LiteLLM's documented prompt-template fields -(`initial_prompt_value` / `roles` / `final_prompt_value`) so chat -messages get flattened into a plain prompt — no role labels, messages -joined by `\n` — before being forwarded to vLLM `/v1/completions`. - -This is **not** a chat tune; the model is still a base model and -prompt formatting still matters for evaluation. Use it only when a -chat-shaped client cannot be changed. The vLLM container is not -restarted, no `--chat-template` is rendered, and the adapter takes -effect after a `litellm`-only restart: +An endpoint's `protocol` is `chat` (the default) or `completions`. It decides +which surface the readiness probe and `infer-stack test` use, so a base model +without a chat template must declare `completions` or its `acquire` never sees +a ready generation: ```bash -infer-stack render -infer-stack restart litellm +infer-stack catalog model add pythia-160m --source hf://EleutherAI/pythia-160m +infer-stack catalog endpoint add --model pythia-160m --protocol completions +infer-stack acquire pythia-160m-1 +infer-stack test pythia-160m-1 # hits /v1/completions ``` -The built-in `pythia-inspect-mmlu-compat` profile is a ready-made -example; see -[`recipies/compose_pythia_inspect_mmlu_compat.md`](recipies/compose_pythia_inspect_mmlu_compat.md). +The gateway forwards `/v1/completions` unchanged, so evaluation clients that +need exact prompt control should call it directly. Open WebUI is a chat UI and +sends chat requests, which a base model cannot answer. ### Images with their own launcher @@ -608,34 +465,26 @@ endpoint's `runtime`; infer-stack has no model-specific code for it: ```yaml runtime: - image: ghcr.io/syv-ai/hyperqwen:sha-684e927 + image: example.org/my-vllm-launcher:1.0 max_model_len: 65536 gpu_memory_utilization: 0.93 command: [single] # replaces `vllm serve MODEL ` env: # container environment PORT: '{port}' - SPEC: dflash2 - CTX: fast - PREFIX_CACHE: 1 MAX_LEN: '{max_model_len}' # filled from the field above GPU_UTIL: '{gpu_memory_utilization}' EXTRA_ARGS: '--served-model-name={served_model_name}' mounts: # persisted under the runtime data dir - /app/models: hyperqwen/qwen3.8-27b/models - /cache: hyperqwen/qwen3.8-27b/cache + /app/models: my-launcher/models ``` -Switching that image to another context profile is a data edit, in the catalog -or the TUI's endpoint editor. HyperQwen's measured 3090 profiles are: - -- fast/default: `max_model_len: 65536`, `SPEC: dflash2`, `CTX: fast`; -- long: `max_model_len: 150000`, `SPEC: mtp`, `CTX: long`; -- huge: `max_model_len: 245760`, `SPEC: dflash2`, `CTX: huge`. - -On an RTX 3090, `catalog suggest` emits the latter two as `-long` and `-huge` -endpoint variants. The hardware check happens only while suggesting: the -resulting catalog contains ordinary explicit runtime data, so `apply`/`acquire` -never silently retunes an endpoint after the fact. +Changing the launcher's mode is a data edit, in the catalog or the TUI's +endpoint editor. The HyperQwen suggestion (see "Related work") is a worked +example: `catalog suggest` uses compute capability and VRAM to emit the serving +profiles appropriate to the detected hardware class (and exact name gates only +for measured exceptions). The hardware check happens only while suggesting; +the resulting catalog holds ordinary explicit runtime data, so `apply`/`acquire` +never retunes an endpoint after the fact. - `{max_model_len}`, `{gpu_memory_utilization}`, `{served_model_name}` and `{port}` are filled in from the endpoint, so a launcher that takes them @@ -650,46 +499,35 @@ never silently retunes an endpoint after the fact. launcher instead. - All of these are deployment identity: endpoints that launch differently never share a process. -- Compose only; KubeAI refuses an endpoint with `command`, `env` or `mounts`. +- `env` works on both backends (on KubeAI it becomes the Model's `spec.env`). + `command` and `mounts` are Compose only; KubeAI refuses an endpoint with + either rather than serve stock vLLM in its place. ### Reasoning / thinking models -Models can declare reasoning support in the catalog: +vLLM separates a reasoning trace from the answer when it is started with a +reasoning parser. Pass the flag through `runtime.extra_args`: ```yaml -reasoning: - enabled: true - parser: qwen3 - expose_to_openwebui: true +endpoints: + qwen3-think: + engine: vllm + model: qwen3-0.6b + runtime: + extra_args: [--reasoning-parser=qwen3] ``` -Profiles can override or set the same field per service. When a -service has `reasoning.enabled: true` and a `parser`, the renderer -adds `--reasoning-parser ` to that vLLM container's command -line — that flag alone enables reasoning extraction in the current -vLLM CLI. You do not need to repeat it by hand in `extra_args`. - -Open WebUI sees reasoning content via two paths: - -1. Inline `...` tags emitted by the model. -2. Structured `reasoning_content` fields when LiteLLM normalizes them. - -The LiteLLM template keeps `merge_reasoning_content_in_choices: true` -on chat-mode entries so Open WebUI can display reasoning in the -streamed response. To test reasoning end-to-end: +The parser name depends on the model family and the vLLM version (`vllm serve +--help` lists them). To test end to end: ```bash -# Non-streaming CLI smoke test: -infer-stack smoke-test \ - --model qwen3.6-35b-a3b \ - --prompt "Think step by step: 17*23" - -# For streaming inspection, read the key with the CLI wrapper: -LITELLM_MASTER_KEY=$(infer-stack env LITELLM_MASTER_KEY) -curl -N http://127.0.0.1:14042/v1/chat/completions \ - -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ +infer-stack test qwen3-think --prompt "Think step by step: 17*23" --max-tokens 512 + +# Streaming, through the gateway: +curl -N "$(infer-stack env OPENAI_BASE_URL)/chat/completions" \ + -H "Authorization: Bearer $(infer-stack env LITELLM_MASTER_KEY)" \ -H 'Content-Type: application/json' \ - -d '{"model":"qwen3.6-35b-a3b","stream":true, + -d '{"model":"qwen3-think","stream":true, "messages":[{"role":"user","content":"Think step by step: 17*23"}]}' ``` @@ -700,403 +538,142 @@ chat settings. ## Backend 2: KubeAI -Use KubeAI when you want Kubernetes-managed serving. - -### Important rules - -1. **Use the same namespace everywhere.** The namespace in `infer-stack setup --namespace ...` must match the namespace where the KubeAI Helm release already exists. -2. **Prefer the repo-driven path.** The normal path is `setup` -> `validate` -> `render` -> `deploy` -> `status`. -3. **`kubectl port-forward` stays in the foreground.** Leave it running in one terminal and send requests from another. -4. **The first request can take a while.** `/openai/v1/models` may work before chat completions work. The first completion may trigger pod creation, image pull, model load, and compile warmup. -5. **On the current repo version, KubeAI still needs a live workaround after deploy.** The renderer currently produces a `Model` spec that needs a small manual patch to work with the KubeAI version used in these notes. - -### KubeAI prerequisites - -You need: - -* a working Kubernetes cluster -* `kubectl` -* Helm - -If you want a quick local single-node cluster, K3s is a good option. - -Install K3s: - -```bash -curl -sfL https://get.k3s.io | sh - -# or pin -curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v1.34.3+k3s1' sh - -``` - -Make `kubectl` usable without `sudo`: - -```bash -sudo mkdir -p /etc/rancher/k3s/config.yaml.d -printf 'write-kubeconfig-mode: "0644"\n' | \ - sudo tee /etc/rancher/k3s/config.yaml.d/10-kubeconfig-mode.yaml >/dev/null -sudo systemctl restart k3s -kubectl get nodes -``` - -Install Helm: - -```bash -curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/83a46119086589a593a62ca544982977a60318ca/scripts/get-helm-4 -chmod 700 get_helm.sh -./get_helm.sh -helm version -``` - -### NVIDIA GPU support - -Install the NVIDIA device plugin and GPU Feature Discovery so Kubernetes can expose GPU resources and labels: - -```bash -helm repo add nvdp https://nvidia.github.io/k8s-device-plugin -helm repo update -helm upgrade -i nvdp nvdp/nvidia-device-plugin \ - --version 0.17.1 \ - --namespace nvidia-device-plugin \ - --create-namespace \ - --set gfd.enabled=true \ - --set runtimeClassName=nvidia -``` - -Check that GPU support is working: - -```bash -kubectl -n nvidia-device-plugin get pods -kubectl get node "$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" \ - -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}' -kubectl get nodes --show-labels | tr ',' '\n' | grep 'nvidia.com/' || true -``` - -You want to see a non-empty `nvidia.com/gpu` count and `nvidia.com/*` labels such as product and memory. - -### KubeAI Helm repository - -```bash -helm repo add kubeai https://www.kubeai.org -helm repo update -``` - -## Determine which namespace to use - -Before doing anything else, discover whether a `kubeai` release already exists and which namespace it uses. - -```bash -KUBEAI_NAMESPACE="$(helm list -A | awk '$1=="kubeai"{print $2; exit}')" -if [ -z "${KUBEAI_NAMESPACE}" ]; then - KUBEAI_NAMESPACE=default -fi -echo "Using KubeAI namespace: ${KUBEAI_NAMESPACE}" -``` - -If a release already exists, **reuse that namespace**. - -Sanity-check the cluster: - -```bash -kubectl get nodes -kubectl get crd models.kubeai.org || true -helm list -A | grep kubeai || true -kubectl -n "${KUBEAI_NAMESPACE}" get pods || true -kubectl get node "$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')" \ - -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}' -``` - ---- - -## Generate the local KubeAI resource-profile file - -Generate a local KubeAI resource-profile file from the labels on this machine. - -For the built-in serving profiles in this repo, keep these names aligned: - -* `gpu-single-default` -* `gpu-tp2-balanced` -* `gpu-tp2-maxctx` - -**Important:** include GPU `requests`, GPU `limits`, and `runtimeClassName: nvidia`. Without those, the model pod can land on the GPU node but still start without `libcuda.so.1` available inside the container. - -```bash -PRODUCT="$(kubectl get nodes -o jsonpath='{.items[0].metadata.labels.nvidia\.com/gpu\.product}')" -MEMORY="$(kubectl get nodes -o jsonpath='{.items[0].metadata.labels.nvidia\.com/gpu\.memory}')" - -cat > values-kubeai-local-gpu.yaml <:` - * append `:1` in the live `Model` spec -* `libcuda.so.1: cannot open shared object file` - * the pod landed on the GPU node without actually requesting a GPU; fix the resource-profile file to include GPU requests, limits, and `runtimeClassName: nvidia` -* `/models` works but completions 404 with `The model ... does not exist.` - * the served model name does not match the public profile name; apply the live args patch above -* startup probe fails with `connection refused` - * the model pod may still be pulling the image, loading the model, or warming up +`--backend kubeai` runs the same leasing verbs against a Kubernetes cluster +running [KubeAI](https://www.kubeai.org): the same catalog, ledger, TTLs, +env file and TUI, with `Model` custom resources in place of compose +services and the cluster scheduler in place of the local GPU planner. The +LiteLLM gateway still fronts everything, so a card sees one `OPENAI_BASE_URL`, +the managed key and the endpoint alias on either backend. + +* Creating/joining a cluster and the distribution boundary: + [docs/cluster-setup.md](docs/cluster-setup.md). +* KubeAI setup, settings and semantics: + [docs/kubeai-backend.md](docs/kubeai-backend.md). +* What matches Compose, what does not yet, and what is deliberate: + [docs/backend-parity.md](docs/backend-parity.md); the plan to close the + rest: [docs/planning/backend-parity-roadmap.md](docs/planning/backend-parity-roadmap.md). + +The short version: + +```bash +infer-stack kube inventory +infer-stack kube doctor +# New local cluster / NVIDIA plugin + GFD prerequisites (review before applying): +infer-stack kube bootstrap --provider=k3s +sudo -v +infer-stack kube bootstrap --provider=k3s --apply +# Automatic profiles from Kubernetes node labels; no temporary values file needed: +infer-stack kube install +infer-stack kube install --apply +infer-stack kube doctor +infer-stack catalog suggest --backend kubeai +infer-stack config set backend kubeai +infer-stack config set kubeai_gateway cluster +infer-stack doctor --backend kubeai +infer-stack acquire --ttl 2h --env-file lease.env --yes +``` + +If the workstation already has a Compose ledger/configuration that you plan to +return to, keep that authority intact rather than switching backend kinds in +the same ledger. The cluster setup guide documents the temporary test pattern: +use `INFER_STACK_BACKEND=kubeai` with a separate KubeAI data root, and use +`infer-stack kube node detach/attach` to hand each physical GPU host between +Kubernetes scheduling and direct Compose use without uninstalling or rejoining +the node. See [docs/cluster-setup.md](docs/cluster-setup.md). + +### Switching an existing recovery ledger to KubeAI + +Stopping the Compose realization retains its recovery backend and historical +ledger. Backend kinds require separate recovery epochs. After releasing every +old lease and stopping the old runtime: + +```bash +infer-stack release --all --backend compose +infer-stack stack down --backend compose +infer-stack config set backend kubeai +infer-stack status # configured kubeai / active recovery compose +infer-stack ledger rotate # verify quiescence and preview the archive +infer-stack ledger rotate --yes # archive old history; initialize KubeAI epoch +infer-stack ledger archives # archive paths + history inspection commands +infer-stack acquire --yes +``` + +Rotation refuses active leases, remaining old runtime objects, and unknown +runtime state. It does not change configuration/catalogs or silently tear down +workloads. Old leases, deployments and the recovery profile remain in SQLite +archives under `/archives/`. Rotation is transactional and +safe to retry; a matching recovery backend makes the command a no-op. `gc` +reports backend mismatches with transition instructions; `gc --forget` remains +a history-only operation. + +### Kubernetes cluster setup and KubeAI integration + +[docs/cluster-setup.md](docs/cluster-setup.md) describes server/worker setup. +`kube inventory` and `kube status` report structured facts; `--json` includes +probe errors without discarding other discoveries. Inventory and catalog +suggestions work before the KubeAI chart or namespace exists. + +`kube doctor` checks Kubernetes, NVIDIA scheduling/discovery, and KubeAI in +dependency order. Top-level `doctor --backend kubeai` remains the operational +preflight for acquiring work. If Helm installation succeeds but the configured +KubeAI API URL is unavailable, doctor reports that remaining routing issue. +Set `kubeai_base_url` to a reachable OpenAI URL for the installed service. + +`kube bootstrap` defaults to a read-only plan; `--apply` (or `--yes`) authorizes +host/cluster changes. K3s is the implemented provisioning provider; inventory, +installation and the backend work with any Kubernetes distribution. Bootstrap +always provisions/reconciles the local K3s server, preserves unrelated selected +contexts and kubeconfigs, installs Helm if missing, +and reconciles NVIDIA device plugin **0.17.1** with GPU Feature Discovery. +Install the host NVIDIA driver and container toolkit first. K3s discovers the +installed runtime on startup; bootstrap restarts local K3s only when runtime +discovery needs repair. It waits for Ready nodes, GPU allocation and GFD labels, and verifies unknown +GPU-node runtime handlers with small node-specific canary pods. Root admin +credentials stay `0600`; a private `0600` copy lives at +`~/.kube/infer-stack-k3s.yaml`. Existing default configs are preserved. Select the +local server explicitly with `export KUBECONFIG=~/.kube/infer-stack-k3s.yaml` +before inventory/install; rerun bootstrap to refresh copied certificates. + +`kube install` shows inferred Helm values; `--apply` uses Helm upgrade/install +and checks readiness afterward. `--dry-run`/`--plan` forces read-only behavior. +`--namespace`, `--release`, `--chart`, `--version` and `--values` allow overrides. +Existing named profiles and custom values are preserved; `HF_TOKEN` overrides +the chart token using a temporary protected file. Generated public values live +at `/generated/kube/kubeai-values.yaml`. + +`kube setup` is a deprecated alias of `kube install`; it no longer owns a +separate prerequisite workflow. `kube k3s bootstrap` provisions only the local +server, `join` verifies requested worker membership, and `status` reports local +membership independently of any stale selected admin context. Setup scripts are compatibility wrappers around the +package commands. + +### Debugging serving failures + +`infer-stack kube status` summarizes live Kubernetes state. `infer-stack acquire` +reports pod failures such as `ImagePullBackOff`, `Unschedulable`, and engine +crashes. Use `infer-stack ps` and `infer-stack logs -f ` to inspect +model serving. If `/models` works but completions return 404, check that clients +use the gateway alias; direct KubeAI requests use the Model name from the env +file (`INFER_STACK_ENDPOINT_*`). Missing `libcuda.so.1` usually indicates that +the model profile did not request GPUs or select the correct runtime. --- ## Which backend should I start with? -Start with **Compose** if you want: - -* the fastest path to a working local server -* easy inspection of generated files -* simple single-host iteration - -Move to **KubeAI** when you want: - -* vLLM runtimes on Kubernetes -* KubeAI’s OpenAI-compatible front door -* profile deployment through Kubernetes artifacts - -KubeAI rendering is vLLM-only for now. Profiles that enable Ollama, LiteLLM, or Open WebUI are rejected for `--backend kubeai`. - -A good workflow is: - -1. inspect a profile with `describe-profile` -2. run it with Compose when you want the simplest local deployment -3. move to KubeAI when you want Kubernetes-backed serving - -Compose is the better fit when you already know which profile you want. KubeAI has more first-request overhead because it may need to create pods, pull images, load the model, and warm up the backend. +**Compose** when one workstation is enough: it is the fastest path to a +working server, everything it renders is a file you can read, and it needs +only Docker. +**KubeAI** when the models must run on more than one machine, or a cluster +already exists. It is Compose plus a scheduler: the same catalog, verbs and +env file, with a cluster and `resourceProfiles` supplied. Expect more +first-request overhead (pod creation, image pull, model load). +A catalog written for Compose runs on KubeAI, with two exceptions: ollama +endpoints and custom container launches (`runtime.command` / `mounts`), +which stay Compose-only. [docs/backend-parity.md](docs/backend-parity.md) +has the full matrix. ## vLLM startup caches @@ -1105,39 +682,63 @@ Triton, and CUDA JIT caches. Warm starts avoid redownloading and redoing many compile/JIT steps, but a vLLM model swap still creates a new engine process and must reload weights into GPU memory. -### Diagnosing profile switches and readiness - -`docker compose` health only means that a container-level healthcheck passed. It -is not the same thing as "the routed model can answer a request through the -active access surface." This matters most when switching between two vLLM -profiles that reuse the same runtime service name: the old vLLM process exits, -Docker starts the replacement process, and LiteLLM may remain up while returning -upstream connection errors until vLLM finishes loading the new model. - -Use the dedicated readiness and diagnostics commands after a switch: - -```bash -infer-stack switch gpt2-single --apply --yes -infer-stack wait-ready --model gpt2 -infer-stack smoke-test --model gpt2 -``` - -For debugging, use: - -```bash -infer-stack diagnose --model gpt2 --generation -infer-stack diagnose --logs --tail 80 -``` - -`diagnose` prints the resolved provider/gateway/frontend graph, rendered Compose -service state, LiteLLM route probes, direct provider probes, and optional recent -logs. It is intended to distinguish an actual LiteLLM outage from the more -common case where LiteLLM is running but its upstream vLLM runtime is still -booting. - -The Compose service-state diagnostics include Docker's exit code, OOM-killed -flag, restart count, and actual container name. This is important because -`litellm exited with code 137` usually means Docker sent SIGKILL, commonly from -an OOM kill or a forced container replacement, whereas LiteLLM returning HTTP -500 with `Cannot connect to host vllm-*` means LiteLLM is still running but the -upstream vLLM runtime is not ready yet. +### Diagnosing readiness + +`docker compose` health only means a container-level healthcheck passed. It +is not "the routed model answers a request through the front door", which is +what `acquire` and `wait` check: a model swap starts a new engine process, +and LiteLLM stays up while returning upstream connection errors until vLLM +has loaded the weights. + +```bash +infer-stack acquire --yes # waits for a real generation +infer-stack wait # after acquire --no-wait +infer-stack test # one generation through the gateway +infer-stack status # desired vs running, per deployment +infer-stack logs --tail 80 # the engine or gateway log +``` + +A crash-looping engine fails the acquire at once with its error quoted, so +the timeout is only for a model that is loading. Reading Docker's own +signals: `litellm exited with code 137` is a SIGKILL (an OOM kill or a forced +replacement), whereas LiteLLM returning HTTP 500 with `Cannot connect to host +vllm-*` means LiteLLM is running and its upstream vLLM is not ready yet. + +### Join and accept GPU workers + +`infer-stack kube k3s onboard namek --server=... --token-file=... --kubeconfig=...` +plans a local GPU worker join and node-specific acceptance; add `--apply` to +execute it. Host NVIDIA drivers/toolkit are prerequisites. It infers the server +version, verifies every locally detected device, then checks actual one-GPU +KubeAI placement and generation on this worker. Admin credentials are explicit +private files, separate from any selected EKS context. + +For an already joined worker, run `infer-stack kube node test namek +--expected-gpus=1 --namespace=default --apply` from the control plane. The same +surface accepts two-device `yardrat` and four-device `aiq-gpu2`; mixed products +are reported separately. Tests preserve leases and unrelated Models, remove +their temporary resources, and retain a reusable node profile. See the +[worker onboarding instructions](docs/cluster-setup.md#join-and-test-a-gpu-worker-in-one-reviewed-operation) +for plan/apply, credentials, interrupted cleanup and heterogeneous-node limits. + +### Kubernetes dashboard + +`infer-stack tui` provides a **Cluster** tab in the expanded runtime pane on the +KubeAI backend. It shows Ready/scheduling state, GPU allocation and scheduled +requests across namespaces, GFD labels, runtime evidence, managed Models and +KubeAI/NVIDIA pod startup/restart state. GPU requests are scheduling facts, not +live GPU utilization; the system pane explicitly describes the local host. + +Cluster monitoring uses three batched API reads, cached for at least 15 seconds +and polled only while the tab is visible. Slow probes cannot overlap or block +keyboard input; **Refresh cluster** forces a sample, and **Doctor** runs the +shared detailed checks on demand. Pod samples also feed the Instances view. + +Select a node for **Detach node** or **Attach node**. Each previews its workload +and scheduling state and requires confirmation; detach drains workloads and +emptyDir data, while attach confirms local Compose workloads have stopped. +These reuse `kube node detach/attach` and preserve cluster membership. The +Control tab provides **Apply** through the leasing controller and confirmed +**Down** for managed Models and the gateway. Leases remain after Down. The +KubeAI endpoint editor selects resource profiles; Kubernetes chooses node/GPU +placement. diff --git a/dev/aivm-pi-bridge.md b/dev/aivm-pi-bridge.md new file mode 100644 index 00000000..7ece6e26 --- /dev/null +++ b/dev/aivm-pi-bridge.md @@ -0,0 +1,226 @@ +# aivm × pi bridge: design notes (temporary home) + +Status: **design only, not implemented.** This document codifies the bridging +logic that makes a user's aivm VM able to run `pi` against this stack's +LiteLLM gateway, so that whoever implements the script (possibly a stronger +agent) does not have to rediscover the ground truth. If this ever graduates +to a real feature, move it to `docs/` and pair it with an ADR; until then it +lives here, next to the temporary script it describes +(`dev/aivm_pi_bridge.sh` or `.py`, to be created). + +## Purpose and ownership split + +After `infer-stack acquire ` on a host, the user wants `pi` on a +managed aivm VM to work against that host's gateway with no manual +configuration. The tools each own one half of the truth: + +- **infer-stack** knows the *endpoint*: the LiteLLM host port, the master + key, the gateway health, and the acquired model ids. +- **aivm** knows the *VM network*: which bridge network a VM is on, its + gateway IP, how to reach into the guest (`aivm ssh`), and the guest's + firewall exceptions (`firewall.allow_tcp_endpoints`). +- **pi** is a passive consumer: it needs a provider registration, a + credential, and a default model in its per-user config. + +The bridge is a small script that composes the two tools' *public* surfaces. +Deliberately not: a first-class `infer-stack aivm` subcommand (feature creep +for a niche pairing — the stack must not know what aivm is), and not an +aivm-side hook that knows about LiteLLM (aivm must not know what +infer-stack is). Promotion criterion: a second consumer (another agent +client, multi-VM fleets, or a second target platform) is what would justify +a generic "clients/connectors" abstraction in infer-stack. + +## Ground truth (verified 2026-10-02) + +### The gateway (infer-stack side) + +- Compose service `litellm`: image `ghcr.io/berriai/litellm:v1.82.3-stable` + (`infer_stack/config.py::PINNED_IMAGES`), container port 4000, host port + 14042 (`DEFAULT_PORTS['litellm']`, user-configurable). +- The published port is a plain `f'{host_port}:4000'` mapping + (`infer_stack/leasing/gateway.py`), so Docker binds **0.0.0.0** on the + host (confirmed live: `0.0.0.0:14042->4000/tcp`). +- The master key is generated with an `sk-` prefix and stored in the + compose project env file: `API_KEY_ENV = 'LITELLM_MASTER_KEY'`, + `set_master_key()` / `master_key()` / `rotate_master_key()` in + `gateway.py`. The rendered LiteLLM config references it as + `os.environ/LITELLM_MASTER_KEY`. +- `Gateway.urls()` advertises `http://127.0.0.1:14042/v1` — + host-local only. A VM client needs the *host's bridge-side IP* instead + (the IP aivm calls the gateway IP). +- There is **no TLS/HTTPS anywhere** in the stack today. An optional nginx + reverse proxy exists (ports 80/443, `DEFAULT_PORTS['reverse_proxy_*']`) + and is the natural TLS-termination point if we ever want one. + +### The client (pi side) + +The provider is the npm extension **`pi-provider-litellm` v3.3.0** +("LiteLLM proxy provider extension for Pi", requires Pi 0.83+; MCP parts +need 0.99+). Verified against a live VM (`~/.pi/agent/`): + +- **Install**: `pi install npm:pi-provider-litellm` — the *personal* + (global) install, writing the package declaration to + `~/.pi/agent/settings.json`. Note: `-l`/`--local` means the *project* + `.pi/settings.json`, which loads only after project trust is granted — + the wrong scope for a VM-wide setup. (The TODO in aivm's pi installer + currently says `pi install -l …`; it should drop the `-l`.) +- **Credential/endpoint config surfaces** (any one works; the first is what + a live, working VM shows): + 1. `~/.pi/agent/auth.json` (mode 0600): + `{"litellm": {"type": "api_key", "key": "sk-…", "env": {"LITELLM_BASE_URL": "http://:"}}}` + — the exact shape the extension's own `/login` persists. + 2. `~/.pi/agent/settings.json`: + `{"litellm": {"providers": {"litellm": {"baseUrl": "http://:", "apiKey": "…", "allowInsecureHttp": true}}}}` + (`apiKey` may be `$ENV` / `!command` / literal). + 3. Env: `LITELLM_BASE_URL` + `LITELLM_API_KEY` (lowest precedence). +- **`allowInsecureHttp`** (default `false`; loopback is exempt) is the + current plain-HTTP workaround for non-loopback `http://` base URLs. + There is **no CA-cert / self-signed option** in the extension's documented + fields; trusting a self-signed cert would have to go through Node's + `NODE_EXTRA_CA_CERTS` (unverified against pi's Node 22) or an upstream + feature request. +- **Model discovery**: `/model/info` (admin) first, falling back to + `/v1/models`, then `/health` + per-route probes. Results persist to + `~/.pi/agent/models-store.json` (`litellmDiscoveryVersion` marker), so + the guest keeps a working catalog even if the extension is later absent. + `/litellm-refresh` re-discovers on demand. The extension also registers + LiteLLM MCP tools and Skills Gateway injection *by default* — fine for a + self-owned proxy, but the design should mention the trust note. +- **Default model/provider**: `settings.json` top-level + `defaultProvider` / `defaultModel` (e.g. `litellm` / + `qwen3.8-27b-dbirks-hyperqwen`). + +### aivm side + +- The machine store (host, `/var/lib/aivm` for fresh 0.6 installs) holds + `[[networks]]` records: `[networks.network]` (`name`, `bridge`, + `gateway_ip`, …) and `[networks.firewall]` (`enabled`, `block_cidrs`, + `allow_tcp_ports`, `allow_tcp_endpoints`, …). `[[vms]]` records carry + `network_name`, so the VM→network mapping is explicit + (`aivm/config_store/models.py::VMEntry`). +- `firewall.allow_tcp_endpoints` takes `IPv4:PORT` strings and generates: + an input-hook accept for traffic addressed to that host, and a + forward-hook accept on the **pre-DNAT** conntrack original tuple + (`aivm/firewall.py::_nft_script`). Because of that, one entry + `:` covers the Docker-published port even + though Docker DNATs it to `:4000` (the pre-v0.6.0 failure + that made people add the *container* port to `allow_tcp_ports`; see + aivm `dev/journals/gpt56.md`, fix commit 42c8b47). +- Programmatic mutation is first-class: + `aivm.config_store.parse.parse_store_toml` → + `aivm.config_store.mutate.upsert_network` → render/write. Applying is + `aivm vm update --yes ` (reconciles the managed nftables table live, + policy-fingerprint driven, no VM restart) or `aivm firewall apply`. +- Guest access is `aivm ssh ` (managed access identity; verify the + exact non-interactive command form before scripting it — the CLI treats + it as a foreground session workflow). Pi itself is provisioned by + `aivm vm provision pi` (managed install under `~/.pi/agent`, official + installer, Node bootstrap). + +### Concrete observed instance + +This design was discovered from a live VM: default route `10.77.0.1` +(gateway), VM address `10.77.0.119`, `settings.json` with +`baseUrl: http://10.77.0.1:14042`, `allowInsecureHttp: true`, and 86 +discovered models in `models-store.json` — i.e. the manual version of this +bridge already works; the script just automates and idempotently repeats it. + +## The bridge algorithm + +Runs on the **host**. Inputs: target VM name, `--model` (optional; default: +a sensible choice from the acquired endpoints), `--dry-run`, `--yes`. +Fail closed at each stage; never echo the master key. + +**Host side** + +1. Read infer-stack state: litellm enabled, host port, master key (env + file in the generated dir), and probe the gateway + (`/health` or `/v1/models` with the key). Abort with a clear message if + the gateway is down. +2. Read the aivm machine store; find the target VM's `network_name`; take + that network's `gateway_ip`. +3. Compute `endpoint = ":"`; merge it into + that network's `firewall.allow_tcp_endpoints` (deduplicated; leave + `allow_tcp_ports` alone). Write the store back through aivm's own + parse/mutate/render round-trip so formatting, schema version, and + sibling records are preserved. +4. Apply: `aivm vm update --yes `; confirm the endpoint shows up + (`aivm firewall status` or `nft list`), and optionally TCP-probe the + endpoint from the guest. + +**Guest side** (via `aivm ssh` or the managed ssh key) + +5. Verify pi is provisioned (managed install present); if not, direct the + user to `aivm vm provision pi` (its installer already carries the TODO + for the plugin step). +6. `pi install npm:pi-provider-litellm` (global; idempotent). +7. Merge `~/.pi/agent/auth.json` (keep 0600): `litellm = {type: api_key, + key: , env: {LITELLM_BASE_URL: http://:}}`. + Re-running after `rotate_master_key` is what resyncs the credential. +8. Merge `~/.pi/agent/settings.json`: `defaultProvider: "litellm"` and + `defaultModel: ` — JSON-merge only; never rewrite unrelated keys, + and skip the merge when values already match (idempotence). +9. End-to-end verification: one-shot non-interactive `pi` print of a trivial + completion through the gateway. Report success/failure with the + concrete commands used (auditability: the script logs what it ran). + +**Shape**: ~150 lines of Python (aivm and infer-stack are both importable +on the host; prefer their Python APIs over parsing their output) or a shell +script if importability turns out to be an assumption. Secrets: write files +directly, log only redacted forms. + +## HTTPS: options and recommendation + +- **Option A — keep plain HTTP (status quo).** Inside the aivm sandbox this + is defensible: the firewall endpoint is scoped to one VM network, every + other private-range destination is blocked, and the traffic is between the + user's own VM and their own host. Residual risk is only the *host-side* + exposure of the 0.0.0.0 publish to whatever LAN the host sits on. Cheap + mitigation without TLS: publish the port bound to the bridge IP instead of + 0.0.0.0 (compose `ports: ["10.77.0.1:14042:4000"]` — Docker supports + `IP:port:port`), so the gateway is reachable only from VM networks. +- **Option B — self-signed TLS terminated at the stack's nginx.** The + optional reverse proxy (443) generates a self-signed cert (SAN = the + bridge IP(s)), terminates TLS, and proxies to `litellm:4000`. The guest + trusts the cert via `NODE_EXTRA_CA_CERTS=` on pi's Node runtime. + **Unverified**: whether pi's Node 22 build honors `NODE_EXTRA_CA_CERTS` + for the `fetch` used by the provider extension (undici historically lagged + on this). Alternatively request a `caCert` field upstream in + pi-provider-litellm. +- **Option C — native TLS in the LiteLLM container.** Check whether the + pinned image passes uvicorn `ssl_certfile`/`ssl_keyfile`; if it does, this + is simpler than B (no proxy hop) but gives the least control over cert + lifecycle. + +**Recommendation**: ship the bridge with Option A (+ the interface-scoped +bind if easy) and treat B as a follow-up once the Node trust-store behavior +is confirmed with a 10-minute experiment on one VM. Do not block the bridge +on TLS: the aivm firewall already does the actual confinement. + +## Open questions / verification checklist + +- Exact non-interactive form of `aivm ssh ` (does it need a TTY? + does `--yes` cover the prompts?) — the script may fall back to plain + `ssh` using aivm's managed identity. +- Fix the aivm pi-installer TODO: `pi install -l …` → `pi install …` + (global, not project). +- `NODE_EXTRA_CA_CERTS` behavior with pi's bundled Node 22 (only for B). +- Docker `IP:port:port` publish on the target host (Option A mitigation). +- Multi-VM on one network: one endpoint entry covers *all* VMs on that + bridge (the entry is network-scoped, not VM-scoped) — say so in output. +- KubeAI backend is out of scope for the bridge (no compose publish; + ingress is a different story). +- What the script does when the user has *other* pi providers configured + (default: only touch `default*` when `--make-default` is passed? decide + during implementation — the observed live setup did set them). + +## Confident / at risk + +- **Confident**: the endpoint math (`gateway_ip:litellm_port`, one entry, + pre-DNAT matching), the pi config shapes (verified against a working + VM), and the aivm mutation/apply path (first-class APIs, live-reconciling + update). The manual procedure demonstrably works today. +- **At risk**: non-interactive `aivm ssh` scripting, `NODE_EXTRA_CA_CERTS` + (Option B), and the store write-back preserving every corner of a user's + hand-edited machine store (mitigate by going through aivm's own + parse/render and by `--dry-run` showing the exact store diff first). diff --git a/dev/e2e_tests/README.md b/dev/e2e_tests/README.md index 89549a5f..5bea5120 100644 --- a/dev/e2e_tests/README.md +++ b/dev/e2e_tests/README.md @@ -25,7 +25,7 @@ Then rsync the printed results dir back; I review `report.md`. ### Before `--gpu` (one-time, so timeouts reflect logic not 10GB pulls) ```bash -docker pull vllm/vllm-openai:v0.19.1 +docker pull vllm/vllm-openai:v0.30.0-cu129 docker pull ghcr.io/berriai/litellm:v1.82.3-stable docker pull ollama/ollama:latest ``` diff --git a/dev/e2e_tests/kubeai-cpu-values.yaml b/dev/e2e_tests/kubeai-cpu-values.yaml new file mode 100644 index 00000000..8d5fb354 --- /dev/null +++ b/dev/e2e_tests/kubeai-cpu-values.yaml @@ -0,0 +1,38 @@ +# KubeAI on a GPU-less machine, for exercising the kubeai backend for real. +# The chart's `cpu` profile runs vLLM's CPU image (needs AVX-512). Sized for +# a sub-1B model; raise the requests for anything larger. +resourceProfiles: + cpu: + imageName: "cpu" + requests: + cpu: 8 + memory: "16Gi" + # Half the guest's 120 cores: only one such Model fits at a time, which is + # how dev/kubeai_e2e.sh's make-room check forces a scheduling conflict. + cpu-half: + imageName: "cpu" + requests: + cpu: 64 + memory: "16Gi" + # Two "GPU sizes" on CPU, for the sizing check (E2E_SIZED=1 in + # dev/kubeai_e2e.sh): each selects the nodes labelled with one fake GPU + # product, as a real profile selects nvidia.com/gpu.product. `sized-80g` + # tolerates dev/k3s_agent_container.sh's taint, so it lands on that node. + sized-24g: + imageName: "cpu" + requests: + cpu: 8 + memory: "16Gi" + nodeSelector: + nvidia.com/gpu.product: "FAKE-24G" + sized-80g: + imageName: "cpu" + requests: + cpu: 8 + memory: "16Gi" + nodeSelector: + nvidia.com/gpu.product: "FAKE-80G" + tolerations: + - key: infer-stack.test/simulated + operator: Exists + effect: NoSchedule diff --git a/dev/external_e2e.sh b/dev/external_e2e.sh new file mode 100755 index 00000000..35f0f07c --- /dev/null +++ b/dev/external_e2e.sh @@ -0,0 +1,270 @@ +#!/usr/bin/env bash +# End-to-end test for external endpoints against a REAL LiteLLM gateway. +# +# No GPU, no model weights: the compose backend runs only its front door, and +# the "external" server is infer-stack's own mock OpenAI server on this host, +# which accepts one bearer key and records every request. Runs on the guest +# VM (docker, the pinned litellm and postgres images). +# +# ./dev/external_e2e.sh +# +# For each routing mode (static, then dynamic): +# 1. an external endpoint whose key has no value is refused before publishing; +# 2. `routes seed` publishes it and the gateway routes the alias to the mock, +# sending the key from the managed .env by name (`os.environ/NAME`); +# 3. the key changes: `env` says an apply is needed, the old key is refused +# by the upstream until `apply`, the new key is sent after it; +# 4. `routes prune` unpublishes it and the alias stops routing; +# 5. `access` of the external endpoint alone: no lease, no lease id in the +# env-file, a client that knows only the env-file reaches it; +# 6. `access` of a managed endpoint alone (llm-d-inference-sim, on CPU): +# one lease, the same env-file shape; +# 7. `access` of a mixed bundle: one lease for the managed member only; +# `release --env-file` releases it and the external route stays; +# 8. the managed alias becomes external after release and eviction: +# same alias, same workflow, no lease. +# +# Knobs (env): +# E2E_MODES routing modes to run (default "static dynamic") +# E2E_PORT the mock's port (default 18911) +# E2E_SIM the simulator image (default ghcr.io/llm-d/llm-d-inference-sim:v0.9.0) +# E2E_BACKEND compose (default) or kubeai: the external route through +# KubeAI's host gateway (phases 1-5; 6-8 need the simulator, +# which runs on Compose only). Needs KUBECONFIG and the chart. +set -euo pipefail + +MODES="${E2E_MODES:-static dynamic}" +PORT="${E2E_PORT:-18911}" +SIM="${E2E_SIM:-ghcr.io/llm-d/llm-d-inference-sim:v0.9.0}" +BACKEND="${E2E_BACKEND:-compose}" +HOST_IP="$(ip -4 -o addr show docker0 | awk '{print $4}' | cut -d/ -f1)" +[ -n "$HOST_IP" ] || { echo 'no docker0 address' >&2; exit 1; } + +fail() { echo "FAIL: $*" >&2; exit 1; } + +rm_work() { # Postgres owns its data dir as its own uid + [ -d "$1/data" ] && docker run --rm -v "$1/data:/d" --entrypoint sh postgres:16.8 \ + -c 'rm -rf /d/postgres-litellm /d/postgres-open-webui' >/dev/null 2>&1 + rm -rf "$1" +} + +MOCK_PID='' +start_mock() { # start_mock KEY + stop_mock + cat > "$WORK/mock.yaml" <"$WORK/mock.log" 2>&1 & + MOCK_PID=$! + for _ in $(seq 50); do + curl -sf "http://127.0.0.1:$PORT/health" >/dev/null && return 0 + sleep 0.2 + done + fail "mock server did not start: $(cat "$WORK/mock.log")" +} +stop_mock() { + if [ -n "$MOCK_PID" ]; then kill "$MOCK_PID" 2>/dev/null || true; wait "$MOCK_PID" 2>/dev/null || true; fi + MOCK_PID='' +} + +last_bearer() { # the Authorization the mock last received + curl -sf "http://127.0.0.1:$PORT/__mock__/requests" | python -c ' +import json, sys +reqs = [r for r in json.load(sys.stdin)["requests"] + if r.get("path", "").endswith("/chat/completions")] +headers = {k.lower(): v for k, v in reqs[-1]["headers"].items()} if reqs else {} +print(headers.get("authorization", ""))' +} + +chat() { # chat ALIAS -> http status; body in $WORK/chat.json + local base key + base="$(run_is env OPENAI_BASE_URL)" + key="$(run_is env LITELLM_MASTER_KEY)" + curl -s -o "$WORK/chat.json" -w '%{http_code}' "$base/chat/completions" \ + -H "Authorization: Bearer $key" -H 'Content-Type: application/json' \ + -d "{\"model\": \"$1\", \"max_tokens\": 8, + \"messages\": [{\"role\": \"user\", \"content\": \"hi\"}]}" || true +} + +chat_env() { # chat_env ENV_FILE ALIAS -> http status, as a card would + ( + # shellcheck disable=SC1090 + source "$1" + var="INFER_STACK_ENDPOINT_$(echo "$2" | tr '[:lower:]-' '[:upper:]_')" + curl -s -o "$WORK/chat.json" -w '%{http_code}' "$OPENAI_BASE_URL/chat/completions" \ + -H "Authorization: Bearer $OPENAI_API_KEY" -H 'Content-Type: application/json' \ + -d "{\"model\": \"${!var}\", \"max_tokens\": 8, + \"messages\": [{\"role\": \"user\", \"content\": \"hi\"}]}" || true + ) +} + +leases_active() { # how many ACTIVE leases the ledger holds + run_is leases --json | python -c ' +import json, sys +print(sum(1 for le in json.load(sys.stdin).get("leases", []) if le.get("state") == "active"))' +} + +chat_until() { # chat_until ALIAS STATUS: routes settle after apply + local got='' + for _ in $(seq 60); do + got="$(chat "$1")" + [ "$got" = "$2" ] && return 0 + sleep 2 + done + fail "chat $1: wanted HTTP $2, got $got: $(cat "$WORK/chat.json")" +} + +run_mode() { + local mode=$1 + WORK="$(mktemp -d /tmp/infer-stack-external-e2e.XXXXXX)" + IS_ENV="INFER_STACK_CONFIG_DIR=$WORK/config INFER_STACK_DATA_DIR=$WORK/data" + run_is() { env $IS_ENV infer-stack "$@"; } + echo "== [$mode] work dir: $WORK" + mkdir -p "$WORK/config" + run_is config set backend "$BACKEND" >/dev/null + if [ "$mode" = dynamic ]; then run_is config set dynamic_routing true >/dev/null; fi + # The runbook's own catalog: nothing external. + cat > "$WORK/config/catalog.yaml" < "$WORK/external.yaml" <"$WORK/out" 2>&1; then + fail 'seed without the key succeeded' + fi + grep -q 'infer-stack env E2E_REMOTE_KEY=' "$WORK/out" || fail "$(cat "$WORK/out")" + + echo "== [$mode] 2. publish; the gateway sends the key by name" + run_is env E2E_REMOTE_KEY=key-one >/dev/null + run_is routes seed "$WORK/external.yaml" --yes + run_is routes list + run_is routes list --json | grep -q '"origin": "external"' || fail 'not listed as external' + if grep -rq --include=docker-compose.yml 'key-one' "$WORK/data"; then + fail 'the key value is in the compose file' + fi + chat_until remote 200 + [ "$(last_bearer)" = 'Bearer key-one' ] || fail "sent $(last_bearer)" + + echo "== [$mode] 3. the key changes" + start_mock key-two # the upstream rotated + chat_until remote 401 + run_is env E2E_REMOTE_KEY=key-two | tee "$WORK/out" + grep -q 'after `infer-stack apply`' "$WORK/out" || fail 'env did not say apply' + run_is apply --yes + chat_until remote 200 + [ "$(last_bearer)" = 'Bearer key-two' ] || fail "sent $(last_bearer)" + + echo "== [$mode] 4. prune unpublishes it" + run_is routes prune --yes + if run_is routes list --json | grep -q '"remote"'; then fail 'still listed after prune'; fi + chat_until remote 400 + + if [ "$BACKEND" != compose ]; then + echo "== [$mode] 5. access an external endpoint: no lease" + cp "$WORK/external.yaml" "$WORK/config/catalog.yaml" + run_is access remote --yes --env-file "$WORK/remote.env" + if grep -q INFER_STACK_LEASE_ID "$WORK/remote.env"; then fail 'a lease id without a lease'; fi + [ "$(chat_env "$WORK/remote.env" remote)" = 200 ] || fail "remote: $(cat "$WORK/chat.json")" + run_is stack down >/dev/null 2>&1 || true + stop_mock + rm_work "$WORK" + echo "== [$mode/$BACKEND] PASS (phases 1-5)" + return 0 + fi + # The runbook's catalog now names a managed endpoint, the external one, + # and a bundle of both. + cat > "$WORK/config/catalog.yaml" < "$WORK/pair.json" + python - "$WORK/pair.json" <<'PY' || fail "pair: $(cat "$WORK/pair.json")" +import json, sys +d = json.load(open(sys.argv[1])) +assert d['lease_id'] and d['external'] == ['remote'], d +assert set(d['descriptor']['endpoints']) == {'local', 'remote'}, d +PY + [ "$(leases_active)" = 1 ] || fail 'a mixed bundle took other than one lease' + [ "$(chat_env "$WORK/pair.env" local)" = 200 ] || fail "pair/local: $(cat "$WORK/chat.json")" + [ "$(chat_env "$WORK/pair.env" remote)" = 200 ] || fail "pair/remote: $(cat "$WORK/chat.json")" + run_is release --env-file "$WORK/pair.env" --yes + [ "$(leases_active)" = 0 ] || fail 'the bundle lease is still held' + [ "$(chat_env "$WORK/pair.env" remote)" = 200 ] || fail 'remote stopped routing after release' + + echo "== [$mode] 8. the managed alias moves to an external server" + run_is evict --all --yes + python - "$WORK/config/catalog.yaml" <<'PY' +import sys, yaml +path = sys.argv[1] +cat = yaml.safe_load(open(path)) +cat['endpoints']['local'] = {'external': dict(cat['endpoints']['remote']['external'])} +yaml.safe_dump(cat, open(path, 'w'), sort_keys=False) +PY + run_is access local --yes --env-file "$WORK/moved.env" + if grep -q INFER_STACK_LEASE_ID "$WORK/moved.env"; then fail 'moved alias took a lease'; fi + [ "$(chat_env "$WORK/moved.env" local)" = 200 ] || fail "moved: $(cat "$WORK/chat.json")" + [ "$(last_bearer)" = 'Bearer key-two' ] || fail "moved sent $(last_bearer)" + + run_is stack down >/dev/null 2>&1 || true + stop_mock + rm_work "$WORK" + echo "== [$mode] PASS" +} + +cleanup() { + status=$? + set +e + stop_mock + if [ -n "${WORK:-}" ] && [ -d "$WORK" ]; then + env $IS_ENV infer-stack stack down >/dev/null 2>&1 + rm_work "$WORK" + fi + exit $status +} +trap cleanup EXIT + +for mode in $MODES; do run_mode "$mode"; done +echo '== external endpoints e2e: PASS' diff --git a/dev/handover/p4_gpu_labels.sh b/dev/handover/p4_gpu_labels.sh new file mode 100755 index 00000000..e47e2cfe --- /dev/null +++ b/dev/handover/p4_gpu_labels.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# P4 on real GPUs: what the CPU-only development cluster cannot show. +# +# Run on a GPU machine that is a k3s node with the NVIDIA device plugin and +# GPU Feature Discovery (README: "NVIDIA GPU support"), with the KubeAI chart +# installed and `kubectl -n kubeai port-forward svc/kubeai 8000:80` running. +# It serves one small model on one GPU for a few minutes: run it when that +# GPU is free. +# +# It checks, in order: +# 1. GPU Feature Discovery labels each GPU node (product, memory) and the +# device plugin advertises nvidia.com/gpu: the facts sizing reads. +# 2. `catalog suggest --backend kubeai` proposes a resource profile for +# each GPU product, from those labels. +# 3. With the proposed profiles installed, an endpoint that declares only +# `min_vram_gib` gets the right profile and answers a request on a GPU. +# 4. `infer-stack measure --record` finds vLLM's memory-profiling +# lines in the pod's log and records a min_vram_gib. +# +# It uses its own config and data directories (never this host's infer-stack +# state). It leaves the proposed profiles in the chart's values: they +# describe this cluster's GPUs and are what min_vram_gib needs. +# +# dev/handover/p4_gpu_labels.sh 2>&1 | tee p4-gpu.log +# +# Knobs: E2E_MODEL (default Qwen/Qwen2.5-0.5B-Instruct), NAMESPACE (kubeai), +# E2E_TIMEOUT (900 s). +set -euo pipefail + +MODEL="${E2E_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}" +NAMESPACE="${NAMESPACE:-kubeai}" +TIMEOUT="${E2E_TIMEOUT:-900}" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/infer-stack-p4.XXXXXX")" +IS_ENV="INFER_STACK_CONFIG_DIR=$WORK/config INFER_STACK_DATA_DIR=$WORK/data" +run_is() { env $IS_ENV infer-stack "$@"; } +pass() { echo "PASS $*"; } +fail() { echo "FAIL $*" >&2; exit 1; } + +cleanup() { + set +e + run_is release --all --yes >/dev/null 2>&1 + run_is stack down >/dev/null 2>&1 + echo "work dir kept for the report: $WORK" +} +trap cleanup EXIT +mkdir -p "$WORK/config" + +echo '== 1. the GPU facts sizing reads' +kubectl get nodes -o json | python3 -c ' +import json, sys +found = 0 +for n in json.load(sys.stdin)["items"]: + labels = n["metadata"].get("labels", {}) + gpus = n["status"].get("allocatable", {}).get("nvidia.com/gpu") + product, memory = labels.get("nvidia.com/gpu.product"), labels.get("nvidia.com/gpu.memory") + print(f" {n[\"metadata\"][\"name\"]}: gpus={gpus} product={product} memory_mib={memory}") + found += bool(gpus and product and memory) +sys.exit(0 if found else 1)' || fail 'no node has nvidia.com/gpu and the GFD product/memory labels' +pass 'GPU nodes are labelled' + +echo '== 2. catalog suggest proposes profiles from the labels' +run_is config set backend kubeai >/dev/null +run_is config set kubeai_namespace "$NAMESPACE" >/dev/null +run_is catalog suggest --backend kubeai > "$WORK/suggest.yaml" 2> "$WORK/suggest.err" \ + || { cat "$WORK/suggest.err" >&2; fail 'catalog suggest failed'; } +python3 - "$WORK/suggest.err" "$WORK/profiles.yaml" <<'PY' || fail 'suggest printed no resourceProfiles' +import sys, yaml +text = open(sys.argv[1]).read() +block = yaml.safe_load(text[text.index('resourceProfiles:'):]) +open(sys.argv[2], 'w').write(yaml.safe_dump(block, sort_keys=False)) +print(yaml.safe_dump(block, sort_keys=False)) +PY +pass 'suggest proposed a profile per GPU product' + +echo '== 3. min_vram_gib picks the proposed profile and serves on a GPU' +version=$(helm list -n "$NAMESPACE" -o json | python3 -c \ + 'import json,sys; print(json.load(sys.stdin)[0]["chart"].rsplit("-", 1)[1])') +helm upgrade kubeai kubeai/kubeai -n "$NAMESPACE" --version "$version" \ + --reuse-values -f "$WORK/profiles.yaml" --wait >/dev/null +cat > "$WORK/config/catalog.yaml" < "$WORK/acquire.log" 2>&1 || true +grep -q 'ready: True' "$WORK/acquire.log" || fail 'p4-small never became ready' +got=$(kubectl -n "$NAMESPACE" get models.kubeai.org -l infer-stack/managed=true \ + -o jsonpath='{.items[0].spec.resourceProfile}') +echo " Model resourceProfile: $got (profiles proposed: $(tr '\n' ' ' < "$WORK/profiles.yaml" | head -c 200))" +case "$got" in nvidia-*:1) pass "min_vram_gib 8 -> $got, and it answers" ;; + *) fail "unexpected resource profile $got" ;; esac + +echo '== 4. measure reads the pod log on a GPU' +run_is measure p4-small --record | tee "$WORK/measure.log" +grep -q 'min_vram_gib' "$WORK/measure.log" || fail 'measure printed no min_vram_gib' +test -s "$WORK/data/leasing/kubeai/measurements.json" || fail 'measure --record wrote nothing' +pass 'measure recorded a min_vram_gib on kubeai' + +echo 'ALL PASS' diff --git a/dev/handover/p5_two_hosts.sh b/dev/handover/p5_two_hosts.sh new file mode 100755 index 00000000..527e4b78 --- /dev/null +++ b/dev/handover/p5_two_hosts.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# P5 across two real machines: what a second node in a container cannot show. +# +# Run on the cluster's first node (kubeconfig working) once a second GPU +# workstation has joined it (docs/kubeai-backend.md, "Add a workstation"), +# with the NVIDIA device plugin and GPU Feature Discovery running and the +# KubeAI chart installed. It serves one small model on the second node's GPU +# for a few minutes: run it when that GPU is free. +# +# It checks, in order: +# 1. two Ready nodes, and the second reports GPUs with GFD's labels; +# 2. with the gateway in the cluster, an endpoint whose resource profile +# selects the second node's GPU product lands there and answers through +# the gateway: pod-to-pod traffic across the real network (flannel); +# 3. the NodePort answers on the second node's own address too, so a card +# on either workstation uses the same env file; +# 4. `secrets rotate` (a Secret and a rollout) with the Model still served. +# +# Own config and data directories; nothing of this host's infer-stack state +# is touched. It adds one resource profile for the second node's GPU product +# to the chart's values and leaves it: that is what the node needs. +# +# dev/handover/p5_two_hosts.sh [SECOND_NODE_NAME] 2>&1 | tee p5-two-hosts.log +set -euo pipefail + +NODE_B="${1:-}" +MODEL="${E2E_MODEL:-Qwen/Qwen2.5-0.5B-Instruct}" +NAMESPACE="${NAMESPACE:-kubeai}" +TIMEOUT="${E2E_TIMEOUT:-900}" +PORT="${NODE_PORT:-30442}" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/infer-stack-p5.XXXXXX")" +IS_ENV="INFER_STACK_CONFIG_DIR=$WORK/config INFER_STACK_DATA_DIR=$WORK/data" +run_is() { env $IS_ENV infer-stack "$@"; } +pass() { echo "PASS $*"; } +fail() { echo "FAIL $*" >&2; exit 1; } + +cleanup() { + set +e + run_is release --all --yes >/dev/null 2>&1 + run_is stack down >/dev/null 2>&1 + echo "work dir kept for the report: $WORK" +} +trap cleanup EXIT +mkdir -p "$WORK/config" + +echo '== 1. two nodes, the second with GPUs' +if [ -z "$NODE_B" ]; then + NODE_B=$(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' \ + -o jsonpath='{.items[0].metadata.name}') +fi +[ -n "$NODE_B" ] || fail 'no second node (pass its name as the first argument)' +kubectl wait --for=condition=Ready "node/$NODE_B" --timeout=30s >/dev/null \ + || fail "$NODE_B is not Ready" +PRODUCT=$(kubectl get node "$NODE_B" -o jsonpath='{.metadata.labels.nvidia\.com/gpu\.product}') +GPUS=$(kubectl get node "$NODE_B" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}') +ADDR_B=$(kubectl get node "$NODE_B" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') +[ -n "$PRODUCT" ] && [ "${GPUS:-0}" -gt 0 ] \ + || fail "$NODE_B reports no GPUs or no nvidia.com/gpu.product label" +pass "$NODE_B at $ADDR_B: $GPUS x $PRODUCT" + +echo '== 2. a Model on the second node, through the in-cluster gateway' +profile="p5-$(echo "$PRODUCT" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9\n' '-')" +cat > "$WORK/profile.yaml" </dev/null +run_is config set backend kubeai >/dev/null +run_is config set kubeai_namespace "$NAMESPACE" >/dev/null +run_is config set kubeai_gateway cluster >/dev/null +cat > "$WORK/config/catalog.yaml" < "$WORK/acquire.log" 2>&1 || true +grep -q 'ready: True' "$WORK/acquire.log" || { tail -30 "$WORK/acquire.log"; fail 'not ready'; } +node=$(kubectl -n "$NAMESPACE" get pods -l infer-stack/managed=true \ + -o jsonpath='{.items[0].spec.nodeName}') +[ "$node" = "$NODE_B" ] || fail "the Model runs on $node, not $NODE_B" +# shellcheck disable=SC1090 +source "$WORK/lease.env" +ask() { # -> a generation for p5-remote + curl -sS --fail-with-body "$1/chat/completions" -H "Authorization: Bearer $OPENAI_API_KEY" \ + -H 'Content-Type: application/json' \ + -d '{"model": "p5-remote", "max_tokens": 4, "messages": [{"role": "user", "content": "say ok"}]}' \ + > "$WORK/ask.json" && grep -q choices "$WORK/ask.json" +} +ask "$OPENAI_BASE_URL" || fail "no generation via $OPENAI_BASE_URL" +pass "the Model on $NODE_B answers via $OPENAI_BASE_URL" + +echo '== 3. the NodePort answers on the second node too' +ask "http://$ADDR_B:$PORT/v1" || fail "no generation via $ADDR_B:$PORT (firewall on the NodePort?)" +pass "a card on $NODE_B can use http://$ADDR_B:$PORT/v1" + +echo '== 4. secrets rotate across the cluster' +run_is secrets rotate --force > "$WORK/rotate.log" 2>&1 || { cat "$WORK/rotate.log"; fail 'rotate'; } +grep -q 'new key accepted, old key rejected' "$WORK/rotate.log" || { cat "$WORK/rotate.log"; fail 'rotate'; } +pass 'secrets rotate: new key accepted, old key rejected' + +echo 'ALL PASS' diff --git a/dev/journals/claude.md b/dev/journals/claude.md index caadd793..ea5d049b 100644 --- a/dev/journals/claude.md +++ b/dev/journals/claude.md @@ -3180,3 +3180,330 @@ labels are stable enough to key residency on. **Takeaway.** Before unifying internals, check what callers see. Two backends that share 90% of their code but answer to different names are two products to the people using them. + +## 2026-09-24 12:54:30 -0400 + +**Intent.** The user approved the backend-unification plan, asked for the +work on a branch (`dev/backend-unification`), and authorised installing k3s +on the guest to test KubeAI for real. Model: Claude Opus 5.5 (1M context). + +**What happened.** K0: k3s plus KubeAI 0.23.4, using the chart's `cpu` +profile. It runs real vLLM on CPU, which beats the simulator: KubeAI builds +vLLM's own command line, and the simulator's CLI would have refused it. The +backend passed unchanged on its first real run since July. K1: the kubeai +backend now owns a gateway-only ComposeBackend and feeds it generic +`upstream` route rows. Before, a card's alias got 404 from KubeAI; after, +the same request answered. K2 and K4: strict pod residency, and the crash +diagnosis moved into a backend-neutral module. A Model whose vLLM rejected a +flag failed in 52 s instead of the 800 s timeout. K5: one served-name rule. +It fixed a real disagreement between the engine and two route builders. +Removed the dead profile-era KubeAI renderer. + +**What I got wrong along the way.** I wrote a second e2e script before +finding `dev/kubeai_e2e.sh`, a duplicate authority created in the middle of +a de-duplication task, and folded it back in. That script's generation check +could never fail (a `curl | grep && echo` list, where `set -e` does not +fire), so it "passed" on a 404. The plan's claim that KubeAI's lenient +`observe()` was a bug was wrong: Compose's is lenient by the same contract. +I corrected the plan. + +**Open: K3 (one acquire path).** It is feasible: only 12 controller tests and +the null backend depend on the old path. It carries a semantic decision the +user should make. On a cluster nothing displaces an idle keep-warm Model, so +the September starvation incident can recur on KubeAI unless admission +evicts idle keep-warm Models when a new one cannot be scheduled. + +**Takeaways.** (1) A test that has never failed is not evidence. Make it +fail on purpose first; GATEWAY=0 is that switch here. (2) "Has method X" is +a poor stand-in for "is backend Y": when a second backend gains X, every +such check silently changes meaning. + +## 2026-09-24 14:39:35 -0400 + +**Intent.** Make infer-stack more elegant, architecture first and the TUI +after. The user agreed that the Compose backend's real seam is gateway vs. +engines. Model: Claude Opus 5.5 (1M context). + +**What I did.** Four steps on `dev/backend-unification`, each green before +the next: naming rules into `naming.py`, the gateway's module code into +`gateway.py`, a `Gateway` class for its state, and `render_front_door`. +The one real design change: the registry merge no longer reads backend +state. Backends supply rows and the gateway only merges them, which removed +the `upstream_routes` side-channel I had added for KubeAI in K1. Settings the +gateway reads became backend properties onto the single `Gateway`, because +`use_profile` and tests set them on the backend after construction. Two +copies would drift. + +**How I checked a pure refactor.** Unit tests alone are weak evidence for +moved code, so I also rendered six stack configurations with the old and new +code and compared bytes (identical), then ran the real Compose gateway, +`secrets rotate` and the KubeAI e2e. + +**What went wrong.** My first byte comparison "passed" on two empty files, +because the script crashed in both runs. I caught it only because the byte +count was 0. More serious: the weight-floor doctest I wrote earlier wrote +4 GiB of real zeros into /tmp on every run and never cleaned up. A day of +test runs filled 49 GB, k3s went into disk pressure and evicted KubeAI. +Fixed with sparse files. + +**Takeaways.** (1) A comparison of outputs must check the outputs exist: +equality of two failures is not evidence. (2) Test fixtures that need large +files should be sparse. Size was the property under test, not the bytes. + +## 2026-09-26 13:05:00 -0400 + +**Intent.** Execute the backend-parity queue (`docs/queue.md`): the roadmap +phases that can be built and verified without a GPU or a second machine, +refactoring duplicate authorities on the way, and not stopping before a +passing UX audit. Model: Claude Opus 5.5 (Claude Code). + +**What landed.** P1b deleted the pre-admission acquire branch: a +`SimpleAdmission` mixin gives the dry-run and test backends the admission +surface, so the controller has one path. P2 put `ps`, `logs`, `status` and +the TUI behind an `Instance` view built from residency, on both backends. +P3 made the gateway in front of a cluster the compose gateway (UI, proxy, +dynamic routing, one approval). P6 is a parity suite, one test per *same* +row. P4 picks a KubeAI resource profile by GPU size from node labels. + +**Decisions.** The review split P1: a `preview` alone would have dropped +every KubeAI lease, because the admission view assumed GPU accounting, so +"does this backend allocate GPUs" became one function. Under dynamic +routing a KubeAI Model is named per deployment with compose's own tail +rule, rather than a new scheme. A profile's size is what its node selector +selects; the chart's selector-less profiles deliberately have none, because +guessing a size for "anywhere" would pick wrong silently. + +**What surprised me.** Three bugs older than this work surfaced only in a +real terminal or a real cluster: the TUI replaced kubeai's kubectl runner +with Docker's (no KUBECONFIG, so every kubectl call failed), a kwconf flag +swallowed the positional after it on every command, and the TUI swapped an +injected runner for the real Docker. Fakes passed throughout; each was +found by running the thing. The e2e also failed once on its own +`grep -q` + pipefail pattern (lessons.md). + +**Risks.** The kubeai recovery profile now nests the gateway's profile; +old profiles (a bare boolean) keep this process's settings. P4 is verified +with fake labels only; `dev/handover/p4_gpu_labels.sh` is the real test. + +**Takeaways.** (1) When a refactor removes a branch, first run the whole +suite with the other branch forced on: the failures list exactly what to +migrate. (2) A seam that wraps a runner must wrap only what it owns; a +wrapper that replaces is a second authority. (3) Put every e2e check's +output in a file before testing it. + +## 2026-09-26 — UX audit passes 5 and 6 + +**Did.** Pass 5 (both audit scripts, the TUI by eye on both backends at +80x24 and 200x50, the README's first run from empty roots, a naming grep) +found eight things, each fixed with a test: instance start times in UTC, +the TUI deriving the gateway URL itself (a duplicate authority; now +`Gateway.urls()`), top tabs unreachable from the keyboard, a sidebar and +log pane that ignored the terminal's size, and a released `reclaim: stop` +deployment reported as missing (now `Controller.keeps_up`). Pass 6 found a +traceback on `release --env-file` for a file never written. + +**Environment.** Pass 6's kubeai run failed because the dev cluster's node +went into disk pressure: my audit roots filled the shared disk. Deleting +them did not clear the taint (minimum reclaim, lessons.md). I wrote +`/etc/rancher/k3s/config.yaml` with a 1Gi minimum reclaim and restarted k3s; +remove the file to undo it. + +**Takeaways.** (1) An audit that passes when its command is missing is not +an audit: the script now refuses to start without `infer-stack` on PATH. +(2) Read a report whole: the one pass-5 finding I nearly missed was a line +my own `sed` range skipped. + +## 2026-09-26 18:30:00 -0400 — review hardening of the backend-parity campaign + +**Intent.** An outside review, in two parts, of the finished queue: fix what +it confirmed (replica residency, the backend protocol, cross-feature tests; +then scheduler reclaim, approval lifetime, secret rotation, route ordering, +coalescing, route seeding, preview purity, decomposition, docs), queueing +the work first. Model: Claude Opus 5.5 (claude-opus-5-5), Claude Code. + +**Did.** Queue items 11-22, each with a poison test that fails on the old +code. The common thread the review named held: several shared concepts +carried less than the controller needed. `Residency` now says whether +several units are replicas or a conflict; `Readiness.needs_room` is only +set for a capacity shortage, and the backend names useful victims; +`apply()` returns how far it got; routes retire before upstreams go. The +protocol the controller takes is the one the backends implement, checked by +`ty` (a deliberate drift fails it). + +**Found on the way.** KubeAI's `apply` dropped its host gateway's result, +so unverified routes behind KubeAI cleared the marker; `routes seed --json` +printed its conflict list into the JSON; a TUI test counted unrelated +refusals. + +**Uncertain.** The scheduler parsing reads Kubernetes' English message +("Insufficient "); a scheduler that words it differently evicts +nothing, which is the safe failure. Required node affinity is not +evaluated, also safe. The replica diagnosis quotes the concatenated pod +logs; with mixed failures it names the first fatal line it finds. + +**Takeaways.** (1) A "same" row is a claim about one path; the bug lived +where two rows met, so a backend feature needs one test with the common +lifecycle. (2) When a boolean crosses a boundary, ask what the caller does +with the cases it merges; `True/False` and `needs_room` each merged cases +that needed opposite actions. (3) Run long e2e passes from a frozen copy +(lessons.md). + +## 2026-09-26 20:23:05 -0400 — re-review: second-order cases + +**Intent.** The re-review closed most of items 11-22 and reopened five, plus +a route-seed race, for states the richer abstractions exposed but did not +carry through; and it classified the optional backend hooks instead of +promoting them all. Model: Claude Opus 5.5 (claude-opus-5-5), Claude Code. + +**Did.** In the order the reviewer asked, each with a test failing on the +code before it: KubeAI settles its host gateway after an interrupted apply +(the safety bug: `settle_snapshot` was an optional probe KubeAI lacked); +rotation keys on "the runtime changed", not "apply was called"; an explicit +re-approval is durable; replicas are judged from their own logs and +`deployment_health` separates serving from residency; reclaim fails closed +on schedulers and constraints it does not model; a race-time seed conflict +leaves no marker; the hooks are required state, common operations, or two +typed nullable capabilities (`recovery_profile`, `FrontDoor`). + +**Mistake.** One commit went in with a failing test: the chain ran pytest +and then committed regardless. Fixed forward in the next commit; later +commits gate on pytest's exit code. + +**Takeaways.** (1) An optional structural probe is a silent default: absent +meant "nothing to settle" and "owns host GPUs", both wrong for a backend +that simply had not implemented them. Required members make the checker +catch it. (2) Physical state and serving state are different questions; +once residency became physical, every reader that meant "is it serving" +needed its own summary. (3) A race test must inject the other writer at the +lock, not before the call; otherwise an earlier unlocked check passes it. + +## 2026-09-27 — campaign 2, item 32: one GatewayRoute, routes derived at render + +**Intent.** Replace the route plumbing (registry rows of three shapes, four +`render_front_door` strategies, `UPSTREAM_ROUTE`, KubeAI's parallel +`upstream_rows`) with one route type, and make the published catalog union +the only store of endpoint definitions (design decision 2). Model: Claude +Opus 5.5 (claude-opus-5-5), Claude Code. + +**Did.** `GatewayRoute` with one renderer; `front_door_routes` derives the +static table or the dynamic set for all three gateways; `routes seed` / +`prune` publish into and unpublish from the union; external endpoints route +to their own servers on every backend, with a dynamic id from their alias. + +**Changed my mind.** The design said deployment routes would be derived only +while live. The first full run failed +`test_converge_to_empty_keeps_the_front_door`: releasing an ad-hoc +deployment then recreated the gateway, a blip for every client. The registry +keeps that one owner (routes of aliases no catalog defines), and the design +doc says so. Also dropped the seed-from-`litellm_config.yaml` migration: +with catalog rows no longer written, reseeding from our own rendered config +would have made every route permanent. + +**Mistakes.** `Gateway.remember` took the converge flock inside a converge +and hung the suite (lesson added). A `pkill -f pytest` pattern matched my +own shell and killed the edit in the same command; the edit had to be +reapplied. + +**Takeaways.** (1) A "derive, do not store" rule needs checking against +every property the store was providing: the registry was also what kept the +gateway byte-stable across an ad-hoc release. (2) Tests that called +`render_compose` with deployments that serve no catalog endpoint were +testing a branch production never took; making the deployments serve the +catalog's aliases made them test the real property. + +## 2026-09-27 — items 33-34: external routes, keys by name, a real-LiteLLM e2e + +**Did.** Survival and replacement tests for external routes; keys by name +on both gateways (host env interpolation + fingerprint, in-cluster Secret + +key-hash); `env` says when a key takes effect; `routes seed` refuses a key +with no value. `dev/external_e2e.sh` drives a real LiteLLM against the mock +OpenAI server on the guest, static and dynamic, through a key rotation. + +**Found by the e2e, not the unit tests.** After `routes seed`, every command +warned that the catalogs drifted: `profile_drift` compared sources by digest, +and merging rewrites the union's sources. It now compares endpoint meaning. +Also: LiteLLM's upstream request lowercases `authorization`, and Postgres +leaves its data dir owned by its own uid (the script removes it through a +container). + +## 2026-09-27 — items 35-40: access above leasing, and the e2e that checked it + +**Did.** `Controller.access` over `acquire` (managed members, one lease) and +`publish_endpoints` (external-only: the same preview, approved marker and +publication, no ledger mutation); typed `ConnectionInfo` / +`request_names()` replace the `access` dict hook; `infer-stack access`, +`run` on top; lease-less env-files; `status` and TUI views; the +`FrontDoorControl` / `RenderedFrontDoor` names. `dev/external_e2e.sh` +covers eight phases in both routing modes on the guest. + +**Found by the e2e.** Redefining a bundle member (managed -> external) left +the published copy of the bundle naming an endpoint its own source had +dropped; every source must validate alone, so `drop_catalog_names` now +drops such bundles. The first external-only access on a fresh ledger +published nothing: freezing the snapshot consumed the candidate. Access now +always publishes. + +**Mistake.** A nested heredoc (`<