From 5c18619e850d2265d9f3c2ec5f327476bd0c5d56 Mon Sep 17 00:00:00 2001 From: Ori Nachum Date: Mon, 13 Jul 2026 19:02:23 +0300 Subject: [PATCH 1/7] =?UTF-8?q?spec:=20lobes=20fits=20the=20machine=20it?= =?UTF-8?q?=20lands=20on=20=E2=80=94=20per-machine=20profiles=20(devague?= =?UTF-8?q?=20/think)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Converged /think frame + exported spec for per-card hardware profiles. Today the packaged templates hardcode one tuning for the DGX Spark GB10 (sm_121). Bringing the fleet up on a Jetson AGX Thor (sm_110) showed it does not transfer: 1 of 4 roles came up correct on first boot, and reaching 3 of 4 took four hand-edits to the generated docker-compose.yml — cortex crash-looped: --kv-cache-dtype=fp8 asserts k_scale > 0, but the NVFP4 MTP checkpoint carries no calibrated KV scales -> auto embedder hung: the auto-picked FLASH_ATTN pooling path never answers a request -> --attention-config={"backend": "TRITON_ATTN"} reranker killed its engine (cudaErrorLaunchFailure) -> Triton + --enforce-eager; still returns wrong rankings (#105, verify on GB10: #106) senses came up clean, no changes The spec makes the profile the machine contract: which roles are feasible, which model serves each, and every machine-dependent knob. Spark (default) + Thor ship supported; Orin / Orin Nano Super are named but unvalidated; an unrecognised card refuses-or-warns rather than silently applying the Spark profile. Every role gains a correctness probe — "healthy" must stop meaning "semantically wrong". Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01RZNRUXYv84HUQ5ULQ5qJgz --- .devague/current | 2 +- ...e-machine-it-lands-on-one-command-det.json | 334 ++++++++++++++++++ CHANGELOG.md | 26 ++ ...the-machine-it-lands-on-one-command-det.md | 70 ++++ pyproject.toml | 2 +- 5 files changed, 432 insertions(+), 2 deletions(-) create mode 100644 .devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json create mode 100644 docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md diff --git a/.devague/current b/.devague/current index 2818b4b..138e621 100644 --- a/.devague/current +++ b/.devague/current @@ -1 +1 @@ -lobes-gateway-relays-sse-streams-frame-by-frame-th +lobes-fits-the-machine-it-lands-on-one-command-det diff --git a/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json b/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json new file mode 100644 index 0000000..1f3f11e --- /dev/null +++ b/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json @@ -0,0 +1,334 @@ +{ + "slug": "lobes-fits-the-machine-it-lands-on-one-command-det", + "title": "lobes fits the machine it lands on: one command detects the card (GB10, Thor, Orin, Orin Nano Super) and serves a profile tuned for THAT box \u2014 not a lowest-common-denominator config that fits none", + "schema_version": 2, + "status": "exported", + "created": "2026-07-13T15:33:49Z", + "updated": "2026-07-13T16:01:29Z", + "claims": [ + { + "id": "c1", + "kind": "announcement", + "text": "lobes fits the machine it lands on: one command detects the card (GB10, Thor, Orin, Orin Nano Super) and serves a profile tuned for THAT box \u2014 not a lowest-common-denominator config that fits none", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h6", + "text": "One command does it: on a supported card, 'lobes init' picks the right profile with no flags, and 'lobes doctor' names which profile it picked and why (detected card + capability + memory).", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c2", + "kind": "before_state", + "text": "The packaged templates hardcode one tuning for the DGX Spark GB10 (sm_121). It does not transfer: on Jetson AGX Thor (sm_110) cortex needed --kv-cache-dtype=auto, embedder needed --attention-config TRITON_ATTN, reranker needed --enforce-eager and STILL returns wrong rankings. Every one of these was found by crashing into it.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h7", + "text": "The three Thor divergences found by hand this session (kv-cache dtype auto, attention backend TRITON_ATTN on the pooling gears, enforce-eager on rerank) are exactly what a Thor profile encodes \u2014 the profile reproduces the working fleet without hand-editing.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c3", + "kind": "why_it_matters", + "text": "A single 'best for all machines' config fits none of them: it either wastes a big box's headroom or fails to boot on a small one.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h8", + "text": "The 'fits none' cost is real and measurable: a single shared config either leaves a big box's headroom unused or fails to boot a small one \u2014 demonstrable by pointing the GB10 profile at Thor (it crashes) and a Thor-safe config at the GB10 (it under-uses it).", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c4", + "kind": "audience", + "text": "Operators running lobes on a specific NVIDIA box \u2014 today DGX Spark GB10 and Jetson AGX Thor; next Jetson AGX Orin and Orin Nano Super \u2014 plus the lobes agent deployed on it.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h9", + "text": "The operator never has to know the card's compute capability to get a correct boot \u2014 the tool knows it; the operator only overrides when they WANT to.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c5", + "kind": "after_state", + "text": "lobes detects the host card and applies a per-machine profile (memory budget, quantization, attention backend, KV dtype, context, eager-vs-cudagraph, which roles are even feasible), so a supported box boots correct-and-tuned on first try.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h10", + "text": "'Correct on first try' is checked, not asserted: every role the profile claims passes its correctness probe on a clean boot, and a role the card cannot serve is absent from capabilities rather than broken-but-advertised.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c6", + "kind": "boundary", + "text": "Not a universal autotuner and not a benchmark-search: a profile is a hand-validated, checked-in config for a named card, not something lobes discovers at runtime by trying settings.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h11", + "text": "Every knob in a profile traces to a human-validated observation on a real card (a boot log, a probe result, a measured budget) \u2014 no value is there because a search loop landed on it, and lobes never mutates a profile at runtime.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [ + { + "id": "q1", + "text": "Does a profile pin exact MODELS per role (so Orin gets a smaller cortex), or only tuning knobs for the models the catalog already picks? The Orin Nano Super case forces model selection into the profile \u2014 which makes 'profile' and 'catalog' overlap. Which owns the model choice?", + "resolved": true, + "blocking": false + } + ], + "links": [], + "instruction": "" + }, + { + "id": "c7", + "kind": "success_signal", + "text": "On a supported box, a clean 'lobes init && lobes serve' brings every role it claims up correct on the FIRST try \u2014 no hand-editing of docker-compose.yml. On an unsupported box lobes says so instead of silently mis-serving.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h5", + "text": "A profile can be proven on a real box: booting a supported profile from a clean ~/.lobes needs zero hand-edits to the generated compose file.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c8", + "kind": "requirement", + "text": "lobes detects the host card (compute capability + total memory + device name) and resolves a named profile; the operator can override the choice explicitly (--profile) and see what was chosen (lobes doctor / lobes status).", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h1", + "text": "Detection is unambiguous on every target: GB10, Thor, AGX Orin and Orin Nano Super are each distinguishable from inside the container/host without guessing. (Note: nvidia-smi reports memory.used=[N/A] on Thor's integrated GPU, so detection cannot rely on nvidia-smi memory fields.)", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c9", + "kind": "requirement", + "text": "A profile owns the machine-dependent knobs, not just memory: GPU mem util per role, max-model-len, quantization, KV-cache dtype, attention backend, enforce-eager, and max-num-seqs.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h2", + "text": "The knob set is sufficient: every divergence found on Thor so far (kv-cache dtype, attention backend, enforce-eager) is expressible as a profile value \u2014 no code fork per card.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c10", + "kind": "requirement", + "text": "A profile also declares which ROLES are feasible on that card and which model serves each. On Orin Nano Super the 27B NVFP4 cortex is not runnable at all, so the profile must select a smaller cortex model or declare cortex unavailable \u2014 capabilities must never advertise a role the box cannot serve.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h3", + "text": "A role can be declared unavailable end-to-end: lobes capabilities / GET /capabilities omit it (or mark it unserved), and the gateway 4xx's a request for it rather than silently routing to a different model.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c11", + "kind": "requirement", + "text": "Every role ships a CORRECTNESS probe, not just a /health check: embed must rank a paraphrase above an unrelated string, rerank must put the relevant document first, cortex must answer a known-answer question. A role that is 'healthy' but semantically wrong must fail the probe.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h4", + "text": "The rerank probe FAILS today on this Thor box (relevant doc does not rank first) \u2014 i.e. the probe catches the bug we actually hit, rather than being a test that only passes.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c12", + "kind": "non_goal", + "text": "Not cross-vendor: no AMD/Intel/CPU-only targets. NVIDIA CUDA boxes only.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c13", + "kind": "assumption", + "text": "A profile is validated by RUNNING it on the physical card \u2014 a profile for a box nobody has booted is marked unvalidated, not shipped as supported.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c14", + "kind": "decision", + "text": "A machine profile pins the MODEL for each role, not just tuning knobs: the profile is the full machine contract (feasible roles + model per role + knobs), and the catalog is the menu it selects from. This is what lets Orin Nano Super downshift cortex to a small model instead of only disabling it.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c15", + "kind": "boundary", + "text": "Only cards we have ACTUALLY BOOTED ship as supported profiles: DGX Spark GB10 (the default) and Jetson AGX Thor (validated this session). Jetson AGX Orin and Orin Nano Super are named future targets \u2014 the design must accommodate them, but lobes does not promise them until someone boots one.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h12", + "text": "lobes is honest about coverage: it ships Spark (default) + Thor as supported, names Orin/Orin Nano Super as unvalidated, and on an unrecognised card it refuses-or-warns rather than silently applying the Spark profile.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c16", + "kind": "decision", + "text": "Profiles are default + overridable, and an operator may keep SEVERAL: lobes ships built-in profiles (Spark default, Thor), auto-detection picks one, --profile overrides it, and an operator can define their own profile rather than being limited to the shipped set.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c17", + "kind": "success_signal", + "text": "Measurable: on each supported card, a clean 'lobes init && lobes serve' from an empty deployment dir brings up 100% of the roles the profile claims, each passing its correctness probe, with ZERO hand-edits to the generated compose. Baseline to beat, measured on Thor this session: the Spark profile scored 0/4 roles on first boot (cortex crash-looped; embedder hung; reranker killed its engine; senses came up), and reaching 3/4 took 4 hand-edits to docker-compose.yml. Target for the Thor profile: 4/4 with 0 edits (rerank gated on #105/#106).", + "origin": "llm", + "status": "rejected", + "honesty_conditions": [ + { + "id": "h13", + "text": "The baseline is real and reproducible: pointing the current (Spark) template at Thor reproduces the 0/4-on-first-boot result, so the 4/4-with-0-edits target is a measured improvement rather than a claim.", + "status": "proposed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c18", + "kind": "success_signal", + "text": "Measurable: on each supported card, a clean 'lobes init && lobes serve' from an empty deployment dir brings up 100% of the roles the profile claims, each passing its correctness probe, with ZERO hand-edits to the generated compose. Measured baseline (Thor, this session, stock Spark-tuned template): 1 of 4 roles correct on first boot \u2014 senses came up clean; cortex crash-looped on an fp8-KV assert; embedder accepted requests and never answered; reranker killed its own engine with cudaErrorLaunchFailure. Reaching 3 of 4 took 4 hand-edits to docker-compose.yml. Target for the Thor profile: 4/4 with 0 edits (the 4th, rerank, is gated on #105/#106).", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h14", + "text": "The baseline is reproducible: a clean 'lobes init' + 'lobes serve' on Thor with the stock template reproduces 1/4-correct-on-first-boot, so 4/4-with-0-edits is a measured improvement, not a claim.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + } + ], + "open_vagueness": [ + { + "id": "v1", + "text": "Is the reranker's wrong ordering Thor-specific, or a pre-existing lobes bug also live on the GB10? Nothing currently checks rerank ordering. TRACKED, not blocking this spec: agentculture/lobes-cli#105 (Thor: wrong rankings + cudaErrorLaunchFailure) and #106 (verify the same ordering probe on the GB10; the result decides whether the fix belongs in a Thor profile or in the shared rerank template).", + "kind": "follow_up", + "claim_id": null + }, + { + "id": "v2", + "text": "Whether Orin (sm_87, no NVFP4/FP4 tensor cores) can serve any useful cortex at all, or is senses/embed-only. Needs a physical Orin to answer.", + "kind": "unknown_nonblocking", + "claim_id": null + }, + { + "id": "v3", + "text": "Whether profiles are resolved at 'lobes init' (baked into .env once) or re-resolved at every 'lobes serve' (so moving a deployment dir between boxes re-tunes).", + "kind": "unknown_nonblocking", + "claim_id": null + } + ], + "scope_entries": [] +} diff --git a/CHANGELOG.md b/CHANGELOG.md index a30bf50..23479c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,32 @@ All notable changes to this project are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.40.2] - 2026-07-13 + +### Added + +- Spec (`/think`, converged): **per-machine hardware profiles** — + `docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md`. + lobes detects the host card and applies a profile tuned for *that* box + (feasible roles + model per role + util / context / quantization / KV dtype / + attention backend / enforce-eager), instead of the single GB10-tuned config + the templates hardcode today. Ships Spark (default) + Thor as supported; + Orin / Orin Nano Super are named but unvalidated; an unrecognised card + refuses-or-warns rather than silently applying the Spark profile. + Every role gains a **correctness** probe, not just `/health` — a role that is + healthy but semantically wrong must fail. + + Motivated by bringing the fleet up on a Jetson AGX Thor (sm_110): the + Spark-tuned template scored **1 of 4 roles correct on first boot** (senses + clean; cortex crash-looped on an fp8-KV assert; embedder accepted requests and + never answered; reranker killed its engine with `cudaErrorLaunchFailure`), and + reaching 3 of 4 took four hand-edits to the generated compose. Rerank is still + wrong there — tracked in #105 / #106. + +### Changed + +### Fixed + ## [0.40.1] - 2026-07-11 ### Added diff --git a/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md b/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md new file mode 100644 index 0000000..a0e9818 --- /dev/null +++ b/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md @@ -0,0 +1,70 @@ +# lobes fits the machine it lands on: one command detects the card (GB10, Thor, Orin, Orin Nano Super) and serves a profile tuned for THAT box — not a lowest-common-denominator config that fits none + +> lobes fits the machine it lands on: one command detects the card (GB10, Thor, Orin, Orin Nano Super) and serves a profile tuned for THAT box — not a lowest-common-denominator config that fits none + +## Audience + +- Operators running lobes on a specific NVIDIA box — today DGX Spark GB10 and Jetson AGX Thor; next Jetson AGX Orin and Orin Nano Super — plus the lobes agent deployed on it. + +## Before → After + +- Before: The packaged templates hardcode one tuning for the DGX Spark GB10 (sm_121). It does not transfer: on Jetson AGX Thor (sm_110) cortex needed --kv-cache-dtype=auto, embedder needed --attention-config TRITON_ATTN, reranker needed --enforce-eager and STILL returns wrong rankings. Every one of these was found by crashing into it. +- After: lobes detects the host card and applies a per-machine profile (memory budget, quantization, attention backend, KV dtype, context, eager-vs-cudagraph, which roles are even feasible), so a supported box boots correct-and-tuned on first try. + +## Why it matters + +- A single 'best for all machines' config fits none of them: it either wastes a big box's headroom or fails to boot on a small one. + +## Requirements + +- lobes detects the host card (compute capability + total memory + device name) and resolves a named profile; the operator can override the choice explicitly (--profile) and see what was chosen (lobes doctor / lobes status). + - honesty: Detection is unambiguous on every target: GB10, Thor, AGX Orin and Orin Nano Super are each distinguishable from inside the container/host without guessing. (Note: nvidia-smi reports memory.used=[N/A] on Thor's integrated GPU, so detection cannot rely on nvidia-smi memory fields.) +- A profile owns the machine-dependent knobs, not just memory: GPU mem util per role, max-model-len, quantization, KV-cache dtype, attention backend, enforce-eager, and max-num-seqs. + - honesty: The knob set is sufficient: every divergence found on Thor so far (kv-cache dtype, attention backend, enforce-eager) is expressible as a profile value — no code fork per card. +- A profile also declares which ROLES are feasible on that card and which model serves each. On Orin Nano Super the 27B NVFP4 cortex is not runnable at all, so the profile must select a smaller cortex model or declare cortex unavailable — capabilities must never advertise a role the box cannot serve. + - honesty: A role can be declared unavailable end-to-end: lobes capabilities / GET /capabilities omit it (or mark it unserved), and the gateway 4xx's a request for it rather than silently routing to a different model. +- Every role ships a CORRECTNESS probe, not just a /health check: embed must rank a paraphrase above an unrelated string, rerank must put the relevant document first, cortex must answer a known-answer question. A role that is 'healthy' but semantically wrong must fail the probe. + - honesty: The rerank probe FAILS today on this Thor box (relevant doc does not rank first) — i.e. the probe catches the bug we actually hit, rather than being a test that only passes. + +## Honesty conditions + +- One command does it: on a supported card, 'lobes init' picks the right profile with no flags, and 'lobes doctor' names which profile it picked and why (detected card + capability + memory). +- The three Thor divergences found by hand this session (kv-cache dtype auto, attention backend TRITON_ATTN on the pooling gears, enforce-eager on rerank) are exactly what a Thor profile encodes — the profile reproduces the working fleet without hand-editing. +- The 'fits none' cost is real and measurable: a single shared config either leaves a big box's headroom unused or fails to boot a small one — demonstrable by pointing the GB10 profile at Thor (it crashes) and a Thor-safe config at the GB10 (it under-uses it). +- The operator never has to know the card's compute capability to get a correct boot — the tool knows it; the operator only overrides when they WANT to. +- 'Correct on first try' is checked, not asserted: every role the profile claims passes its correctness probe on a clean boot, and a role the card cannot serve is absent from capabilities rather than broken-but-advertised. +- Every knob in a profile traces to a human-validated observation on a real card (a boot log, a probe result, a measured budget) — no value is there because a search loop landed on it, and lobes never mutates a profile at runtime. +- A profile can be proven on a real box: booting a supported profile from a clean ~/.lobes needs zero hand-edits to the generated compose file. +- lobes is honest about coverage: it ships Spark (default) + Thor as supported, names Orin/Orin Nano Super as unvalidated, and on an unrecognised card it refuses-or-warns rather than silently applying the Spark profile. +- The baseline is reproducible: a clean 'lobes init' + 'lobes serve' on Thor with the stock template reproduces 1/4-correct-on-first-boot, so 4/4-with-0-edits is a measured improvement, not a claim. + +## Success signals + +- On a supported box, a clean 'lobes init && lobes serve' brings every role it claims up correct on the FIRST try — no hand-editing of docker-compose.yml. On an unsupported box lobes says so instead of silently mis-serving. +- Measurable: on each supported card, a clean 'lobes init && lobes serve' from an empty deployment dir brings up 100% of the roles the profile claims, each passing its correctness probe, with ZERO hand-edits to the generated compose. Measured baseline (Thor, this session, stock Spark-tuned template): 1 of 4 roles correct on first boot — senses came up clean; cortex crash-looped on an fp8-KV assert; embedder accepted requests and never answered; reranker killed its own engine with cudaErrorLaunchFailure. Reaching 3 of 4 took 4 hand-edits to docker-compose.yml. Target for the Thor profile: 4/4 with 0 edits (the 4th, rerank, is gated on #105/#106). + +## Scope / boundaries + +- Not a universal autotuner and not a benchmark-search: a profile is a hand-validated, checked-in config for a named card, not something lobes discovers at runtime by trying settings. +- Only cards we have ACTUALLY BOOTED ship as supported profiles: DGX Spark GB10 (the default) and Jetson AGX Thor (validated this session). Jetson AGX Orin and Orin Nano Super are named future targets — the design must accommodate them, but lobes does not promise them until someone boots one. + +## Non-goals + +- Not cross-vendor: no AMD/Intel/CPU-only targets. NVIDIA CUDA boxes only. + +## Assumptions + +- A profile is validated by RUNNING it on the physical card — a profile for a box nobody has booted is marked unvalidated, not shipped as supported. + +## Decisions + +- A machine profile pins the MODEL for each role, not just tuning knobs: the profile is the full machine contract (feasible roles + model per role + knobs), and the catalog is the menu it selects from. This is what lets Orin Nano Super downshift cortex to a small model instead of only disabling it. +- Profiles are default + overridable, and an operator may keep SEVERAL: lobes ships built-in profiles (Spark default, Thor), auto-detection picks one, --profile overrides it, and an operator can define their own profile rather than being limited to the shipped set. + +## Hard questions + +- Does a profile pin exact MODELS per role (so Orin gets a smaller cortex), or only tuning knobs for the models the catalog already picks? The Orin Nano Super case forces model selection into the profile — which makes 'profile' and 'catalog' overlap. Which owns the model choice? + +## Open / follow-up + +- Is the reranker's wrong ordering Thor-specific, or a pre-existing lobes bug also live on the GB10? Nothing currently checks rerank ordering. TRACKED, not blocking this spec: agentculture/lobes-cli#105 (Thor: wrong rankings + cudaErrorLaunchFailure) and #106 (verify the same ordering probe on the GB10; the result decides whether the fix belongs in a Thor profile or in the shared rerank template). diff --git a/pyproject.toml b/pyproject.toml index 50927ae..4eefec5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "lobes-cli" -version = "0.40.1" +version = "0.40.2" description = "lobes — run, assess, and switch the local vLLM model." readme = "README.md" license = "Apache-2.0" From a932d16a6f7b7b4a81ddf3e2eeb8712e6a2333ac Mon Sep 17 00:00:00 2001 From: Ori Nachum Date: Mon, 13 Jul 2026 19:24:52 +0300 Subject: [PATCH 2/7] =?UTF-8?q?plan:=20per-machine=20profiles=20=E2=80=94?= =?UTF-8?q?=2010=20tasks,=204=20file-disjoint=20waves=20(devague=20/spec-t?= =?UTF-8?q?o-plan)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Converged plan for the per-machine-profiles spec. Waves are file-disjoint so they parallelise at merge, not just on paper: w1 t1 profile schema + spark/thor profiles lobes/profiles/ t2 card detection lobes/runtime/_detect.py t3 parameterise the compose templates lobes/templates/ t7 per-role correctness probes lobes/assess.py w2 t4 init applies the resolved profile cli/_commands/init.py t6 role feasibility -> capabilities/gw lobes/gateway/ w3 t5 doctor/status report the profile cli/_commands/{doctor,status}.py t10 validate the Thor profile on the box evidence w4 t11 docs + honest support table docs/ t7 deliberately has no dependency on the profile work: the probes are worth having even if profiles slip, and t7's acceptance criterion is that the rerank probe FAILS on Thor today. A probe that only ever passes is theatre. On Thor the reranker stays served and advertised — it runs, it is just not yet correct. Its ordering probe is a known failure pointing at #105/#106, which is where the fix lands; the role is not hidden to make the plan look green. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01RZNRUXYv84HUQ5ULQ5qJgz --- .devague/current_plan | 2 +- ...e-machine-it-lands-on-one-command-det.json | 397 ++++++++++++++++++ CHANGELOG.md | 21 + ...the-machine-it-lands-on-one-command-det.md | 99 +++++ pyproject.toml | 2 +- 5 files changed, 519 insertions(+), 2 deletions(-) create mode 100644 .devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json create mode 100644 docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md diff --git a/.devague/current_plan b/.devague/current_plan index 2818b4b..138e621 100644 --- a/.devague/current_plan +++ b/.devague/current_plan @@ -1 +1 @@ -lobes-gateway-relays-sse-streams-frame-by-frame-th +lobes-fits-the-machine-it-lands-on-one-command-det diff --git a/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json b/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json new file mode 100644 index 0000000..02d4b16 --- /dev/null +++ b/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json @@ -0,0 +1,397 @@ +{ + "slug": "lobes-fits-the-machine-it-lands-on-one-command-det", + "title": "lobes fits the machine it lands on: one command detects the card (GB10, Thor, Orin, Orin Nano Super) and serves a profile tuned for THAT box \u2014 not a lowest-common-denominator config that fits none", + "frame_slug": "lobes-fits-the-machine-it-lands-on-one-command-det", + "schema_version": 2, + "status": "exported", + "created": "2026-07-13T16:09:59Z", + "updated": "2026-07-13T16:24:28Z", + "targets": [ + { + "id": "c1", + "kind": "announcement", + "text": "lobes fits the machine it lands on: one command detects the card (GB10, Thor, Orin, Orin Nano Super) and serves a profile tuned for THAT box \u2014 not a lowest-common-denominator config that fits none" + }, + { + "id": "h6", + "kind": "honesty", + "text": "One command does it: on a supported card, 'lobes init' picks the right profile with no flags, and 'lobes doctor' names which profile it picked and why (detected card + capability + memory)." + }, + { + "id": "c2", + "kind": "before_state", + "text": "The packaged templates hardcode one tuning for the DGX Spark GB10 (sm_121). It does not transfer: on Jetson AGX Thor (sm_110) cortex needed --kv-cache-dtype=auto, embedder needed --attention-config TRITON_ATTN, reranker needed --enforce-eager and STILL returns wrong rankings. Every one of these was found by crashing into it." + }, + { + "id": "h7", + "kind": "honesty", + "text": "The three Thor divergences found by hand this session (kv-cache dtype auto, attention backend TRITON_ATTN on the pooling gears, enforce-eager on rerank) are exactly what a Thor profile encodes \u2014 the profile reproduces the working fleet without hand-editing." + }, + { + "id": "c3", + "kind": "why_it_matters", + "text": "A single 'best for all machines' config fits none of them: it either wastes a big box's headroom or fails to boot on a small one." + }, + { + "id": "h8", + "kind": "honesty", + "text": "The 'fits none' cost is real and measurable: a single shared config either leaves a big box's headroom unused or fails to boot a small one \u2014 demonstrable by pointing the GB10 profile at Thor (it crashes) and a Thor-safe config at the GB10 (it under-uses it)." + }, + { + "id": "c4", + "kind": "audience", + "text": "Operators running lobes on a specific NVIDIA box \u2014 today DGX Spark GB10 and Jetson AGX Thor; next Jetson AGX Orin and Orin Nano Super \u2014 plus the lobes agent deployed on it." + }, + { + "id": "h9", + "kind": "honesty", + "text": "The operator never has to know the card's compute capability to get a correct boot \u2014 the tool knows it; the operator only overrides when they WANT to." + }, + { + "id": "c5", + "kind": "after_state", + "text": "lobes detects the host card and applies a per-machine profile (memory budget, quantization, attention backend, KV dtype, context, eager-vs-cudagraph, which roles are even feasible), so a supported box boots correct-and-tuned on first try." + }, + { + "id": "h10", + "kind": "honesty", + "text": "'Correct on first try' is checked, not asserted: every role the profile claims passes its correctness probe on a clean boot, and a role the card cannot serve is absent from capabilities rather than broken-but-advertised." + }, + { + "id": "c6", + "kind": "boundary", + "text": "Not a universal autotuner and not a benchmark-search: a profile is a hand-validated, checked-in config for a named card, not something lobes discovers at runtime by trying settings." + }, + { + "id": "h11", + "kind": "honesty", + "text": "Every knob in a profile traces to a human-validated observation on a real card (a boot log, a probe result, a measured budget) \u2014 no value is there because a search loop landed on it, and lobes never mutates a profile at runtime." + }, + { + "id": "c7", + "kind": "success_signal", + "text": "On a supported box, a clean 'lobes init && lobes serve' brings every role it claims up correct on the FIRST try \u2014 no hand-editing of docker-compose.yml. On an unsupported box lobes says so instead of silently mis-serving." + }, + { + "id": "h5", + "kind": "honesty", + "text": "A profile can be proven on a real box: booting a supported profile from a clean ~/.lobes needs zero hand-edits to the generated compose file." + }, + { + "id": "c8", + "kind": "requirement", + "text": "lobes detects the host card (compute capability + total memory + device name) and resolves a named profile; the operator can override the choice explicitly (--profile) and see what was chosen (lobes doctor / lobes status)." + }, + { + "id": "h1", + "kind": "honesty", + "text": "Detection is unambiguous on every target: GB10, Thor, AGX Orin and Orin Nano Super are each distinguishable from inside the container/host without guessing. (Note: nvidia-smi reports memory.used=[N/A] on Thor's integrated GPU, so detection cannot rely on nvidia-smi memory fields.)" + }, + { + "id": "c9", + "kind": "requirement", + "text": "A profile owns the machine-dependent knobs, not just memory: GPU mem util per role, max-model-len, quantization, KV-cache dtype, attention backend, enforce-eager, and max-num-seqs." + }, + { + "id": "h2", + "kind": "honesty", + "text": "The knob set is sufficient: every divergence found on Thor so far (kv-cache dtype, attention backend, enforce-eager) is expressible as a profile value \u2014 no code fork per card." + }, + { + "id": "c10", + "kind": "requirement", + "text": "A profile also declares which ROLES are feasible on that card and which model serves each. On Orin Nano Super the 27B NVFP4 cortex is not runnable at all, so the profile must select a smaller cortex model or declare cortex unavailable \u2014 capabilities must never advertise a role the box cannot serve." + }, + { + "id": "h3", + "kind": "honesty", + "text": "A role can be declared unavailable end-to-end: lobes capabilities / GET /capabilities omit it (or mark it unserved), and the gateway 4xx's a request for it rather than silently routing to a different model." + }, + { + "id": "c11", + "kind": "requirement", + "text": "Every role ships a CORRECTNESS probe, not just a /health check: embed must rank a paraphrase above an unrelated string, rerank must put the relevant document first, cortex must answer a known-answer question. A role that is 'healthy' but semantically wrong must fail the probe." + }, + { + "id": "h4", + "kind": "honesty", + "text": "The rerank probe FAILS today on this Thor box (relevant doc does not rank first) \u2014 i.e. the probe catches the bug we actually hit, rather than being a test that only passes." + }, + { + "id": "c15", + "kind": "boundary", + "text": "Only cards we have ACTUALLY BOOTED ship as supported profiles: DGX Spark GB10 (the default) and Jetson AGX Thor (validated this session). Jetson AGX Orin and Orin Nano Super are named future targets \u2014 the design must accommodate them, but lobes does not promise them until someone boots one." + }, + { + "id": "h12", + "kind": "honesty", + "text": "lobes is honest about coverage: it ships Spark (default) + Thor as supported, names Orin/Orin Nano Super as unvalidated, and on an unrecognised card it refuses-or-warns rather than silently applying the Spark profile." + }, + { + "id": "c18", + "kind": "success_signal", + "text": "Measurable: on each supported card, a clean 'lobes init && lobes serve' from an empty deployment dir brings up 100% of the roles the profile claims, each passing its correctness probe, with ZERO hand-edits to the generated compose. Measured baseline (Thor, this session, stock Spark-tuned template): 1 of 4 roles correct on first boot \u2014 senses came up clean; cortex crash-looped on an fp8-KV assert; embedder accepted requests and never answered; reranker killed its own engine with cudaErrorLaunchFailure. Reaching 3 of 4 took 4 hand-edits to docker-compose.yml. Target for the Thor profile: 4/4 with 0 edits (the 4th, rerank, is gated on #105/#106)." + }, + { + "id": "h14", + "kind": "honesty", + "text": "The baseline is reproducible: a clean 'lobes init' + 'lobes serve' on Thor with the stock template reproduces 1/4-correct-on-first-boot, so 4/4-with-0-edits is a measured improvement, not a claim." + } + ], + "tasks": [ + { + "id": "t1", + "summary": "Profile schema + built-in profiles (lobes/profiles/): a Profile declares, per role, whether it is feasible, which model serves it, and every machine knob (gpu util, max-model-len, quantization, kv-cache dtype, attention backend, enforce-eager, max-num-seqs). Ships spark.yaml (default, from today's template values) and thor.yaml (from the 4 edits validated this session). Loader resolves built-ins + operator-defined profiles and an explicit --profile override.", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "A Profile round-trips: load -> serialise -> identical; an unknown role or unknown knob is a load error, not a silent drop", + "thor.yaml encodes exactly the 4 validated Thor divergences (cortex kv_cache_dtype=auto; embedder+reranker attention_backend=TRITON_ATTN; reranker enforce_eager=true) and spark.yaml reproduces today's shipped template values byte-for-byte", + "An operator-defined profile in the deployment dir is discovered and can override a built-in of the same name; nothing in the loader mutates a profile at runtime" + ], + "deps": [], + "covers": [ + "c6", + "h11", + "c9", + "c10" + ], + "instruction": "OWNS: lobes/profiles/ (new package: schema.py, loader, builtin/spark.yaml, builtin/thor.yaml) + tests/test_profiles.py. Touch NOTHING else. thor.yaml must encode the 4 divergences validated live this session: cortex kv_cache_dtype=auto; embedder attention_backend=TRITON_ATTN; reranker attention_backend=TRITON_ATTN + enforce_eager=true. spark.yaml must reproduce the CURRENT template values exactly (util 0.30/0.14/0.06/0.06, cortex 131072, senses 32768) so GB10 behaviour is unchanged." + }, + { + "id": "t2", + "summary": "Card detection (lobes/runtime/_detect.py): identify the host card by compute capability + device name + total memory, returning a named card id (gb10-spark, jetson-agx-thor, ...) or UNKNOWN. Must NOT read nvidia-smi memory fields \u2014 they report [N/A] on Thor's integrated GPU.", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "Detection distinguishes GB10 (sm_121) from Thor (sm_110) on real hardware, and returns UNKNOWN rather than guessing for a card it does not know", + "Detection never consults nvidia-smi memory.used/memory.total; a test asserts the code path does not depend on those fields (they are [N/A] on Thor)", + "Detection works without importing torch (the CLI has no torch dependency)" + ], + "deps": [], + "covers": [ + "c8", + "h1", + "h9", + "c4" + ], + "instruction": "OWNS: lobes/runtime/_detect.py (new) + tests. Touch NOTHING else. Detect via device name + compute capability + total memory. HARD CONSTRAINT: nvidia-smi reports memory.used=[N/A] on Thor's integrated GPU \u2014 do not read nvidia-smi memory fields. /proc/device-tree/model yields 'NVIDIA Jetson AGX Thor Developer Kit' on this box; 'nvidia-smi --query-gpu=name,compute_cap' yields 'NVIDIA Thor'. No torch import (the CLI has no torch dep). UNKNOWN is a first-class result, never a silent Spark fallback." + }, + { + "id": "t3", + "summary": "Parameterise the compose templates so every machine-dependent knob flows from the profile via .env: per-gear kv-cache-dtype, attention-config, enforce-eager, gpu util, max-model-len, quantization, max-num-seqs, and the model per role. Includes deleting the dead MULTIMODAL_ATTENTION_BACKEND env (VLLM_ATTENTION_BACKEND no longer exists in the pinned nightly; --attention-config is the only honoured knob).", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "Every one of the 4 Thor divergences is expressible as a profile value with NO code fork per card \u2014 proven by rendering thor.yaml and diffing against the hand-edited ~/.lobes/docker-compose.yml that works today", + "Rendering spark.yaml produces the current shipped compose (no behaviour change for existing GB10 deployments)", + "The dead VLLM_ATTENTION_BACKEND/MULTIMODAL_ATTENTION_BACKEND env is gone; the attention backend is set via --attention-config only" + ], + "deps": [], + "covers": [ + "c9", + "h2", + "c2", + "h7" + ], + "instruction": "OWNS: lobes/templates/ (fleet/docker-compose.yml, fleet/env.example, docker-compose.yml, env.example). Touch NO python. Ground truth for the Thor knobs is the working hand-edited /home/thor/.lobes/docker-compose.yml on this box \u2014 diff against it. Note --attention-config takes JSON and MUST be single-quoted in YAML ('--attention-config={\"backend\": \"TRITON_ATTN\"}') or compose fails to parse. Delete the dead MULTIMODAL_ATTENTION_BACKEND env: VLLM_ATTENTION_BACKEND no longer exists in the pinned nightly (0.23.1rc1.dev672)." + }, + { + "id": "t4", + "summary": "lobes init applies the resolved profile: detect card -> pick profile -> render .env + compose. Adds --profile to override, and REFUSES-OR-WARNS on an unrecognised card rather than silently applying the Spark profile.", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "On a supported card, a bare 'lobes init' picks the right profile with no flags and needs zero hand-edits to the generated compose", + "'lobes init --profile ' overrides detection, including forcing a profile onto a card it was not validated for (with a warning)", + "On an UNKNOWN card, init refuses (or warns loudly and requires an explicit --profile) \u2014 it never silently falls back to Spark" + ], + "deps": [ + "t1", + "t2", + "t3" + ], + "covers": [ + "c1", + "h6", + "c5", + "c15", + "h12" + ], + "instruction": "OWNS: lobes/cli/_commands/init.py + the profile-resolution glue in lobes/cli/_runtime_ops.py. Coordinate with t6 if it also needs _runtime_ops. Keep the existing dry-run-by-default mutation-safety contract: init still requires --apply." + }, + { + "id": "t5", + "summary": "lobes doctor / lobes status report the profile: which profile was chosen, which card was detected (name + compute capability + memory), and whether the profile is validated-on-this-card or forced.", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "'lobes doctor' names the detected card and the chosen profile, and flags a forced/unvalidated combination as a warning", + "The operator never needs to know the card's compute capability to get a correct boot \u2014 it is reported to them, not required from them" + ], + "deps": [ + "t4" + ], + "covers": [ + "c8", + "h9" + ], + "instruction": "OWNS: lobes/cli/_commands/doctor.py + status.py. Depends on t4's resolution glue \u2014 do not re-implement detection, call it." + }, + { + "id": "t6", + "summary": "Role feasibility is honoured end-to-end: a role the profile declares unavailable is omitted (or marked unserved) in 'lobes capabilities' and GET /capabilities, and the gateway 4xx's a request for it instead of silently routing it to a different model.", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "A profile with cortex unavailable produces capabilities that do NOT advertise cortex, and a POST with model=cortex returns 4xx rather than being served by another gear", + "No role is ever advertised that the card cannot serve (extends the #92 'never advertise what you cannot serve' invariant to the hardware dimension)" + ], + "deps": [ + "t1" + ], + "covers": [ + "c10", + "h3" + ], + "instruction": "OWNS: lobes/gateway/ + the capabilities command. This extends the existing #92 invariant ('lobes never advertises a capability it cannot serve') to the HARDWARE dimension \u2014 reuse that machinery rather than inventing a parallel path." + }, + { + "id": "t7", + "summary": "Per-role CORRECTNESS probes (not just /health): cortex answers a known-answer question; embedder ranks a paraphrase above an unrelated string; reranker puts the relevant document first. A role that is healthy but semantically wrong must FAIL.", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "The rerank ordering probe FAILS on this Thor box today (the relevant doc does not rank first) \u2014 the probe catches the real bug rather than being a test that only passes", + "The embed probe passes on Thor today (cos(paraphrase) > cos(unrelated)) and would have caught the FLASH_ATTN hang, which /health did not", + "Probes are wired into a read-only verb and never mutate the deployment" + ], + "deps": [], + "covers": [ + "c11", + "h4", + "h10" + ], + "instruction": "OWNS: lobes/assess.py (or a new lobes/probes.py) + lobes/cli/_commands/assess.py. Read-only verb; must not mutate the deployment. The three probes are known-answer generate, embed paraphrase-beats-unrelated, and rerank relevant-doc-ranks-first. On this Thor box today the rerank probe MUST FAIL and the embed probe MUST PASS \u2014 that is the acceptance test for the probes themselves." + }, + { + "id": "t8", + "summary": "Validate the Thor profile on the physical Thor: from an EMPTY deployment dir, 'lobes init && lobes serve' with zero hand-edits, then run every correctness probe and record the score.", + "origin": "llm", + "status": "rejected", + "acceptance_criteria": [ + "Clean init+serve on Thor brings up 100% of the roles the thor profile CLAIMS, with zero hand-edits to the generated compose (rerank is either fixed per #105 or declared unavailable in the profile \u2014 it is never advertised-but-broken)", + "The 1-of-4-on-first-boot baseline is reproduced with the stock Spark template on Thor, so the improvement is measured rather than asserted", + "Pointing the Spark profile at Thor still crashes (demonstrating the 'fits none' cost is real), and the Thor profile does not under-use the GB10 by being applied to it" + ], + "deps": [ + "t4", + "t6", + "t7" + ], + "covers": [ + "c7", + "h5", + "c18", + "h14", + "c3", + "h8" + ], + "instruction": "Validation, run ON the Thor box. Move the current hand-edited ~/.lobes aside first (it carries 4 edits that would mask a template regression), then init from empty. Record the score as evidence. Blocked on #106 for whether rerank can be claimed at all." + }, + { + "id": "t9", + "summary": "Document the profile surface: docs/machine-profiles.md (how detection resolves, how to write a profile, the knob reference), plus 'lobes explain profiles', and an honest support table \u2014 Spark (default, validated) + Thor (validated); Orin / Orin Nano Super named but UNVALIDATED.", + "origin": "llm", + "status": "rejected", + "acceptance_criteria": [ + "The support table states plainly which cards are validated and which are aspirational; no doc claims Orin support", + "Every knob in the reference traces to a human-validated observation on a real card (boot log, probe result, measured budget)" + ], + "deps": [ + "t8", + "t10" + ], + "covers": [ + "c15", + "h12", + "c6", + "h11" + ], + "instruction": "OWNS: docs/machine-profiles.md (new), lobes/explain/, README/CLAUDE.md support table. Docs only, no source. Do not claim Orin support." + }, + { + "id": "t10", + "summary": "Validate the Thor profile on the physical Thor: from an EMPTY deployment dir, 'lobes init && lobes serve' with zero hand-edits, then run every correctness probe and record the score. The reranker RUNS (serves requests without killing its engine) but is KNOWN-INCORRECT \u2014 its ordering probe is an expected failure tracked in #105/#106, which is where the fix lands. The profile neither hides that nor blocks on it.", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "Clean init+serve on Thor brings up 100% of the roles the thor profile claims \u2014 including reranker, which must SERVE (respond, no cudaErrorLaunchFailure) \u2014 with zero hand-edits to the generated compose", + "The rerank ordering probe is recorded as a KNOWN/EXPECTED failure referencing #105/#106: it neither passes silently nor blocks the profile work, and it flips to a hard failure once those issues close", + "The 1-of-4-on-first-boot baseline is reproduced with the stock Spark template on Thor, so the improvement is measured rather than asserted", + "Pointing the Spark profile at Thor still crashes (the 'fits none' cost is real), and the Thor profile is not applied to the GB10 where it would under-use it" + ], + "deps": [ + "t4", + "t6", + "t7" + ], + "covers": [ + "c7", + "h5", + "c18", + "h14", + "c3", + "h8" + ], + "instruction": "Validation, run ON the Thor box. Move the current hand-edited ~/.lobes aside first (its 4 edits would mask a template regression), then init from empty. rerank STAYS SERVED: eager mode stops the engine-killing cudaErrorLaunchFailure, so it responds \u2014 it is simply not yet correct. Record the probe scores as evidence." + }, + { + "id": "t11", + "summary": "Document the profile surface: docs/machine-profiles.md (how detection resolves, how to write a profile, the knob reference), plus 'lobes explain profiles', and an honest support table \u2014 Spark (default, validated) + Thor (validated); Orin / Orin Nano Super named but UNVALIDATED. Thor's reranker is documented as served-but-known-incorrect (#105/#106), not quietly omitted.", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "The support table states plainly which cards are validated and which are aspirational; no doc claims Orin support", + "Every knob in the reference traces to a human-validated observation on a real card (boot log, probe result, measured budget)", + "Thor's known-incorrect reranker is documented with a pointer to #105/#106 rather than being hidden" + ], + "deps": [ + "t10" + ], + "covers": [ + "c15", + "h12", + "c6", + "h11" + ], + "instruction": "OWNS: docs/machine-profiles.md (new), lobes/explain/, README/CLAUDE.md support table. Docs only, no source. Do not claim Orin support. Be explicit that Thor's rerank runs but is not yet correct." + } + ], + "risks": [ + { + "id": "r1", + "text": "Reranker returns deterministically wrong orderings on Thor and MAY be wrong on the GB10 too (nothing checks ordering today). Tracked in lobes-cli#105 / #106. Until #106 answers, t8 cannot claim 4/4 on Thor \u2014 the thor profile must either fix rerank or declare it unavailable rather than advertise it broken.", + "kind": "follow_up", + "task_id": "t8" + }, + { + "id": "r2", + "text": "Jetson AGX Orin (sm_87, no NVFP4) and Orin Nano Super are UNVALIDATED \u2014 no physical board in hand. The schema must accommodate them (smaller cortex model, roles disabled) but no profile ships as supported until someone boots one.", + "kind": "unknown_nonblocking", + "task_id": null + }, + { + "id": "r3", + "text": "Detection runs on the HOST (the lobes CLI), while the kernels that actually differ run INSIDE the container. If the two ever disagree about the card, the profile is resolved against the wrong truth.", + "kind": "unknown_nonblocking", + "task_id": "t2" + }, + { + "id": "r4", + "text": "DECISION (supersedes the framing of r1): on Thor the reranker stays SERVED and advertised \u2014 it runs, it is just not yet correct. Correctness is fixed in lobes-cli#105/#106, not by hiding the role. The correctness probe records it as a known failure until then.", + "kind": "follow_up", + "task_id": "t10" + } + ] +} diff --git a/CHANGELOG.md b/CHANGELOG.md index 23479c7..5b50b8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,27 @@ All notable changes to this project are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.40.3] - 2026-07-13 + +### Added + +- Plan (`/spec-to-plan`, converged): **per-machine hardware profiles** — + `docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md`. + Ten tasks over four dependency waves, each wave file-disjoint so it can be + built in parallel: profile schema + spark/thor profiles, card detection, + template parameterisation, and per-role correctness probes (wave 1); `init` + applies the profile and role-feasibility reaches capabilities/gateway (wave 2); + `doctor`/`status` report the profile, and the Thor profile is validated on the + physical board (wave 3); docs + an honest support table (wave 4). + + On Thor the reranker stays **served and advertised** — it runs, it is simply + not yet correct; its ordering probe is recorded as a known failure pointing at + #105 / #106, rather than the role being hidden. + +### Changed + +### Fixed + ## [0.40.2] - 2026-07-13 ### Added diff --git a/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md b/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md new file mode 100644 index 0000000..1fb1625 --- /dev/null +++ b/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md @@ -0,0 +1,99 @@ +# Build Plan — lobes fits the machine it lands on: one command detects the card (GB10, Thor, Orin, Orin Nano Super) and serves a profile tuned for THAT box — not a lowest-common-denominator config that fits none + +slug: `lobes-fits-the-machine-it-lands-on-one-command-det` · status: `exported` · from frame: `lobes-fits-the-machine-it-lands-on-one-command-det` + +> lobes fits the machine it lands on: one command detects the card (GB10, Thor, Orin, Orin Nano Super) and serves a profile tuned for THAT box — not a lowest-common-denominator config that fits none + +## Tasks + +### t1 — Profile schema + built-in profiles (lobes/profiles/): a Profile declares, per role, whether it is feasible, which model serves it, and every machine knob (gpu util, max-model-len, quantization, kv-cache dtype, attention backend, enforce-eager, max-num-seqs). Ships spark.yaml (default, from today's template values) and thor.yaml (from the 4 edits validated this session). Loader resolves built-ins + operator-defined profiles and an explicit --profile override + +- instruction: OWNS: lobes/profiles/ (new package: schema.py, loader, builtin/spark.yaml, builtin/thor.yaml) + tests/test_profiles.py. Touch NOTHING else. thor.yaml must encode the 4 divergences validated live this session: cortex kv_cache_dtype=auto; embedder attention_backend=TRITON_ATTN; reranker attention_backend=TRITON_ATTN + enforce_eager=true. spark.yaml must reproduce the CURRENT template values exactly (util 0.30/0.14/0.06/0.06, cortex 131072, senses 32768) so GB10 behaviour is unchanged. +- covers: c6, h11, c9, c10 +- acceptance: + - A Profile round-trips: load -> serialise -> identical; an unknown role or unknown knob is a load error, not a silent drop + - thor.yaml encodes exactly the 4 validated Thor divergences (cortex kv_cache_dtype=auto; embedder+reranker attention_backend=TRITON_ATTN; reranker enforce_eager=true) and spark.yaml reproduces today's shipped template values byte-for-byte + - An operator-defined profile in the deployment dir is discovered and can override a built-in of the same name; nothing in the loader mutates a profile at runtime + +### t2 — Card detection (lobes/runtime/_detect.py): identify the host card by compute capability + device name + total memory, returning a named card id (gb10-spark, jetson-agx-thor, ...) or UNKNOWN. Must NOT read nvidia-smi memory fields — they report [N/A] on Thor's integrated GPU + +- instruction: OWNS: lobes/runtime/_detect.py (new) + tests. Touch NOTHING else. Detect via device name + compute capability + total memory. HARD CONSTRAINT: nvidia-smi reports memory.used=[N/A] on Thor's integrated GPU — do not read nvidia-smi memory fields. /proc/device-tree/model yields 'NVIDIA Jetson AGX Thor Developer Kit' on this box; 'nvidia-smi --query-gpu=name,compute_cap' yields 'NVIDIA Thor'. No torch import (the CLI has no torch dep). UNKNOWN is a first-class result, never a silent Spark fallback. +- covers: c8, h1, h9, c4 +- acceptance: + - Detection distinguishes GB10 (sm_121) from Thor (sm_110) on real hardware, and returns UNKNOWN rather than guessing for a card it does not know + - Detection never consults nvidia-smi memory.used/memory.total; a test asserts the code path does not depend on those fields (they are [N/A] on Thor) + - Detection works without importing torch (the CLI has no torch dependency) + +### t3 — Parameterise the compose templates so every machine-dependent knob flows from the profile via .env: per-gear kv-cache-dtype, attention-config, enforce-eager, gpu util, max-model-len, quantization, max-num-seqs, and the model per role. Includes deleting the dead MULTIMODAL_ATTENTION_BACKEND env (VLLM_ATTENTION_BACKEND no longer exists in the pinned nightly; --attention-config is the only honoured knob) + +- instruction: OWNS: lobes/templates/ (fleet/docker-compose.yml, fleet/env.example, docker-compose.yml, env.example). Touch NO python. Ground truth for the Thor knobs is the working hand-edited /home/thor/.lobes/docker-compose.yml on this box — diff against it. Note --attention-config takes JSON and MUST be single-quoted in YAML ('--attention-config={"backend": "TRITON_ATTN"}') or compose fails to parse. Delete the dead MULTIMODAL_ATTENTION_BACKEND env: VLLM_ATTENTION_BACKEND no longer exists in the pinned nightly (0.23.1rc1.dev672). +- covers: c9, h2, c2, h7 +- acceptance: + - Every one of the 4 Thor divergences is expressible as a profile value with NO code fork per card — proven by rendering thor.yaml and diffing against the hand-edited ~/.lobes/docker-compose.yml that works today + - Rendering spark.yaml produces the current shipped compose (no behaviour change for existing GB10 deployments) + - The dead VLLM_ATTENTION_BACKEND/MULTIMODAL_ATTENTION_BACKEND env is gone; the attention backend is set via --attention-config only + +### t4 — lobes init applies the resolved profile: detect card -> pick profile -> render .env + compose. Adds --profile to override, and REFUSES-OR-WARNS on an unrecognised card rather than silently applying the Spark profile + +- instruction: OWNS: lobes/cli/_commands/init.py + the profile-resolution glue in lobes/cli/_runtime_ops.py. Coordinate with t6 if it also needs _runtime_ops. Keep the existing dry-run-by-default mutation-safety contract: init still requires --apply. +- depends on: t1, t2, t3 +- covers: c1, h6, c5, c15, h12 +- acceptance: + - On a supported card, a bare 'lobes init' picks the right profile with no flags and needs zero hand-edits to the generated compose + - 'lobes init --profile ' overrides detection, including forcing a profile onto a card it was not validated for (with a warning) + - On an UNKNOWN card, init refuses (or warns loudly and requires an explicit --profile) — it never silently falls back to Spark + +### t5 — lobes doctor / lobes status report the profile: which profile was chosen, which card was detected (name + compute capability + memory), and whether the profile is validated-on-this-card or forced + +- instruction: OWNS: lobes/cli/_commands/doctor.py + status.py. Depends on t4's resolution glue — do not re-implement detection, call it. +- depends on: t4 +- covers: c8, h9 +- acceptance: + - 'lobes doctor' names the detected card and the chosen profile, and flags a forced/unvalidated combination as a warning + - The operator never needs to know the card's compute capability to get a correct boot — it is reported to them, not required from them + +### t6 — Role feasibility is honoured end-to-end: a role the profile declares unavailable is omitted (or marked unserved) in 'lobes capabilities' and GET /capabilities, and the gateway 4xx's a request for it instead of silently routing it to a different model + +- instruction: OWNS: lobes/gateway/ + the capabilities command. This extends the existing #92 invariant ('lobes never advertises a capability it cannot serve') to the HARDWARE dimension — reuse that machinery rather than inventing a parallel path. +- depends on: t1 +- covers: c10, h3 +- acceptance: + - A profile with cortex unavailable produces capabilities that do NOT advertise cortex, and a POST with model=cortex returns 4xx rather than being served by another gear + - No role is ever advertised that the card cannot serve (extends the #92 'never advertise what you cannot serve' invariant to the hardware dimension) + +### t7 — Per-role CORRECTNESS probes (not just /health): cortex answers a known-answer question; embedder ranks a paraphrase above an unrelated string; reranker puts the relevant document first. A role that is healthy but semantically wrong must FAIL + +- instruction: OWNS: lobes/assess.py (or a new lobes/probes.py) + lobes/cli/_commands/assess.py. Read-only verb; must not mutate the deployment. The three probes are known-answer generate, embed paraphrase-beats-unrelated, and rerank relevant-doc-ranks-first. On this Thor box today the rerank probe MUST FAIL and the embed probe MUST PASS — that is the acceptance test for the probes themselves. +- covers: c11, h4, h10 +- acceptance: + - The rerank ordering probe FAILS on this Thor box today (the relevant doc does not rank first) — the probe catches the real bug rather than being a test that only passes + - The embed probe passes on Thor today (cos(paraphrase) > cos(unrelated)) and would have caught the FLASH_ATTN hang, which /health did not + - Probes are wired into a read-only verb and never mutate the deployment + +### t10 — Validate the Thor profile on the physical Thor: from an EMPTY deployment dir, 'lobes init && lobes serve' with zero hand-edits, then run every correctness probe and record the score. The reranker RUNS (serves requests without killing its engine) but is KNOWN-INCORRECT — its ordering probe is an expected failure tracked in #105/#106, which is where the fix lands. The profile neither hides that nor blocks on it + +- instruction: Validation, run ON the Thor box. Move the current hand-edited ~/.lobes aside first (its 4 edits would mask a template regression), then init from empty. rerank STAYS SERVED: eager mode stops the engine-killing cudaErrorLaunchFailure, so it responds — it is simply not yet correct. Record the probe scores as evidence. +- depends on: t4, t6, t7 +- covers: c7, h5, c18, h14, c3, h8 +- acceptance: + - Clean init+serve on Thor brings up 100% of the roles the thor profile claims — including reranker, which must SERVE (respond, no cudaErrorLaunchFailure) — with zero hand-edits to the generated compose + - The rerank ordering probe is recorded as a KNOWN/EXPECTED failure referencing #105/#106: it neither passes silently nor blocks the profile work, and it flips to a hard failure once those issues close + - The 1-of-4-on-first-boot baseline is reproduced with the stock Spark template on Thor, so the improvement is measured rather than asserted + - Pointing the Spark profile at Thor still crashes (the 'fits none' cost is real), and the Thor profile is not applied to the GB10 where it would under-use it + +### t11 — Document the profile surface: docs/machine-profiles.md (how detection resolves, how to write a profile, the knob reference), plus 'lobes explain profiles', and an honest support table — Spark (default, validated) + Thor (validated); Orin / Orin Nano Super named but UNVALIDATED. Thor's reranker is documented as served-but-known-incorrect (#105/#106), not quietly omitted + +- instruction: OWNS: docs/machine-profiles.md (new), lobes/explain/, README/CLAUDE.md support table. Docs only, no source. Do not claim Orin support. Be explicit that Thor's rerank runs but is not yet correct. +- depends on: t10 +- covers: c15, h12, c6, h11 +- acceptance: + - The support table states plainly which cards are validated and which are aspirational; no doc claims Orin support + - Every knob in the reference traces to a human-validated observation on a real card (boot log, probe result, measured budget) + - Thor's known-incorrect reranker is documented with a pointer to #105/#106 rather than being hidden + +## Risks + +- [follow_up] Reranker returns deterministically wrong orderings on Thor and MAY be wrong on the GB10 too (nothing checks ordering today). Tracked in lobes-cli#105 / #106. Until #106 answers, t8 cannot claim 4/4 on Thor — the thor profile must either fix rerank or declare it unavailable rather than advertise it broken. (task t8) +- [unknown_nonblocking] Jetson AGX Orin (sm_87, no NVFP4) and Orin Nano Super are UNVALIDATED — no physical board in hand. The schema must accommodate them (smaller cortex model, roles disabled) but no profile ships as supported until someone boots one. +- [unknown_nonblocking] Detection runs on the HOST (the lobes CLI), while the kernels that actually differ run INSIDE the container. If the two ever disagree about the card, the profile is resolved against the wrong truth. (task t2) +- [follow_up] DECISION (supersedes the framing of r1): on Thor the reranker stays SERVED and advertised — it runs, it is just not yet correct. Correctness is fixed in lobes-cli#105/#106, not by hiding the role. The correctness probe records it as a known failure until then. (task t10) diff --git a/pyproject.toml b/pyproject.toml index 4eefec5..90639cd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "lobes-cli" -version = "0.40.2" +version = "0.40.3" description = "lobes — run, assess, and switch the local vLLM model." readme = "README.md" license = "Apache-2.0" From 38ad914055a1e07090d790903985eff69af8d095 Mon Sep 17 00:00:00 2001 From: Ori Nachum Date: Mon, 13 Jul 2026 20:16:34 +0300 Subject: [PATCH 3/7] spec+plan: correct the before-state, add the per-chip strategy pattern MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections to the converged spec/plan, both from review: 1. The spec's before-state was WRONG. It claimed lobes had no machine-profile concept. It does: lobes/profiles.py already ships MachineProfile / MACHINE_PROFILES (spark/thor/blackwell/generic) and detect_machine(), wired to VLLM_MACHINE and used by switch/benchmark. The real gaps — now stated honestly — are that it is one knob-set per MACHINE not per ROLE; it lacks the knobs that actually mattered on Thor (kv-cache dtype, enforce-eager, model-per-role, role feasibility); the FLEET compose (the default path) ignores it and hardcodes the Spark values; its thor row is an unvalidated guess (status="configured": flashinfer / 32768 / util 0.6) that live Thor testing contradicts; and detect_machine() silently falls back to "generic" rather than admitting it does not know the card. 2. Per-chip knowledge goes behind a STRATEGY PATTERN (new task t12, wave 1, foundational). One module per chip under lobes/machines/ owning its own detection signature, per-role knobs and provenance, plus a small shared registry. Adding a chip = one new file + one registration line — never an edit to a shared table, and a change for one chip cannot break another. Nothing is deleted: the legacy API keeps working, rebuilt FROM the registry. The honesty condition on t12 is that extensibility is PROVEN, not asserted: a test registers a synthetic chip and shows detection, profile resolution and knob rendering all pick it up with zero edits to profiles.py / _detect.py / init.py, with every pre-existing test passing unmodified. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01RZNRUXYv84HUQ5ULQ5qJgz --- ...e-machine-it-lands-on-one-command-det.json | 40 ++++++++++++++++++- ...e-machine-it-lands-on-one-command-det.json | 39 +++++++++++++----- CHANGELOG.md | 34 +++++++++++++--- ...the-machine-it-lands-on-one-command-det.md | 9 +++++ ...the-machine-it-lands-on-one-command-det.md | 6 ++- 5 files changed, 107 insertions(+), 21 deletions(-) diff --git a/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json b/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json index 1f3f11e..a4f3f3d 100644 --- a/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json +++ b/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json @@ -4,7 +4,7 @@ "schema_version": 2, "status": "exported", "created": "2026-07-13T15:33:49Z", - "updated": "2026-07-13T16:01:29Z", + "updated": "2026-07-13T17:15:13Z", "claims": [ { "id": "c1", @@ -29,7 +29,7 @@ "kind": "before_state", "text": "The packaged templates hardcode one tuning for the DGX Spark GB10 (sm_121). It does not transfer: on Jetson AGX Thor (sm_110) cortex needed --kv-cache-dtype=auto, embedder needed --attention-config TRITON_ATTN, reranker needed --enforce-eager and STILL returns wrong rankings. Every one of these was found by crashing into it.", "origin": "user", - "status": "confirmed", + "status": "rejected", "honesty_conditions": [ { "id": "h7", @@ -308,6 +308,42 @@ "hard_questions": [], "links": [], "instruction": "" + }, + { + "id": "c19", + "kind": "before_state", + "text": "lobes ALREADY has a machine-profile axis \u2014 lobes/profiles.py ships MachineProfile + MACHINE_PROFILES (spark/thor/blackwell/generic) and detect_machine(), wired to VLLM_MACHINE and used by switch/benchmark. But it is (a) ONE knob-set per machine, not per role; (b) missing the knobs that actually matter \u2014 kv-cache dtype, enforce-eager, model-per-role, role feasibility; (c) IGNORED by the fleet compose (the default path), which hardcodes DGX Spark GB10 values; (d) its thor row is an unvalidated GUESS (status 'configured': flashinfer / 32768 / util 0.6) that live testing on a physical Thor CONTRADICTS; and (e) detect_machine() silently falls back to 'generic' instead of admitting it does not know the card. On Thor (sm_110) the shipped fleet scored 1 of 4 roles correct on first boot; cortex needed --kv-cache-dtype=auto, embedder needed --attention-config TRITON_ATTN, reranker needed --enforce-eager and STILL returns wrong rankings.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h15", + "text": "The claim is checkable in the code as it stands: lobes/profiles.py really does define MachineProfile/MACHINE_PROFILES/detect_machine (grep it), the fleet compose really does hardcode the Spark values rather than read them, the thor row really is status='configured' with attention_backend='flashinfer', and detect_machine really does fall back to 'generic'. If any of those five is false, this before_state is wrong and the spec must be re-cut.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c20", + "kind": "decision", + "text": "ARCHITECTURE: per-chip knowledge lives behind a STRATEGY PATTERN \u2014 one module per chip (lobes/machines/.py) owning its own detection signature, per-role knobs and provenance, plus a small shared registry. Adding a chip = one new file + one registration line; it must NOT mean editing shared tables in profiles.py / _detect.py / init.py, and a change for one chip must not be able to break another. Old code is NOT deleted: MachineProfile, MACHINE_PROFILES, detect_machine() and their switch/benchmark callers keep working, rebuilt FROM the registry rather than duplicated.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h16", + "text": "Extensibility is PROVEN, not asserted: a test registers a synthetic new chip strategy and shows detection, profile resolution and knob rendering all pick it up with ZERO edits to profiles.py / _detect.py / init.py \u2014 and every pre-existing test still passes UNMODIFIED (if an existing test's expectations must change, legacy behaviour was broken and the refactor has failed).", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" } ], "open_vagueness": [ diff --git a/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json b/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json index 02d4b16..293fdb9 100644 --- a/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json +++ b/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json @@ -5,7 +5,7 @@ "schema_version": 2, "status": "exported", "created": "2026-07-13T16:09:59Z", - "updated": "2026-07-13T16:24:28Z", + "updated": "2026-07-13T17:15:49Z", "targets": [ { "id": "c1", @@ -17,16 +17,6 @@ "kind": "honesty", "text": "One command does it: on a supported card, 'lobes init' picks the right profile with no flags, and 'lobes doctor' names which profile it picked and why (detected card + capability + memory)." }, - { - "id": "c2", - "kind": "before_state", - "text": "The packaged templates hardcode one tuning for the DGX Spark GB10 (sm_121). It does not transfer: on Jetson AGX Thor (sm_110) cortex needed --kv-cache-dtype=auto, embedder needed --attention-config TRITON_ATTN, reranker needed --enforce-eager and STILL returns wrong rankings. Every one of these was found by crashing into it." - }, - { - "id": "h7", - "kind": "honesty", - "text": "The three Thor divergences found by hand this session (kv-cache dtype auto, attention backend TRITON_ATTN on the pooling gears, enforce-eager on rerank) are exactly what a Thor profile encodes \u2014 the profile reproduces the working fleet without hand-editing." - }, { "id": "c3", "kind": "why_it_matters", @@ -136,6 +126,16 @@ "id": "h14", "kind": "honesty", "text": "The baseline is reproducible: a clean 'lobes init' + 'lobes serve' on Thor with the stock template reproduces 1/4-correct-on-first-boot, so 4/4-with-0-edits is a measured improvement, not a claim." + }, + { + "id": "c19", + "kind": "before_state", + "text": "lobes ALREADY has a machine-profile axis \u2014 lobes/profiles.py ships MachineProfile + MACHINE_PROFILES (spark/thor/blackwell/generic) and detect_machine(), wired to VLLM_MACHINE and used by switch/benchmark. But it is (a) ONE knob-set per machine, not per role; (b) missing the knobs that actually matter \u2014 kv-cache dtype, enforce-eager, model-per-role, role feasibility; (c) IGNORED by the fleet compose (the default path), which hardcodes DGX Spark GB10 values; (d) its thor row is an unvalidated GUESS (status 'configured': flashinfer / 32768 / util 0.6) that live testing on a physical Thor CONTRADICTS; and (e) detect_machine() silently falls back to 'generic' instead of admitting it does not know the card. On Thor (sm_110) the shipped fleet scored 1 of 4 roles correct on first boot; cortex needed --kv-cache-dtype=auto, embedder needed --attention-config TRITON_ATTN, reranker needed --enforce-eager and STILL returns wrong rankings." + }, + { + "id": "h15", + "kind": "honesty", + "text": "The claim is checkable in the code as it stands: lobes/profiles.py really does define MachineProfile/MACHINE_PROFILES/detect_machine (grep it), the fleet compose really does hardcode the Spark values rather than read them, the thor row really is status='configured' with attention_backend='flashinfer', and detect_machine really does fall back to 'generic'. If any of those five is false, this before_state is wrong and the spec must be re-cut." } ], "tasks": [ @@ -366,6 +366,23 @@ "h11" ], "instruction": "OWNS: docs/machine-profiles.md (new), lobes/explain/, README/CLAUDE.md support table. Docs only, no source. Do not claim Orin support. Be explicit that Thor's rerank runs but is not yet correct." + }, + { + "id": "t12", + "summary": "Per-chip STRATEGY PATTERN (foundational \u2014 do this before the other tasks build on top of the current shape): one module per chip under lobes/machines/ (CardStrategy: its own detection signature + per-role knobs + provenance) plus a small shared registry. profiles.py / _detect.py / init.py stop carrying per-chip tables and derive from the registry instead. Nothing is deleted: MachineProfile, MACHINE_PROFILES, detect_machine() and their switch/benchmark callers keep working, rebuilt FROM the registry. Also re-cuts the false premise the spec originally carried (that lobes had no machine-profile axis at all \u2014 it does).", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "Adding a new chip = ONE new file + ONE registration line, proven by a test that registers a synthetic chip strategy and shows detection, profile resolution and knob rendering all pick it up with ZERO edits to profiles.py / _detect.py / init.py", + "Every pre-existing test passes UNMODIFIED \u2014 if an existing test's expectations must change, legacy behaviour was broken and the refactor has failed", + "The existing thor row (status='configured': flashinfer / 32768 / util 0.6 \u2014 an unvalidated guess that live Thor testing contradicts) is replaced with the measured values and marked load-tested; detect_machine()'s silent 'generic' fallback is preserved for its legacy callers but is NOT the source of truth for init" + ], + "deps": [], + "covers": [ + "c19", + "h15" + ], + "instruction": "OWNS: lobes/machines/ (new), and the registry-derivation edits to lobes/profiles.py + lobes/runtime/_detect.py. Foundational: land this BEFORE t1/t2/t4 build per-chip tables that would then have to be unpicked. Do NOT delete the legacy API (MachineProfile / MACHINE_PROFILES / detect_machine / resolve_serve_config) \u2014 derive it from the registry. stdlib only (dependencies = []); an explicit registry with imports in machines/__init__.py, no plugin/entry-point machinery." } ], "risks": [ diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b50b8f..f92a2dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,17 +10,39 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Plan (`/spec-to-plan`, converged): **per-machine hardware profiles** — `docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md`. - Ten tasks over four dependency waves, each wave file-disjoint so it can be - built in parallel: profile schema + spark/thor profiles, card detection, - template parameterisation, and per-role correctness probes (wave 1); `init` - applies the profile and role-feasibility reaches capabilities/gateway (wave 2); + Eleven tasks over four file-disjoint dependency waves: the **per-chip strategy + pattern** + profile schema + spark/thor profiles, card detection, template + parameterisation and per-role correctness probes (wave 1); `init` applies the + profile and role-feasibility reaches capabilities/gateway (wave 2); `doctor`/`status` report the profile, and the Thor profile is validated on the physical board (wave 3); docs + an honest support table (wave 4). + Per-chip knowledge goes behind a **strategy pattern** — one module per chip + (`lobes/machines/.py`) owning its own detection signature, per-role knobs + and provenance, plus a small shared registry. Adding a chip is one new file and + one registration line; it must not mean editing shared tables, and a change for + one chip must not be able to break another. No existing code is deleted: + `MachineProfile`, `MACHINE_PROFILES` and `detect_machine()` keep working, + rebuilt *from* the registry rather than duplicated. + On Thor the reranker stays **served and advertised** — it runs, it is simply not yet correct; its ordering probe is recorded as a known failure pointing at #105 / #106, rather than the role being hidden. +### Fixed + +- Spec correction: the first cut of this spec claimed lobes had no machine-profile + concept at all. It does — `lobes/profiles.py` already ships `MachineProfile` / + `MACHINE_PROFILES` (spark/thor/blackwell/generic) and `detect_machine()`, wired + to `VLLM_MACHINE` and used by `switch`/`benchmark`. The real gaps, now stated + honestly in the spec's before-state: it is one knob-set **per machine, not per + role**; it lacks the knobs that actually mattered on Thor (KV-cache dtype, + enforce-eager, model-per-role, role feasibility); the **fleet** compose (the + default path) ignores it entirely and hardcodes the Spark values; its `thor` row + is an unvalidated guess (`status="configured"`: flashinfer / 32768 / util 0.6) + that live Thor testing **contradicts**; and `detect_machine()` silently falls + back to `generic` instead of admitting it does not know the card. + ### Changed ### Fixed @@ -33,8 +55,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). `docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md`. lobes detects the host card and applies a profile tuned for *that* box (feasible roles + model per role + util / context / quantization / KV dtype / - attention backend / enforce-eager), instead of the single GB10-tuned config - the templates hardcode today. Ships Spark (default) + Thor as supported; + attention backend / enforce-eager), instead of the fleet compose's hardcoded + GB10 values. Ships Spark (default) + Thor as supported; Orin / Orin Nano Super are named but unvalidated; an unrecognised card refuses-or-warns rather than silently applying the Spark profile. Every role gains a **correctness** probe, not just `/health` — a role that is diff --git a/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md b/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md index 1fb1625..627f277 100644 --- a/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md +++ b/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md @@ -91,6 +91,15 @@ slug: `lobes-fits-the-machine-it-lands-on-one-command-det` · status: `exported` - Every knob in the reference traces to a human-validated observation on a real card (boot log, probe result, measured budget) - Thor's known-incorrect reranker is documented with a pointer to #105/#106 rather than being hidden +### t12 — Per-chip STRATEGY PATTERN (foundational — do this before the other tasks build on top of the current shape): one module per chip under lobes/machines/ (CardStrategy: its own detection signature + per-role knobs + provenance) plus a small shared registry. profiles.py / _detect.py / init.py stop carrying per-chip tables and derive from the registry instead. Nothing is deleted: MachineProfile, MACHINE_PROFILES, detect_machine() and their switch/benchmark callers keep working, rebuilt FROM the registry. Also re-cuts the false premise the spec originally carried (that lobes had no machine-profile axis at all — it does) + +- instruction: OWNS: lobes/machines/ (new), and the registry-derivation edits to lobes/profiles.py + lobes/runtime/_detect.py. Foundational: land this BEFORE t1/t2/t4 build per-chip tables that would then have to be unpicked. Do NOT delete the legacy API (MachineProfile / MACHINE_PROFILES / detect_machine / resolve_serve_config) — derive it from the registry. stdlib only (dependencies = []); an explicit registry with imports in machines/__init__.py, no plugin/entry-point machinery. +- covers: c19, h15 +- acceptance: + - Adding a new chip = ONE new file + ONE registration line, proven by a test that registers a synthetic chip strategy and shows detection, profile resolution and knob rendering all pick it up with ZERO edits to profiles.py / _detect.py / init.py + - Every pre-existing test passes UNMODIFIED — if an existing test's expectations must change, legacy behaviour was broken and the refactor has failed + - The existing thor row (status='configured': flashinfer / 32768 / util 0.6 — an unvalidated guess that live Thor testing contradicts) is replaced with the measured values and marked load-tested; detect_machine()'s silent 'generic' fallback is preserved for its legacy callers but is NOT the source of truth for init + ## Risks - [follow_up] Reranker returns deterministically wrong orderings on Thor and MAY be wrong on the GB10 too (nothing checks ordering today). Tracked in lobes-cli#105 / #106. Until #106 answers, t8 cannot claim 4/4 on Thor — the thor profile must either fix rerank or declare it unavailable rather than advertise it broken. (task t8) diff --git a/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md b/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md index a0e9818..d407c13 100644 --- a/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md +++ b/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md @@ -8,7 +8,7 @@ ## Before → After -- Before: The packaged templates hardcode one tuning for the DGX Spark GB10 (sm_121). It does not transfer: on Jetson AGX Thor (sm_110) cortex needed --kv-cache-dtype=auto, embedder needed --attention-config TRITON_ATTN, reranker needed --enforce-eager and STILL returns wrong rankings. Every one of these was found by crashing into it. +- Before: lobes ALREADY has a machine-profile axis — lobes/profiles.py ships MachineProfile + MACHINE_PROFILES (spark/thor/blackwell/generic) and detect_machine(), wired to VLLM_MACHINE and used by switch/benchmark. But it is (a) ONE knob-set per machine, not per role; (b) missing the knobs that actually matter — kv-cache dtype, enforce-eager, model-per-role, role feasibility; (c) IGNORED by the fleet compose (the default path), which hardcodes DGX Spark GB10 values; (d) its thor row is an unvalidated GUESS (status 'configured': flashinfer / 32768 / util 0.6) that live testing on a physical Thor CONTRADICTS; and (e) detect_machine() silently falls back to 'generic' instead of admitting it does not know the card. On Thor (sm_110) the shipped fleet scored 1 of 4 roles correct on first boot; cortex needed --kv-cache-dtype=auto, embedder needed --attention-config TRITON_ATTN, reranker needed --enforce-eager and STILL returns wrong rankings. - After: lobes detects the host card and applies a per-machine profile (memory budget, quantization, attention backend, KV dtype, context, eager-vs-cudagraph, which roles are even feasible), so a supported box boots correct-and-tuned on first try. ## Why it matters @@ -29,7 +29,6 @@ ## Honesty conditions - One command does it: on a supported card, 'lobes init' picks the right profile with no flags, and 'lobes doctor' names which profile it picked and why (detected card + capability + memory). -- The three Thor divergences found by hand this session (kv-cache dtype auto, attention backend TRITON_ATTN on the pooling gears, enforce-eager on rerank) are exactly what a Thor profile encodes — the profile reproduces the working fleet without hand-editing. - The 'fits none' cost is real and measurable: a single shared config either leaves a big box's headroom unused or fails to boot a small one — demonstrable by pointing the GB10 profile at Thor (it crashes) and a Thor-safe config at the GB10 (it under-uses it). - The operator never has to know the card's compute capability to get a correct boot — the tool knows it; the operator only overrides when they WANT to. - 'Correct on first try' is checked, not asserted: every role the profile claims passes its correctness probe on a clean boot, and a role the card cannot serve is absent from capabilities rather than broken-but-advertised. @@ -37,6 +36,8 @@ - A profile can be proven on a real box: booting a supported profile from a clean ~/.lobes needs zero hand-edits to the generated compose file. - lobes is honest about coverage: it ships Spark (default) + Thor as supported, names Orin/Orin Nano Super as unvalidated, and on an unrecognised card it refuses-or-warns rather than silently applying the Spark profile. - The baseline is reproducible: a clean 'lobes init' + 'lobes serve' on Thor with the stock template reproduces 1/4-correct-on-first-boot, so 4/4-with-0-edits is a measured improvement, not a claim. +- The claim is checkable in the code as it stands: lobes/profiles.py really does define MachineProfile/MACHINE_PROFILES/detect_machine (grep it), the fleet compose really does hardcode the Spark values rather than read them, the thor row really is status='configured' with attention_backend='flashinfer', and detect_machine really does fall back to 'generic'. If any of those five is false, this before_state is wrong and the spec must be re-cut. +- Extensibility is PROVEN, not asserted: a test registers a synthetic new chip strategy and shows detection, profile resolution and knob rendering all pick it up with ZERO edits to profiles.py / _detect.py / init.py — and every pre-existing test still passes UNMODIFIED (if an existing test's expectations must change, legacy behaviour was broken and the refactor has failed). ## Success signals @@ -60,6 +61,7 @@ - A machine profile pins the MODEL for each role, not just tuning knobs: the profile is the full machine contract (feasible roles + model per role + knobs), and the catalog is the menu it selects from. This is what lets Orin Nano Super downshift cortex to a small model instead of only disabling it. - Profiles are default + overridable, and an operator may keep SEVERAL: lobes ships built-in profiles (Spark default, Thor), auto-detection picks one, --profile overrides it, and an operator can define their own profile rather than being limited to the shipped set. +- ARCHITECTURE: per-chip knowledge lives behind a STRATEGY PATTERN — one module per chip (lobes/machines/.py) owning its own detection signature, per-role knobs and provenance, plus a small shared registry. Adding a chip = one new file + one registration line; it must NOT mean editing shared tables in profiles.py / _detect.py / init.py, and a change for one chip must not be able to break another. Old code is NOT deleted: MachineProfile, MACHINE_PROFILES, detect_machine() and their switch/benchmark callers keep working, rebuilt FROM the registry rather than duplicated. ## Hard questions From 6268487e1a55a98c2903e76ff3980932de92b394 Mon Sep 17 00:00:00 2001 From: Ori Nachum Date: Mon, 13 Jul 2026 20:17:53 +0300 Subject: [PATCH 4/7] chore: sync uv.lock to the bumped version (0.40.1 -> 0.40.3) The version-bump script updates pyproject.toml but not uv.lock, so the lock kept claiming 0.40.1 and every `uv run` re-dirtied the working tree. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01RZNRUXYv84HUQ5ULQ5qJgz --- uv.lock | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/uv.lock b/uv.lock index c6e1163..9945725 100644 --- a/uv.lock +++ b/uv.lock @@ -440,7 +440,7 @@ wheels = [ [[package]] name = "lobes-cli" -version = "0.40.1" +version = "0.40.3" source = { editable = "." } [package.optional-dependencies] From cd2dbb7cbc3f386a79cec38b2ce2b3ec90dbf932 Mon Sep 17 00:00:00 2001 From: Ori Nachum Date: Mon, 13 Jul 2026 20:20:03 +0300 Subject: [PATCH 5/7] fix(changelog): one '### Fixed' per release (markdownlint MD024) The 0.40.3 entry had two '### Fixed' headings under the same parent. MD024 is configured siblings_only, so duplicates across releases are fine but duplicates within one are not. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01RZNRUXYv84HUQ5ULQ5qJgz --- CHANGELOG.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f92a2dd..b743148 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). not yet correct; its ordering probe is recorded as a known failure pointing at #105 / #106, rather than the role being hidden. +### Changed + ### Fixed - Spec correction: the first cut of this spec claimed lobes had no machine-profile @@ -43,10 +45,6 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). that live Thor testing **contradicts**; and `detect_machine()` silently falls back to `generic` instead of admitting it does not know the card. -### Changed - -### Fixed - ## [0.40.2] - 2026-07-13 ### Added From bdd9dbb9a35ce770407ffaf3f9c7ef7b253cdf1e Mon Sep 17 00:00:00 2001 From: Ori Nachum Date: Mon, 13 Jul 2026 20:46:39 +0300 Subject: [PATCH 6/7] fix(changelog): the plan has ten tasks, not eleven (Qodo #2); frame: proposed rework claims (traits, #107 base profile, golden renders) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01TJc5yvfweHP2AEccKNeaVd --- ...e-machine-it-lands-on-one-command-det.json | 79 ++++++++++++++++++- CHANGELOG.md | 3 +- 2 files changed, 80 insertions(+), 2 deletions(-) diff --git a/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json b/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json index a4f3f3d..1a0c17f 100644 --- a/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json +++ b/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json @@ -4,7 +4,7 @@ "schema_version": 2, "status": "exported", "created": "2026-07-13T15:33:49Z", - "updated": "2026-07-13T17:15:13Z", + "updated": "2026-07-13T17:45:51Z", "claims": [ { "id": "c1", @@ -344,6 +344,71 @@ "hard_questions": [], "links": [], "instruction": "" + }, + { + "id": "c21", + "kind": "decision", + "text": "Per-machine knowledge is keyed by causal capability TRAIT, not board name: each knob divergence names the fact that causes it \u2014 the sm_110 FLASH_ATTN pooling hang -> TRITON_ATTN for embed+rerank; the sm_110 CUDA-graph classify fault -> enforce-eager for rerank; the unscaled-fp8-KV assert (checkpoint x vLLM nightly, per the hand-edit comment \u2014 NOT the board) -> kv-cache-dtype=auto. A machine profile is then a named validated BUNDLE: detection signature + memory budget + model-per-role + the traits that apply. Evidence: 3 of the 4 Thor divergences trace to sm_110 and 1 to the checkpoint; NONE traces to 'Jetson AGX Thor' as a board.", + "origin": "llm", + "status": "proposed", + "honesty_conditions": [ + { + "id": "h17", + "text": "Every knob in thor.yaml carries a trait provenance naming its cause, and detection resolves compute capability + total memory as first-class facts even when the board name is unknown \u2014 so an unrecognised sm_110 board still renders the TRITON_ATTN pooling fix without anyone writing it a profile.", + "status": "proposed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c22", + "kind": "requirement", + "text": "Issue #107 folds in: the default/base profile is conservative and actually runnable \u2014 a small generate model plus the two 0.6B pooling gears, no 27B. Detection UPGRADES a recognised big box (spark, thor) to the 27B cortex; an unrecognised card gets a loud warning plus the small base tuned by whatever traits detection could read (compute capability, total memory) \u2014 never a refusal into nothing, never a silent Spark.", + "origin": "llm", + "status": "proposed", + "honesty_conditions": [ + { + "id": "h18", + "text": "A rendered compose contains the 27B only when a validated big-box profile or an explicit --profile put it there; a clean install on an unknown card boots a generate model that answers its known-answer probe, with a warning naming the detected facts and the assumption made.", + "status": "proposed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c23", + "kind": "requirement", + "text": "Cross-machine no-breakage is ENFORCED, not asserted: every shipped profile checks a golden rendered artifact (docker-compose.yml + .env) into the test suite, and CI fails any change that alters a DIFFERENT machine's rendering than the one the PR claims to touch \u2014 so the Thor box and the Spark box can both change lobes without regressing each other. A deliberate cross-machine change must update the other machine's golden explicitly, in the same PR, where the diff is the review surface.", + "origin": "llm", + "status": "proposed", + "honesty_conditions": [ + { + "id": "h19", + "text": "Proven by test: editing the thor bundle or an sm_110 trait leaves the spark golden byte-identical (and vice versa); rendering every shipped profile is a pure function of (profile, template) with no host state, so the goldens are runnable on any dev box, not just the hardware.", + "status": "proposed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c24", + "kind": "decision", + "text": "Packaging (#107): all profiles always ship in the wheel \u2014 pip extras (lobes-cli[thor]) are REJECTED as the mechanism: profiles are pure data with zero dependencies, and card selection is a runtime concern, not an install-time one. Extras return only if a card ever needs a genuine extra dependency (vendor SDK), and then the dependency, not the profile data, is what goes behind the extra.", + "origin": "llm", + "status": "proposed", + "honesty_conditions": [], + "hard_questions": [], + "links": [], + "instruction": "" } ], "open_vagueness": [ @@ -364,6 +429,18 @@ "text": "Whether profiles are resolved at 'lobes init' (baked into .env once) or re-resolved at every 'lobes serve' (so moving a deployment dir between boxes re-tunes).", "kind": "unknown_nonblocking", "claim_id": null + }, + { + "id": "v4", + "text": "fp8-KV attribution is UNVERIFIED: the Thor hand-edit comment blames the checkpoint (kv_cache_quant_algo: null -> assert k_scale > 0.0 on the pinned nightly), not sm_110 \u2014 yet the GB10 template default is fp8 on the same checkpoint. Verify on the GB10 whether the pinned nightly still boots with --kv-cache-dtype=fp8: if it crashes there too, the fix is the SHARED template default (auto), not a thor trait. Sibling of #106; needs its own tracked issue.", + "kind": "follow_up", + "claim_id": null + }, + { + "id": "v5", + "text": "Deleting the MULTIMODAL_ATTENTION_BACKEND / VLLM_ATTENTION_BACKEND env (plan t3) is grounded only in Thor observation; confirm on the GB10's pinned image that the env is truly ignored there before deleting it, else t3 could silently regress spark \u2014 exactly the cross-machine breakage this spec exists to prevent.", + "kind": "follow_up", + "claim_id": null } ], "scope_entries": [] diff --git a/CHANGELOG.md b/CHANGELOG.md index b743148..ad65391 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,8 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Plan (`/spec-to-plan`, converged): **per-machine hardware profiles** — `docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md`. - Eleven tasks over four file-disjoint dependency waves: the **per-chip strategy + Ten tasks over four file-disjoint dependency waves (twelve drafted, two + rejected during convergence): the **per-chip strategy pattern** + profile schema + spark/thor profiles, card detection, template parameterisation and per-role correctness probes (wave 1); `init` applies the profile and role-feasibility reaches capabilities/gateway (wave 2); From 5a63662e9ab471fcf78489e87f8137bf0f1337fc Mon Sep 17 00:00:00 2001 From: Ori Nachum Date: Mon, 13 Jul 2026 21:02:32 +0300 Subject: [PATCH 7/7] spec+plan rework: trait-keyed knowledge, golden renders, unknown-card small base (#107), upgrade compat MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-converged the frame and re-exported spec + plan on the user's decisions: - Per-machine knowledge keys on the CAUSAL TRAIT (sm_110, unscaled-fp8-KV), not the board name — 3 of the 4 Thor hand-edits trace to sm_110, 1 to the checkpoint, none to 'Thor'; a machine profile is a validated bundle (detection signature + memory budget + model-per-role + traits). - t13: golden rendered compose/.env per shipped profile, byte-diffed in CI — a Thor-side change cannot silently alter Spark's rendering (and vice versa). - t14: an UNKNOWN card warns and serves a conservative small base (no 27B) instead of refusing — the unknown-card slice of #107; the rest stays deferred. - t15: upgrading lobes-cli never breaks an existing scaffold (zero bytes changed in the deployment dir; old env names honoured; re-init is diffed and --apply-gated). - Parked as tracked risks on t3: verify on the GB10 whether the fp8-KV crash is checkpoint-driven (shared fix, not a thor trait) and whether VLLM_ATTENTION_BACKEND is truly dead on its pinned image. - Packaging per #107 recorded: profiles ship in the wheel; no pip extras. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01TJc5yvfweHP2AEccKNeaVd --- ...e-machine-it-lands-on-one-command-det.json | 54 +++++++-- ...e-machine-it-lands-on-one-command-det.json | 106 +++++++++++++++++- CHANGELOG.md | 21 +++- ...the-machine-it-lands-on-one-command-det.md | 32 ++++++ ...the-machine-it-lands-on-one-command-det.md | 11 ++ 5 files changed, 213 insertions(+), 11 deletions(-) diff --git a/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json b/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json index 1a0c17f..6a6e338 100644 --- a/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json +++ b/.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json @@ -4,7 +4,7 @@ "schema_version": 2, "status": "exported", "created": "2026-07-13T15:33:49Z", - "updated": "2026-07-13T17:45:51Z", + "updated": "2026-07-13T17:54:47Z", "claims": [ { "id": "c1", @@ -283,7 +283,7 @@ { "id": "h13", "text": "The baseline is real and reproducible: pointing the current (Spark) template at Thor reproduces the 0/4-on-first-boot result, so the 4/4-with-0-edits target is a measured improvement rather than a claim.", - "status": "proposed", + "status": "rejected", "instruction": "" } ], @@ -350,12 +350,12 @@ "kind": "decision", "text": "Per-machine knowledge is keyed by causal capability TRAIT, not board name: each knob divergence names the fact that causes it \u2014 the sm_110 FLASH_ATTN pooling hang -> TRITON_ATTN for embed+rerank; the sm_110 CUDA-graph classify fault -> enforce-eager for rerank; the unscaled-fp8-KV assert (checkpoint x vLLM nightly, per the hand-edit comment \u2014 NOT the board) -> kv-cache-dtype=auto. A machine profile is then a named validated BUNDLE: detection signature + memory budget + model-per-role + the traits that apply. Evidence: 3 of the 4 Thor divergences trace to sm_110 and 1 to the checkpoint; NONE traces to 'Jetson AGX Thor' as a board.", "origin": "llm", - "status": "proposed", + "status": "confirmed", "honesty_conditions": [ { "id": "h17", "text": "Every knob in thor.yaml carries a trait provenance naming its cause, and detection resolves compute capability + total memory as first-class facts even when the board name is unknown \u2014 so an unrecognised sm_110 board still renders the TRITON_ATTN pooling fix without anyone writing it a profile.", - "status": "proposed", + "status": "confirmed", "instruction": "" } ], @@ -368,12 +368,12 @@ "kind": "requirement", "text": "Issue #107 folds in: the default/base profile is conservative and actually runnable \u2014 a small generate model plus the two 0.6B pooling gears, no 27B. Detection UPGRADES a recognised big box (spark, thor) to the 27B cortex; an unrecognised card gets a loud warning plus the small base tuned by whatever traits detection could read (compute capability, total memory) \u2014 never a refusal into nothing, never a silent Spark.", "origin": "llm", - "status": "proposed", + "status": "rejected", "honesty_conditions": [ { "id": "h18", "text": "A rendered compose contains the 27B only when a validated big-box profile or an explicit --profile put it there; a clean install on an unknown card boots a generate model that answers its known-answer probe, with a warning naming the detected facts and the assumption made.", - "status": "proposed", + "status": "rejected", "instruction": "" } ], @@ -386,12 +386,12 @@ "kind": "requirement", "text": "Cross-machine no-breakage is ENFORCED, not asserted: every shipped profile checks a golden rendered artifact (docker-compose.yml + .env) into the test suite, and CI fails any change that alters a DIFFERENT machine's rendering than the one the PR claims to touch \u2014 so the Thor box and the Spark box can both change lobes without regressing each other. A deliberate cross-machine change must update the other machine's golden explicitly, in the same PR, where the diff is the review surface.", "origin": "llm", - "status": "proposed", + "status": "confirmed", "honesty_conditions": [ { "id": "h19", "text": "Proven by test: editing the thor bundle or an sm_110 trait leaves the spark golden byte-identical (and vice versa); rendering every shipped profile is a pure function of (profile, template) with no host state, so the goldens are runnable on any dev box, not just the hardware.", - "status": "proposed", + "status": "confirmed", "instruction": "" } ], @@ -404,11 +404,47 @@ "kind": "decision", "text": "Packaging (#107): all profiles always ship in the wheel \u2014 pip extras (lobes-cli[thor]) are REJECTED as the mechanism: profiles are pure data with zero dependencies, and card selection is a runtime concern, not an install-time one. Extras return only if a card ever needs a genuine extra dependency (vendor SDK), and then the dependency, not the profile data, is what goes behind the extra.", "origin": "llm", - "status": "proposed", + "status": "confirmed", "honesty_conditions": [], "hard_questions": [], "links": [], "instruction": "" + }, + { + "id": "c25", + "kind": "requirement", + "text": "On an UNRECOGNISED card lobes warns and serves a conservative small base \u2014 a small generate model plus the two 0.6B pooling gears, tuned by the traits it could read (compute capability, total memory) \u2014 instead of refusing into nothing or silently applying Spark. The broader change (a bare default install assumes a small box on EVERY card) stays deferred to #107; this spec only replaces refuse-or-warn with warn-and-serve-small for the unknown-card case.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h20", + "text": "A rendered compose on an unknown card never contains the 27B; the warning names the detected facts (device name, compute capability, total memory) and the assumption made; a recognised spark/thor box is entirely unaffected by this path.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" + }, + { + "id": "c26", + "kind": "requirement", + "text": "Updating the lobes-cli version must not break an existing setup on Spark or default: a deployment scaffolded by an earlier version keeps working with the new CLI \u2014 the upgrade itself changes zero bytes in the deployment dir, every verb keeps operating the old scaffold, and adopting a new template/profile is an explicit, diffed, --apply'd re-init, never a side effect of upgrading.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h21", + "text": "Proven by test: the new CLI operates a deployment dir scaffolded by the previous version without re-init (status/serve/stop against the old .env + compose succeed, old env var names stay honoured), and 'lobes init' over an existing dir shows a diff and requires --apply \u2014 pip install -U alone changes zero bytes in ~/.lobes.", + "status": "confirmed", + "instruction": "" + } + ], + "hard_questions": [], + "links": [], + "instruction": "" } ], "open_vagueness": [ diff --git a/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json b/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json index 293fdb9..3b58c44 100644 --- a/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json +++ b/.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json @@ -5,7 +5,7 @@ "schema_version": 2, "status": "exported", "created": "2026-07-13T16:09:59Z", - "updated": "2026-07-13T17:15:49Z", + "updated": "2026-07-13T18:00:22Z", "targets": [ { "id": "c1", @@ -136,6 +136,36 @@ "id": "h15", "kind": "honesty", "text": "The claim is checkable in the code as it stands: lobes/profiles.py really does define MachineProfile/MACHINE_PROFILES/detect_machine (grep it), the fleet compose really does hardcode the Spark values rather than read them, the thor row really is status='configured' with attention_backend='flashinfer', and detect_machine really does fall back to 'generic'. If any of those five is false, this before_state is wrong and the spec must be re-cut." + }, + { + "id": "c23", + "kind": "requirement", + "text": "Cross-machine no-breakage is ENFORCED, not asserted: every shipped profile checks a golden rendered artifact (docker-compose.yml + .env) into the test suite, and CI fails any change that alters a DIFFERENT machine's rendering than the one the PR claims to touch \u2014 so the Thor box and the Spark box can both change lobes without regressing each other. A deliberate cross-machine change must update the other machine's golden explicitly, in the same PR, where the diff is the review surface." + }, + { + "id": "h19", + "kind": "honesty", + "text": "Proven by test: editing the thor bundle or an sm_110 trait leaves the spark golden byte-identical (and vice versa); rendering every shipped profile is a pure function of (profile, template) with no host state, so the goldens are runnable on any dev box, not just the hardware." + }, + { + "id": "c25", + "kind": "requirement", + "text": "On an UNRECOGNISED card lobes warns and serves a conservative small base \u2014 a small generate model plus the two 0.6B pooling gears, tuned by the traits it could read (compute capability, total memory) \u2014 instead of refusing into nothing or silently applying Spark. The broader change (a bare default install assumes a small box on EVERY card) stays deferred to #107; this spec only replaces refuse-or-warn with warn-and-serve-small for the unknown-card case." + }, + { + "id": "h20", + "kind": "honesty", + "text": "A rendered compose on an unknown card never contains the 27B; the warning names the detected facts (device name, compute capability, total memory) and the assumption made; a recognised spark/thor box is entirely unaffected by this path." + }, + { + "id": "c26", + "kind": "requirement", + "text": "Updating the lobes-cli version must not break an existing setup on Spark or default: a deployment scaffolded by an earlier version keeps working with the new CLI \u2014 the upgrade itself changes zero bytes in the deployment dir, every verb keeps operating the old scaffold, and adopting a new template/profile is an explicit, diffed, --apply'd re-init, never a side effect of upgrading." + }, + { + "id": "h21", + "kind": "honesty", + "text": "Proven by test: the new CLI operates a deployment dir scaffolded by the previous version without re-init (status/serve/stop against the old .env + compose succeed, old env var names stay honoured), and 'lobes init' over an existing dir shows a diff and requires --apply \u2014 pip install -U alone changes zero bytes in ~/.lobes." } ], "tasks": [ @@ -383,6 +413,68 @@ "h15" ], "instruction": "OWNS: lobes/machines/ (new), and the registry-derivation edits to lobes/profiles.py + lobes/runtime/_detect.py. Foundational: land this BEFORE t1/t2/t4 build per-chip tables that would then have to be unpicked. Do NOT delete the legacy API (MachineProfile / MACHINE_PROFILES / detect_machine / resolve_serve_config) \u2014 derive it from the registry. stdlib only (dependencies = []); an explicit registry with imports in machines/__init__.py, no plugin/entry-point machinery." + }, + { + "id": "t13", + "summary": "Golden rendered artifacts per shipped profile: tests hold the rendered docker-compose.yml + .env for EVERY shipped profile (spark, thor, base) and diff byte-for-byte; rendering is a pure function of (profile, template) with no host state, so the goldens run on any dev box. A change for one machine that alters another machine's rendering fails CI unless that other golden is deliberately updated in the same PR", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "Editing the thor bundle or an sm_110 trait leaves the spark golden byte-identical (and vice versa), proven by a test that renders both", + "CI fails when a change alters any profile's rendered compose/.env without updating that profile's checked-in golden", + "Goldens render with no GPU or host state \u2014 the suite passes on a GPU-less CI runner" + ], + "deps": [ + "t1", + "t3" + ], + "covers": [ + "c23", + "h19" + ], + "instruction": "OWNS: tests/test_profile_goldens.py + tests/goldens// (new). Touch NO source. Goldens are committed artifacts regenerated by a documented one-liner; the test re-renders every shipped profile via t1/t3's render API and byte-diffs. Rendering must be a pure function of (profile, template) \u2014 no GPU, no host reads \u2014 so the suite passes on GPU-less CI. This is the enforcement for 'a change for one chip cannot break another'." + }, + { + "id": "t14", + "summary": "Unknown-card path: detection returns UNKNOWN -> 'lobes init' WARNS (naming device name, compute capability, total memory, and the assumption made) and renders the conservative small base \u2014 a small generate model plus the two 0.6B pooling gears, NO 27B \u2014 tuned by the traits it could read. Replaces t4's refuse-or-warn branch; the broader default-install change stays deferred to #107", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "A rendered compose on an unknown card never contains the 27B, proven by a rendering test with a synthetic unknown card", + "The warning names the detected facts (device name, compute capability, total memory) and the assumption made", + "A recognised spark/thor box is entirely unaffected \u2014 its golden rendering is byte-identical before/after this change" + ], + "deps": [ + "t1", + "t2", + "t3", + "t4" + ], + "covers": [ + "c25", + "h20" + ], + "instruction": "OWNS: the unknown-card branch in lobes/cli/_commands/init.py + lobes/profiles/builtin/base.yaml (new). Extends t4's resolution glue \u2014 do NOT re-implement detection. The base profile serves a small generate model from the catalog plus the two 0.6B pooling gears, never the 27B; knobs still come from readable traits (compute capability, total memory). Scope guard: the default on RECOGNISED cards is unchanged \u2014 the broader small-default-everywhere change stays in #107." + }, + { + "id": "t15", + "summary": "Upgrade never breaks an existing scaffold: 'pip install -U lobes-cli' changes zero bytes in the deployment dir; every verb keeps operating a deployment scaffolded by the PREVIOUS version (old env var names stay honoured); adopting a new template/profile is an explicit, diffed, --apply'd re-init, never a side effect of upgrading \u2014 protects the live Spark box and any default ~/.lobes", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "A test scaffolds a deployment dir from the pre-profile template (current main), then runs the new CLI's status/serve/stop dry-run paths against it without re-init \u2014 all succeed", + "Every env var name the current template reads stays honoured by the new compose/CLI (or a compat shim maps it), proven by test", + "'lobes init' over an existing dir shows a diff and requires --apply; it never silently rewrites the operator's files" + ], + "deps": [ + "t3", + "t4" + ], + "covers": [ + "c26", + "h21" + ], + "instruction": "OWNS: tests/test_upgrade_compat.py (new) + any env-name compat shim in lobes/runtime/_env.py. Fixture: a deployment dir scaffolded from current main's template, vendored into tests. Assert the new CLI's status/serve/stop dry-run paths operate it without re-init, every env var name main's template reads stays honoured, and init-over-existing shows a diff and requires --apply. Prefer keeping env names stable over shimming." } ], "risks": [ @@ -409,6 +501,18 @@ "text": "DECISION (supersedes the framing of r1): on Thor the reranker stays SERVED and advertised \u2014 it runs, it is just not yet correct. Correctness is fixed in lobes-cli#105/#106, not by hiding the role. The correctness probe records it as a known failure until then.", "kind": "follow_up", "task_id": "t10" + }, + { + "id": "r5", + "text": "fp8-KV attribution UNVERIFIED: the Thor hand-edit blames the checkpoint (kv_cache_quant_algo: null -> assert k_scale > 0.0), not sm_110 \u2014 yet the GB10 template default is fp8 on the same checkpoint. Verify on the GB10 whether the pinned nightly still boots with --kv-cache-dtype=fp8; if it crashes there too, kv-cache-dtype=auto is the SHARED template default, not a thor trait. Sibling of #106; needs its own tracked issue", + "kind": "follow_up", + "task_id": "t3" + }, + { + "id": "r6", + "text": "Deleting MULTIMODAL_ATTENTION_BACKEND / VLLM_ATTENTION_BACKEND (t3) is grounded only in Thor observation \u2014 confirm on the GB10's pinned image that the env is truly ignored there BEFORE deleting, else t3 silently regresses spark: exactly the cross-machine breakage t13's goldens exist to catch", + "kind": "follow_up", + "task_id": "t3" } ] } diff --git a/CHANGELOG.md b/CHANGELOG.md index ad65391..10a25c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Plan (`/spec-to-plan`, converged): **per-machine hardware profiles** — `docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md`. - Ten tasks over four file-disjoint dependency waves (twelve drafted, two + Thirteen tasks over four file-disjoint dependency waves (fifteen drafted, two rejected during convergence): the **per-chip strategy pattern** + profile schema + spark/thor profiles, card detection, template parameterisation and per-role correctness probes (wave 1); `init` applies the @@ -30,6 +30,25 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). not yet correct; its ordering probe is recorded as a known failure pointing at #105 / #106, rather than the role being hidden. +- Spec + plan rework (same frame, re-converged): per-machine knowledge is keyed + by **causal capability trait**, not board name — of the four Thor hand-edits, + three trace to `sm_110` (the FLASH_ATTN pooling hang, the CUDA-graph classify + fault) and one to the checkpoint's missing KV scales; none traces to "Thor the + board". A machine profile becomes a named validated *bundle* (detection + signature + memory budget + model-per-role + applicable traits), so an + unrecognised board sharing a trait inherits its fix. Three new plan tasks: + **golden rendered compose/.env per shipped profile** byte-diffed in CI, so a + change for one machine cannot silently alter another's rendering (t13); an + **unknown card warns and serves a conservative small base** — no 27B — instead + of refusing, folding the unknown-card slice of #107 in (t14); and **upgrading + lobes-cli never breaks an existing scaffold** — zero bytes changed in the + deployment dir, old env names honoured, re-init always diffed + `--apply` + (t15). Two GB10 verifications are parked as tracked risks: whether the fp8-KV + crash is checkpoint-driven (shared fix) rather than Thor-specific, and whether + the `VLLM_ATTENTION_BACKEND` env is truly dead on the GB10's pinned image + before t3 deletes it. Packaging per #107: profiles ship in the wheel, no pip + extras. + ### Changed ### Fixed diff --git a/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md b/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md index 627f277..b4297de 100644 --- a/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md +++ b/docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md @@ -100,9 +100,41 @@ slug: `lobes-fits-the-machine-it-lands-on-one-command-det` · status: `exported` - Every pre-existing test passes UNMODIFIED — if an existing test's expectations must change, legacy behaviour was broken and the refactor has failed - The existing thor row (status='configured': flashinfer / 32768 / util 0.6 — an unvalidated guess that live Thor testing contradicts) is replaced with the measured values and marked load-tested; detect_machine()'s silent 'generic' fallback is preserved for its legacy callers but is NOT the source of truth for init +### t13 — Golden rendered artifacts per shipped profile: tests hold the rendered docker-compose.yml + .env for EVERY shipped profile (spark, thor, base) and diff byte-for-byte; rendering is a pure function of (profile, template) with no host state, so the goldens run on any dev box. A change for one machine that alters another machine's rendering fails CI unless that other golden is deliberately updated in the same PR + +- instruction: OWNS: tests/test_profile_goldens.py + tests/goldens// (new). Touch NO source. Goldens are committed artifacts regenerated by a documented one-liner; the test re-renders every shipped profile via t1/t3's render API and byte-diffs. Rendering must be a pure function of (profile, template) — no GPU, no host reads — so the suite passes on GPU-less CI. This is the enforcement for 'a change for one chip cannot break another'. +- depends on: t1, t3 +- covers: c23, h19 +- acceptance: + - Editing the thor bundle or an sm_110 trait leaves the spark golden byte-identical (and vice versa), proven by a test that renders both + - CI fails when a change alters any profile's rendered compose/.env without updating that profile's checked-in golden + - Goldens render with no GPU or host state — the suite passes on a GPU-less CI runner + +### t14 — Unknown-card path: detection returns UNKNOWN -> 'lobes init' WARNS (naming device name, compute capability, total memory, and the assumption made) and renders the conservative small base — a small generate model plus the two 0.6B pooling gears, NO 27B — tuned by the traits it could read. Replaces t4's refuse-or-warn branch; the broader default-install change stays deferred to #107 + +- instruction: OWNS: the unknown-card branch in lobes/cli/_commands/init.py + lobes/profiles/builtin/base.yaml (new). Extends t4's resolution glue — do NOT re-implement detection. The base profile serves a small generate model from the catalog plus the two 0.6B pooling gears, never the 27B; knobs still come from readable traits (compute capability, total memory). Scope guard: the default on RECOGNISED cards is unchanged — the broader small-default-everywhere change stays in #107. +- depends on: t1, t2, t3, t4 +- covers: c25, h20 +- acceptance: + - A rendered compose on an unknown card never contains the 27B, proven by a rendering test with a synthetic unknown card + - The warning names the detected facts (device name, compute capability, total memory) and the assumption made + - A recognised spark/thor box is entirely unaffected — its golden rendering is byte-identical before/after this change + +### t15 — Upgrade never breaks an existing scaffold: 'pip install -U lobes-cli' changes zero bytes in the deployment dir; every verb keeps operating a deployment scaffolded by the PREVIOUS version (old env var names stay honoured); adopting a new template/profile is an explicit, diffed, --apply'd re-init, never a side effect of upgrading — protects the live Spark box and any default ~/.lobes + +- instruction: OWNS: tests/test_upgrade_compat.py (new) + any env-name compat shim in lobes/runtime/_env.py. Fixture: a deployment dir scaffolded from current main's template, vendored into tests. Assert the new CLI's status/serve/stop dry-run paths operate it without re-init, every env var name main's template reads stays honoured, and init-over-existing shows a diff and requires --apply. Prefer keeping env names stable over shimming. +- depends on: t3, t4 +- covers: c26, h21 +- acceptance: + - A test scaffolds a deployment dir from the pre-profile template (current main), then runs the new CLI's status/serve/stop dry-run paths against it without re-init — all succeed + - Every env var name the current template reads stays honoured by the new compose/CLI (or a compat shim maps it), proven by test + - 'lobes init' over an existing dir shows a diff and requires --apply; it never silently rewrites the operator's files + ## Risks - [follow_up] Reranker returns deterministically wrong orderings on Thor and MAY be wrong on the GB10 too (nothing checks ordering today). Tracked in lobes-cli#105 / #106. Until #106 answers, t8 cannot claim 4/4 on Thor — the thor profile must either fix rerank or declare it unavailable rather than advertise it broken. (task t8) - [unknown_nonblocking] Jetson AGX Orin (sm_87, no NVFP4) and Orin Nano Super are UNVALIDATED — no physical board in hand. The schema must accommodate them (smaller cortex model, roles disabled) but no profile ships as supported until someone boots one. - [unknown_nonblocking] Detection runs on the HOST (the lobes CLI), while the kernels that actually differ run INSIDE the container. If the two ever disagree about the card, the profile is resolved against the wrong truth. (task t2) - [follow_up] DECISION (supersedes the framing of r1): on Thor the reranker stays SERVED and advertised — it runs, it is just not yet correct. Correctness is fixed in lobes-cli#105/#106, not by hiding the role. The correctness probe records it as a known failure until then. (task t10) +- [follow_up] fp8-KV attribution UNVERIFIED: the Thor hand-edit blames the checkpoint (kv_cache_quant_algo: null -> assert k_scale > 0.0), not sm_110 — yet the GB10 template default is fp8 on the same checkpoint. Verify on the GB10 whether the pinned nightly still boots with --kv-cache-dtype=fp8; if it crashes there too, kv-cache-dtype=auto is the SHARED template default, not a thor trait. Sibling of #106; needs its own tracked issue (task t3) +- [follow_up] Deleting MULTIMODAL_ATTENTION_BACKEND / VLLM_ATTENTION_BACKEND (t3) is grounded only in Thor observation — confirm on the GB10's pinned image that the env is truly ignored there BEFORE deleting, else t3 silently regresses spark: exactly the cross-machine breakage t13's goldens exist to catch (task t3) diff --git a/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md b/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md index d407c13..7eda540 100644 --- a/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md +++ b/docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md @@ -25,6 +25,12 @@ - honesty: A role can be declared unavailable end-to-end: lobes capabilities / GET /capabilities omit it (or mark it unserved), and the gateway 4xx's a request for it rather than silently routing to a different model. - Every role ships a CORRECTNESS probe, not just a /health check: embed must rank a paraphrase above an unrelated string, rerank must put the relevant document first, cortex must answer a known-answer question. A role that is 'healthy' but semantically wrong must fail the probe. - honesty: The rerank probe FAILS today on this Thor box (relevant doc does not rank first) — i.e. the probe catches the bug we actually hit, rather than being a test that only passes. +- Cross-machine no-breakage is ENFORCED, not asserted: every shipped profile checks a golden rendered artifact (docker-compose.yml + .env) into the test suite, and CI fails any change that alters a DIFFERENT machine's rendering than the one the PR claims to touch — so the Thor box and the Spark box can both change lobes without regressing each other. A deliberate cross-machine change must update the other machine's golden explicitly, in the same PR, where the diff is the review surface. + - honesty: Proven by test: editing the thor bundle or an sm_110 trait leaves the spark golden byte-identical (and vice versa); rendering every shipped profile is a pure function of (profile, template) with no host state, so the goldens are runnable on any dev box, not just the hardware. +- On an UNRECOGNISED card lobes warns and serves a conservative small base — a small generate model plus the two 0.6B pooling gears, tuned by the traits it could read (compute capability, total memory) — instead of refusing into nothing or silently applying Spark. The broader change (a bare default install assumes a small box on EVERY card) stays deferred to #107; this spec only replaces refuse-or-warn with warn-and-serve-small for the unknown-card case. + - honesty: A rendered compose on an unknown card never contains the 27B; the warning names the detected facts (device name, compute capability, total memory) and the assumption made; a recognised spark/thor box is entirely unaffected by this path. +- Updating the lobes-cli version must not break an existing setup on Spark or default: a deployment scaffolded by an earlier version keeps working with the new CLI — the upgrade itself changes zero bytes in the deployment dir, every verb keeps operating the old scaffold, and adopting a new template/profile is an explicit, diffed, --apply'd re-init, never a side effect of upgrading. + - honesty: Proven by test: the new CLI operates a deployment dir scaffolded by the previous version without re-init (status/serve/stop against the old .env + compose succeed, old env var names stay honoured), and 'lobes init' over an existing dir shows a diff and requires --apply — pip install -U alone changes zero bytes in ~/.lobes. ## Honesty conditions @@ -38,6 +44,7 @@ - The baseline is reproducible: a clean 'lobes init' + 'lobes serve' on Thor with the stock template reproduces 1/4-correct-on-first-boot, so 4/4-with-0-edits is a measured improvement, not a claim. - The claim is checkable in the code as it stands: lobes/profiles.py really does define MachineProfile/MACHINE_PROFILES/detect_machine (grep it), the fleet compose really does hardcode the Spark values rather than read them, the thor row really is status='configured' with attention_backend='flashinfer', and detect_machine really does fall back to 'generic'. If any of those five is false, this before_state is wrong and the spec must be re-cut. - Extensibility is PROVEN, not asserted: a test registers a synthetic new chip strategy and shows detection, profile resolution and knob rendering all pick it up with ZERO edits to profiles.py / _detect.py / init.py — and every pre-existing test still passes UNMODIFIED (if an existing test's expectations must change, legacy behaviour was broken and the refactor has failed). +- Every knob in thor.yaml carries a trait provenance naming its cause, and detection resolves compute capability + total memory as first-class facts even when the board name is unknown — so an unrecognised sm_110 board still renders the TRITON_ATTN pooling fix without anyone writing it a profile. ## Success signals @@ -62,6 +69,8 @@ - A machine profile pins the MODEL for each role, not just tuning knobs: the profile is the full machine contract (feasible roles + model per role + knobs), and the catalog is the menu it selects from. This is what lets Orin Nano Super downshift cortex to a small model instead of only disabling it. - Profiles are default + overridable, and an operator may keep SEVERAL: lobes ships built-in profiles (Spark default, Thor), auto-detection picks one, --profile overrides it, and an operator can define their own profile rather than being limited to the shipped set. - ARCHITECTURE: per-chip knowledge lives behind a STRATEGY PATTERN — one module per chip (lobes/machines/.py) owning its own detection signature, per-role knobs and provenance, plus a small shared registry. Adding a chip = one new file + one registration line; it must NOT mean editing shared tables in profiles.py / _detect.py / init.py, and a change for one chip must not be able to break another. Old code is NOT deleted: MachineProfile, MACHINE_PROFILES, detect_machine() and their switch/benchmark callers keep working, rebuilt FROM the registry rather than duplicated. +- Per-machine knowledge is keyed by causal capability TRAIT, not board name: each knob divergence names the fact that causes it — the sm_110 FLASH_ATTN pooling hang -> TRITON_ATTN for embed+rerank; the sm_110 CUDA-graph classify fault -> enforce-eager for rerank; the unscaled-fp8-KV assert (checkpoint x vLLM nightly, per the hand-edit comment — NOT the board) -> kv-cache-dtype=auto. A machine profile is then a named validated BUNDLE: detection signature + memory budget + model-per-role + the traits that apply. Evidence: 3 of the 4 Thor divergences trace to sm_110 and 1 to the checkpoint; NONE traces to 'Jetson AGX Thor' as a board. +- Packaging (#107): all profiles always ship in the wheel — pip extras (lobes-cli[thor]) are REJECTED as the mechanism: profiles are pure data with zero dependencies, and card selection is a runtime concern, not an install-time one. Extras return only if a card ever needs a genuine extra dependency (vendor SDK), and then the dependency, not the profile data, is what goes behind the extra. ## Hard questions @@ -70,3 +79,5 @@ ## Open / follow-up - Is the reranker's wrong ordering Thor-specific, or a pre-existing lobes bug also live on the GB10? Nothing currently checks rerank ordering. TRACKED, not blocking this spec: agentculture/lobes-cli#105 (Thor: wrong rankings + cudaErrorLaunchFailure) and #106 (verify the same ordering probe on the GB10; the result decides whether the fix belongs in a Thor profile or in the shared rerank template). +- fp8-KV attribution is UNVERIFIED: the Thor hand-edit comment blames the checkpoint (kv_cache_quant_algo: null -> assert k_scale > 0.0 on the pinned nightly), not sm_110 — yet the GB10 template default is fp8 on the same checkpoint. Verify on the GB10 whether the pinned nightly still boots with --kv-cache-dtype=fp8: if it crashes there too, the fix is the SHARED template default (auto), not a thor trait. Sibling of #106; needs its own tracked issue. +- Deleting the MULTIMODAL_ATTENTION_BACKEND / VLLM_ATTENTION_BACKEND env (plan t3) is grounded only in Thor observation; confirm on the GB10's pinned image that the env is truly ignored there before deleting it, else t3 could silently regress spark — exactly the cross-machine breakage this spec exists to prevent.