spec+plan: lobes fits the machine it lands on — per-machine hardware profiles

[OriNachum]

Docs only — spec and plan, no implementation. This PR is the converged /think spec and /spec-to-plan plan for per-machine hardware profiles. The code is deliberately not here: it was built, reviewed, and then reverted so the design can be agreed before implementation lands.

Why

The fleet was brought up on a Jetson AGX Thor (aarch64, sm_110, 122 GB unified, CUDA 13.0) — not the DGX Spark GB10 (sm_121) the templates are tuned for. The shipped configuration scored 1 of 4 roles correct on first boot:

role	first boot on Thor	fix
senses (Gemma 4 12B)	came up clean	none
cortex (27B NVFP4 + MTP)	crash-loop: assert layer.k_scale > 0.0	--kv-cache-dtype=auto — the checkpoint carries no calibrated KV scales (kv_cache_quant_algo: null), so fp8 cannot work
embedder	hung — request accepted, never answered, never logged; /health stayed green throughout	--attention-config={"backend": "TRITON_ATTN"} — the auto-picked FLASH_ATTN pooling path hangs
reranker	killed its own engine: CUDA error: unspecified launch failure	Triton + --enforce-eager stops the crash; it still returns wrong rankings (#105, verify on GB10: #106)

Reaching 3 of 4 took four hand-edits to the generated compose. Every one was found by crashing into it.

The cortex fp8 failure is not Thor-specific — it would fail identically on a GB10 with this vLLM nightly.

What the spec commits to

• The profile is the machine contract: which roles are feasible, which model serves each (so a future Orin Nano Super can downshift cortex rather than only disabling it), and every machine knob — util, context, quantization, KV dtype, attention backend, enforce-eager, max-num-seqs.
• Per-chip strategy pattern: one module per chip (lobes/machines/<chip>.py) owning its own detection signature, knobs and provenance, plus a small registry. Adding a chip = one new file + one registration line — never an edit to a shared table, and a change for one chip cannot break another. Nothing is deleted: MachineProfile / MACHINE_PROFILES / detect_machine() keep working, rebuilt from the registry.
• Honest coverage: Spark (default) and Thor ship supported because both were actually booted. Orin / Orin Nano Super are named but unvalidated. An unrecognised card refuses-or-warns rather than silently applying the Spark profile — which is precisely what produced the crash-loop above.
• Correctness probes, not liveness checks: embed must rank a paraphrase above an unrelated string; rerank must put the relevant document first; cortex must answer a known-answer question. A role that is /health-green but semantically wrong must fail. Today nothing in lobes checks rerank ordering — which is exactly how a reranker that ranks "Bananas are yellow" above the correct document passes every check we have.
• Measurable target: on a supported card, 100% of the roles the profile claims come up correct with zero hand-edits, against the measured 1-of-4 baseline.

Correction in this PR

The first cut of the spec was wrong and the third commit fixes it. It claimed lobes had no machine-profile concept. It does — lobes/profiles.py already ships MachineProfile / MACHINE_PROFILES (spark/thor/blackwell/generic) and detect_machine(). The real gaps are narrower and now stated honestly: one knob-set per machine, not per role; missing exactly the knobs Thor needed; the fleet compose ignores it entirely; its thor row is an unvalidated guess (status="configured": flashinfer / 32768 / util 0.6) that live testing contradicts; and detect_machine() silently falls back to generic instead of admitting it doesn't know the card.

Plan

Eleven tasks, four file-disjoint waves. t12 (strategy pattern) is foundational and lands first. Its honesty condition is that extensibility is proven, not asserted — a test registers a synthetic chip and shows detection, profile resolution and knob rendering all pick it up with zero edits to profiles.py / _detect.py / init.py, with every pre-existing test passing unmodified.

The Thor reranker stays served and advertised — it runs, it is simply not yet correct. Its ordering probe is recorded as a known failure pointing at #105 / #106, rather than the role being hidden to make the plan look green.

Related

• reranker returns wrong rankings on Jetson AGX Thor (sm_110) — and crashes the engine under CUDA graphs #105 — reranker returns wrong rankings on Thor (and crashes the engine under CUDA graphs)
• verify reranker ORDERING on the DGX Spark GB10 — nothing currently checks it, it may be wrong there too #106 — verify rerank ordering on the GB10; nothing checks it today, so it may be broken there too. The result decides whether the fix belongs in a Thor profile or the shared template.
• default install should assume a SMALL box, not a 128 GB Blackwell — conservative base profile + a weak default model #107 — default install should assume a small box, not a 128 GB Blackwell (and why packaging extras are the wrong mechanism for shipping profiles)

Test plan

Docs only: no source changes. Full suite green at the 1188 baseline; afi cli doctor --strict passes; version bumped 0.40.1 → 0.40.3.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RZNRUXYv84HUQ5ULQ5qJgz

[qodo-code-review]

PR Summary by Qodo

Docs: spec + plan for per-machine hardware profiles (Spark/Thor first)

📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add converged spec + build plan for per-machine hardware profiles and detection.
• Define strategy-pattern registry for per-chip tuning and role feasibility/model selection.
• Bump version and document the change in the changelog.

Diagram

graph TD
  HW{{"NVIDIA GPU / Board"}} --> DET["Card detection"] --> RES["Profile resolution"] --> INIT["lobes init"] --> RENDER["Compose + .env render"] --> SERVE["lobes serve"]
  SERVE --> CAP["Capabilities + gateway"]
  SERVE --> PROBE["Correctness probes"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Central table-driven profiles (single file map)

• ➕ Simpler to discover/edit all profiles in one place
• ➕ Less import/registration plumbing
• ➖ Higher risk of cross-chip breakage when editing shared structures
• ➖ Harder to enforce “add a chip without touching shared tables”

2. Per-role profiles first (role contract, not machine contract)

• ➕ Matches current mental model of 'roles' as the primary unit
• ➕ Could minimize changes to gateway/capabilities semantics initially
• ➖ Doesn’t solve the core problem: many knobs are machine-coupled (memory, attention backend quirks, KV dtype)
• ➖ Makes “which roles are feasible on this box” harder to reason about and validate

3. Plugin/entry-point based chip strategies

• ➕ Third parties can add chip support without repo changes
• ➕ Potentially cleaner optionality
• ➖ More complexity and packaging surface area than needed right now
• ➖ Harder to keep behavior deterministic and testable across environments

Recommendation: The proposed machine-contract model plus per-chip strategy registry is the best fit for preventing silent misconfiguration and making “supported hardware” explicit. Keeping strategies as explicit in-repo modules (not plugins) aligns with the plan’s testable extensibility requirement (synthetic chip registration without touching legacy call sites). The main thing to watch in implementation is the host-vs-container detection boundary called out in risks: ensure the resolved profile is based on a consistent, observable hardware truth.

Files changed (8) +1036 / -3

Documentation (5) +1033 / -0

lobes-fits-the-machine-it-lands-on-one-command-det.jsonAdd exported devague frame JSON for per-machine profiles spec +370/-0

Add exported devague frame JSON for per-machine profiles spec

• Adds the exported frame capturing the problem statement, requirements, decisions, and honesty conditions for per-machine hardware profiles and correctness probes.

.devague/frames/lobes-fits-the-machine-it-lands-on-one-command-det.json

lobes-fits-the-machine-it-lands-on-one-command-det.jsonAdd exported devague plan JSON with task waves and acceptance criteria +414/-0

Add exported devague plan JSON with task waves and acceptance criteria

• Adds the exported implementation plan with 11 tasks, dependencies, acceptance criteria, and explicit risks around reranker correctness and detection boundaries.

.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json

CHANGELOG.mdDocument spec/plan additions and correct earlier spec premise +69/-0

Document spec/plan additions and correct earlier spec premise

• Adds entries for 0.40.2 and 0.40.3 describing the per-machine profile spec/plan and explicitly correcting the earlier incorrect premise about existing machine-profile support.

CHANGELOG.md

2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.mdAdd human-readable build plan for per-machine profiles +108/-0

Add human-readable build plan for per-machine profiles

• Introduces the build plan markdown summarizing tasks t1–t12, their ownership boundaries, dependencies, acceptance checks, and key risks.

docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md

2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.mdAdd human-readable spec for per-machine hardware profiles +72/-0

Add human-readable spec for per-machine hardware profiles

• Introduces the spec markdown defining before/after, requirements (detection, knobs, feasibility, probes), and the strategy-pattern architecture decision.

docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md

Other (3) +3 / -3

currentPoint devague 'current' pointer to the new spec slug +1/-1

Point devague 'current' pointer to the new spec slug

• Updates the devague tracking pointer to reference the new exported spec/frame slug for per-machine hardware profiles.

.devague/current

current_planPoint devague 'current_plan' pointer to the new plan slug +1/-1

Point devague 'current_plan' pointer to the new plan slug

• Updates the devague plan pointer to reference the new exported plan slug for per-machine hardware profiles.

.devague/current_plan

pyproject.tomlBump project version to 0.40.3 +1/-1

Bump project version to 0.40.3

• Updates the package version from 0.40.1 to 0.40.3 to reflect the spec/plan publication.

pyproject.toml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

Context used

✅ Tickets: 🎫 verify reranker ORDERING on the DGX Spark GB10 — nothing currently checks it, it may be wrong there too 🎫 reranker returns wrong rankings on Jetson AGX Thor (sm_110) — and crashes the engine under CUDA graphs 🎫 default install should assume a SMALL box, not a 128 GB Blackwell — conservative base profile + a weak default model

✅ Compliance rules (platform): 27 rules

✅ Skills: doc-test-alignment, cicd

1. Duplicate changelog heading ✓ Resolved 🐞 Bug ☼ Reliability

Description

The 0.40.3 CHANGELOG entry contains two sibling "### Fixed" headings, which triggers markdownlint
MD024 (duplicate headings) and will fail the repo’s markdownlint-cli2 CI step.

Code

CHANGELOG.md[R46-49]

+### Changed
+
+### Fixed
+

Relevance

⭐⭐⭐ High

Changelog heading/markdown compliance fixes have been accepted before (Keep-a-Changelog subsection
heading cleanup in PR #33).

PR-#33

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The changelog includes two ### Fixed headings within the same 0.40.3 section; markdownlint is
configured to flag duplicate sibling headings and is executed in CI.

CHANGELOG.md[32-50]
.markdownlint-cli2.yaml[6-16]
.github/workflows/tests.yml[69-73]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`CHANGELOG.md` has two sibling `### Fixed` headings under the `0.40.3` release entry. With markdownlint MD024 configured as `siblings_only: true`, this is a deterministic lint violation and breaks CI.

### Issue Context
CI runs `markdownlint-cli2 "**/*.md"` and does not ignore `CHANGELOG.md`, so the duplicate heading is merge-blocking.

### Fix Focus Areas
- CHANGELOG.md[32-49]

### Suggested fix
Delete the empty trailing `### Fixed` stub (and optionally the empty `### Changed` stub), or merge content so there is only one `### Fixed` section for `0.40.3`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Wrong task count stated ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

The 0.40.3 changelog says the plan has "Eleven tasks", but the exported plan markdown lists 10 tasks
(t1–t7, t10–t12) and the plan JSON contains 12 task records with 2 marked rejected, so the release
note count is inconsistent with the shipped artifacts.

Code

CHANGELOG.md[R11-18]

+- Plan (`/spec-to-plan`, converged): **per-machine hardware profiles** —
+  `docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md`.
+  Eleven tasks over four file-disjoint dependency waves: the **per-chip strategy
+  pattern** + profile schema + spark/thor profiles, card detection, template
+  parameterisation and per-role correctness probes (wave 1); `init` applies the
+  profile and role-feasibility reaches capabilities/gateway (wave 2);
+  `doctor`/`status` report the profile, and the Thor profile is validated on the
+  physical board (wave 3); docs + an honest support table (wave 4).

Relevance

⭐⭐⭐ High

Team has accepted correcting inaccurate CHANGELOG claims (e.g., reworded misleading release-note
statement in PR #23).

PR-#23

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The changelog’s stated task count conflicts with both the referenced exported plan markdown (10 task
headings) and the plan JSON (12 tasks with t8/t9 rejected), so "Eleven" matches neither artifact.

CHANGELOG.md[11-18]
docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md[7-102]
.devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json[278-307]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`CHANGELOG.md` claims the per-machine profiles plan has "Eleven tasks", but the exported plan markdown enumerates 10 tasks, and the source plan JSON includes 12 tasks with two rejected. This makes release notes internally inconsistent.

### Issue Context
Readers will use the changelog line to understand the plan’s scope; it should match either the exported markdown’s enumerated task count, or explicitly describe the counting convention (e.g., "12 tasks total, 10 active, 2 rejected").

### Fix Focus Areas
- CHANGELOG.md[11-18]
- docs/plans/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md[7-102]
- .devague/plans/lobes-fits-the-machine-it-lands-on-one-command-det.json[278-307]

### Suggested fix
Update the changelog sentence to match the exported plan ("Ten tasks") or clarify total vs active vs rejected tasks.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[OriNachum]

Fixed in cd2dbb7 — the empty ### Changed / ### Fixed stubs under 0.40.3 are gone; one ### Fixed per release, markdownlint MD024 passes.

• lobes (Claude)

[OriNachum]

Fixed in bdd9dbb — the changelog now says ten tasks over four waves (twelve drafted, two rejected during convergence), matching the exported plan markdown and the plan JSON.

• lobes (Claude)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud