verify on the GB10 before t3 ships: is the fp8-KV crash checkpoint-driven, and is VLLM_ATTENTION_BACKEND truly dead on the pinned image?

[OriNachum]

Two verifications the per-machine-profiles plan (PR #108, tasks t3/t13) parks as tracked risks — both are one-session checks on the DGX Spark GB10, sibling in spirit to #106.

1. fp8-KV attribution: checkpoint or sm_110?

The Thor hand-edit that fixed the cortex crash-loop blames the checkpoint, not the board:

fp8 KV needs calibrated k_scale/v_scale in the checkpoint; this NVFP4 re-export ships none (kv_cache_quant_algo: null), so fp8 trips assert layer.k_scale > 0.0 on this vLLM nightly.

Yet the shipped fleet template defaults to --kv-cache-dtype=fp8 and the GB10 reportedly boots it — on the same checkpoint. Both cannot be fully true unless the assert is arch- or backend-conditional (e.g. only hit on the sm_110 / TRITON path).

Check: on the GB10, clean-boot the cortex gear with the pinned nightly and --kv-cache-dtype=fp8. If it crashes with the same assert, the fix is the shared template default (auto), not a Thor trait — and thor.yaml gets one divergence smaller.

2. Is VLLM_ATTENTION_BACKEND dead on the GB10's pinned image?

Plan task t3 deletes the MULTIMODAL_ATTENTION_BACKEND env (VLLM_ATTENTION_BACKEND=… on the senses gear) on the grounds that the env no longer exists in the pinned nightly (--attention-config is the only honoured knob). That observation comes from Thor only.

Check: confirm on the GB10's pinned image that the env is ignored there too (senses boots identically with and without it). If the GB10 deployment is on an older image where the env still works, deleting it from the template regresses spark on the next re-init — exactly the cross-machine breakage the new golden-render tests (t13) exist to catch.

Outcome routing

• fp8 crashes on GB10 too → move kv-cache-dtype=auto to the shared template default in t3.
• fp8 boots on GB10 → keep it as an sm_110-conditional trait with provenance.
• env dead on GB10 → t3 deletes it as planned.
• env live on GB10 → t3 keeps it until the GB10 image is repinned.

Spec: docs/specs/2026-07-13-lobes-fits-the-machine-it-lands-on-one-command-det.md (Open / follow-up). Related: #105, #106, #107.

• lobes (Claude)

[OriNachum]

t10 validation evidence from the physical Thor box (2026-07-13, feat/per-machine-profiles) updates question 1:

The fp8 k_scale assert did NOT reproduce on the pinned nightly (0.23.1rc1.dev672). With clean memory, the 27B MTP checkpoint boots with --kv-cache-dtype=fp8 past weight-load, emitting warnings instead of asserting:

WARNING [kv_cache.py:148] Using KV cache scaling factor 1.0 for fp8_e4m3. / Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues.

So on Thor today, unscaled-fp8-KV is an accuracy risk, not a crash — the crash the original session attributed to the assert did not recur. What DOES reproducibly kill first boots on Thor (spark AND thor profiles alike, 4/4 attempts) is a bring-up memory race: each gear's vLLM memory-profiling window sees the co-resident gears' concurrent weight loads (Jetson unified memory counts page cache as occupied), failing either the free-memory gate or "No available memory for the cache blocks". Sequential bring-up (primary first, then the rest) boots clean and all three correctness probes pass, rerank included. Post-teardown, memory stays "used" until sync; echo 3 > /proc/sys/vm/drop_caches.

Implications for this issue's checks:

• The GB10 fp8 check (question 1) is still worth one clean boot for the record, but the sharper question is now accuracy: whether scale-1.0 fp8-KV degrades output quality on either box vs auto.

• Question 2 (is VLLM_ATTENTION_BACKEND dead on the GB10 image) stands unchanged; the env stays in the template until it's answered.

• New, tracked in the plan as risk r7: boot ordering/serialization is a per-machine divergence the per-gear knob env cannot express — likely needs compose depends_on sequencing or a serve-side staggered bring-up, and belongs with the profiles follow-up rather than here.

• lobes (Claude)

[OriNachum]

Both checks verified on the GB10 — read-only, from the live deployment. No reboot was needed: the running fleet is itself the boot evidence (cortex engine init completed on the pinned image and has served healthily since; lobes 0.40.1, all 8 containers healthy).

1. fp8-KV: the crash is (checkpoint × sm_110 backend path) — keep auto as a Thor trait

• The GB10 cortex is running --kv-cache-dtype=fp8 on the identical pinned nightly as Thor (0.23.1rc1.dev672+g93d8f834d, vllm/vllm-openai@sha256:7c5a10e9…) with the same scale-less checkpoint (kv_cache_quant_algo: null).

• Boot log shows a warning, not the assert:

  INFO  [cache.py:280] Using fp8 data type to store kv cache. …
  WARNING [kv_cache.py:134] Checkpoint does not provide a q scaling factor. Setting it to k_scale.
          This only matters for FP8 Attention backends (flash-attn or flashinfer).
  WARNING [kv_cache.py:148] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended,
          verify that k/v_scale scaling factors are properly set in the checkpoint.
  

• The warning names the conditionality itself: missing scales "only matter for FP8 Attention backends (flash-attn or flashinfer)" — backends sm_110 doesn't take (it is forced onto the TRITON path, where assert layer.k_scale > 0.0 trips).

Routing: fp8 boots on GB10 → kv-cache-dtype=auto stays an sm_110-conditional trait with this provenance; the shared template default remains fp8. The Thor hand-edit's checkpoint-only attribution was incomplete: the missing scales are real on both boxes, but only the sm_110 backend path asserts on them. (One accepted caveat: fp8-KV on the GB10 runs with scale 1.0 — a potential accuracy cost, but a separate question from the boot attribution this issue asked about.)

2. VLLM_ATTENTION_BACKEND is dead on the GB10's pinned image too → t3 deletes as planned

• The senses image (lobes/vllm-gemma4:nightly-audio, locally built) ships the same vLLM 0.23.1rc1.dev672+g93d8f834d.

• vllm.envs does not declare the variable ('VLLM_ATTENTION_BACKEND' in envs.environment_variables → False), and the live senses log prints on every poll:

  WARNING [envs.py:2035] Unknown vLLM environment variable detected: VLLM_ATTENTION_BACKEND
  

• Senses lands on TRITON_ATTN anyway, via model-side forcing, not the env:

  INFO [config.py:232] Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512).
       FA4 not available, forcing TRITON_ATTN backend.
  

Routing: env dead on GB10 → t3 deletes MULTIMODAL_ATTENTION_BACKEND / VLLM_ATTENTION_BACKEND as planned — zero regression risk on Spark; the GB10 senses backend is model-forced regardless.

Both outcome routes are now decided with provenance; closing. t3 picks these up as: keep fp8 shared default + auto as an sm_110 trait; delete the dead env.

• lobes (Claude)