Generate king and challenger concurrently on separate devices (throughput, not consensus)

[vex0209-bt]

Summary

King and challenger generate sequentially today — even on an 8×H100 box where seven GPUs sit idle for the length of every duel. Clips are independent and both duelists use the same per-clip seed, so they can generate at the same time on separate devices. This makes that concurrent, roughly halving duel wall-clock when enabled.

Why this is a throughput setting, not a consensus one

Generation was already conceded to be non-bit-exact across GPU architectures — delta_threshold exists precisely to absorb that noise regardless of its source. Whether two duelists happen to share a physical card or run on two different (but identical) cards in the same box is exactly that kind of generation-side noise, not a fact about the exam. So this is:

• Resolved from the eval box's own environment (LEOMA_CONCURRENT_GENERATION, LEOMA_KING_DEVICE, LEOMA_CHALLENGER_DEVICE) — never sent by the validator, never hashed into consensus_digest.
• Off by default. Today's sequential single-device behavior is completely unchanged unless explicitly enabled, and it falls back automatically with fewer than 2 CUDA devices (or if two overrides collide on the same device) rather than silently generating "concurrently" on one GPU's queue.
• Diagnosable. The device each duelist ran on is recorded in the verdict's audit.generation block, so a distance disagreement between validators can be checked against a device asymmetry before anyone assumes a real bug.

⚠️ Operator note

Enabling this changes what noise delta_threshold has to absorb — even two identical H100s in one node are not provably bit-identical generators. Re-run leoma calibrate with it enabled before relying on it in production; you want to calibrate against the runtime configuration you actually intend to run, not the one you happened to measure before turning this on. (The testnet runbook, docs/TESTNET_RUNBOOK.md, lives in #10 — I'll fold this cross-reference in there once that merges, rather than duplicate the file on this branch.)

What changed

• leoma/eval/devices.py (new) — resolve_duel_devices(), a pure function deciding where each duelist generates: disabled → both None (today's behavior); fewer than 2 devices → falls back with an explanation; 2+ devices → cuda:0/cuda:1 (or explicit overrides, trusted even against a low reported device count); a collision (explicit or via a default) → falls back rather than reporting false concurrency.
• leoma/eval/video_runner.py — run_duel(..., concurrent=False): when True, each clip's two raw generation calls are submitted to a 2-worker ThreadPoolExecutor and both awaited before validation/scoring proceeds exactly as before. Only the raw pipeline call is threaded — digesting, scoring, the freeze gate, copy-of-king, and the bootstrap all stay single-threaded and unchanged. The executor is created once per duel and torn down in finally, so it can't leak across an early-stop return, an exception, or a cancellation.
• leoma/eval_server.py — resolves devices once per duel from the box's env (never per-request), passes them into both load_video_pipeline calls, threads concurrent into duel(), and records the decision in the audit block. The king's warm-pipeline cache is unaffected: device assignment is fixed for the life of the process, and the cache is too.
• env.eval.example — documents the three new env vars and the recalibration requirement.

Tests

526 passing, up from 504. The headline test is test_concurrent_and_sequential_produce_byte_identical_verdicts — it runs the same duel both ways and asserts every distance, digest, and verdict field matches exactly; if that one ever fails, this feature has become a consensus bug. Also: a real wall-clock speedup test (using time.sleep, which releases the GIL, so it's a reliable way to prove overlap without a GPU), exception propagation from either side, cancellation between clips still works, degenerate-generation detection still fires, and no thread leak across repeated runs (including on the exception path). resolve_duel_devices has its own full unit-test file covering every fallback and override case.

Verified the wheel ships eval/devices.py and imports it from a clean venv outside the source tree.

---

Update: two bugs found and fixed in adversarial review

1. A dropped exception could hide a poisoned CUDA context. _generate_pair called king_future.result() before chall_future.result() — if the king's generation raised a benign error, the challenger's exception was never retrieved at all (an exception nobody calls .result()/.exception() on is simply gone). If that dropped exception was CUDA-fatal, the eval server's self-kill check never saw it, released the lock, and handed the next challenger a poisoned GPU. Fixed: both futures are always retrieved, and whichever side is CUDA-fatal is what propagates. is_cuda_fatal moved to eval/errors.py so the eval server and the duel runner check the same token list rather than risking two definitions drifting apart.
2. Device pinning was needlessly coupled to the concurrency flag. resolve_duel_devices discarded explicit device overrides whenever LEOMA_CONCURRENT_GENERATION was off — but pinning ("which devices to load onto") and concurrency ("do the two generations overlap in time") are different questions. An operator running several eval-server processes on one multi-GPU box (one per EVAL_SERVER_URLS entry, from the sibling PR) wants each process's king+challenger spread across its own device pair regardless of whether that process also runs within-duel concurrency. Overrides are now honored for loading either way.

33 new/updated tests for both fixes (up to 537 total), including a direct reproduction of the review's exact failure scenario (king benign, challenger CUDA-fatal) and confirmation that the propagated exception is one is_cuda_fatal() actually recognizes.