Anti-abuse hardening from the Teutonic audit: OCI copy detection, CUDA self-kill, stuck-duel guillotine, and the cross-GPU noise harness

[vex0209-bt]

Summary

Stacked on #8. After the Teutonic consensus audit (96 agents, 60 mechanisms verified against source), five gaps were where Teutonic's incident-hardened defensive layer is genuinely ahead of Leoma, or where the audit's #1 open risk had no tooling at all. This PR closes them. Four are direct ports adapted to Leoma's domain; one is the harness for the risk Leoma's own chain.toml names as "bigger than any bug."

1. Catch repackaged king copies + displace front-run authors (191bfb9)

The exact-digest copy check missed the copy that matters: identical weights repackaged with a changed README or tokenizer — a new top-level digest, but the same per-layer safetensor digests. Under the old gate that model was handed a full multi-hour duel before tying the king and losing. For a subnet whose entire thesis is that GPU time is scarce, that was the wrong gate to have built narrow.

check_model_copy (ported from Teutonic's validator.py:738) compares per-layer OCI safetensor digests from one manifest fetch — a few KB, no weight download. On a content-addressed registry, identical layer digests mean identical bytes. And it adds the earliest-author path Leoma lacked entirely: a byte-identical model pushed to the registry earlier than the king is the true original, front-run by whoever got crowned first, and it displaces the king with a synthetic crown — no duel, because there is nothing to evaluate.

Both decisions rest only on the registry's own push time (Harbor push_time / manifest Last-Modified), never a client annotation a miner could backdate — so validators agree. They fail safe (reject, not crown, when timestamps are unavailable) and fail open (let the model duel when the registry can't be reached).

2. Self-kill the eval box on a poisoned CUDA context (62c6e2d)

An illegal memory access or device-side assert corrupts the CUDA allocator and every stream on the process; every later load/generate keeps failing against the dead context. Leoma's watchdog only fires when a duel stalls — a duel that fails fast on a CUDA fault would release the lock and hand the next challenger a poisoned box, wasting hours mis-rejecting an honest model as broken. (Teutonic ported this after a real 2.5h box degradation; it bites harder for Leoma because a poisoned box wastes hours, not minutes.)

The box now reports the error cleanly (so the validator classifies it cuda_fatal → TRANSIENT and retries against the restarted box) and schedules an os._exit for a supervisor restart with a fresh context, with a threading.excepthook covering CUDA faults that escape a daemon thread. OOM is deliberately not fatal — it is recoverable, and self-killing on it would be a denial-of-service on the fleet.

3. Abandon a duel stuck in flight (2996097)

Leoma moved the wall-clock bound entirely onto the eval server, so a box that keeps reporting running without ever tripping a phase budget — a disabled watchdog, a lying box, a partition where poll succeeds but the box is wedged — would hold the single in-flight slot forever and starve every other challenger. settle_inflight now abandons a duel past a generous block bound (~18h, so only a genuinely hung box trips it), sends a best-effort DELETE to stop the GPU, and frees the slot. The stuck box is charged transient, never a strike, so a one-off wedge retries while a model that hangs every time still exhausts its budget and quarantines.

4. The cross-GPU noise-measurement harness (b65f2b3)

This is the one that matters most. delta_threshold is the single number between Leoma's consensus converging and forking, and it is a hopeful constant set against a noise floor nobody has measured. The audit named this the subnet's largest open risk; chain.toml:97 already says it is "bigger than any bug."

The harness makes the constant a measured multiple of a known floor. Generate one real model on the calibration clips on each GPU type (leoma calibrate generate, on the box, using the real duel path). Then for every pair of boxes, the per-clip distance difference d_i = dist_A[i] − dist_B[i] is the cross-GPU noise — the model, clips, seed and truth are all held identical, so only the hardware differs. A self-duel between two honest boxes has a true mean advantage of zero, so whatever the bootstrap finds is the bias a marginal verdict could suffer. leoma calibrate analyze reports the floor across all pairs and flags a delta_threshold set below it:

H100 vs A100: |Δ| max=4.4e-02 p99=4.3e-02  mu_hat=-4.4e-03 lcb=-1.5e-02 ucb=+6.6e-03
recommended delta_threshold (>= 3.0x): 0.045408
FAIL: current delta_threshold 0.0025 is BELOW the measured noise floor — two honest
validators can fork. Raise it to >= 0.045408

The analysis is pure and unit-tested (13 tests); the generate step runs the real path on a GPU.

5. Corrected the rate-limit rationale (1bbe564)

The audit caught a factually wrong justification in my own rate_limit.py docstring — it claimed Teutonic's hotkey-burn "does not transfer: their eval is cheap," but a permanent burn is independent of eval cost and would in fact be more spam-proof. The real reason Leoma uses a softer cooldown+cap is a deliberate product choice (keying the seen-set on hotkey|digest lets a miner iterate). The comment now owns that tradeoff.

Tests

504 passing, up from 491. Highlights:

• test_a_repackaged_copy_never_reaches_the_GPU, test_an_earlier_author_takes_the_crown_with_no_duel, test_missing_timestamps_REJECT_never_crown
• test_a_cuda_fatal_duel_reports_its_error_AND_self_kills, test_an_oom_does_NOT_self_kill
• test_a_duel_past_the_bound_is_abandoned_and_the_slot_freed, test_the_stuck_box_is_never_blamed_on_the_miner
• test_a_delta_below_the_floor_FAILS_loudly, test_the_floor_is_the_WORST_pair_not_the_average

Verified by building a wheel and importing every new module from a clean venv outside the source tree.

What this does not fix

The harness measures the floor; it does not lower it. If calibration shows the real cross-GPU noise is larger than any workable delta_threshold, that is a signal the metric itself (LPIPS on diffusion-generated frames) is too noisy for cross-hardware consensus, and the answer is a structural change — a more reproducible metric, or restricting the fleet to one GPU class — not another constant. This PR gives you the number to make that call with; it can't make the number small.