feat(competitions): GPU slices for submission runners

[sleep3r]

Why

Competitions could not use a GPU. Verified on a rented VM (2 × RTX 4090, driver 580.95.05) on 0.21.0: a competition on base-gpu ran CUDA torch without a card (torch 2.14.1+cu130 | available False | devices 0, no nvidia-smi), and nobody was told; on Kubernetes the broker refused every GPU image and the app showed it as «упала метрика». The owner asked for runners that hold a slice of a card so neural-network submissions can train.

What

Cards for submissions (Resources → «Видеокарты для посылок»). The owner gives whole cards (KERNEL_GPUS entries) to submission runners and sets runs per card (1..8). A card given to submissions leaves the rooms' pool, so a class and submissions never share a card. The panel lists every card with its model and memory (probed through docker from inside the app container, at start and every 10 min), who holds it, and the resulting share. «Как поделена машина» gets a cards bar. COMPETITION_GPUS / COMPETITION_GPU_PER_DEVICE are the env fallbacks (colloq-host, compose, .env.example).

The share is held from two sides. share = card memory / runs per card.

• torch: the harness ships site/sitecustomize.py; with COLLOQ_GPU_MEMORY_MB (share − 512 MB for the CUDA context) torch's allocator is capped at its share; a probe of torch before the real import no longer disables it. CatBoost's gpu_ram_part defaults to the share (it preallocates ~95 % of the card otherwise — measured 22.9 GB of 24 GB; with the share 6.3 GB). xgboost grows on demand.
• outside: a watchdog samples nvidia-smi in the run container as root every 3 s (a uid-1000 participant cannot kill it — checked on the card); above share + 256 MB on two samples in a row the run is stopped with our own words; if the share cannot be read several times in a row on a shared card the run is stopped too (fail closed).
• A run killed by a neighbour's burst on the same card is put back in the queue once instead of being judged.

Per competition. A «Видеокарта: нет / срез ≈ N ГБ» switch, only on CUDA environments (the server refuses the combination otherwise). With the switch on and no card given, uploads and the baseline check are refused at the door with a reason (gpu_unavailable); metric checks and rescores still work (the metric never gets a card). Existing competitions keep gpu: false and behave as before.

Queue. The take binds a card and the run's share/per-card count on the queue row; a card's load is counted in shares, so raising runs per card mid-class cannot over-commit it. A GPU job waiting for a card never blocks CPU jobs. Deleting a competition stops its runs first. Rooms never get a card the queue has bound, even before the run's container is up. Places and ETAs simulate the card shares.

Environment kaggle-gpu = base-gpu + the kaggle-base pins + seaborn 0.13.2, opencv-python-headless 5.0.0.93, timm 1.0.30 (built on a card: 12.7 GB, pip check clean, torch/xgboost/catboost/timm on CUDA). base-gpu now pins torch 2.14.1 / torchvision 0.29.1 (CUDA 13.0 wheels, driver 580+); the Vast rental scripts take CUDA 13 hosts only. GPU images stay local builds (publish.yml unchanged).

Kubernetes. GPU notebook jobs request gpu.competitions.resource × count (the cluster's device plugin defines the unit: time-slicing, MPS, MIG), with the GPU RuntimeClass/selector/tolerations and a /dev/shm volume; GPU images run every other job kind on the CPU; at most gpu.competitions.slots GPU notebooks at once so CPU work is not starved; API refusals (quota, RuntimeClass) come back as a gpu deferral; GPU Pods get a pull allowance. Helm values/schema/templates/README, release.py.

UI (Paper page «15 · Видеокарты для посылок», G1–G4): the Resources block and bar, the editor's GPU row with its states, «GPU 0» chips and «видеокарты N из M» on the console, the card in «Работают сейчас», the participant's «Видеокарта» condition, the stop and OOM texts (with a hint: «У посылки срез видеокарты 6 ГБ: уменьшите batch_size или модель.»).

Verified on hardware (before the review fixes)

Letters competition (CNN seminar), reference notebook, 2 CPU / 4 GB / 600 s, kaggle-gpu, VM 61 vCPU / 98 GB / 2 × RTX 4090:

• probe: one card as cuda:0, torch capped at 5.50 GiB for a 6141 MB share; a run that lifted the cap and took 10.4 GB was stopped by the watchdog; an 8 GB allocation failed inside the cell with the hint; a class on kaggle-gpu got the other card; all cards given to submissions → the class refused with the new reason.
• 25 simultaneous submissions:

setup	run (median)	verdict median / last
CPU only (0.21.0, 22 × 2 cores)	159 s	162 / 235 s
1 card × 4 (6 GB)	25 s	102 / 175 s
2 cards × 4 (6 GB)	29 s	58 / 88 s
1 card × 8 (3 GB)	29 s	60 / 117 s
2 cards × 8 (3 GB)	31 s	36 / 62 s

A mixed load (10 GPU + 4 CPU) showed the CPU submissions do not wait behind the GPU ones.

• After the review the mechanisms the fixes rely on were checked on the card (root docker exec nvidia-smi in the hardened container; uid 1000 cannot kill it; CatBoost with gpu_ram_part stays within the share). The fixed tree itself was not rerun on a card (the rented VM ran out of credit; the owner chose to release without a second hardware run).

Review

Seven-dimension adversarial review: 45 findings, all confirmed by an independent judge (19 important, 20 minor, 6 nits), fixed in six lanes plus a seams pass, each fix with a regression test that fails without it.

Tests

npm run typecheck: 0 errors. npm test (Node 22, helm 3.19.0): 5337 tests, 5295 pass, 41 skipped, 1 fail — tests/class-zip.test.mts, the known local-only failure (reads PUBLIC_URL from the repo's .env); CI is unaffected.