CLOSED training on unet3d/b200 forces ~1 client machine per accelerator on 100 Gbps hardware — please re-admit h100/a100 to CLOSED

[gaikwadabhishek]

CLOSED training is restricted to b200/mi355 (ACCELERATORS_CLOSED in
mlpstorage/config.py). For unet3d, the b200 emulation demands ~5.9 GiB/s
of read bandwidth per accelerator, which is essentially the entire practical
throughput of a single 100 Gbps NIC. Since most cloud VM/BM shapes ship one
100 Gbps NIC (and commonly one GB200 per instance), a submitter can feed only
one b200 per client machine — so scaling the accelerator count N means
adding one whole (expensive) client node per accelerator.

We'd like the WG to consider re-admitting h100 and/or a100 to CLOSED for
training (the workload configs already ship). Their lower per-accelerator
demand makes CLOSED submissions feasible and affordable on commodity cloud
hardware.

The numbers (derived from configs/dlio/workload/*.yaml)

unet3d — record 146,600,628 B, batch_size 7 ⇒ 978.66 MiB read per step
per accelerator; demand = that ÷ computation_time:

Accelerator	computation_time	Demand / accelerator	Accelerators per 100 Gbps NIC*
b200	0.162 s	5.90 GiB/s	~1
h100	0.323 s	2.96 GiB/s	~2
a100	0.636 s	1.50 GiB/s	~3–4

* Based on the ~5.7–7 GiB/s a single 100 Gbps NIC delivers in practice on these
workloads (see measured below); theoretical line rate is 11.6 GiB/s but real
efficiency on 140 MiB random-ish object reads is ~50–60%.

retinanet is not bandwidth-bound and is unaffected: b200 (batch 24 /
0.04755 s) and mi355 (batch 36 / 0.071325 s) both demand only ~0.152 GiB/s per
accelerator (small-file/IOPS-bound, ~505 samples/s/accel, 315 KB records).

Why this matters for submissions

• Most publicly available cloud instances (and many on-prem BM shapes) expose a
  single 100 Gbps NIC, and GB200 instances commonly present one GB200.
• CLOSED unet3d therefore caps at 1 accelerator per client machine, so
  raising N requires linearly more client nodes — each an expensive, separately
  provisioned instance.
• With h100/a100 admitted, one 100 Gbps client could feed 2–4 accelerators,
  letting submitters demonstrate meaningful scale without a large client fleet.

[dslik]

Won't these two configurations (4x A100, 2x H100 vs. 1x GB200) place generate approximately the same throughput that is placed on the storage system, since the NIC is the per-client bottleneck? Maximizing the number of accelerators isn't the benchmark goal, so I don't understand the benefit.

[gaikwadabhishek]

@dslik yeah, the goal is samples/s — and to get there I have to find the highest
N I can while keeping AU ≥ 90%.

That's exactly where GB200 bites me: a single b200 hits the AU target but only
pulls ~half of my client node's 100 Gbps NIC — the other ~40-50% just sits idle,
purely because of b200's batch/compute params. And I can't add a second b200 to
use that headroom (it overshoots the NIC and tanks AU).

So the only way for me to push samples/s higher is to add more client nodes. And
since the dataset rule is 5× per-host memory, every node I add balloons the
required dataset — which means a lot more datagen time and cost just to run the
test. There should have some better way to use 100% of my client machines.

[dslik]

That makes sense - It's a really unfortunate misalignment between the NIC bandwidth and what the GB200 will use. @FileSystemGuy, what is your thoughts on this?

[FileSystemGuy]

@gaikwadabhishek @dslik I agree that the chunkiness is unfortunate. IMHO at it's core this comes from our historical position that we are simulating accelerators, whole accelerators. Back when accelerators were "slow" (compared to today's beasts), the bin-packing problem of how many accelerators a client host could run and therefore how much resource was wasted, was a much smaller issue.

A statistical model with a generic workload generator that used that model would not be simulating whole accelerators. It would be applying the load it was told to apply, which would correspond to an arbitrary set of whole and/or fractional accelerators. Yet more reason to move to new workload generators in v4.0.

I don't think we can do anything about the bin-packing problem in v3.0, and not just because of the lateness in the process.