DLIO local_changes: Make `skip_head=True` the default for large-record object-storage training (unet3d)

[bissantz]

Summary.
_s3_stream_s3dlio() hardcodes skip_head=False for records ≥ 16 MiB
(dlio_benchmark/reader/_s3_iterable_mixin.py#L407). The HEAD supplies the object size
that activates s3dlio's concurrent range-split engine (own threshold: 32 MiB, S3DLIO_RANGE_THRESHOLD_MB), so every unet3d sample (139.8 MiB) becomes 1 HEAD + ~4–5 concurrent range-GETs instead of one whole-object GET.

Impact.
For the same configuration (same read_threads,prefetch_window, dataset), the forced HEAD+range path costs ~20 AU points vs whole-object GETs — the difference between unet3d passing and failing its 90% AU bar.

Actual finding
The comment above the heuristic justifies the HEAD as enabling range-split from epoch 1 instead of epoch 2+ (via s3dlio's size cache). Per-epoch AU shows otherwise: with skip_head=True both epochs run at identical AU (no automatic epoch-2 range-split — the per-process size cache does not survive DLIO's per-epoch DataLoader worker re-spawn), and with skip_head=False both epochs pay the full penalty. In practice the flag selects range-split always vs never, not epoch-1 vs epoch-2+. DLIO already parallelizes across objects (read_threads × prefetch_window); intra-object range parallelism only multiplies the request count ~5× in the case for unet3d and contends with the pipeline's own in-flight window. Since skip_head is not user-tunable and is in neither CLOSED nor OPEN allowed params, this default alone determines what every object-storage submission measures.
My suspicion is that the clearing caches call in between epochs the mlpstorage executable reverts the DLIO optimization and therefore the caches stay cold.

Proposed change.
In dlio_benchmark/reader/_s3_iterable_mixin.py, line 407 (mlcommons/DLIO_local_changes @ f6796ed): default to whole-object GETs for all record sizes; keep HEAD+range-split as an opt-in for stores where per-stream bandwidth is the bottleneck.

-        # skip_head controls whether s3dlio issues a HEAD before each GET.
-        # Default (True): skip HEAD, do plain GET, cache the size — epoch 2+
-        # will range-split correctly for large objects automatically.
-        # False: issue HEAD first so range splitting can fire from epoch 1.
-        # We override to False only when we know objects are >= 16 MiB, since
-        # those benefit most from parallel range GETs on the very first epoch.
-        record_bytes = getattr(args, "record_length", 0) or 0
-        skip_head = not (record_bytes >= 16 * 1024 * 1024)
+        skip_head = True

[FileSystemGuy]

Thanks for the detailed writeup — the diagnosis is credible and the code sites all check out. Since this default alone determines what every object-storage training submission measures, I think the useful framing is when each choice is the better model of the real workload, not just what the current code does.

When skip_head=False (HEAD + range-split) is the right default

The intent of forcing HEAD on large records is to expose intra-object parallelism within a single record read. Because s3dlio needs the object size to decide whether to range-split (own threshold: 32 MiB), doing HEAD first lets it fire the range-split engine on the first read rather than after a size-cache warmup. This is a real win when:

• Single-stream bandwidth to the store is the bottleneck. A single whole-object GET is limited by the throughput of one TCP stream / one backend. Range-split gives ~4–5 concurrent streams pulling non-overlapping byte ranges of the same object, and that concurrency can outrun the single-stream cap.
• Per-record request latency dominates. For very large objects on backends where GET latency scales with object size (some tiered / cold stores), splitting into parallel ranges reduces wall-clock even at the same aggregate byte rate.
• DLIO's outer-loop parallelism is not saturating the store yet. If read_threads × prefetch_window is small relative to what the store can serve, adding intra-object parallelism is pure win.

When skip_head=True (plain whole-object GET) is the right default

• DLIO's outer-loop parallelism already saturates the store. In unet3d's regime, read_threads × prefetch_window typically drives dozens-to-hundreds of in-flight GETs across objects. Once the store is saturated at the object level, layering ~5× per-object range parallelism on top just multiplies the request count without adding throughput, and pushes against per-endpoint request-rate limits (SlowDown / 429 territory at scale — see DLIO checkpoint discovery: all-ranks LIST throttles and aborts intact reads #667 for a nearby example).
• Per-connection bandwidth is not the constraint. Modern object stores (S3, MinIO, GCS, Azure Blob, gateway-backed NFS/POSIX bridges) typically serve single-stream reads at rates well above what a single training worker can consume when the outer loop is also pulling other objects concurrently.
• You want request count minimized. One GET per record is O(N) requests for N objects; HEAD+range is O(N) HEADs + O(5N) GETs. For 512-rank submissions this compounds fast.

What the reporter's per-epoch measurement is actually telling us

The comment above the heuristic justifies the override as "HEAD enables range-split from epoch 1 instead of epoch 2+ (via the s3dlio per-process size cache)." The reporter's empirical finding is that per-epoch AU is flat under skip_head=True on this DataLoader setup — i.e., the promised epoch-2+ optimization does not materialize. The most likely mechanism is DataLoader worker respawn between epochs: PyTorch's default is persistent_workers=False, so each epoch gets fresh worker processes with an empty s3dlio in-process size cache. If that's the actual behavior (which matches the code I can see — I couldn't find persistent_workers=True set anywhere in dlio_benchmark/reader/), then the current default isn't "range-split from epoch 1 vs epoch 2+" — it's "range-split every read, forever, on every epoch". That reframes the tradeoff considerably.

The comparability cost, stated bluntly

If we flip the default, every prior CLOSED result on object-storage unet3d becomes non-comparable to results measured under the new default. A ~20 AU delta is well outside run-to-run noise, so this isn't a fuzzy "results may vary slightly" — it's a hard break in the comparison. Every submitter who has already run unet3d over object storage would need to re-run.

The offsetting upside is that the change would apparently raise everyone's AU on the same hardware — likely welcomed given that unet3d's 90% AU bar has been the wall a lot of object-storage bring-ups have hit. That doesn't erase the re-run cost, but it does make the re-run land in a friendlier place for the submitters doing the extra work.

What would strengthen the case before we act

Before flipping the default, I think it's worth asking two or three other submitters to reproduce the delta on their own object-storage bring-ups (Milvus/S3, MinIO, gateway-backed NFS, etc.) so the "~20 AU points" number isn't a single-lab result. If two or three independent environments see the same directionality — even if the magnitude varies — the argument that the current default is empirically the wrong model of the workload becomes hard to argue against, and the invalidation cost becomes a one-time acceptable price for a spec that better matches how the workload actually runs in production.

Independent of the default choice, the code comment on lines 402–406 (as of DLIO_local_changes f6796ed) is describing behavior that doesn't appear to happen in practice on this codebase's DataLoader setup — it's worth correcting even if we keep the heuristic.

[russfellows]

Thanks for the detailed report, and to @FileSystemGuy for the thorough tradeoff analysis. I'm the maintainer of s3dlio — a couple of facts from the library side that reframe this a bit.

s3dlio's own default is already skip_head=True (no HEAD, ever) — added in v0.9.100. The skip_head = not (record_bytes >= 16 * 1024 * 1024) logic quoted here lives in _s3_iterable_mixin.py, in this repo's DLIO_local_changes, not in s3dlio. That code is overriding s3dlio's default to force HEAD+range-split on for records ≥16 MiB. So this isn't a case of s3dlio doing the wrong thing by default — it's DLIO_local_changes opting back into HEAD for large records, on the theory that it buys range-split parallelism from epoch 1.

There's already a zero-code-change way to get the behavior this issue is asking for, without touching _s3_iterable_mixin.py at all: set S3DLIO_SKIP_HEAD=1 in the process environment before the benchmark starts. Mechanically, skip_head is backed by a process-wide latch in s3dlio (s3_utils.rs) — once anything sets it true (a loader option or this env var), it stays true for that process's lifetime. The env var is checked lazily on the very first GET, so if it's set, it wins over _s3_iterable_mixin.py's hardcoded skip_head=False for large records — every object in that process skips HEAD, full stop. Submitters can adopt this today in their run config, with no dependency on a DLIO_local_changes patch landing or a re-release of anything.

That also sidesteps the CLOSED/OPEN-params question raised above — it's a process environment setting, not a benchmark harness parameter, so it doesn't need to go through the params allowlist any more than AWS_REGION would.

I've also just pushed a doc fix on the s3dlio side (S3DLIO_SKIP_HEAD entry) — this env var existed and was documented in our data-loader and changelog docs, but was missing from our environment-variable reference, which is plausibly why it wasn't obvious as an option here. Full env-var reference: https://github.com/russfellows/s3dlio/blob/main/docs/Environment_Variables.md

The separate question of whether _s3_iterable_mixin.py's default should change (and the worker-respawn / persistent_workers analysis above) is a DLIO_local_changes / MLCommons Storage WG call, not something I can speak to from the library side — but from s3dlio's side, the lever to make skip_head=True win for large objects already exists and ships in the current release (v0.9.106).

My recommendation: no code change needed in mlperf-storage or DLIO_local_changes to fix this. S3DLIO_SKIP_HEAD=1 set in the run environment fully resolves the reported behavior today, without touching the harness, without a params-allowlist change, and without any risk of breaking the size-conditional logic for anyone who wants HEAD+range-split on large single-epoch jobs (they simply don't set the var).

[russfellows]

Also, to more directly answer @FileSystemGuy very good questions. The multi-part download is VERY object storage system dependent. Many object storage systems get significantly better performance when range byte get (aka multi-part download) is enabled. So @bissantz , it seems that your system does not react in this way. That is fine, and that is what the environment variable settings are for, but it is NOT a bug as is, since every object storage system I have tested to date has benefited from this setting.