Tracking: DLIO removes persistent_workers=True between epochs (root cause of #499 cosmetic traceback, also adds per-epoch respawn cost)

[FileSystemGuy]

Summary

Tracking issue for an upstream-in-DLIO behavior change that is the root cause of the intermittent OSError: [Errno 39] Directory not empty: '/tmp/pymp-*' traceback reported in #499. Verified to have zero impact on AU / throughput scoring, but still has real (small) side effects worth fixing upstream when convenient. Filed here rather than against mlcommons/DLIO_local_changes because that's a downstream fork that doesn't accept issue creation, and the further-upstream repo is too far removed from storage-WG concerns.

What changed (DLIO PR mlcommons/DLIO_local_changes#26)

PR #26 (merged 2026-06-19) intentionally disabled persistent_workers=True on the resharded branch of TorchDataLoader.read():

# dlio_benchmark/data_loader/torch_data_loader.py @ 17d982a3, line ~761
kwargs={'multiprocessing_context': self._args.multiprocessing_context,
        'prefetch_factor': prefetch_factor}
# persistent_workers=False: workers re-spawn each epoch to pick up
# resharded file lists from updated serial_args.

The other two TorchDataLoader DataLoader-construction sites in the same method (lines 693, 727) still set persistent_workers=True. The disabled branch is the one used by mlpstorage's training path. The trade-off PR #26 chose: respawn workers every epoch so they re-pickle the resharded file list via worker_init, instead of pushing updates to live workers.

Consequences

1. Intermittent ENOTEMPTY traceback (storage#499)

With persistent_workers=False, the DataLoader tears down its worker pool at end-of-epoch and spawns a fresh one for the next. Each worker uses Python multiprocessing's per-process temp dir (/tmp/pymp-<rand>). Worker shutdown registers a _remove_temp_dir finalizer that rmtree's that directory. Occasionally a parent-side write (Queue flush, tensor handle, shm cleanup) lands in the temp dir AFTER rmtree walked it but BEFORE os.rmdir runs → ENOTEMPTY. Race depends on parent queue drain order vs. child rmtree walk; reshard alltoall (new in PR #26) widens the gap.

2. Per-epoch worker respawn cost (NEW)

Even when the cleanup race doesn't fire, every epoch transition now incurs:

• Tear down all DataLoader workers (8 per rank by default)
• Spawn fresh workers: process fork + Python interpreter init + module import + dataset re-init + pickle.loads(serial_args)

In the storage#499 reporter's log, "Worker pre-warm complete for epoch 5" arrives ~2.3s after "Ending epoch 4". With persistent_workers=True (pre-PR#26 behavior), that gap was sub-second.

3. /tmp leak when the race fires

The orphaned /tmp/pymp-XXXX/ and its contents leak until reboot. Negligible on most setups; can matter on bare-metal with constrained /tmp tmpfs over very long runs.

4. User confusion

Reporters see a scary traceback at every epoch boundary on some runs and may file duplicate issues / lose trust in the run output.

Verified: NO impact on AU / throughput scoring

Investigated explicitly. Inter-epoch wall-clock time does NOT enter any reported metric:

• Submission checker (mlpstorage_py/submission_checker/checks/training_checks.py:341-342) reads exactly train_au_mean_percentage and train_au_meet_expectation.
• DLIO utils/statscounter.py:465 computes per-epoch AU as total_compute_time / total_time, where total_time = end_timestamp - start_timestamp (reset inside each epoch's training loop, minus excluded warm-up/cool-down step durations). Run-level AU is np.mean(train_au) across per-epoch values (line 195). The respawn/reshard gap sits OUTSIDE both start_timestamp and end_timestamp of every epoch.
• DLIO's whole-run total_elapsed_time = end_run_timestamp - start_run_timestamp is computed at statscounter.py:189 but never written into summary metrics or referenced elsewhere (grep confirms one usage only — the assignment). Dead variable.
• Rules.md §3.3.2 defines AU = (total_compute_time / total_benchmark_running_time) * 100. The implementation matches this in spirit: total_benchmark_running_time is the per-epoch training-loop time accumulated across epochs, not start-to-finish wall-clock.
• mlpstorage_py/run_summary.py and mlpstorage_py/rules/ don't derive any wall-clock-based metric on top of DLIO's summary.json.

Fix options (rank-ordered, safest first)

1. Restore persistent_workers=True on the resharded branch; deliver resharded file list to live workers via shared per-epoch file + worker_init_fn, or dataset.set_epoch(...). Eliminates spawn/teardown cycle, the race, and the per-epoch respawn cost. Highest impact, cleanest. Recommended.
2. Explicit drain before pre-warm: in DLIO main.py after loader.finalize(), deterministically dispose old iterator (train_loader._iterator = None; gc.collect(); join _MultiProcessingDataLoaderIter._workers). Keeps the respawn model but closes the cleanup race window.
3. Swallow ENOTEMPTY: monkey-patch multiprocessing.util._remove_temp_dir to retry on ENOTEMPTY. Pure workaround — hides the traceback but doesn't address the respawn cost or leak.
4. Switch start method to forkserver: requires CUDA + mpi4py-init validation; out of scope as a quick fix.

Reproducer

In a dev branch of DLIO (local checkout at mlcommons/DLIO_local_changes):

1. Flip persistent_workers=True on the changed branch in dlio_benchmark/data_loader/torch_data_loader.py (~line 761).
2. Re-run storage#499 reporter's workload.
3. Traceback should disappear AND inter-epoch latency should drop sub-second.

Priority

Low. No score impact, no run failure, no data corruption. Worth fixing for cleanliness and to reduce per-epoch overhead, but not blocking any submission.

References

• storage#499 — original symptom report
• Improve large scale training file lists with distributed approach DLIO_local_changes#26 — the triggering change (merged)
• Fix issues #449 #450 #451 #464 #466 #472 #475: large-scale listing, S3 object storage, epoch-2+ AU, TFRecord via s3dlio DLIO_local_changes#27 — extra epoch-start logging / HEAD validation shifted timing and made the race more visible (no direct touch of torch_data_loader.py)
• Rules.md §3.3.2 — AU definition
• mlpstorage_py/submission_checker/checks/training_checks.py:325-352 — AU pass/fail check

[FileSystemGuy]

I believe this has been fixed along the way.