A multi-client ... training <model> run object run over an S3 backend (NVIDIA
AIStore here, but any S3/MinIO target applies) never starts training. There
are two independent failures, hit in sequence:
-
(A) — mlcommons/storage, fix included below. The S3 credentials/endpoint/
tunables that s3dlio reads from each rank's environment are not forwarded to
remote MPI ranks. OpenMPI does not propagate env to remote ranks, and the
mpirun prefix only opts-in DLIO_DROP_CACHES_TIMEOUT. So ranks on every host
except the launcher have no AWS_ENDPOINT_URL/creds/S3DLIO_* and cannot reach
the backend. Single-host runs hide this (all ranks inherit the launcher's shell).
-
(B) — deeper blocker, appears to be in the DLIO engine
(mlcommons/DLIO_local_changes). Even after (A) is fixed and every rank can
reach S3, the run deadlocks in DLIO's initialize(): the launcher-host rank
reaches the step-count allreduce and blocks forever, while the remote-host ranks
are stuck earlier in initialize() and never reach that collective.
Single-host runs of the same workload work correctly. This makes multi-client
object-mode training (the headline scale-out path) currently unusable.
Environment
- mlpstorage
3.0.25 (origin/main),
- s3dlio
0.9.102, DLIO mlcommons/DLIO_local_changes rev 252a54b18d113541e1e8e24832921fac0a7f2b96
- OpenMPI over TCP:
OMPI_MCA_pml=ob1 btl=tcp,self,vader mtl=^ofi btl_tcp_if_include=ens300np0
- Backend: NVIDIA AIStore S3 endpoint
http://<ip>:51080/s3/
Reproduction
# .env: BUCKET, AWS_ENDPOINT_URL, AWS creds, STORAGE_LIBRARY=s3dlio, S3DLIO_FOLLOW_REDIRECTS=1
mlpstorage open training unet3d run object \
--accelerator-type b200 --num-accelerators 3 \
--params dataset.num_files_train=56000 \
--client-host-memory-in-gb 500 --data-dir data/unet3d \
--hosts <3 IPs> --num-client-hosts 3 \
--dlio-bin-path <venv>/bin --systemname <name> --results-dir <init'd-dir> \
--skip-validation
(1 rank/host = the minimal multi-host case; 4/host etc. behave identically.)
Problem A — S3 env not forwarded to remote ranks
Root cause
mlpstorage_py/benchmarks/dlio.py, where the mpirun prefix is assembled (~L575-585):
mpi_prefix = generate_mpi_prefix_cmd(...)
# Forward DLIO_DROP_CACHES_TIMEOUT to ranks ... (mlcommons/storage #487)
if 'DLIO_DROP_CACHES_TIMEOUT' in os.environ:
mpi_prefix += " -x DLIO_DROP_CACHES_TIMEOUT"
cmd = f"{mpi_prefix} {cmd}"Only DLIO_DROP_CACHES_TIMEOUT is opted in via -x. But the same file documents
that the S3 backend reads its config straight from each rank's env (~L136):
Credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) and the endpoint
(AWS_ENDPOINT_URL) are read directly from the environment by obj_store_lib.py
OpenMPI does not forward arbitrary env to remote ranks, so on a multi-host run
the ranks on every non-launcher host start with no AWS_* / S3DLIO_* /
STORAGE_LIBRARY / BUCKET, and their first S3 op fails/hangs.
Symptom
The remote-host ranks have none of the storage env; inspecting a remote rank's
/proc/<pid>/environ shows AWS_ENDPOINT_URL etc. absent. (After the fix below,
the same inspection shows them present, and the generated mpirun line carries the
-x AWS_… -x S3DLIO_… -x STORAGE_LIBRARY -x BUCKET flags.)
Proposed fix (mlcommons/storage, verified locally)
Opt-in the storage env vars present in the launcher environment, mirroring the
existing DLIO_DROP_CACHES_TIMEOUT idiom:
if 'DLIO_DROP_CACHES_TIMEOUT' in os.environ:
mpi_prefix += " -x DLIO_DROP_CACHES_TIMEOUT"
# Object storage: s3dlio reads creds/endpoint/tunables from each rank's env, but
# OpenMPI does not forward env to remote ranks, so a multi-host object run would
# hang on the remote ranks' first S3 op. Opt-in the storage vars from the launcher.
for _v in sorted(os.environ):
if (_v.startswith('AWS_') or _v.startswith('S3DLIO_')
or _v in ('STORAGE_LIBRARY', 'BUCKET')):
mpi_prefix += f" -x {_v}"This is harmless for local-fs runs (the vars are simply absent) and only opts-in
present vars. Verified: with the patch, -x AWS_… -x S3DLIO_… appear in the
mpirun command and remote ranks receive the full S3 env.
Problem B — DLIO deadlocks in initialize() on multi-host (the real blocker)
With (A) fixed, the run gets further but deadlocks before the first training
step, every time, on every multi-host config. The log freezes right after:
Streamed file sharding: 56000 train files across 3 ranks via round-robin (rank 0 shard: 18666 files)
and 0 MB/s is observed on all client NICs indefinitely.
What the stacks show (py-spy, default config)
Launcher-host rank (rank 0): blocked in the step-count allreduce, waiting for
all ranks that never arrive —
allreduce_min (dlio_benchmark/utils/utility.py:371) # self.comm().allreduce(value, op=MPI.MIN)
derive_configurations (dlio_benchmark/utils/config.py:762) # self.training_steps = allreduce_min(local_train_steps)
initialize (dlio_benchmark/main.py:505)
Remote-host rank: stuck earlier in initialize(), in the
streamed-sharding/listing region, and never reaches allreduce_min —
__tz_convert (libc-2.28.so) # native; py-spy --native
initialize (dlio_benchmark/main.py:270)
wrapper (python/common.py:504)
run_benchmark (dlio_benchmark/main.py:517)
The remote rank is in state R at ~100–115% CPU, the Python line is pinned at
main.py:270 across the entire observation window (5/5 samples over 25 s), with
0 connections to the S3 endpoint and 0 MPI traffic. So it is busy-spinning
in initialize() — not doing S3 I/O, not in a collective — and never reaches the
allreduce_min rendezvous the other rank is blocked on → classic collective
deadlock.
What it is NOT (ruled out)
- Not MPI/fabric. A standalone mpi4py test across all 3 hosts (manual
MPI.Init(), then Barrier, an 8 MB Allreduce, and lowercase
bcast/allreduce(MIN)/gather/allgather) completes in <0.05 s, exit 0.
- Not rank/identity detection. Remote ranks have correct
OMPI_COMM_WORLD_RANK/SIZE (e.g. 1/3), and DLIOMPI's own init
(Split_type/Split/bcast) completes.
- Not config-dependent. Reproduces identically with
reader.read_threads 0
vs 4, dataset.skip_listing True vs False (False hangs even earlier, right
after "Running DLIO with N processes"), and MPI4PY_RC_INITIALIZE 0 vs 1.
- Not env (Problem A). Reproduces with the full S3 env confirmed present on
every rank.
- Not the data. The same dataset trains fine on a single host (reads sustain
at ~NIC line rate, valid AU).
Where to look
files_pre_sharded is set unconditionally for do_train (main.py ~L260), so the
run always takes the streamed-sharding path that ends in the
derive_configurations → allreduce_min rendezvous. The remote ranks stall
somewhere in main.py initialize() between the sharding loop (~L260) and that
rendezvous (~L505) on multi-host only. Suggested next step for a DLIO maintainer:
add flush-prints through initialize() 260–505 to locate the exact line where
remote ranks stall before allreduce_min. (The persistent __tz_convert native
frame may be a profiling/log-timestamp hot path — python/common.py:504 wrapper is
a decorator around initialize() — but that is unconfirmed.)
This likely belongs in mlcommons/DLIO_local_changes, not mlpstorage.
Related
A multi-client
... training <model> run objectrun over an S3 backend (NVIDIAAIStore here, but any S3/MinIO target applies) never starts training. There
are two independent failures, hit in sequence:
(A) —
mlcommons/storage, fix included below. The S3 credentials/endpoint/tunables that
s3dlioreads from each rank's environment are not forwarded toremote MPI ranks. OpenMPI does not propagate env to remote ranks, and the
mpirun prefix only opts-in
DLIO_DROP_CACHES_TIMEOUT. So ranks on every hostexcept the launcher have no
AWS_ENDPOINT_URL/creds/S3DLIO_*and cannot reachthe backend. Single-host runs hide this (all ranks inherit the launcher's shell).
(B) — deeper blocker, appears to be in the DLIO engine
(
mlcommons/DLIO_local_changes). Even after (A) is fixed and every rank canreach S3, the run deadlocks in DLIO's
initialize(): the launcher-host rankreaches the step-count
allreduceand blocks forever, while the remote-host ranksare stuck earlier in
initialize()and never reach that collective.Single-host runs of the same workload work correctly. This makes multi-client
object-mode training (the headline scale-out path) currently unusable.
Environment
3.0.25(origin/main),0.9.102, DLIOmlcommons/DLIO_local_changesrev252a54b18d113541e1e8e24832921fac0a7f2b96OMPI_MCA_pml=ob1 btl=tcp,self,vader mtl=^ofi btl_tcp_if_include=ens300np0http://<ip>:51080/s3/Reproduction
(1 rank/host = the minimal multi-host case; 4/host etc. behave identically.)
Problem A — S3 env not forwarded to remote ranks
Root cause
mlpstorage_py/benchmarks/dlio.py, where the mpirun prefix is assembled (~L575-585):Only
DLIO_DROP_CACHES_TIMEOUTis opted in via-x. But the same file documentsthat the S3 backend reads its config straight from each rank's env (~L136):
OpenMPI does not forward arbitrary env to remote ranks, so on a multi-host run
the ranks on every non-launcher host start with no
AWS_*/S3DLIO_*/STORAGE_LIBRARY/BUCKET, and their first S3 op fails/hangs.Symptom
The remote-host ranks have none of the storage env; inspecting a remote rank's
/proc/<pid>/environshowsAWS_ENDPOINT_URLetc. absent. (After the fix below,the same inspection shows them present, and the generated
mpirunline carries the-x AWS_… -x S3DLIO_… -x STORAGE_LIBRARY -x BUCKETflags.)Proposed fix (
mlcommons/storage, verified locally)Opt-in the storage env vars present in the launcher environment, mirroring the
existing
DLIO_DROP_CACHES_TIMEOUTidiom:This is harmless for local-fs runs (the vars are simply absent) and only opts-in
present vars. Verified: with the patch,
-x AWS_… -x S3DLIO_…appear in thempirun command and remote ranks receive the full S3 env.
Problem B — DLIO deadlocks in
initialize()on multi-host (the real blocker)With (A) fixed, the run gets further but deadlocks before the first training
step, every time, on every multi-host config. The log freezes right after:
and
0 MB/sis observed on all client NICs indefinitely.What the stacks show (py-spy, default config)
Launcher-host rank (rank 0): blocked in the step-count allreduce, waiting for
all ranks that never arrive —
Remote-host rank: stuck earlier in
initialize(), in thestreamed-sharding/listing region, and never reaches
allreduce_min—The remote rank is in state
Rat ~100–115% CPU, the Python line is pinned atmain.py:270across the entire observation window (5/5 samples over 25 s), with0 connections to the S3 endpoint and 0 MPI traffic. So it is busy-spinning
in
initialize()— not doing S3 I/O, not in a collective — and never reaches theallreduce_minrendezvous the other rank is blocked on → classic collectivedeadlock.
What it is NOT (ruled out)
MPI.Init(), thenBarrier, an 8 MBAllreduce, and lowercasebcast/allreduce(MIN)/gather/allgather) completes in <0.05 s, exit 0.OMPI_COMM_WORLD_RANK/SIZE(e.g.1/3), and DLIOMPI's own init(
Split_type/Split/bcast) completes.reader.read_threads0vs 4,
dataset.skip_listingTrue vs False (False hangs even earlier, rightafter "Running DLIO with N processes"), and
MPI4PY_RC_INITIALIZE0 vs 1.every rank.
at ~NIC line rate, valid AU).
Where to look
files_pre_shardedis set unconditionally fordo_train(main.py~L260), so therun always takes the streamed-sharding path that ends in the
derive_configurations→allreduce_minrendezvous. The remote ranks stallsomewhere in
main.py initialize()between the sharding loop (~L260) and thatrendezvous (~L505) on multi-host only. Suggested next step for a DLIO maintainer:
add flush-prints through
initialize()260–505 to locate the exact line whereremote ranks stall before
allreduce_min. (The persistent__tz_convertnativeframe may be a profiling/log-timestamp hot path —
python/common.py:504 wrapperisa decorator around
initialize()— but that is unconfirmed.)This likely belongs in
mlcommons/DLIO_local_changes, not mlpstorage.Related
DLIO_DROP_CACHES_TIMEOUTforwarding via-x(the existing single-varprecedent this fix extends).
(adjacent object-mode plumbing).
object(S3) mode —statvfsruns on thes3://URI #568 [closed] / fix(#568): skip CAP-01 statvfs for object-mode DLIO runs #579 [closed] — CAP-01 object-modestatvfsskip(another object-mode gate fixed recently).