Corroborating #682 with an independent reproduction on different hardware. Please treat as a duplicate/confirmation of #682.
Summary
We independently reproduce the #682 checkpoint save regression on llama3-8b, POSIX / local_fs (a FUSE-backed POSIX filesystem over a single storage node). On latest main (3.0.34, DLIO 59f9480) the writer subprocess uses forkserver (PR #650); our save throughput drops ~17% vs our pre-#650 (fork) result, with load far less affected — matching #682's writer-subprocess root cause.
Data
| Phase |
pre-#650 (fork) |
3.0.34 (forkserver, #650) |
Δ |
| save |
4.89 GiB/s |
4.05 GiB/s (4.348 GB/s) |
−17.2% |
| load |
5.61 GiB/s |
5.26 GiB/s (5.646 GB/s) |
−6.3% |
Environment
- Model/mode:
llama3-8b, checkpointing, num_checkpoints_write=10, num_processes=8, single client host.
- Storage-under-test: a FUSE-backed POSIX filesystem,
storage_type: local_fs (POSIX), o_direct: false, fsync: true.
- Client: r5n.24xlarge (96 vCPU, 768 GB). Worker: i3en.12xlarge. AWS us-west-1.
- Tool: mlpstorage
main @ 3.0.34; DLIO pinned 59f9480 (forkserver) vs prior f6796ed (fork).
Root cause (same as #682)
PR #650 (db4e4f4) applied forkserver unconditionally, including the backend='file' / local_fs path that never had the Tokio-fork deadlock #642 was fixing. A conditional context (fork for file-mode, forkserver only for the object-store path) would restore file-mode throughput.
Summary
We independently reproduce the #682 checkpoint save regression on
llama3-8b, POSIX /local_fs(a FUSE-backed POSIX filesystem over a single storage node). On latestmain(3.0.34, DLIO59f9480) the writer subprocess usesforkserver(PR #650); our save throughput drops ~17% vs our pre-#650 (fork) result, with load far less affected — matching #682's writer-subprocess root cause.Data
fork)forkserver, #650)streaming_checkpoint.py_writer_mp_context).local_fs(mlpstorage 3.0.26 → 3.0.33) - Materially change of results #682's 41%, as expected: our client is r5n.24xlarge (96 vCPU) vs the reporter's 192-vCPU dual-socket. The forkserver penalty (loss of copy-on-write warm state + NUMA locality) scales with core/socket topology, so a smaller client shows less.Environment
llama3-8b, checkpointing,num_checkpoints_write=10,num_processes=8, single client host.storage_type: local_fs(POSIX),o_direct: false,fsync: true.main@ 3.0.34; DLIO pinned59f9480(forkserver) vs priorf6796ed(fork).Root cause (same as #682)
PR #650 (
db4e4f4) appliedforkserverunconditionally, including thebackend='file'/local_fspath that never had the Tokio-fork deadlock #642 was fixing. A conditional context (forkfor file-mode,forkserveronly for the object-store path) would restore file-mode throughput.