Checkpoint write throughput regresses ~41% on POSIX/`local_fs` (mlpstorage 3.0.26 → 3.0.33) - Materially change of results

[wolfgang-desalvador]

Severity: High — regression landed inside the submission window and materially changes reported results.

Summary

llama3-8b checkpoint save throughput dropped ~41% between mlpstorage
3.0.26 and 3.0.33 on an identical cluster, workload, and DLIO checkpoint
code path (POSIX filesystem, storage_type: local_fs).

Version	Commit	Checkpoint save I/O (GiB/s)
3.0.26	6576710	~12.4
3.0.33	75414e4	~7.4

Δ = ~−41% on the same hardware and workload.

Environment

• Model / mode: llama3-8b, checkpointing, num_checkpoints_write=10
• Ranks: num_processes=8 (2 × 4), dual-socket 192 vCPU clients
• Target: POSIX filesystem — storage_type: local_fs, o_direct: false,
  fsync: true

Root cause

AI Analysis below, which has been proven to be correct on real tests.

The regression is caused by PR #650 / fix(#642), which changed the
streaming-checkpoint writer subprocess context from fork to forkserver
in mlpstorage_py/checkpointing/streaming_checkpoint.py:

# 3.0.26 (~12.4 GiB/s)
ctx = mp.get_context('fork')

# 3.0.33 (#642, ~7.4 GiB/s)
ctx = _writer_mp_context()   # prefers forkserver, falls back to spawn

#642 was a correct fix for a fork-after-live-Tokio deadlock on the
object-store path (the parent starts a Tokio runtime via
ObjStoreLibStorage._preflight() before forking). However it was applied
unconditionally, including the backend='file' / local_fs path, which
never starts a parent-side Tokio runtime and therefore never had the
deadlock. forkserver writers spawn from a clean daemon started before
MPI/NUMA placement, losing the parent's copy-on-write warm state and CPU/NUMA
locality with the shared-memory buffers, consistent with the ~41% drop.

Profiling evidence

Per-checkpoint profiling on the post-#650 (forkserver) code shows the write
pipeline is starved: total time is dominated by the generation/writer split
rather than the storage device (close/fsync is negligible). Illustrative
profile output:

================================================================================
RESULTS
================================================================================
Generation:  4.1571s @ 0.45 GB/s
I/O:         1.3122s @ 1.42 GB/s
  - write:   1.3049s
  - close:   0.0073s (fsync/finalize)
Total:       4.3688s

Throughput ratio: 0.3x (gen/io)
Pipeline overhead: 4.8%
Bottleneck: Generation
Chunks: 60

Submission-window impact (why this invalidates results)

The performance-affecting change was merged to main inside the submission
window:

Change	PR	Commit	Arrived on main
fork → forkserver writer context (root cause)	#650 (fix #642)	db4e4f4	2026-07-02

Because a performance-affecting code change (db4e4f4, 2026-07-02) landed on
main during the submission window, results captured against the pre-#650 code
(3.0.26, ~12.4 GiB/s) and results captured against post-#650 code (3.0.33,
~7.4 GiB/s) are not comparable, and any submission spanning this boundary is
invalid: the same run configuration yields a ~41% different headline metric
depending solely on which side of db4e4f4 the benchmark code sits.
.

[russfellows]

Yes, this is certainly an issue, with no simple solution apparent. @FileSystemGuy , @bbelgodere , @dslik or @wolfgang-desalvador , do any of you have any suggestions? This isn't just a "let's change the code again" issue, which we can certainly do, but more important and to Wolfgang's point, this is during the submission window so changes need to not impact performance.

[dslik]

My recommendation is that we need a consistent codebase that all the submitters are required to use. Typically there would be a GitHub release (like what we did with 2.0 https://github.com/mlcommons/storage/releases/tag/v2.0 ), and this is what all the submitters should be running against.

Given that this feels like a "must-fix", this will likely require extending the submission window for checkpoint submissions. Good thing this was caught early.

[wolfgang-desalvador]

Related to this, even with the commit before this issue, I am observing a generation bottleneck as soon as I start scaling my storage system large enough. @russfellows let me know if this makes sense and you have any troubleshooting I should look into -> #689

[russfellows]

Wolfgang, David, et al. Unfortunately, I have no insights, given I have never run any of this code on Host >= 2 before. Everything has been single VM runs. I would imagine it is one of a few things though: 1) Coordination between MPI processes. Is this really the problem though given that MPI was designed for this? Possibly. 2) The code not designed correctly to handle multiple systems. This is more likely, and should be discernible from a decent code review 3) Something more esoteric, like your findings on the differences between spawn and fork. Not sure why this has such a big impact, as IMHO it should not. Given that my day job is consuming 90% of my time over the last 2 weeks, and coming 2 weeks, I really have very few cycles to devote to this I am sorry to say. Regards, —Russ

…

On Jul 6, 2026, at 6:58 AM, Wolfgang De Salvador ***@***.***> wrote: wolfgang-desalvador left a comment (mlcommons/storage#682) <#682 (comment)> Related to this, even with the commit before this issue, I am observing a generation bottleneck as soon as I start scaling my storage system large enough. @russfellows <https://github.com/russfellows> let me know if this makes sense and you have any troubleshooting I should look into -> #689 <#689> — Reply to this email directly, view it on GitHub <#682?email_source=notifications&email_token=AF64UJ63FGIZ7TAKWVWHTSD5DOO63A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBZGMYDIOJUGY4KM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4893049468>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF64UJ4CSNFJX34Y2EGXMYT5DOO63AVCNFSNUABFKJSXA33TNF2G64TZHM2DKOJXGYZTMMBTHNEXG43VMU5TIOBRGMYDCNRQGEY2C5QC>. You are receiving this because you were mentioned.

[russfellows]

O forgot to say, I agree with David in that we SHOULD create a tagged release. The “current” release is still the 1 year old v2.0 tag. Also, agree it feels like BOTH a “Must fix” and “Must extend” sort of issue. Regards, —Russ

…

On Jul 6, 2026, at 3:28 AM, David Slik ***@***.***> wrote: dslik left a comment (mlcommons/storage#682) <#682 (comment)> My recommendation is that we need a consistent codebase that all the submitters are required to use. Typically there would be a GitHub release (like what we did with 2.0 https://github.com/mlcommons/storage/releases/tag/v2.0 ), and this is what all the submitters should be running against. Given that this feels like a "must-fix", this will likely require extending the submission window for checkpoint submissions. Good thing this was caught early. — Reply to this email directly, view it on GitHub <#682?email_source=notifications&email_token=AF64UJ32I3P34Y4OWNOBPGT5DNWMLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBZGEZDANRSHEYKM4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4891206290>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF64UJYFCVW42NJJWYFCR735DNWMLAVCNFSNUABFKJSXA33TNF2G64TZHM2DKOJXGYZTMMBTHNEXG43VMU5TIOBRGMYDCNRQGEY2C5QC>. You are receiving this because you commented.

[wolfgang-desalvador]

Added comment on #689 for the other issue (potential) root cause

[zhenghh04]

How about we make the thread model: fork vs fork-server an option for people to select?

[FileSystemGuy]

@wolfgang-desalvador @zhenghh04 I believe this has been fixed in PR#700. Can you confirm so that we can closed this Issue?