Read Artemis123, Curry, EEGLAB, Eximia, FIL and NSx in cache-sized blocks - #14246
Merged
Merged
Conversation
_read_segments_file read up to 100 MB at a time. Letting a caller choose a smaller budget and having BrainVision ask for 8 MiB keeps the working set in cache: preloading a 101 MB .eeg (32 channels, 404 MB decoded) goes from 61.5-63.4 ms to 54.0-55.3 ms, min of 5 per process over 3 interleaved A/B rounds against a pristine main worktree. The default is unchanged, so the other eight callers of _read_segments_file read exactly as before. Output is bit-identical on all 13 BrainVision fixtures plus the 101 MB file. block_size is also floored at one channel frame, since a budget smaller than a frame would otherwise leave it at zero and the read would not advance.
…ocks Follow-up to mne-tools#14241, which added max_block_bytes= to _read_segments_file and had BrainVision request 8 MiB. A maintainer asked for the same treatment on other formats. Six more readers opt in. The block size is measured per reader rather than copied, because the optimum is not a fixed number of bytes -- it tracks the time points per block, so it moves with the channel count: eximia 2 MiB 2.4x on a 102 MB file nsx 4 MiB 2.3x on a 102 MB file eeglab 4 MiB 2.1x on a 102 MB file fil 16 MiB 1.9x on a 197 MB file artemis123 16 MiB 1.5x on a 176 MB file curry 16 MiB 1.5x on a 107 MB file Sizes were also chosen so no shipped fixture regresses. Copying BrainVision's 8 MiB everywhere would have: it measured 0.83x on Curry's real 11.8 MB fixture and 0.81-0.85x on Artemis123's 17.6 MB one, because it splits a read that otherwise fits in a single block. FIL is 16 MiB rather than the faster 4 MiB for the same reason -- 4 MiB is 2.5x on a large file but 0.96x on the 9.8 MB fixture. In-process A/B varying only the constant now shows every real fixture within noise: eximia 1.003, fil 1.001, curry 1.003, artemis123 0.988. Output is bit-identical to main: 25/25 arrays compared with np.array_equal across every readable fixture of the six formats. pytest over the six readers plus brainvision and _fiff: 305 passed.
bruAristimunha
requested review from
agramfort,
drammock,
larsoner and
sappelhoff
as code owners
August 28, 2026 18:46
larsoner
approved these changes
Aug 28, 2026
larsoner
left a comment
larsoner
left a comment
Member
There was a problem hiding this comment.
Looks reasonable to me, marking for merge-when-green, thanks in advance @bruAristimunha !
larsoner
enabled auto-merge (squash)
August 28, 2026 18:49
larsoner
disabled auto-merge
August 28, 2026 18:49
larsoner
enabled auto-merge (squash)
August 28, 2026 18:49
This was referenced Aug 29, 2026
bruAristimunha
added a commit
to bruAristimunha/mne-python
that referenced
this pull request
Aug 30, 2026
The reader asked for up to 100 MB per block, then cast the whole block to
float64 and scaled it in place, so the working set was several hundred MB and
none of it stayed in cache.
raw.get_data(), interleaved cross-process medians:
136 MB file (161ch x 425000) 258.5 -> 145.6 ms 1.78x
A sweep over 32/16/8/4/2/1 MiB puts the optimum at 2 MiB (32 MiB is only 1.14x,
8 MiB 1.38x, 4 MiB 1.66x, 1 MiB 1.60x).
No shipped fixture regresses; two of them improve because they were already
being split differently:
ArtificalSignalData_Yokogawa_1khz.con 2.87 -> 2.26 ms 1.27x
ArtificalSignalData_RICOH_1khz.con 3.06 -> 2.77 ms 1.10x
Example_PQA160C_1001-export.con 0.99x
010409_Motor_task_coregist-export.con 0.98x
Output is bit-identical on every readable KIT fixture. The large file was
synthesised by tiling the raw-data section of data_berlin.con -- it is the last
section in the file -- and patching n_samples in the acquisition header.
Same treatment as mne-tools#14241 and mne-tools#14246; the constant differs because the optimum
tracks time points per block, so it moves with the channel count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference issue (if any)
Follow-up to #14241, where a maintainer asked for the same treatment on other
formats. Stacked on #14241 — that PR adds the
max_block_bytes=parameterthis one uses, so please merge it first; this diff is only the per-reader opt-ins.
What does this implement/fix?
Six more readers ask
_read_segments_file()for a cache-sized block instead ofthe 100 MB default.
The size is measured per reader rather than copied from BrainVision, because the
optimum is not a fixed number of bytes — it tracks the number of time points per
block, so it moves with the channel count:
Each was swept (default / 32 / 16 / 8 / 4 / 2 / 1 MiB) with interleaved A/B
against a pristine
mainworktree, medians of repeated rounds.Sizes were also chosen so that no shipped fixture regresses
This is the part worth reviewing. Copying BrainVision's 8 MiB everywhere would
have made real files slower: it measures 0.83x on Curry's 11.8 MB fixture
(10/10 rounds) and 0.81-0.85x on Artemis123's 17.6 MB one, because it splits a
read that otherwise fits in a single block. FIL is shipped at 16 MiB rather than
the faster 4 MiB for the same reason — 4 MiB is 2.5x on a large file but 0.96x
on the 9.8 MB fixture.
An in-process A/B that varies only the constant now shows every real fixture
within noise of the default: eximia 1.003, fil 1.001, curry 1.003,
artemis123 0.988.
Correctness
main: 25/25 arrays compared withnp.array_equalacross every readable fixture of the six formats.
pytest mne/io/{eximia,nsx,fil,eeglab,artemis123,curry,brainvision} mne/_fiff:305 passed.
Caveats a reviewer should weigh
a CPU/cache effect in decode plus calibration, not disk I/O, so the numbers
will move on other hardware — but the shape (large files gain, small files
unaffected) should hold.
so CI cannot demonstrate the win. Large fixtures were synthesised by tiling the
shipped binaries (valid for these formats — sample counts come from file size,
or from a sidecar that was patched to match).
found the optimum tracks ~8192 time points per block, not a byte budget:
reinterpreting the same bytes as 128 and 256 channels moved the peak. A
max_block_samples=parameter would express this better. Happy to go that wayinstead if you prefer.
Additional information
AI disclosure: I directed the work and reviewed and tested everything; Claude
Code (Claude Opus 5) ran the per-reader sweeps and bit-exactness checks and made
the edits under my direction.