Persistent managed memmap caches + preload="memmap" sentinel

[bruAristimunha]

Reference issue (if any)

None. Part 5/5 of the IO-speedup series. Depends on #14215
(→ #14214 → #14213 → #14212).

What does this implement/fix?

• preload="memmap" on any reader: selects an automatically managed
  memory-map cache keyed by source path/mtime/size. First read decodes into
  it; later reads, including from fresh processes, mmap it directly.
• load_data(memmap=<path>) caches are reused when valid instead of always
  re-decoding.
• BaseRaw.__del__ no longer deletes memmap-backed files; cache lifetime is
  owned by the caller. Behavior change, noted in the changelog fragment.

Fresh-process open + first access of a 944 MB float64 recording:
~450 ms decode → 11–21 ms. Public-API windows: ~84 µs.

Additional information

AI disclosure: same as #14212.
CI note: held until the parent PR merges.

[larsoner]

Okay for this one... I'm thinking the real fix is to fix preload=<str> in another PR, then in this one add preload="auto" or similar which means "create for me this auto memmap file and cache it"

[bruAristimunha]

hey @larsoner, bonjour!

This is my proposal to really speed up! I completely pivot from the initial trick, which I plan to address in another PR. This way, we can have both worlds, nothing is breaking, and we consume the. _ data object more seriously, only changing one flag, the preload="auto".

I will still iterate this across the day, but I feel that this will unlock a lot of power with this base memmap representation.

It will not be used without the person knowing the flag preload="auto"

[larsoner]

I agree a preload="auto" could have some benefits, but the implementation is much too big to be maintainable.

Would hashing on the filename + mtime of the file be enough, rather than needing to triage based on internal file stuff?

If not, is there some simpler idea that could work?

[bruAristimunha]

@larsoner, could you please take another look at a138fb7, especially at whether this is now a maintainable/minimal direction?

I followed your simplification suggestion quite aggressively:

• mne/io/_preload_cache.py: 628 -> 108 lines
• mne/io/tests/test_preload_cache.py: 679 -> 161 lines
• one deterministic <sha256>.data entry, populated through a stable temporary file and os.replace
• one FileLock only on cache misses; warm hits do not import filelock
• no manifests, generations, cache scavenger, JSON, or platform-specific file identity code

I could not safely use only (filename, size, mtime): for example, read_raw_brainvision(..., scale=1) and scale=2 have identical source stats but different decoded values. The remaining key is therefore source path/size/mtime plus reader type, _raw_extras, _cals, dtype, and shape. There is a regression test for that scale case. Does this seem like the right minimal identity, or would you prefer a different boundary?

One other design question: the simplification treats the configured MNE cache directory as trusted/private and consequently removed the previous regular-file and symlink validation. Is that the contract you would expect, or should I retain a small explicit validation check?

Here is the complete microbenchmark I used (run from the PR checkout with an empty CACHE_DIR for a true miss):

# python bench.py RECORDING CACHE_DIR
import os
import sys
from pathlib import Path
from time import perf_counter

import mne

fname, cache_dir = map(Path, sys.argv[1:3])
cache_dir.mkdir(parents=True, exist_ok=True)
os.environ["MNE_CACHE_DIR"] = str(cache_dir)

print(mne.__file__)
for label in ("miss", "hit"):
    start = perf_counter()
    raw = mne.io.read_raw(fname, preload="auto", verbose="error")
    opened = perf_counter()
    raw.get_data(stop=min(1024, raw.n_times))
    accessed = perf_counter()
    print(
        f"{label}: open={(opened - start) * 1e3:.1f} ms, "
        f"1024 samples={(accessed - opened) * 1e3:.3f} ms"
    )

I executed this exact snippet against mne/io/edf/tests/data/test.edf with an isolated empty cache:

miss: open=476.3 ms, 1024 samples=0.227 ms
hit: open=9.5 ms, 1024 samples=0.198 ms

For the separate interleaved 900 MiB format benchmark, median warm opens were 16.1 ms (FIF), 14.6 ms (EDF), 15.7 ms (BDF), and 24.3 ms (BrainVision), with matching decoded-data checksums.

[larsoner]

Windows failure looks real:

================================== FAILURES ===================================
____________________________ test_auto_preload_api ____________________________
[gw0] win32 -- Python 3.14.7 C:\hostedtoolcache\windows\Python\3.14.7\x64\python.exe
mne\io\tests\test_preload_cache.py:58: in test_auto_preload_api
    lazy.load_data(memmap="auto")
<decorator-gen-301>:12: in load_data
    ???
mne\io\base.py:633: in load_data
    self._preload_data(memmap if memmap is not None else True)
mne\io\base.py:652: in _preload_data
    self._data = self._read_segment(data_buffer=data_buffer)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
<decorator-gen-300>:12: in _read_segment
    ???
mne\io\base.py:449: in _read_segment
    data = _allocate_data(data_buffer, data_shape, dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
mne\io\base.py:2739: in _allocate_data
    data = np.memmap(str(preload), mode="w+", dtype=dtype, shape=shape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\hostedtoolcache\windows\Python\3.14.7\x64\Lib\site-packages\numpy\_core\memmap.py:235: in __new__
    f_ctx = open(
E   OSError: [Errno 22] Invalid argument: 'auto'

[larsoner]

Approach seems much simpler, thanks! Now I'm curious to see if we can generalize it. Moving it to baseraw will require looking to see if a fname-like arg is passed and if it's not file-like, but it should buy us support for this for almost all formats (and any new ones).

And thus some smoke test for it would ideally maybe be added to the def _test_raw function, which is run for every I/O format

[larsoner]

Why just these three? Can't we build this into the BaseRaw class like we already do for strings?

[bruAristimunha]

@larsoner CI is green now, Windows included.

The Windows failure was in the test, not the cache: it created a Raw with preload=Path("auto"), kept that w+ memmap alive, and then reopened the same filename. Windows refuses to truncate a file that still has a mapping (ERROR_USER_MAPPED_FILE -> OSError errno 22); Linux allows it, which is why it only showed up there. The test no longer reuses that name.

Two other things while I was in there:

• This PR had promoted filelock from optional to a required dependency, which isn't needed. Entries are now published through a per-process temporary plus os.replace, which is atomic on POSIX and Windows, so concurrent misses at worst decode the same entry twice instead of blocking. filelock is back in full-no-qt (mne/utils/config.py already soft-imports it). test_auto_preload_concurrent_misses still passes, which was the lock's reason for existing.
• load_data(memmap="auto") used to mean a file literally named auto, while preload="auto" meant the cache. Same word, opposite meaning. Both now resolve the same sentinel; Path("auto") remains the literal-filename escape.

On your changelog question: the cache isn't reader-specific, _raw_preload_auto works for any file-backed Raw (the only reader-specific bit is a FIF guard rejecting file-like inputs, which have no stable identity to key on). The entry now says that instead of naming three formats.

[larsoner]

Awesome, thanks @bruAristimunha !