Skip to content

Preserve caller-owned memory-mapped Raw preload files - #14213

Merged
larsoner merged 6 commits into
mne-tools:mainfrom
bruAristimunha:pr/2-fif-mmap
Aug 26, 2026
Merged

Preserve caller-owned memory-mapped Raw preload files#14213
larsoner merged 6 commits into
mne-tools:mainfrom
bruAristimunha:pr/2-fif-mmap

Conversation

@bruAristimunha

@bruAristimunha bruAristimunha commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Reference issue (if any)

None.

What does this implement/fix?

This fixes the ownership semantics of caller-named memory-mapped preload files.

MNE already supports storing decoded data in a memory-mapped file through preload=<path> and raw.load_data(memmap=<path>). Because that path is supplied by the caller, the resulting file should remain
under caller ownership. Previously, BaseRaw.__del__ could unlink the backing file when the Raw object was garbage-collected.

This PR removes that automatic unlinking and clarifies the existing contract:

  • A caller-supplied path is freshly created or overwritten; it is not reused as a cache.
  • The file remains available after Raw.close() and after the Raw object is destroyed.
  • The caller is responsible for removing the file when it is no longer needed.
  • Caller-supplied memory-mapped arrays passed to RawArray follow the same ownership rule.

This keeps the existing preload concepts unified: passing a path means “decode into this caller-owned memory-mapped file.” It does not introduce source-file memory mapping, persistent file handles, cache
reuse, or a new preload mode.

Additional information

Tests cover:

  • construction with preload=<path>;
  • load_data(memmap=<path>);
  • overwriting and truncating an existing destination;
  • survival across close() and garbage collection;
  • reopening and replacing the destination;
  • concatenation, copying, and appending;
  • direct RawArray memory maps;
  • signal-bearing EDF and FIF inputs.

Validation performed locally:

  • focused ownership tests: 7 passed, 4 testing-dataset skips;
  • adjacent Raw, EDF, and FIF tests: 174 passed, 81 testing-dataset skips;
  • docstring parameter tests: 19 passed;
  • pre-commit hooks passed.

@larsoner

Copy link
Copy Markdown
Member

I'm going to kill CIs on this and other PRs that depend on previous ones... otherwise no other PRs will have runners to test.

@bruAristimunha

Copy link
Copy Markdown
Contributor Author

Makes sense — thanks. Holding all pushes to the stacked branches until #14212 merges so runners aren't consumed by re-runs; downstream drafts (#14214#14216) likewise won't be pushed until their parents land.

@larsoner larsoner moved this from Ideas to In Progress in 2026 MNE-Python Maintainers Sprint Aug 25, 2026
@bruAristimunha
bruAristimunha marked this pull request as ready for review August 25, 2026 11:19
@bruAristimunha
bruAristimunha marked this pull request as draft August 25, 2026 11:28
@bruAristimunha
bruAristimunha force-pushed the pr/2-fif-mmap branch 5 times, most recently from 3995d9a to faa6a45 Compare August 25, 2026 11:56
FIF raw segments are read as byte-offset views into a PID-keyed memory
map of the file instead of open/seek/read per call; buffer entries are
selected with searchsorted on the sorted bounds. gzip, file-like objects,
and non-simple tag types keep the legacy path. The generic memory-map
cache in _read_segments_file also serves the other binary readers.
Comment thread mne/_fiff/utils.py Outdated
@larsoner

Copy link
Copy Markdown
Member

I think this should already work in theory:

>>> import mne
>>> raw = mne.io.read_raw_edf("/Users/larsoner/python/MNE-testing-data/EDF/SC4001EC-Hypnogram.edf", preload="test.tmp")
<python-input-1>:1: RuntimeWarning: Header information is incorrect for record length. Default record length set to 1.
It is possible that this file only contains annotations and no signals. In that case, please use mne.read_annotations() to load these annotations.
<python-input-1>:1: RuntimeWarning: Omitted 153 annotation(s) that were outside data range.
<python-input-1>:1: RuntimeWarning: Limited 1 annotation(s) that were expanding outside the data range.

But it maybe "just" needs to be fixed? Because this is not the right result...

>>> raw._data
memmap([], shape=(0, 2054), dtype=float64)

It should look more like this

>>> raw = mne.io.read_raw_fif(mne.datasets.sample.data_path() / "MEG" / "sample" / "sample_audvis_raw.fif", preload="test.tmp2")
>>> raw._data
memmap([[ 9.64355481e-12,  0.00000000e+00,  0.00000000e+00, ...,
         -1.92871096e-12,  2.89306644e-12,  3.85742192e-12],
        [-4.82177740e-12, -2.89306644e-12, -9.64355481e-13, ...,
         -9.64355481e-13, -9.64355481e-13, -1.92871096e-12],
        [ 1.01074222e-13,  6.31713890e-14,  7.58056668e-14, ...,
         -4.80102556e-13, -6.06445334e-13, -5.93811056e-13],
        ...,
        [ 3.88542173e-05,  4.07510373e-05,  4.09957883e-05, ...,
          6.72453304e-05,  6.68782039e-05,  6.91421504e-05],
        [ 6.58391126e-05,  6.80025648e-05,  6.81779798e-05, ...,
          8.51932390e-05,  8.58948991e-05,  8.89938982e-05],
        [ 2.85661012e-04,  2.83699953e-04,  2.80431520e-04, ...,
          2.64089357e-04,  2.62781984e-04,  2.57552492e-04]],
       shape=(376, 166800))

@bruAristimunha bruAristimunha changed the title Read FIF raw tags through a memory map Preserve caller-owned memory-mapped Raw preload files Aug 26, 2026
@bruAristimunha

Copy link
Copy Markdown
Contributor Author

hey @larsoner, can you please check again?

@larsoner
larsoner marked this pull request as ready for review August 26, 2026 12:57

@larsoner larsoner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay for me!

@larsoner
larsoner enabled auto-merge (squash) August 26, 2026 12:58
@larsoner
larsoner merged commit f1b170f into mne-tools:main Aug 26, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants