[dev] Use datalad-fuse's remfile backend and all other dev branches

[CodyCBakerPhD]

Stacked on #1956. This tries the development version of datalad-fuse from datalad/datalad-fuse#131, which adds pluggable backends, including remfile for HDF5-based files.

Changes

• Adapter choice: get_adapter() uses datalad_fuse.adapter.RemoteFilesystemAdapter (dandiset.yaml: should become READONLY locally, with a comment, no metadata uploaded from it #131) when it is available, and falls back to the released datalad_fuse.fsspec.FsspecAdapter (0.5.1) otherwise.
  • With dandiset.yaml: should become READONLY locally, with a comment, no metadata uploaded from it #131 and remfile installed, NWB and other HDF5 files are read through remfile; other files still go through fsspec.
  • The backends can be configured with DataLad's datalad.fusefs.backends setting.
• S3 export fallback: get_annexed_readable() also tries the S3 exporttree URLs from dandiset.yaml: should become READONLY locally, with a comment, no metadata uploaded from it #131 when git-annex lists no URL, for older datasets that lack versioned S3 URLs.
• Docs: a note in the validate docs.
• Tests: test_annexed_readable_remfile checks that an NWB file is opened through datalad-fuse's RemfileWrapper and read with h5py over the local HTTP range server. It is skipped unless both dandiset.yaml: should become READONLY locally, with a comment, no metadata uploaded from it #131 and remfile are installed.
• CI: a new datalad-dev job installs git-annex, remfile and datalad-fuse from refs/pull/131/head. The existing datalad job keeps testing the PyPI release.

Locally, the datalad-fuse and validate stream tests pass with both datalad-fuse 0.5.1 (where the remfile test is skipped) and #131 plus remfile.

Once #131 is released, the CI install can switch to that release and the datalad extra's lower bound can be raised.

Follow-up branch: remfile instead of the workarounds

Branch claude/use-dev-datalad-fuse-remfile (c9b2d02) builds on this PR. It has no PR of its own yet. It reads NWB and other HDF5-based files with datalad-fuse's remfile backend instead of fsspec with a custom cache. It uses the remfile-dev branch of CodyCBakerPhD/datalad-fuse, which is datalad/datalad-fuse#131 plus init_annex from CodyCBakerPhD/datalad-fuse#4.

Removed from dandi/support/datalad_fuse.py	Replaced by
_coalescing_block_cache() and _read_in_small_blocks()	remfile. To read the structure of a 16 MB NWB file, remfile downloads 0.6 MB; fsspec with its default cache downloads 64 MB.
annex_initialized()	init_annex=False and annex_initialized in datalad-fuse
_GCPausedFile for every streamed file	_GCPausedFile only for files read with fsspec

Why _GCPausedFile stays for fsspec, and why the h5py fork is not used:

• remfile fetches data in the reading thread, so no finalizer can run in another thread while h5py holds its lock waiting for that data. No h5py change is needed.
• With fsspec, a patched h5py (CodyCBakerPhD/h5py defer-dealloc-when-locked) still deadlocks. In fsspec's I/O thread, a garbage collection runs HDMFIO.__del__ (hdmf 6.2.0). That calls is_open(), i.e. bool(h5py.File), which waits for h5py's lock, held by the reading thread.

CI on the follow-up branch:

• remfile is added to the datalad extra.
• The datalad-dev job installs the remfile-dev branch.
• The datalad job keeps the released datalad-fuse, where streaming is skipped.

Results so far (local only):

• Tests: the datalad-fuse, validate and cmd_validate tests pass:

  • with remfile: 102 passed;
  • without remfile: 100 passed, 2 skipped;
  • with released datalad-fuse 0.5.1: 86 passed, 16 skipped.

• Real data: the 53 NWB files of 000006 with -J 8 and stock h5py 3.16 gave the same results as the fsspec path. Nothing was downloaded into the clone.

  Path	Time	Downloaded
  remfile	33–61 s	340 MiB
  fsspec, coalescing cache and garbage collection paused (this PR)	16–18 s	179 MiB

  For HDF5 files, remfile trades some bandwidth for not needing a custom cache, a garbage collection pause or an h5py patch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

---

Generated by Claude Code

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 94.59459% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.14%. Comparing base (336ec0b) to head (e0aac05).
⚠️ Report is 5 commits behind head on claude/validate-stream-via-datalad-fuse.

Files with missing lines	Patch %	Lines
dandi/support/datalad_fuse.py	87.50%	2 Missing ⚠️

Additional details and impacted files

@@                             Coverage Diff                              @@
##           claude/validate-stream-via-datalad-fuse    #1957       +/-   ##
============================================================================
+ Coverage                                    66.82%   91.14%   +24.32%     
============================================================================
  Files                                           94       94               
  Lines                                        14835    14866       +31     
============================================================================
+ Hits                                          9913    13550     +3637     
+ Misses                                        4922     1316     -3606     

Flag	Coverage Δ
unittests	91.14% <94.59%> (+24.32%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.