Stream validation in parallel

[CodyCBakerPhD]

Stacked on #1955.

Stream dandi validate through datalad-fuse adapters (not FUSE mounts).

Try this out with

mkdir dandi-stream-demo && cd dandi-stream-demo

python3 -m venv venv && . venv/bin/activate
pip install git-annex con-duct \
    "dandi[datalad] @ git+https://github.com/dandi/dandi-cli@claude/validate-stream-via-datalad-fuse"
git annex version | head -1                     # expect 10.2025xxxx or newer

datalad clone https://github.com/dandisets/000006
find 000006 -name '*.nwb' -xtype l | wc -l      # expect 53 (content not fetched)

# Cold run (cache cleared first), then warm run (results come from the cache)
DANDI_CACHE=clear duct -m "stream 000006 -J8 cold" \
    dandi validate --missing-file-content=stream -J 8 -o 000006-cold.jsonl 000006
duct -m "stream 000006 -J8 warm" \
    dandi validate --missing-file-content=stream -J 8 -o 000006-warm.jsonl 000006

and then con-duct ls -F message wall_clock_time peak_rss average_pcpu exit_code should give something like

PREFIX                               MESSAGE                     WALL_CLOCK_TIME PEAK_RSS AVERAGE_PCPU EXIT_CODE
.duct/logs/2026.10.15T21.17.34-3016_ 1956 stream 000006 -J8 cold 30.585 sec      1.8 GB   422.91%      1
.duct/logs/2026.10.15T21.18.05-7156_ 1956 stream 000006 -J8 warm 5.073 sec       1.5 GB   754.05%      1

and then dandi validate --load 000006-cold.jsonl --summary should give

--- Validation Summary ---
Total issues: 893  
By severity:
  ERROR: 576
  WARNING: 317
By validator:
  nwbinspector: 469
  pynwb: 424
By standard:
  N/A: 469
  NWB: 424

🤖 AI text below here 🤖

Details

Changes

• New stream policy for --missing-file-content:
  • Annexed files whose content isn't present locally are opened through AnnexedReadableFile, and pynwb and nwbinspector run on the streamed content.
  • Each streamed file yields an INFO result, DANDI.FILE_CONTENT_STREAMED, naming the first URL git-annex reported for its content.
  • A file that can't be streamed yields a DANDI.FILE_CONTENT_MISSING error.
  • In a clone where git-annex isn't initialized, no git annex command runs, so the clone isn't modified, and every file is reported as missing.
• Pipeline support, carried over from Enable dandi validate to operate on Datalad datasets through streaming #1933:
  • LocalFileAsset.content_source lets an asset read its content from a Readable.
  • pynwb_utils.validate(..., readable=...) validates content from a Readable.
  • nwbinspector inspects the streamed file object.
  • BIDS errors that need file content are suppressed for streamed files, as under only-non-data.
• Caching:
  • pynwb validation results and NWB Inspector results are cached under the annex key, via annex_fingerprint from Stream missing annexed content with remfile, cached by annex key #1955.
  • Re-running skips the work for files whose key hasn't changed.
  • Results aren't cached when a streamed read fails.
• -J/--jobs N: NWB files outside BIDS datasets are validated in N processes. Results are yielded in the same order as with 1 (the default), and the workers' log messages are forwarded to the main process.
• Progress: one INFO log message per validated file, e.g. [12/53] sub-01/sub-01_ses-1.nwb: 3 issues.
• Requirements: the policy requires datalad-fuse with remfile (pip install "dandi[datalad]") and git-annex, and raises an error if either is missing. Until datalad-fuse releases its remfile backend, the extra pins the head commit of ENH: add remfile backend with pluggable backend architecture datalad/datalad-fuse#131 (see Stream missing annexed content with remfile, cached by annex key #1955).
• Docs: "Validating DataLad Dandisets Remotely" in the validate docs explains the policy, using datalad clone.
• Tests: the stream tests from Enable dandi validate to operate on Datalad datasets through streaming #1933 are rewritten to use a real git-annex repository and the local HTTP range server from Stream missing annexed content with remfile, cached by annex key #1955. They cover:
  • the results match a local copy of the file;
  • caching by key;
  • the unstreamable and unreadable-URL cases;
  • the missing-requirements error;
  • --jobs and progress output.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qn5WBSiQgoZoL4fytF6nEr

[codecov]

Codecov Report

❌ Patch coverage is 98.55491% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.32%. Comparing base (b75c824) to head (63ee181).

Files with missing lines	Patch %	Lines
dandi/validate/_core.py	97.01%	4 Missing ⚠️
dandi/files/bases.py	97.91%	1 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                        @@
##           claude/datalad-fuse-readable    #1956      +/-   ##
================================================================
+ Coverage                         91.10%   91.32%   +0.22%     
================================================================
  Files                                96       96              
  Lines                             14648    14956     +308     
================================================================
+ Hits                              13345    13659     +314     
+ Misses                             1303     1297       -6     

Flag	Coverage Δ
unittests	91.32% <98.55%> (+0.22%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[CodyCBakerPhD]

Note: test on https://dandiarchive.org/dandiset/001769 (BIDS with nifti's; @yarikoptic had concern about BIDS validator fetching content)