Skip to content

feat(tools): neuralset — wrap Meta FAIR neuro-AI pipeline - #10

Merged
cagataycali merged 4 commits into
mainfrom
feature/neuralset-tool
Apr 21, 2026
Merged

feat(tools): neuralset — wrap Meta FAIR neuro-AI pipeline#10
cagataycali merged 4 commits into
mainfrom
feature/neuralset-tool

Conversation

@cagataycali

Copy link
Copy Markdown
Owner

🧠 NeuralSet tool

Wraps the Meta FAIR NeuralSet project as a single DevDuck tool.

Pipeline actions

setup → list → info → download → preprocess → features → align → encode → decode → evaluate → leaderboard → cache

Coverage

  • Modalities: EEG / MEG / fMRI / iEEG / fNIRS / Spikes
  • Backbones: CLIP · DINOv2 · ResNet50 · V-JEPA · wav2vec2 · HuBERT · EnCodec · LLaMA · GPT-2 · BERT · SONAR
  • Metrics: pearson / r² / accuracy

Design

  • Lazy heavy imports — no startup cost
  • dry_run=True on every stage for preview
  • DEVDUCK_NEURALSET_{REPO,REPO_DIR,CACHE} env-var overrides
  • Standard Strands return shape {status, content:[{text}]}
  • Fallback dataset list when manifest isn't downloaded yet
  • Cache helpers (stats / list / clear)

Example e2e

neuralset(action='setup')
neuralset(action='download',   dataset='things-eeg2')
neuralset(action='preprocess', dataset='things-eeg2')
neuralset(action='features',   dataset='things-eeg2', backbone='clip')
neuralset(action='encode',     dataset='things-eeg2', backbone='clip')
neuralset(action='evaluate',   dataset='things-eeg2', backbone='clip')

Notes

  • Heavy deps (torch / mne / nibabel) remain optional — only pulled in when setup is called.
  • The tool shells out to the neuralset CLI for stages that require it; returns a clear error if CLI is missing.

🦆 Autonomous devduck generation — Telegram-triggered.

cagataycali and others added 2 commits April 21, 2026 17:08
Single tool covering the full NeuralSet pipeline:
  setup, list, info, download, preprocess, features, align,
  encode, decode, evaluate, leaderboard, cache, help

- modalities: EEG / MEG / fMRI / iEEG / fNIRS / Spikes
- backbones:  CLIP / DINOv2 / ResNet / V-JEPA / wav2vec2 / HuBERT /
              EnCodec / LLaMA / GPT-2 / BERT / SONAR
- lazy imports, dry_run preview, cache stats, fallback dataset list
- respects DEVDUCK_NEURALSET_{REPO,REPO_DIR,CACHE} env vars
- standard Strands {status, content:[{text}]} return shape

Ref: https://facebookresearch.github.io/neuroai/neuralset/
… matrix

Bug: 7 stage functions (download, preprocess, features, align, encode,
decode, evaluate) returned error when CLI not found, even with dry_run=True.
This made the pipeline impossible to preview/test without installing.

Fix: skip CLI installation check when dry_run is True. Uses 'neuralset' as
placeholder command name so _run() can emit '[dry-run] neuralset ...'.

Tests: tests/test_neuralset.py exercises all 13 actions across 42 cases
(modality filters, required-param errors, dry-run simulations, sub_actions,
invalid actions). 42/42 passing.
@cagataycali

Copy link
Copy Markdown
Owner Author

🧪 Autonomous Coverage Matrix — 42/42 ✅

Ran every branch of neuralset() via tests/test_neuralset.py (added in 52cde57).

Results

# Action Case Expected Got
T01 help default success success
T02–T08 list all + 6 modalities (eeg/meg/fmri/ieeg/fnirs/spikes) success success
T09 list bad modality (xray) error error
T10–T12 info known / missing / unknown ok/err/hint ok/err/hint
T13–T14 setup dry + force-dry success success
T15–T17 download dry / subjects / missing ok/ok/err ok/ok/err
T18–T20 preprocess dry / config / missing ok/ok/err ok/ok/err
T21–T24 features dry / layer / 2× missing ok ok err err ok ok err err
T25–T26 align dry / missing ok/err ok/err
T27–T29 encode dry / mlp / missing ok/ok/err ok/ok/err
T30–T33 decode stim / category / embed / missing ok×3+err ok×3+err
T34–T37 evaluate dry / r2 / acc / missing ok×3+err ok×3+err
T38–T39 leaderboard view / submit-missing ok/err ok/err
T40–T41 cache stats / list success success
T42 * invalid action error error

🐛 Bug found & fixed in this PR

7/13 stage functions (download, preprocess, features, align, encode, decode, evaluate) returned error when the neuralset CLI wasn't installed — even with dry_run=True. That made the whole pipeline impossible to preview without a full install.

Fix (52cde57): skip the CLI-presence check when dry_run=True; use "neuralset" as a placeholder command so _run() emits the simulated [dry-run] neuralset ... line.

Before: 26/42 ✅ → After: 42/42

Coverage summary

  • Actions: 13/13 (help, setup, list, info, download, preprocess, features, align, encode, decode, evaluate, leaderboard, cache)
  • Modalities: 6/6 (eeg, meg, fmri, ieeg, fnirs, spikes)
  • Error paths: 10 (missing params, bad modality, bad action, missing submission)
  • Dry-run paths: 16 (all pipeline stages)

Run locally: python3 tests/test_neuralset.py

cagataycali and others added 2 commits April 21, 2026 17:46
Resolves checkout failure: 'fatal: could not read Username for https://github.com'
when the PAT_TOKEN secret is empty/unconfigured. The workflow now falls back to
the auto-provided GITHUB_TOKEN so CI runs on PRs without requiring an extra secret.
Root cause of 'could not read Username for https://github.com':
The PAT_TOKEN secret is set but expired/invalid. Passing it explicitly
to actions/checkout@v4 causes the checkout to try (and fail) with bad
credentials. The `secrets.PAT_TOKEN || secrets.GITHUB_TOKEN` fallback
only works when PAT_TOKEN is empty, not when it's invalid.

Fix: Remove the explicit `token:` parameter from checkout step. actions/
checkout uses the auto-provided GITHUB_TOKEN by default, which is always
valid for the workflow's own repository.

PAT_TOKEN is still passed as env var to the agent itself (with same
fallback) for cross-repo API calls.
@cagataycali
cagataycali merged commit 6eadb2a into main Apr 21, 2026
@github-actions

Copy link
Copy Markdown

🤖 Autonomous Agent Verification

Workflow Run: #66
Branch: feature/neuralset-tool
Timestamp: 2026-04-21 22:08:24 UTC

✅ Verification Results

  • Merge Status: Successfully merged to main
  • Tool Location: devduck/tools/neuralset.py (737 additions)
  • Import Test: Passed
  • Integration: Neuralset tool is now available in DevDuck

📋 Tool Capabilities

The neuralset tool wraps the Meta FAIR NeuralSet neuro-AI pipeline, providing actions for:

  • Dataset discovery (EEG/MEG/fMRI/iEEG/fNIRS/Spikes)
  • Neural encoding/decoding benchmarks
  • Full pipeline: setup → download → preprocess → encode → decode → evaluate

🎯 Next Steps

The feature branch can be safely deleted as the work has been successfully integrated into main.


Autonomous verification by DevDuck GitHub Agent

cagataycali added a commit that referenced this pull request Apr 21, 2026
Ruff flagged 2 f-strings without placeholders in tests/test_neuralset.py:
  - line 128: print(f"COVERAGE MATRIX - PR #10 (neuralset)")
  - line 150: print(f"\nDetailed results: /tmp/neuralset_coverage.json")

Fix auto-applied via 'ruff check --fix'. Verified locally in a fresh venv:
  python3 -m venv .venv && .venv/bin/pip install ruff
  .venv/bin/ruff check devduck/tools/neuralset.py tests/test_neuralset.py
  → All checks passed!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant