An EEG brain–computer interface that decodes imagined movement of a third hand, a novel limb, alongside imagined movement of their own left and right hands, and drives it in real time.
The repository covers the whole path: raw .xdf recordings → preprocessing →
decoding → group-level time-frequency statistics, and the same
preprocessing → a live LSL loop that pushes control markers to the effector.
conda env create -f environment.yml && conda activate 3rd-arm-mi
python -m src.live --replay Recordings/<file>.xdf --dry-runThat second command runs the actual real-time decoder — acquisition, artifact gating, preprocessing, classification — against a recording instead of an amplifier, so it works with no hardware attached.
Motor imagery BCIs decode imagined movement from the user's existing motor repertoire, relying on an existing cortical representation. A supernumerary limb has no such representation. The question is whether imagined movement of a third hand produces a sensorimotor signature that is separable from the natural hands — and stable enough to drive a device.
Paradigm. Four classes: MiddleHand (the supernumerary hand), LeftHand,
RightHand, FixatedRest (control class). 64-channel EEG at 500 Hz (LiveAmp,
FCz online reference).
Data collection is ongoing and results are in preparation. Accuracy figures, cohort statistics and plots will be published soon.
The decoding analyses share one preprocessing contract and one epoch cache
(src/analysis_common.py):
| Module | Question |
|---|---|
full_epoch_batch |
Given the whole trial, how well does it classify? |
windowed_batch |
How much information do we need for reliable classification? Sliding window, cross-validated at every position. Relevant for real-time performance, as well as BCI-induced neuroplasticity. |
On top of that, a time-frequency stack (tfr_batch → tfr_group → tfr_stats)
computes ERD/ERS per participant and tests it in two tiers:
- Confirmatory. One value per participant per condition from an a-priori sensorimotor ROI × band (Mu 8–13 Hz, Beta 13–30 Hz) × active window (0.5–3.5 s), tested with an exact paired sign-flip permutation t.
- Exploratory. Cluster-based permutation over channels × frequencies × times with real 3-D adjacency (Maris & Oostenveld 2007), for effects that were not pre-specified, with FWER control across the whole space.
The contrast family is pre-specified in default_stats_params() and recorded in
the output, because a contrast chosen after looking at the maps is a different
test with a different error rate. Note that the exact floor for the sign-flip
test is 2 / 2**N, not 1 / 2**N: flipping every participant's sign negates t
and leaves the two-tailed statistic unchanged, so the two extreme permutations
are not independent.
- Real-Time Decoding: Achieved a mean online accuracy of 76% ± 6% SE (2 s sliding windows, 250 ms step) during real-time closed-loop object manipulation.
- Distinct ERD Topography / Sensorimotor Signature: Third-hand imagery elicited distinct Mu (8–13 Hz) and Beta (13–30 Hz) event-related desynchronization (ERD) with a central sensorimotor and bilateral posterior distribution, distinct from the classic contralateral pattern observed during natural hand imagery.
- Embodiment & Agency: Participants reported significant ownership and agency over the virtual effector during concurrent natural- and third-arm tasks.
python -m src.live --list # available decoder bundles
python -m src.live --replay REC.xdf --dry-run # no hardware
python -m src.live --replay REC.xdf --realtime --seconds 60
python -m src.live --model <stem> # live → LSL markersThe loop is acquire → buffer → montage & reference → filter → gate → predict, at a ~250 ms cadence over a 2 s window, providing LSL string markers
that the Unity effector subscribes to.
Some noteworthy design feature decisions:
Offline and online share one preprocessing path. src/live/loop.py does not
reimplement referencing — it calls the same apply_montage_and_reference and
select_electrodes the offline pipeline calls. Divergence there would not raise;
it would just make the model quietly worse online than it tested offline.
Filtering is causal. phase='forward' throughout. A zero-phase filter needs
future samples, which do not exist online, and using one offline makes validation
optimistic in a way that does not survive deployment.
The decoder enforces a channel contract. A classifier handed the right number
of channels in the wrong order returns confident nonsense, and online there is no
ground truth to catch it. So a model is never loaded alone: it comes bundled with
the channel list it was fit on and the params_dict that produced them, and the
live montage is asserted against that list on the first chunk — membership and
order.
Replay as a debugging tool. The .xdf was recorded from the same LSL stream —
same 67 channels, same order, same units — so --replay feeds the identical loop.
Bugs in preprocessing, gating or the channel contract reproduce under it, which
is what makes it a debugging tool rather than a demo.
The amplifier records against FCz, so FCz is a real electrode that never appears
in the data. params_dict['AddRefChannel'] rebuilds it by adding a zero-filled
channel before average referencing, leaving it holding -Σ(others) — the
estimated potential at the reference site.
The step order is load-bearing and fails silently: add_reference_channels must
run before set_montage, or MNE 1.6 leaves FCz at a NaN location with no
catchable warning, and you find out only when CSD dies several steps later. The
regression check is therefore on the channel's coordinates, not on a warning.
Full detail in docs/ARCHITECTURE.md.
The frontend is built in Unity (C#), providing a spatially registered, first-person augmented reality environment where the third arm extends from the user's mid-torso.
- LSL Subscription: A dedicated listener consumes the LSL string markers published by the Python backend at ~100 ms intervals. These markers trigger state transitions for the virtual limb.
- Embodiment via Haptics: To drive ownership and agency, the Unity environment interfaces with the bHaptics SDK. Physical interactions with virtual objects trigger spatially mapped tactile feedback via a haptic vest.
- Concurrent Task Design: The physics interactions allow for simultaneous manipulation, requiring the user to seamlessly coordinate the BCI-driven supernumerary limb alongside their naturally tracked hands to successfully grasp and transport objects.
| Path | Contents |
|---|---|
src/preprocessing.py |
XDF import, montage, referencing, filtering, epoching, CSD, ICA helpers |
src/training.py |
Classifier stack: Covariances(oas) → FGDA → TangentSpace → LDA |
src/evaluation.py |
Metrics, permutation tests, tangent-space interpretability |
src/analysis_common.py |
Shared contract for the three decoding analyses |
src/{windowed,full_epoch,session}_batch.py |
The analyses themselves |
src/tfr_batch.py, tfr_group.py, tfr_stats.py |
ERD/ERS, group aggregates, statistics |
src/live/ |
The real-time decoder |
src/subject_ids.py |
Pseudonym ↔ recording registry |
notebooks/ |
Thin drivers over the src/ modules |
tests/ |
python -m pytest tests/ -q |
docs/ |
ARCHITECTURE.md, QUICK_REFERENCE.md |
The notebooks are drivers, not implementations — Windowed_Analysis,
Full_Epoch_Analysis, Session_Analysis and TFR_Analysis are a handful of
cells each. Main_Experiment.ipynb is the per-participant training workflow,
including the co-adaptive block-weighted retraining used between recording
blocks. Main_Experiment_ICA.ipynb is the same workflow through the ICA path, and
Live_Stream.ipynb is the interactive version of the online loop - the same
steps as src/live/, run cell by cell against a live amplifier.
conda env create -f environment.yml
conda activate 3rd-arm-mior pip install -r requirements.txt. The deep-learning baselines in
src/training.py need requirements-optional.txt as well; nothing else does.
MNE is pinned to 1.6.1 — the FCz reconstruction depends on referencing behaviour that differs between releases.
To run against your own recordings, copy configs/subjects.example.json to
configs/subjects.local.json and add an entry per participant.
Recordings are not distributed. Participants are identified only as S01, S02,
… throughout the code, the notebooks and every output; the map to real
identifiers lives in configs/subjects.local.json, which is gitignored and stays
on the acquisition machine. No figures, trained models or result summaries are
committed — they are either rebuildable from the pipeline or carry
per-participant results held back pending publication.
MIT — see LICENSE.