Liquid State XR

Sound becomes spikes — a liquid state machine you can explore in Meta Quest passthrough.

Launch on GitHub Pages · Launch on Build Nebulytic · Model and scientific notes

Live microphone audio passes through a Lyon cochlear model and BSA spike encoder into a recurrent spiking reservoir. Watch the cochleogram, neuron activity, curved synaptic paths, and spike histories together in 3D. The recovered TI-46 spoken-digit classifier and a separate personal readout expose their guesses alongside the reservoir.

The app is a static website. GitHub Pages serves the same browser application independently of Build Nebulytic; no backend, API key, account, CDN, or audio upload is required.

From audio to activity

Pipeline diagram. The cover image above is generated from the actual saved recurrent graph. Open either launch link to explore the interactive scene and floating panels.

Try it

1. Open either launch link in Meta Quest Browser or a desktop browser with WebGL2.
2. Choose Play sound demo to explore without microphone access, or Start microphone and grant permission to use your own voice.
3. On Quest, enter passthrough. The floating panels include microphone controls, waveform, the 77-channel cochlea, input raster, output guesses, and teaching controls.
4. Point and trigger/pinch to inspect a neuron. Squeeze to move the workspace; the left thumbstick rotates it and the right thumbstick changes its size.

Desktop supports orbit/zoom, local audio files, and Spatial controls to preview the floating panels. The size slider and floating smaller/larger buttons scale the entire workspace from 20% to 300%.

WebXR passthrough needs an immersive-AR-capable browser and headset. GitHub Pages supplies HTTPS, which is required for microphone and WebXR access outside localhost. Desktop browsers retain the interactive 3D view when immersive AR is unavailable.

What is inside

Stage	Implementation
Audio	AudioWorklet capture, bounded message queue, anti-aliased resampling to 12 kHz
Cochlea	77-channel Lyon cascade, rectification, four AGC stages, channel difference and decimation to 1 kHz
Spike encoder	BSA matching pursuit, 33-tap kernel, threshold 0.955, 32 ms lookahead
Reservoir	125 leaky integrate-and-fire neurons, 1,150 recurrent and 308 input synapses
Saved readout	10 spiking neurons using the recovered TI-46 classifier weights
Personal readout	Separate online recursive ridge classifier trained from labelled utterances
Rendering	Three.js / WebGL2, instanced neurons, shared axon trunks, curved branches and GPU spike pulses
Histories	Per-neuron raster strips showing 2.048 seconds; larger strips at the output

The graph layout relaxes drawing coordinates into irregular clusters. Every saved neuron ID, connection, weight, and solver parameter remains unchanged. A shared outgoing axon points toward the mean of its targets, then branches into individual connections and dendritic ends. This is a schematic visualization, not reconstructed biological anatomy.

The encoder runs in WebAssembly with float64 state. The sparse CPU reservoir is the default. Optional WebGPU simulation must pass numerical and timing checks on the viewing device and falls back if it cannot sustain the audio budget. Rendering uses the GPU in either case.

Spoken-digit recognition

The included inference model comes from TI-46 checkpoint runTI46_test23.mat, fold 1, saved epoch 200. Its reservoir, input connections, neuron types, timing parameters, and 10 × 125 readout weights are included together. This is not a randomly generated reservoir paired with unrelated classifier weights.

Live speech recognition remains experimental: microphone acoustics, utterance segmentation, state resets, and a correction to the original cochlear wrapper differ from the historical evaluation. Output counts are spikes, not calibrated confidence percentages. No historical dataset accuracy is claimed for browser use. See the full provenance and limitations.

To teach a personal readout:

1. Start the microphone, then select a digit on the output panel or an output neuron.
2. Say the digit once and pause for half a second.
3. Repeat with several examples of each digit. Teach at least two different digits before expecting a guess.

Training updates only the separate personal classifier; the original reservoir and TI-46 readout remain intact. Training restores gain 1, recurrent strength 1, and the CPU solver. Guesses are restricted to labels you have taught.

Up to 200 feature vectors and labels are saved locally, bound to a fingerprint of the model and encoder settings. Raw audio is not stored or uploaded. Personal profiles belong to the current browser and website origin, so the GitHub Pages and Build Nebulytic versions keep separate profiles. Clear personal samples removes the local profile.

Run locally

Install Node.js 22 or later and Python 3.10 or later. Three.js is vendored and there are no npm dependencies to install.

git clone https://github.com/ajinkyagorad/liquid-state-xr.git
cd liquid-state-xr
npm test
npm run build
npm run dev

Open http://127.0.0.1:8179/. The local server serves dist/; rebuild and reload after edits. Opening index.html with file:// will not provide the required worker and microphone environment.

Host on GitHub Pages

This repository deploys dist/ using GitHub Actions on pushes to main. The workflow runs the numerical tests and builds the site before deployment. GitHub Pages provides static HTTPS hosting; all neural processing stays on the device.

For a fork, enable Settings → Pages → Build and deployment → GitHub Actions, then run the workflow or push to main. A project site works under /<repository-name>/ because asset, worker, module and model URLs are relative. No separate service or custom domain is needed. To use another static host, publish the contents of dist/ with JavaScript and WebAssembly MIME types.

Source and validation

Path	Purpose
src/dsp.js, src/encoder.c	Reference JS and WebAssembly Lyon/BSA implementations
src/engine-worker.js, src/capture-worklet.js	Streaming audio pipeline and speech segmentation
src/reservoir.js, src/gpu.js	Sparse CPU and optional WebGPU neural solvers
src/personal-readout.js	Separate personal classifier and model-bound profile
src/scene.js, src/layout.js, src/spatial-dashboard.js	Neuron geometry, WebXR interaction and floating panels
public/models/gorad-ti46.json	Inference-only checkpoint export
tests/	Numerical, topology, streaming, classifier and worker checks

Fifteen automated checks cover independent SciPy references, model dimensions, neural dynamics, cochlear filtering, BSA boundaries, silence, alias rejection, refractory periods, WebAssembly parity, geometry invariance, personal learning and worker behavior. Test audio is synthetic; no TI-46 recordings or speaker records are included. Physical Quest performance and live speech accuracy still require on-device evaluation.

The compiled encoder is included. To rebuild it with Clang and the WebAssembly linker on Linux:

clang --target=wasm32 -O3 -nostdlib -fno-builtin -ffp-contract=off \
  -Wl,--no-entry -Wl,--export-memory -Wl,-z,stack-size=131072 \
  -Wl,--initial-memory=1048576 -Wl,--max-memory=1048576 \
  src/encoder.c -o public/encoder.wasm
npm test
npm run build

tools/reference_fixture.py CHECKPOINT.mat regenerates the independent synthetic reference using NumPy and SciPy when you have the original research checkpoint. The checkpoint itself is not needed to build, test, or run the app.

Research and attribution

• Gorad, Saraswat and Ganguly: Predicting Performance using Approximate State Space Model for Liquid State Machines.
• Hardware-Friendly Synaptic Orders and Timescales in Liquid State Machines for Speech Classification.
• Malcolm Slaney: Auditory Toolbox, Version 2.
• Three.js: vendored with its MIT license.

See third-party notices and scientific notes for attribution and implementation details. A public repository does not change the licenses of its research materials or dependencies.