Audio Awareness V1: live audio → Hearer brain → HUD

[wdwd720]

Audio Awareness V1

Adds a real (semi-real) audio pipeline on top of the existing Hearer simulator brain. Same priority engine, same cue compressor, same action router, same HUD — fed now by mic / tab+system audio / file uploads / synthetic fixtures instead of only scripted scenarios.

audio source → features → classifier → cooldown → signal mapper
   → liveAudioDecision → priority/cue/action engines → HudOutputAdapter

The simulator, scenario panel, auto-demo, and engine layer from the merged MVP PR are untouched.

What's in here

Audio layer (src/audio/)

File	Role
audioTypes.ts	AudioFrame, AudioFeatures, AudioClassification, AudioDetectionEvent, LiveContextOverride, controller snapshot.
audioSource.ts	Web Audio mic, getDisplayMedia tab/system audio (audio-track required, video track dropped immediately), decodeAudioData for files plus an offline Goertzel-based coarse spectrum so file uploads exercise the same feature path.
audioFeatureExtractor.ts	RMS, peak, ZCR, dominant frequency, energy bands, tonality, beep/siren/speech/transient scores. Linear conversion from AnalyserNode.getFloatFrequencyData dBFS magnitudes.
audioClassifier.ts	Pluggable AudioClassifier interface. RuleBasedAudioClassifier (always available) tracks inter-frame frequency variation for siren detection and a burst counter for timer beeps. PretrainedAudioClassifierStub is the YAMNet plug point — intentionally not wired to TF.js / a remote model in this build.
audioSignalMapper.ts	Per-label routing thresholds, maps a classification into a Hearer AudioSignal (added knock and speech_nearby).
audioController.ts	Stateful orchestrator: source management, cooldown (1.2 s urgent / 3 s default), debounce, auto-route toggle, manual route button, fixture injection. Emits snapshots to React.
audioFixtures.ts	Synthetic beeps/sirens/horn/knock/doorbell/applause/speech-like Float32 buffers with a WAV encoder so fixtures flow through the same decode pipeline as a real upload.

Glasses boundary (src/glasses/)

• hudOutputAdapter.ts — interface.
• simulatorHudAdapter.ts — active in this build.
• evenG2AdapterStub.ts — wire-format only: { text, priority, actionType, ttlMs }. No BLE / SDK / device integration in this build. The stub sendCue() only logs the payload it would send. The boundary exists so a future Even Hub / Even G2 SDK swap is a one-file change.

Engine extensions

• types.ts — AudioEvent now includes "knock" and "speech_nearby".
• liveAudioDecision.ts — new entry point that takes a single AudioSignal plus a LiveContextOverride and runs the same priority engine / cue compressor / action router. Never hallucinates a transcript from speech-like audio.
• cueCompressor.ts — new live-audio compressors: timer-unknown, alarm, loud-alert-unknown, doorbell-unknown, knock, speech-nearby, applause, laughter.
• priorityEngine.ts — live_* scenario rules.
• actionRouter.ts — live_* routing (vibration sims for physical alerts; awareness only for speech/crowd).

UI (src/components/audio/)

• AudioAwarenessPanel — source controls (mic / tab+system / stop / file drop), test fixtures, audio level meter, classifier status chip, live context override (location + activity dropdowns and toggles for cooking routine / delivery expected / event mode), auto-route toggle, manual route button, last-detection card, error banner.
• AudioLevelMeter, AudioFileDropzone, AudioClassifierStatus, AudioDetectionLog.

App wiring

App.tsx keeps refs for memory and override so the audio controller always sees the latest state, subscribes to its decisions, and fans them out through both the simulator HUD adapter and the Even G2 stub. Reset stops audio capture.

Live → HUD examples

Action	Live context	HUD cue	Priority / Type
Beep ×5 fixture (or real timer audio)	kitchen + cooking + cooking routine	Kitchen timer beeping. / Check stove.	urgent / physical
Beep ×5 fixture	unknown	Timer beeping. / Check nearby.	high / physical
Siren fixture	street + walking	Road alert: / siren nearby.	urgent / physical
Siren fixture	unknown	Loud alert nearby. / Look around.	high / physical
Doorbell fixture	home	Doorbell. / Possible delivery.	high / physical
Knock fixture	home	Knock detected. / Check door.	high / physical
Speech-like fixture	event mode	Speech nearby. / Stay aware.	low / awareness — no transcript inferred

Tests

npm test → 23 / 23 passing across cue compressor, priority engine, action router, audio signal mapper, rule-based classifier, and the live audio decision pipeline (kitchen vs unknown context, siren on street, knock at home, speech_nearby never hallucinates a transcript).

npm run build → success (~207 kB JS gzipped to ~64 kB).

Privacy & safety

• Audio is processed entirely in-browser. No server, no upload, no recording. Raw audio is held only as a rolling Float32 window and discarded after features are extracted.
• The rule-based classifier is honest about its confidence — every detection is logged with explanation and route note.
• No medical / emergency reliability claims. Trusted-contact escalation is referenced only as opt-in.

Constraints honoured

• Did not rebuild the existing app — the simulator, scenarios, scenario panel, auto-demo, and engine layer are untouched.
• No backend, no auth, no database.
• No remote model download / TF.js dependency.
• No real BLE / Even G2 integration — only the adapter boundary.
• HUD cues stay tiny (two short lines, ≤60 chars) and never dump transcripts.