Glasses-first runtime + Memory API + dataset spine

[wdwd720]

Glasses-first

Pivots Hearer to a glasses-first architecture and lands the supporting layers:

Even G2 microphone hears the environment
       ↓
Even Hub / phone runtime receives glasses audio
       ↓
Hearer brain classifies + updates world state
       ↓
Hearer combines sound + time + routine + memory + context
       ↓
Hearer decides whether this matters right now
       ↓
Hearer sends one tiny cue to the G2 HUD

Browser mic, tab/system audio, file uploads, and generated fixtures are now clearly dev fallbacks, not the product surface. The simulator HUD and the real Even G2 HUD adapter receive the same payload — { text, priority, actionType, ttlMs } — so the dashboard always reflects what the glasses would show.

What's in here

Glasses-first runtime (src/glasses/)

• g2RuntimeTypes.ts — WorldAudioInputAdapter (even_g2_mic is isPrimaryTarget: true; browser mic / file / fixtures are fallbacks) and HudOutputAdapter (even_g2_hud primary, simulator fallback).
• g2RuntimeController.ts — owns the active input/output adapters, probes the Even Hub bridge once on boot, routes G2 PCM frames through the same feature → classifier → world-state → cue pipeline as everything else.
• evenHub/ — bridge probe (dynamic, SDK-optional, no hard dependency on @evenrealities/even_hub_sdk), Even G2 audio input adapter (audioControl + onEvenHubEvent → PCM16 LE → AudioFrame), Even G2 HUD output adapter (showText / updateText / ensureCuePage+setCuePageText), and evenHubPcm.ts (PCM16 LE → Float32, robust event-shape extraction).
• simulator/ — browser-mic / display-audio / file fallback + simulator HUD that mirrors the same payload.

World-state engine (src/engine/worldStateEngine.ts)

The actual intelligence. Same beep produces a different cue depending on:

• saved cooking routine,
• live context (location / activity / cooking toggle),
• time of day.

Stays honest about certainty — Check stove, never Stove is on. Plugs into decideFromLiveAudio and bumps priority to urgent when a saved cooking routine matches.

Memory API + brain integration

• server/ — Fastify + better-sqlite3 (SQLite) with JSON fallback. Routes for profile / people / items / locations / routines / commitments / events / cues / audio-detections / cue-feedback / memory-extract / memory-reset.
• shared/ — types and rule-based memory extractor used by both server and browser.
• src/api/hearerApi.ts + src/memory/{api,local,createMemory}Client.ts — frontend probes the API and falls back to localStorage cleanly.
• decide() and decideFromLiveAudio() now consume persistedRoutines and surface Saved routine matched in the reasoning trail. The leaving-home compressor uses the saved routine action label when present (e.g. bring laptop instead of keys + laptop).

UI

• G2RuntimePanel — input/output target rows with primary/fallback chips, bridge-detected status, Start/Stop/Send-test-cue/Use-fallback buttons.
• MemoryCommandBox — "Tell Hearer what to remember…" with example one-click buttons; submit triggers /api/memory/extract and a tiny HUD confirmation cue.
• SystemPipelineCard — 4-step glasses-first flow.
• DatasetModelCard — training/eval status; explicitly not the runtime.
• AudioAwarenessPanel re-labelled Audio fallback (dev); mic/tab buttons say (fallback); fixtures labelled "Deterministic test inputs". Adds a direction selector for simulated 360 awareness.
• MemoryPanel reads the persisted summary (routines / commitments / items) and falls back to in-engine demo memory.
• ActionPanel adds cue-feedback buttons (helpful / too_much / wrong / missed) that POST to /api/cue-feedback.

ML / dataset spine (training/eval ONLY — never the runtime)

• ml/hearer_labels.yaml — Hearer label taxonomy V1 with cueTemplates, defaultPriority, actionType, minConfidenceForInterrupt, and ESC-50 / UrbanSound8K / FSD50K source mappings.
• ml/scripts/{prepare_esc50, prepare_urbansound8k, prepare_fsd50k, build_hearer_manifest, validate_manifest, summarize_manifest, export_label_map}.py — build a unified JSONL manifest, validate, and generate src/audio/hearerLabelMap.generated.ts.
• ml/training/{train_audio_baseline, eval_audio_baseline, export_model_stub}.py — CPU-friendly log-mel + sklearn baseline.
• ml/requirements.txt and ml/README.md cover setup. Datasets are never auto-downloaded.

Audio model adapter boundary

src/audio/modelAdapters/audioModelAdapter.ts ships:

• RuleBasedModelAdapter (active),
• LocalTrainedModelAdapterStub (waits for ml/training/ artifacts),
• PretrainedModelAdapterStub (YAMNet/TF.js plug point).

Tests

npm test → 46 / 46 passing across 13 suites:

• evenHubPcm — PCM16 LE → Float32, Int16Array passthrough, empty/odd inputs, AudioFrame at 16 kHz with the right duration, event-shape extraction across audioEvent / sysEvent.audioEvent / pcm / samples[].
• g2RuntimeController — simulator HUD active when bridge unavailable; start/stop don't crash without a bridge.
• g2AudioDecision — G2 mic + cooking memory → Kitchen timer beeping. / Check stove. (urgent, physical); same beep without context → softer cue; speech_nearby never hallucinates a transcript.
• worldStateEngine — cooking routine match → urgent + check-stove; unknown context → softer; never asserts hazard certainty.
• hudOutputAdapter — payload is just text + priority + actionType + ttlMs; simulator and Even G2 adapters agree on payload.
• memory extractor — routine, commitment, pharmacy routine, ambiguous-input notes.
• hearerLabelMap.generated — core labels present; ESC-50 / UrbanSound8K source mappings round-trip.
• All 23 prior tests (cue compressor, priority engine, action router, audio mapper, classifier, live audio decision) still pass unchanged.

npm run build → success (~249 kB JS gzipped to ~75 kB).

What is not in here

• No real BLE / Even G2 firmware code. The Even Hub adapters are version-tolerant probes that honestly report unavailable when the SDK isn't present. The boundary is in place; a real SDK swap is a single-file change.
• No remote model download / TF.js dependency. Pretrained adapter is a stub. The runtime classifier remains the rule-based local one.
• No medical / emergency reliability claims. Hearer suggests checks (Check stove), never asserts hazards (Stove is on).

How to run

npm install
npm run dev          # http://localhost:5173
npm run server       # http://localhost:8788  (SQLite by default)
npm run dev:all      # both at once
npm run build
npm test

Killer demo

1. In Teach Hearer, click "Usually after dinner I cook, so if you hear beeping remind me to check the stove." and press Save. The Memory & routines panel shows a Cooking timer safety cue routine.
2. In Audio fallback (dev) click Beep ×5.
3. The HUD shows Kitchen timer beeping. / Check stove. — URGENT / PHYSICAL — because the world-state engine matched the saved cooking routine.
4. Switch the live context to location: unknown and rerun the fixture. The same audio softens to Timer beeping. / Check nearby..