Emotion-Aware Voice Assistant

"Hands-Free, Low-Latency, Empathetic Agentic Interaction"

A sophisticated, real-time AI Voice Assistant designed for emotion and coaching. Unlike standard chatbots, it detects user emotion and intent in parallel, and handles interruptions instantly (Barge-In) via Silero ONNX VAD. It runs a fully local audio pipeline, faster-whisper for STT and Kokoro for TTS, with a fault-tolerant FastAPI backend.

Docs

Doc	Purpose
docs/STATUS.md	Handoff status
docs/setup.md	Setup
AGENTS.md	Agent guidance

V2.0 Update: The system now runs a fully localized audio pipeline (Whisper + Kokoro), ensuring zero API latency for voice generation and absolute data privacy.

🧠 Why Silero VAD? (System Design)

In voice AI, latency breaks the "illusion of conversation." We utilize Silero VAD running via ONNX Runtime Web for three critical reasons:

1. "Barge-In" Capability: Analyzes audio every 30-50ms. The moment human speech energy is detected, it kills the AI's audio player instantly, mimicking natural human conversation.
2. Absolute Privacy: Speech detection and transcription happen entirely locally. No audio leaves your hardware.
3. Zero-Latency Triggering: ONNX Runtime Web runs outside the main JS thread, avoiding garbage collection pauses found in JavaScript, ensuring smooth audio buffer processing even during heavy UI rendering.

🏗️ Robust Architecture (The Singleton Pattern)

To prevent "State Fragmentation" and memory leaks, the system uses a Global Reference (Singleton) pattern:

• Resource Management: Uses useRef to manage the Audio Player and Microphone Stream globally.
• Interruption Logic: Ensures the VAD engine (running in an effect scope) has direct access to the Audio Player to hit the "Stop" button the millisecond an interruption is detected.

🚀 The 5 Pillars

1. The Brain (FastAPI + Groq Llama-3): Uses Async Parallelism to process Emotion Analysis, Intent Detection, and Fact Extraction simultaneously.
2. The Ears (Silero VAD + faster-whisper): High-performance ONNX model for <50ms voice detection, paired with local Whisper transcription.
3. The Memory (MongoDB + Fact Profiler): Distinguishes between short-term context and long-term "Fact Profiles" (e.g., recalling specific user life details across sessions).
4. The Router (Semantic Intent Layer): Classifies input into Venting (Validation), Coaching (Action), or Crisis (Safety) modes.
5. The Voice (Kokoro TTS + Emotional Physics): Dynamically adjusts speed and voice profiles locally for empathetic feedback without cloud API latency.

🛠️ Tech Stack

• Frontend: React 18, Tailwind CSS, Web Audio API, ONNX Runtime Web (Silero VAD)
• Backend: Python 3.11, FastAPI, Motor (Async MongoDB), Groq SDK
• Local AI Models: faster-whisper (STT), kokoro (TTS)

🔮 Future Roadmap

• Crisis Guardrails: Integration of Nvidia NeMo Guardrails for semantic vector-based safety checks.
• Dynamic Temperature: Intent-based LLM temperature scaling (Venting: 0.9 for warmth | Coaching: 0.2 for precision).
• "Double-Talk" Handling: Fine-tune Web Audio API echoCancellation and implement a "Debounce" timer in the VAD wrapper to prevent the AI from triggering its own VAD.