🎙️ Oxla — Multimodal Voice AI Executive Assistant & Desktop Agent

Oxla is a privacy-preserving, always-listening multimodal voice AI executive assistant engineered with sub-second WebRTC speech streaming, dual biometric identity verification, a 3-layer security guardrail engine, and an authenticated 3D web control panel.

---

🌟 Key Features & Highlights

• 🎙️ Sub-Second Voice WebRTC Engine: Real-time bidirectional voice streaming built on LiveKit Agents and Google Gemini Realtime API (gemini-3.1-flash-live-preview) with natural speech synthesis (Aoede voice).
• 🛡️ 3-Layer Security Guardrail Architecture:
  • Input Guardrail: Detects and blocks prompt injection attacks and malicious keywords before tool execution.
  • Output Guardrail: Sanitizes and redacts secrets (AIzaSy...), passwords, and private tokens ([REDACTED_SECRET]) before speech or logging.
  • Tool Execution Guardrail: Enforces app whitelisting (Cursor, Chrome, VS Code, Notepad, WhatsApp, File Explorer) and rate limiting (max 15 requests/min).
• 👤 Dual Biometric Authentication:
  • OpenCV LBPH Face Verification: Biometric face identity validation.
  • Voice Biometrics Engine: Local cosine-similarity voice verification (master_voice.enc).
• 🏢 Dual Operational Modes:
  • Standard Mode: Warm, conversational voice assistant tier for general Q&A and daily inquiries.
  • Executive Mode: Biometrically guarded tier granting administrative control over dev tools, file access, and PC automation.
• 📊 Authenticated 3D Web Control Panel:
  • Bound strictly to 127.0.0.1:8765 with 32-character random session token authentication (SESSION_AUTH_TOKEN).
  • Real-time 3D Quantum Plasma audio visualizer, live telemetry gauges (CPU/RAM, Disk free space, Power), and scrolling transcript logs.
• 💻 Developer & Workspace Automation:
  • Git Codebase Inspector: Voice-driven git status and recent commit log inspection.
  • Code Syntax Tester: Automated Python syntax compilation verification.
  • Voice PDF Summarizer: Multi-page PDF text extraction (pypdf) with 3-bullet AI briefings stored to encrypted memory.
  • Smart Clipboard Assistant: Security-redacted clipboard reading and code snippet saving.
  • Proactive System Telemetry Daemon: Background monitoring for battery (<15%) and RAM (>90%) voice alerts.
• 🔐 Fernet 256-Bit Encrypted Memory: Persistent cross-session facts and research notes stored safely in memory.enc.

---

🏗️ System Architecture

                                 ┌───────────────────────────┐
                                 │   Voice Input (Mic / UI)  │
                                 └─────────────┬─────────────┘
                                               │
                                 ┌─────────────▼─────────────┐
                                 │  Input Guardrail Filter   │
                                 └─────────────┬─────────────┘
                                               │
               ┌───────────────────────────────┴───────────────────────────────┐
               │                                                               │
┌──────────────▼──────────────┐                                 ┌──────────────▼──────────────┐
│ Standard Mode (General Tier)│                                 │ Executive Mode (Biometric)  │
│ • General Voice Answers     │                                 │ • App Control & Dev Tools   │
│ • Daily Briefing & Alarms   │                                 │ • File Manager & Memory     │
└─────────────────────────────┘                                 │ • Email, PDF & Git Tools    │
                                                                └──────────────┬──────────────┘
                                                                               │
                                                                ┌──────────────▼──────────────┐
                                                                │ Tool Execution & Sandbox    │
                                                                └──────────────┬──────────────┘
                                                                               │
                                                                ┌──────────────▼──────────────┐
                                                                │ Output Guardrail (Redactor) │
                                                                └──────────────┬──────────────┘
                                                                               │
                                                                ┌──────────────▼──────────────┐
                                                                │   Audio Response (Speaker)  │
                                                                └─────────────────────────────┘

---

📁 Repository Structure

├── agent.py                 # Core LiveKit Agent & Tool Server (45+ Production Tools)
├── instruction.py           # System Prompt Directives (Standard vs. Executive Tiers)
├── karg.html                # Authenticated 3D Web Control Panel & Telemetry Dashboard
├── setup_secrets.py         # Utility to encrypt secrets into Fernet-protected .env.enc
├── generate_keys.py         # Fernet 256-bit Key Generator
├── capture_face.py          # OpenCV Face Capture Tool
├── train_face_model.py      # OpenCV LBPH Model Trainer
├── face_ident.py            # OpenCV Biometric Face Recognition Engine
├── enroll_voice.py          # Local Biometric Voice Enrollment Utility
├── wakeword_listener.py     # Local Privacy-Preserving Wake Word Daemon ("Hey Oxla")
├── test_guardrails.py       # Security Guardrails Unit Test Suite
├── test_voice_biometrics.py # Voice Biometrics Test Suite
├── .env.example             # Environment Variables Template
├── .gitignore               # Security Rules (Blocks secrets, keys, models, & biometrics)
├── requirements.txt         # Project Dependencies
└── README.md                # Documentation & Architecture Overview

---

⚙️ Quick Start & Setup

1. Clone & Set Up Environment

git clone https://github.com/YOUR_USERNAME/Oxla-AI-Agent.git
cd Oxla-AI-Agent

python -m venv venv
# On Windows:
venv\Scripts\activate
# On Linux/macOS:
source venv/bin/activate

pip install -r requirements.txt

2. Configure Environment Variables

Copy .env.example to .env and populate your API credentials:

cp .env.example .env

Required keys in .env:

GOOGLE_API_KEY=your_gemini_api_key
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
OXLA_ENV_KEY=your_fernet_encryption_key

Encrypt your secrets safely:

python setup_secrets.py

3. Enroll Biometrics (Optional)

# Face Verification Enrollment:
python capture_face.py
python train_face_model.py

# Voice Biometrics Enrollment:
python enroll_voice.py

4. Run Oxla Agent

python agent.py console

---

🗣️ Voice Commands Showcase

Category	Command Example	Action
☀️ Briefing	"Oxla, give me my daily briefing."	Speaks time, RAM/Disk status, unread emails, and task alarms.
💻 App Control	"Oxla, open Cursor."	Launches Cursor IDE on desktop (whitelisted execution).
📁 File Manager	"Oxla, open File Explorer."	Launches Windows File Explorer.
💻 Git Assistant	"Oxla, check my git status."	Inspects codebase git changes and recent commit history.
📄 PDF Summarizer	"Oxla, summarize report.pdf."	Extracts PDF text, speaks a 3-bullet briefing, and saves to memory.
🔬 AI Research	"Oxla, research OpenCV LBPH face recognition."	Generates technical briefing and saves to encrypted memory (memory.enc).
📋 Smart Clipboard	"Oxla, read my clipboard."	Redacts secrets and explains copied code or text out loud.

---

🛡️ Security & Privacy Architecture

• Zero Secret Leaks: .gitignore strictly excludes .env, .env.enc, memory.enc, master_voice.enc, face training datasets, and log files.
• Dynamic Portability: Local application paths use OS environment expansion (USERPROFILE / LOCALAPPDATA) with zero hardcoded OS usernames.
• Local WebSocket Binding: Control panel bridge binds strictly to 127.0.0.1:8765 requiring random token authentication (close code 1008 on unauthorized connections).

---

📄 License

Distributed under the MIT License. See LICENSE for more information.