PROMETHEUS is a highly personalized, local-first personal AI assistant. It integrates a fine-tuned local Large Language Model (LLM) with a cloned voice, a real-time UE5 avatar, and a strictly permission-scoped, sandboxed tool-calling execution layer.
Designed with security, privacy, and true local ownership in mind, PROMETHEUS gives you an assistant that sounds like you, remembers your context across modalities, and can take real actions on your behalf—safely, transparently, and entirely under your control.
- Personalization via Digital DNA: Ingests your private chat exports and writing samples, applies a third-party PII redaction pass via spaCy NER, and fine-tunes an 8B base model using LoRA.
- Voice Cloning & Provenance: Clones your voice using XTTS v2. Every synthesized voice clip is locally logged with a destination tag to establish a permanent provenance audit trail, ensuring transparent and traceable usage of your digital likeness.
- Strictly Sandboxed Tool Registry: Actions (e.g., executing Python code, browsing, or file I/O) are defined in a Pydantic-validated
ToolRegistry. The agent does not have free-form shell execution rights. - Confirmation Queue: Any tool requiring
confirmpermissions (like sandboxed execution or home automation) halts the ReAct planner until the user explicitly approves the JSON payload via the frontend UI. - Full-Stack Interface: Features a beautiful Next.js glassmorphism frontend that talks to a FastAPI backend over WebSockets, maintaining real-time status of agent thinking, speaking, and pending tool confirmations.
- Opt-In Perception: Incorporates explicit toggles for Audio (Whisper), Vision (YOLOv8 + CLIP), and Screen OCR. No silent background sensing.
- UE5 Metahuman Integration: Interfaces with Unreal Engine 5 via an OSC UDP bridge (
python-osc) to drive real-time Live Link Face blend shapes and lip-syncing.
PROMETHEUS operates in discrete layers, explicitly decoupling planning from execution, and perception from transmission.
┌─────────────────────────────────────────────────────────────────────────────┐
│ AVATAR & PRESENCE LAYER │
│ Unreal Engine 5 Metahuman (Live Link Face + Audio2Face-driven lip sync) │
└─────────────────────────────────────────────────────────────────────────────┘
│ OSC / local IPC
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ VOICE SYNTHESIS LAYER │
│ XTTS v2 / StyleTTS 2 (local voice clone) │
│ Every synthesized clip logged to the provenance record │
└─────────────────────────────────────────────────────────────────────────────┘
▲
│
┌─────────────────────────────────────────────────────────────────────────────┐
│ MULTIMODAL PERCEPTION LAYER │
│ Vision (YOLOv8) · Audio (Whisper) · Screen (OCR) │
│ All perception is explicitly toggle-controlled in the UI │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ COGNITIVE CORE (Model Router + Memory) │
│ Hot: 8B Llama 3.1 Instruct (conversation, low-latency tool routing) │
│ Cold: 70B Llama 3.1 Instruct (on-demand deep reasoning, mmap offload) │
│ Memory: Working buffer -> Episodic & Semantic (ChromaDB) │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ TOOL LAYER (Closed set, Pydantic-validated, Sandboxed) │
│ - File I/O - Sandboxed Code Execution - Home Automation │
│ - Browser Control - Communication Drafting (Never Auto-Sent) │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ HUMAN REVIEW & AUDIT LAYER (UI) │
│ Confirmation Queue blocks ReAct planner on restricted tool execution. │
└─────────────────────────────────────────────────────────────────────────────┘
PROMETHEUS abandons the flawed "prompt engineering" approach to safety and implements a structurally enforced sandbox:
- No Free-Form Execution: The LLM cannot emit raw shell commands. It must return a JSON payload matching a Pydantic schema in the
ToolRegistry. - No Autonomous Outbound Communiation: Network-capable tools (
draft_communication) are stripped of sending capabilities at the AST (Abstract Syntax Tree) level. The agent drafts; humans send. - Queue Halting: The
ConfirmationQueueintercepts high-risk tool calls. The planner thread strictly awaits humanapproveorrejectWebSocket signals before releasing the sandbox. - Third-Party Consent: Training data is routed through a
third_party_redactorpipeline to anonymize non-owner PII prior to any LoRA training runs. - No Internet Dependencies: Core capabilities run 100% locally.
- Python 3.11+
- Node.js 18+
- NVIDIA GPU with at least 8-12GB VRAM (for the 8B hot model). Note: To run the UE5 Avatar concurrently, you must account for an additional 4-6GB of VRAM overhead.
Clone the repository and install the Python dependencies:
git clone https://github.com/HarshkumarG007/PROMETHEUS.git
cd PROMETHEUS
# Install the backend package and dependencies
pip install -e .
pip install fastapi uvicorn[standard] python-oscInstall the Next.js dependencies:
cd src/prometheus/ui
npm installPROMETHEUS runs as a decoupled full-stack application. You must start the Python FastAPI backend and the Next.js frontend separately.
Open a terminal in the root PROMETHEUS directory and run:
uvicorn prometheus.server.main:app --reloadThe WebSocket endpoint will be exposed on ws://localhost:8000/ws/chat.
Open a second terminal, navigate to the UI directory, and run the development server:
cd src/prometheus/ui
npm run devNavigate your browser to http://localhost:3000.
The safety mechanisms of PROMETHEUS are load-bearing. A comprehensive test suite ensures the sandboxing, registry mapping, and hardware fallbacks behave correctly.
Run the entire test suite using pytest:
pytest tests/tests/safety: Validates the adversarial boundaries of the tool registry, AST communication checks, and queue halting.tests/avatar: Validates the OSC blend-shape routing and the graceful GPU-headroom degradation.tests/server: Validates the WebSocket lifecycle and approval queue streaming.tests/redaction: Ensures third-party PII is successfully anonymized from synthetic datasets.
PROMETHEUS is rigorously quantified to ensure real-world latency targets are hit under realistic consumer hardware constraints.
Hardware: RTX 4090 24GB + 64GB DDR5
- 8B Hot Tier (Q5_K_M): Real-time interactive latency, maintaining <2s to first token. Fully VRAM resident (~6GB footprint).
- 70B Cold Tier (Q4_K_M): On-demand mmap load with CPU offloading. Reserved for batch/reasoning tasks (latency budgeted at minutes-per-response).
Safety cannot introduce bottleneck latency. We benchmarked the routing overheads:
- Tool Schema Validation: < 0.001 ms per tool call. The Pydantic strict-validation layer introduces virtually zero latency before catching adversarial payloads.
- Third-Party PII Redaction: ~0.005 ms per message turn (regex/spaCy hybrid pass). The ingestion pipeline can safely strip real identities from millions of chat logs in seconds, ensuring your LoRA fine-tuning is untainted by unconsenting third-party PII.
MIT License. See the LICENSE file for more information.