A local-first AI character engine. Give your character a voice, a face, memory, and screen vision; all running on your hardware. Stream them on Twitch with OBS, or just talk. Technically speaking, it can run in your machine, otherwise this app provides cloud provider options (OpenRouter mostly).
DEMO VIDEO: Youtube
You talk, it listens, it talks back. The whole voice loop runs locally — mic → VAD → STT → LLM → TTS → speaker. It's real-time, streaming, and you can interrupt it mid-sentence.
-
Character Cards — SillyTavern V2 compatible. Import PNG cards straight from Chub or SillyTavern (avatar + lorebook included), export your characters as Tavern Cards, or make your own from scratch.
-
Memory — ChromaDB vector store with RAG across three tiers: global (cross-character, user-curated), per-character general (durable facts), and per-session (auto-generated reflection flash cards and summaries). Retrieval uses composite scoring — relevance, recency decay, importance rating, and access frequency — with configurable weights and tier multipliers. Reflection prompts are fully user-editable.
-
Screen Vision — Takes screenshots and describes them via VLM. Can run as a dedicated local server, a cloud API, or piggyback on your main LLM if it supports vision, so far I only tested with Gemma3 models for local multi-modal vision. Note: The image is based on what is viewable in your main display.
-
Avatar — Standalone Tauri 2 + Three.js desktop overlay. Loads VRM models, does lipsync, tracks your cursor, and reacts with facial expressions and body animations based on what it's saying. Emotion classifier picks the mood, which drives both per-character expression composites (custom blend shape recipes) and base animation slots (idle, happy, sad, angry, curious) using Mixamo FBX clips retargeted to VRM skeletons. Assign different VRM models and animation sets per character.
-
Stream Subtitles — Separate Tauri 2 window for OBS compositing. Typewriter text synced to TTS duration, chroma key backgrounds, configurable fade. Just window-capture it in OBS.
-
Stimuli — The character doesn't just wait for you. It'll start talking on its own if you go idle, and it reads Twitch chat. Weighted lottery scheduling picks which stimulus fires — modules set priority weights, fired stimuli decay, idle stimuli recover. You can write your own stimulus modules too.
-
Function Calling — The LLM can call tools mid-conversation. Built-in tools: screen vision (describe what's on screen), game state query (ask about the current game), and invoke hack (send commands to a connected game). Tool executor supports forced invocation from lorebook triggers, with per-call timeouts to prevent dead air on provider failures.
-
Game Bridge — TCP bridge to game processes via SpindL Game Bridge. The character receives real-time game events (combat, dialogue, inventory, chapter transitions) as stimulus and can send commands back (e.g., solving puzzle grids). Bidirectional newline-delimited JSON on a local socket. Currently supports Pragmata via REFramework.
-
Twitch EventSub — Channel follow notifications via Twitch EventSub WebSocket. Approved chat messages emit overlay events for OBS compositing.
-
Chat Overlay — Lightweight OBS browser source (
/approved-chat.html) that displays only Twitch messages approved by the selection pass. Connects to SpindL's Socket.IO server and renders a filtered chat feed — no raw spam, no rejected messages, just the curated picks the character actually responds to. Point an OBS browser source atlocalhost:3000/approved-chat.html. -
Twitch Audience Memory — Persistent per-stream audience transcript (sidecar JSONL) with self-recall. The character remembers viewers by username across the whole stream — both their messages and her own replies — so she doesn't re-answer questions or forget who's been talking. Configurable rolling window (25–300 messages) and per-message char cap (50–500). Every user-role turn carries a structural source tag (
[Message Type - Voice | Direct Keyboard | Twitch Chat | Stimuli]) that persists into history, so the character can tell where any past message came from. -
Stream Deck — Standalone Tauri 2 overlay with hold-to-activate buttons. Signal when you're talking to chat, mods, Discord, or someone in the room — the character suppresses responses while held and gets context-aware prompting on release. Multiple named contexts, each with its own button and custom prompt. Dynamic add/remove from the dashboard.
-
Prompt Workshop — Block-based prompt editor. See exactly how many tokens each section costs, reorder them, override individual blocks with your own text, wrap them with injection prefixes/suffixes.
-
Generation Control — Temperature, top-p, max tokens, repeat penalty, repeat window, frequency penalty, presence penalty, and history mode (splice role-array vs. flatten-into-system) — all adjustable from the dashboard mid-conversation. Values persist across restarts. History mode is also pickable pre-launch from the Launcher page.
-
Runtime Swapping — Switch LLM or VLM providers mid-conversation from the dashboard. No restart needed.
-
Chat Interface — Text and voice in one view. Message history persists across sessions.
-
Web Dashboard — Next.js control panel. Character portrait with audio-reactive glow, real-time pipeline status, config, prompt editing, memory curation, session browser.
flowchart TD
A[Mic Input] -->|audio| B[Voice Pipeline]
B -->|text| C[LLM Pipeline]
C -->|response| D[TTS]
D -->|audio| E[Speaker Output]
C -->|events| F[Stimuli Engine]
C -->|mood| G[Avatar Bridge]
G -->|Socket.IO| H[SpindL Avatar]
G -->|Socket.IO| I[SpindL Subtitles]
G -->|Socket.IO| M[SpindL Stream Deck]
J[Web Dashboard] -->|"text + controls"| C
M -->|"addressing start/stop"| C
C -.->|query| K[Memory]
C -.->|capture| L[Screen Vision]
C -.->|tool calls| N[Tool Executor]
N -.->|commands| O[Game Bridge]
O -->|"TCP events"| F
One launcher script starts everything. One config file controls it all.
Hardware:
- NVIDIA GPU with at least 12GB VRAM (LLM + TTS + STT need to fit somewhere)
- More VRAM = bigger models. Two GPUs is nice if you have them
Developed on:
- CPU: Intel Core i9-13900K
- RAM: 64GB DDR5
- GPU: NVIDIA RTX 4090 (24GB) + RTX 3090 (24GB)
- OS: Windows 11, Python 3.12, CUDA 12.x
Software:
- Python 3.10+
- Node.js 20.9+ (for the web dashboard — required by Next.js 16)
- llama.cpp (pre-built binary or build from source)
- A GGUF model file (Qwen3, Llama 3, etc.)
Optional:
- Rust 1.75+ (required for SpindL Avatar, SpindL Subtitles, and SpindL Stream Deck — Tauri 2 apps. First-time build via Install button in Settings, ~6.5GB disk for shared Cargo workspace)
- A VLM-capable model (Gemma 3, LLaVA, etc.) or cloud VLM API key
- An embedding model (for memory/RAG — e.g., nomic-embed-text-v1.5)
# Clone the repository
git clone https://github.com/YOUR_USERNAME/spindl.git
cd spindl
# Create a virtual environment (conda or venv)
conda create -n spindl python=3.12
conda activate spindl
# Install dependencies
pip install -e ".[dev]"cd gui
npm install# Copy the example config
cp config/spindl.yaml.example config/spindl.yaml
# Copy the example .env for API keys (if using cloud providers)
cp .env.example .envEdit config/spindl.yaml to point to your model files, set ports, and pick your providers. API keys go in .env, not the YAML (use ${ENV_VAR} syntax).
Or skip the YAML entirely — launch the dashboard and configure everything from the Launcher page.
python scripts/dev.pyOpen http://localhost:3000 → use the Launcher page to configure and start services. Ctrl+C gracefully terminates everything.
For headless mode, avatar setup, subtitles, tests, and more — see the Usage Guide.
spindl/
├── src/spindl/ # Backend (Python, ~45,500 lines, 162 files)
│ ├── audio/ # Mic capture, speaker playback, Silero VAD
│ ├── avatar/ # Emotion classifier (DistilBERT/ONNX), tool mood mapping
│ ├── characters/ # SillyTavern V2 card models, import/export
│ ├── codex/ # Lorebook activation & management
│ ├── config/ # YAML config loading, env var resolution
│ ├── core/ # State machine, event bus, context manager
│ ├── gui/ # Socket.IO server (domain-split: 8 handler modules), response models
│ ├── history/ # JSONL conversation persistence, prompt snapshots
│ ├── launcher/ # Service process management & health checks
│ ├── llm/ # Prompt building, LLM providers, plugin pipeline
│ │ ├── builtin/ # llama.cpp, DeepSeek, OpenRouter providers
│ │ ├── providers/ # Pipeline content providers (history, persona, etc.)
│ │ └── plugins/ # History, budget, codex, reasoning, TTS cleanup (dual-output)
│ ├── memory/ # ChromaDB store, RAG injector, reflection, summaries
│ ├── orchestrator/ # Central voice agent loop, config (Pydantic v2), service capabilities
│ ├── personas/ # Persona loading
│ ├── stimuli/ # Autonomous behavior engine (weighted lottery, idle, Twitch, game state)
│ │ └── game_state/ # Game bridge consumer — TCP events, dialogue buffer, summarizer
│ ├── stt/ # Speech-to-text providers (Whisper, Parakeet)
│ ├── tools/ # Function calling framework + tool executor
│ │ └── builtin/ # screen_vision, game_state_query, invoke_hack
│ ├── tts/ # Text-to-speech providers (Kokoro, Qwen3)
│ ├── utils/ # Shared utilities
│ ├── vision/ # Screen capture + VLM providers (local, cloud, unified)
│ └── vts/ # VTubeStudio WebSocket driver
├── gui/ # Frontend (Next.js + React + TypeScript, ~38,200 lines, 166 files)
│ └── src/
│ ├── app/ # 8 pages (dashboard, launcher, characters, settings, etc.)
│ ├── components/ # 90+ feature components + Radix UI design system
│ └── lib/ # 13 Zustand stores, Socket.IO client, Zod schemas
├── spindl-avatar/ # Standalone avatar renderer (Tauri 2 + Three.js + VRM)
├── spindl-subtitles/ # Stream subtitle overlay (Tauri 2, OBS-compositable)
├── spindl-stream-deck/ # Addressing-others button overlay (Tauri 2, hold-to-activate)
├── Cargo.toml # Workspace root — shared target/ across all Tauri apps
├── tests/ # 106 unit test modules (~33,500 lines)
├── tests_e2e/ # 8 E2E test modules (Playwright, 5-config matrix)
├── scripts/ # Launcher, migration, standalone GUI
├── config/ # spindl.yaml.example template
├── characters/ # User character cards (gitignored)
├── stt/server/ # External STT server scripts (NeMo Parakeet)
└── tts/server/ # External TTS server scripts
- Local-first. Everything runs on your machine. Cloud providers are optional if you want them.
- One config file.
spindl.yamlcontrols the whole system.${ENV_VAR}syntax keeps secrets out of version control. - Pluggable. Every backend (LLM, STT, TTS, VLM) uses a registry. Adding a new provider = implement an ABC, register it, done.
- Hot-swappable. Switch LLM/VLM providers mid-conversation from the dashboard. No restarts.
- SillyTavern compatible. V2 character cards with PNG import/export. Drag a card from Chub into the import dialog and it just works — avatar, lorebook, and all.
- Validated. Pydantic v2 on the backend, Zod on the frontend. Config gets validated before it touches disk.
| Component | Local | Cloud |
|---|---|---|
| LLM | llama.cpp (any GGUF model) | DeepSeek, OpenRouter (200+ models) |
| TTS | Kokoro, Qwen3 | — |
| STT | Whisper, Parakeet | — |
| VLM | llama.cpp (Gemma 3, LLaVA, etc.) or unified mode (LLM handles vision) | Any OpenAI-compatible API (xAI, OpenAI, etc.) or unified mode (multimodal cloud LLM) |
| Embeddings | llama.cpp (--embedding mode) |
— |
Want to add a provider? Implement the ABC, register it, and you're good.
- kimjammer/Neuro — The project that proved a local Neuro-Sama was possible on consumer hardware. SpindL's stimuli system and overall architecture were heavily informed by studying this codebase.
- Klaa/V1R4 — Tauri + Three.js + VRM avatar with lipsync and mood expressions. Studied as a reference implementation for SpindL's standalone avatar renderer.
- SillyTavern — Character card V2 specification.
- ggerganov/llama.cpp — The LLM inference engine that makes local models practical.
- VRoid Hub — AvatarSample_B bundled as the default VRM model (redistribution permitted, no attribution required).
MIT — Copyright 2026 jubbydubby