The brain of the Cortex wearable system. Runs on an Orange Pi Zero 2W (Allwinner H616, 2GB RAM) with a WhisPlay board (display, speaker, gamepad). Receives commands from AI agents via WiFi HTTP (preferred) or BLE (fallback via ESP32 dongle), and stores everything in a local SQLite database.
AI Agent ──MCP──> cortex-mcp ──WiFi HTTP──> [this code] Orange Pi Zero 2W (preferred)
──serial──> ESP32 ──BLE──> [this code] (fallback)
├── SQLite DB (10 tables)
├── HTTP API (port 8420)
├── BLE client (bleak)
├── On-device LLM (llama-server, overseer router)
├── Voice STT (Vosk)
├── Audio recorder
└── Display UI (PIL, 240x280 ST7789)
cortex-core/
├── src/ # Core Python source (runtime + plugin host)
│ ├── main.py # Entry point — state machine + main loop
│ ├── config.py # All configuration constants
│ ├── ble_client.py # BLE central client (bleak, background thread)
│ ├── cortex_protocol.py # CMD: protocol handler + chunk reassembly
│ ├── cortex_db.py # SQLite persistence (10 tables)
│ ├── http_server.py # HTTP API server (WiFi transport, port 8420)
│ ├── plugin_api.py # Plugin host: discovery, lifecycle, route registry
│ ├── display.py # 240x280 ST7789 display renderer (PIL)
│ ├── recorder.py # Audio recording via arecord
│ ├── stt.py # Speech-to-text engine (Vosk)
│ ├── button.py # Single-button input handler
│ ├── led.py # RGB LED status patterns
│ ├── logger.py # JSONL activity logger
│ └── power.py # WiFi power management
├── plugins/ # First-class plugins (auto-discovered, hot-loaded)
│ └── overseer/ # Memory-upkeep agent (Slice 3 + 4)
│ ├── __init__.py # Plugin lifecycle + ~30 HTTP routes
│ ├── overseer_db.py # Schema + CRUD (15+ tables)
│ ├── llm_router.py # OpenRouter / LM Studio routing
│ ├── core_memory_ro.py # Read-only access to cortex.db
│ ├── loop.py # Background tick loop (8 steps)
│ ├── claude_jsonl.py # Claude Code .jsonl parser + extended stats
│ ├── chat.py # Overseer chat (Opus 4.7 default)
│ ├── insight_scan.py # Pattern + drift detection
│ ├── distill_corrections.py # Slice 3i: corrections → blindspots
│ ├── blindspots.py # Meta-honesty layer (Slice 3f.5)
│ ├── dialectic.py # Paired Opus + Gemma generation (Slice 3f)
│ ├── journal.py # Append-only overseer reflection
│ ├── notifications.py # Bell rules engine
│ ├── automation_rollup.py # Per-project daily rollups (Slice 3e)
│ ├── question_routing.py # Route gists → open_questions
│ ├── project_summary.py # Slice 4 CP1a: per-project rollup data
│ ├── project_narrative.py # Slice 4 CP1b: Sonnet narrative generator
│ ├── pricing.py # Anthropic price table (as_of: 2026-05-02)
│ ├── temporal.py # Slice 5: local-TZ helpers, 22:00 trigger
│ ├── temporal_narrative.py # Slice 5: daily/weekly/monthly Sonnet rollups
│ ├── detail.py # Token-based drill-down (Slice 3g)
│ └── data/ # overseer.db
├── scripts/
│ ├── deploy.sh # scp + restart service on Pi
│ ├── setup_llama_server.sh # Build llama-server (-j1 for 2GB boards)
│ ├── setup_power.sh # Pi power optimization
│ └── backfill_session_stats.py # Slice 4: one-shot per-session enrichment
├── systemd/
│ └── cortex-core.service # systemd unit file
├── llama-server.service # systemd unit for LLM inference
├── docs/
│ ├── BLE_PROTOCOL.md
│ ├── 8BITDO_MICRO_SETUP.md
│ ├── UX_DESIGN_BRIEF.md
│ └── WM8960_ORANGE_PI_KNOWN_ISSUES.md
├── requirements.txt
└── README.md
The runtime hosts plugins under plugins/. Each plugin owns:
- A
plugin.tomlmanifest (name, version, dependencies, optional[llm]config) - A subclass of
plugin_api.Pluginwith anon_load(self)lifecycle hook - Its own SQLite at
plugin_data/(the runtime gives a typeddbhandle) - A
routeslist —Route("GET"|"POST", "/path", handler)mounted under/plugins/<name>/...
One ships in-tree today:
- overseer — Memory-upkeep agent. Reads notes / sessions / imported Claude Code conversations and produces interpretive layers via OpenRouter (Opus 4.7 + Sonnet 4.6 dialectic). Background loop with 9 tick steps as of v0.17 cycle (Slice 5 added the temporal cadence step). Slice 3 + Slice 4 + Slice 5 — full feature set documented in cortex-desktop/CLAUDE.md. Locked principle (Slice 5): the Overseer stays a quiet, lightweight memory layer. Captures, surfaces, connects. Not a journaling app or life coach.
The Tamagotchi pet was extracted to the cortex-pet sister repo (Slice 11). It can still load out-of-tree by cloning that repo into plugins/pet/; all pet code, assets, and docs live there now.
Edit locally, deploy to Pi:
# Edit code with Claude Code (or any editor) on your PC
# Then deploy to the Pi:
bash scripts/deploy.sh
# Deploy without restarting the service:
bash scripts/deploy.sh --no-restart# 1. Clone the repo on your PC
git clone https://github.com/turfptax/cortex-core.git
# 2. Install on Pi (creates dirs, copies code, installs + starts service)
bash scripts/deploy.sh --install
# 3. Install Python deps on Pi
ssh turfptax@10.0.0.25 "pip install -r ~/cortex-core/requirements.txt"# Live service logs
ssh turfptax@10.0.0.25 "sudo journalctl -u cortex-core -f"
# Recent errors only
ssh turfptax@10.0.0.25 "sudo journalctl -u cortex-core --no-pager -p err"The test suite runs without hardware dependencies:
ssh turfptax@10.0.0.25 "cd ~/cortex-core/src && python3 test_cortex.py"- Board: Orange Pi Zero 2W (Allwinner H616, 2GB RAM) with WhisPlay HAT (ST7789 display, WM8960 codec, gamepad)
- Gamepad: 8BitDo Micro — Bluetooth auto-pairing via evdev
- BLE: Connects to ESP32-S3 KeyMaster dongle as BLE central
- Storage: SQLite database at
~/cortex.db - Audio: 16kHz mono WAV recording via
arecord - STT: Vosk offline speech recognition (push-to-talk)
The HTTP server (http_server.py) runs on port 8420 alongside BLE — both transports are active simultaneously. Bearer token authentication is auto-generated and stored at ~/cortex-http.secret.
| Method | Path | Auth | Description |
|---|---|---|---|
| GET | /health |
No | Health check (for auto-detection) |
| POST | /api/cmd |
Yes | Execute any CMD: protocol command |
| GET | /files/<category> |
Yes | List files in category |
| GET | /files/<category>/<name> |
Yes | Download a file |
| POST | /files/uploads |
Yes | Upload file (raw body + X-Filename header) |
| DELETE | /files/<category>/<name> |
Yes | Delete file (recordings/uploads only) |
| GET | /files/db |
Yes | Download cortex.db snapshot |
File categories: recordings, notes, logs, uploads.
BLE auto-discovery: After connecting to the ESP32 over BLE, the Pi sends a DISCOVER: message containing its IP, HTTP port, and auth token to the computer. This enables automatic WiFi transport setup with no manual configuration.
A small LLM (Qwen3.5-0.8B base, fine-tuned bloom-21 in production) runs locally on the Pi via llama-server (HTTP backend, port 8081). The overseer's LLM router uses it as the on-device tier.
The model runs as a separate llama-server systemd service. Always build with -j1 on 2GB boards — using -j4 will OOM and crash the system.
# Full setup: build llama-server + download model + install service (~25 min)
# Run this ON the Pi (or via SSH):
bash scripts/setup_llama_server.sh
# Or step by step:
bash scripts/setup_llama_server.sh --build-only # build llama-server (~20-30 min)
bash scripts/setup_llama_server.sh --model-only # download Qwen3.5-0.8B (~533 MB)
bash scripts/setup_llama_server.sh --cleanup # remove build dir (~1.5 GB)The build uses static linking (-DBUILD_SHARED_LIBS=OFF) so there are no shared library dependencies. The resulting binary is fully self-contained.
-j1. The -j4 flag will exhaust RAM, trigger OOM, and freeze the board requiring a power cycle.
Execute shell commands on the Pi remotely via the protocol. Useful for deploying, debugging, and managing the Pi through the MCP bridge.
| Command | Payload | Response |
|---|---|---|
shell_exec |
{"command": "ls -la", "timeout": 30, "cwd": "/home/turfptax"} |
RSP:shell_exec:{"exit_code":0,"stdout":"...","stderr":"..."} |
- timeout: Max seconds (default 30, max 120)
- max_output: Max chars per stream (default 10000, max 50000)
- cwd: Working directory (defaults to home)
Cortex stores state in two SQLite files:
~/cortex.db(user-owned) — core memory the user authored: sessions, notes, activities, searches, projects, computers, people, files.~/cortex-core/plugins/overseer/data/overseer.db(root-owned, plugin-managed) — the overseer plugin's own working memory: imported AI conversation history (ChatGPT, Claude Code, Grok), gist + temporal summaries, the overseer's journal, dialectic questions, biometric streams, phone/voicemail logs.
A full walkthrough — including the source taxonomy (which source= values exist for ChatGPT vs Claude Code vs Twitter vs Grok), the per-conversation .jsonl import pattern, and a recipe for adding a new data source — lives in docs/DATA_ARCHITECTURE.md. Read that first if you're forking cortex-core to import your own data.
- sessions — AI conversation sessions (id, platform, hostname, started_at, summary)
- notes — Timestamped notes with tags, project, and type (note/decision/bug/reminder/idea/todo/context/tweet/tweet-reply/tweet-retweet/voice)
- activities — Program and file tracking (what was being worked on)
- searches — Research query history
- projects — Project registry (tag, status, priority, description)
- computers — Registered machines (hostname, OS, hardware)
- people — Collaborator directory
- files — File metadata for AI-discoverable sharing (filename, category, description, tags)
- time_entries, organizations, training_examples, training_ledger — supporting tables
- cortex — MCP server, CLI, and daemon (runs on PC)
- cortex-link — ESP32-S3 USB-BLE bridge firmware
- cortex-pet — extracted Tamagotchi pet plugin (loads out-of-tree under
plugins/pet/)
MIT