diff --git a/README.md b/README.md index ea9ccd2..0bf87f2 100644 --- a/README.md +++ b/README.md @@ -1,371 +1,118 @@ -# Politician MCP Server +# Polytician -A local AI agent that stores and converts concepts between three representations: **vectors**, **markdown**, and **ThoughtForm** (structured JSON). Built as a Model Context Protocol (MCP) server for seamless integration with Claude Desktop and other MCP-compatible clients. +[![CI](https://github.com/johnnyclem/polytician/actions/workflows/ci.yml/badge.svg)](https://github.com/johnnyclem/polytician/actions/workflows/ci.yml) +[![npm version](https://img.shields.io/npm/v/polytician.svg)](https://www.npmjs.com/package/polytician) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) +[![Node](https://img.shields.io/badge/node-%3E%3D20-brightgreen.svg)](#requirements) + +**Local-first semantic memory for AI agents.** Polytician is a [Model Context Protocol](https://modelcontextprotocol.io) server that gives Claude Desktop (or any MCP client) a persistent, searchable memory: every concept can be stored β€” and freely converted between β€” a **384-dim vector**, human-readable **markdown**, and structured **ThoughtForm** JSON. + +Everything runs on your machine. Embeddings are generated in-process; there's no external API call on the hot path unless you explicitly wire one in for LLM-assisted conversions. --- -## Key Features +## Why Polytician -- **Multi-Representation Storage**: Store concepts as 384-dimensional vectors, human-readable markdown, or structured ThoughtForm JSON -- **Bidirectional Conversion**: Convert between any two representations via the `convert_concept` tool -- **Semantic Search**: Find similar concepts using sqlite-vec (or pgvector) similarity search -- **In-Process Embeddings**: Text embedding via `@xenova/transformers` (all-MiniLM-L6-v2) β€” no external service required -- **Local-First**: All processing happens locally - no external API calls required -- **MCP Native**: Full integration with Anthropic's Model Context Protocol +- 🧠 **One concept, three shapes** β€” save once, read back as a vector, markdown, or structured JSON, and convert between them on demand via a single `convert_concept` tool +- πŸ” **Semantic search** β€” cosine similarity search over `sqlite-vec` (default) or Postgres/`pgvector`, scoped to a namespace by default with explicit opt-in for cross-namespace queries +- πŸ”’ **Local-first embeddings** β€” `@xenova/transformers` runs `all-MiniLM-L6-v2` in-process (384 dimensions); no network round-trip, no API key required +- πŸ—‚οΈ **Namespaces + optimistic concurrency** β€” isolate concepts per agent/tenant and guard concurrent writes with an `expectedVersion` check +- πŸ”Œ **Pluggable LLM + NLP** β€” bring your own provider (Anthropic, OpenAI, MCP sampling, or [AgentVault](#agentvault-integration)) for the conversions that need one (`markdownβ†’thoughtform`, `vectorβ†’markdown`, `vectorβ†’thoughtform`) +- 🧳 **Portable backups** β€” snapshot and restore your entire memory as a single signed JSON bundle (optionally AES-256-GCM encrypted), with an optional Arweave archival path via AgentVault +- πŸš€ **Deploys anywhere** β€” a single Node process (SQLite by default), with first-class Docker Compose and Kubernetes manifests for a distributed, Postgres-backed, multi-node setup --- ## Architecture ``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ MCP Server (TypeScript) β”‚ -β”‚ @modelcontextprotocol/sdk β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ Tools: save/read/delete/list/batch/search/convert/embed/stats β”‚ -β”‚ + agentvault_backup and optional vault_* tools β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ Embeddings: @xenova/transformers (all-MiniLM-L6-v2, 384-dim) β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ Storage: better-sqlite3 + sqlite-vec (WAL mode), or Postgres β”‚ -β”‚ + pgvector via POLYTICIAN_DB_BACKEND=postgres β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ HTTP (optional) - β–Ό -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Python Sidecar (Flask, optional helper) β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ β€’ FAISS index rebuild after bundle restore (/rebuild-index) β”‚ -β”‚ β€’ PolyVault bundle serialize/deserialize endpoints β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ MCP Server (TypeScript, stdio) β”‚ +β”‚ @modelcontextprotocol/sdk β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Tools: save/read/delete/list/batch/search/convert/embed/ β”‚ +β”‚ health_check/get_stats/agentvault_backup + vault_* (opt.) β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Embeddings: @xenova/transformers (all-MiniLM-L6-v2, 384-dim, β”‚ +β”‚ in-process β€” no external call) β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Storage: better-sqlite3 + sqlite-vec (WAL mode, default) β”‚ +β”‚ β€” or Postgres + pgvector via POLYTICIAN_DB_BACKEND β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ HTTP: GET /health on POLYTICIAN_HEALTH_PORT (default 8787) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ HTTP (optional, best-effort) + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Python Sidecar (Flask, optional helper) β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ β€’ FAISS index rebuild after a PolyVault bundle restore β”‚ +β”‚ β€’ PolyVault bundle serialize/deserialize endpoints β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` +The Node server is fully self-contained for everyday use β€” save, read, search, and non-LLM conversions all work with nothing but `npm start`. The Python sidecar is an optional helper for FAISS index rebuilds and PolyVault bundle operations; it is **not** required and is never auto-spawned by the server. If `POLYTICIAN_SIDECAR_URL` is unset, those code paths simply skip and log rather than fail. + --- ## Table of Contents -- [Quick Start](#quick-start) +- [Why Polytician](#why-polytician) +- [Architecture](#architecture) - [Requirements](#requirements) -- [Installation](#installation) -- [Build Guide](#build-guide) +- [Quick Start](#quick-start) +- [Claude Desktop Integration](#claude-desktop-integration) - [Configuration](#configuration) +- [Tools Reference](#tools-reference) +- [Concepts](#concepts) - [Deployment](#deployment) - - [Local Development](#local-development) - - [Claude Desktop Integration](#claude-desktop-integration) - - [Docker Deployment](#docker-deployment) - - [systemd Service](#systemd-service-linux) - - [PM2 Process Manager](#pm2-process-manager) - - [Cloud Deployment](#cloud-deployment) -- [Usage Manual](#usage-manual) - - [Core Concepts](#core-concepts) - - [Tutorial 1: Store Your First Concept](#tutorial-1-store-your-first-concept) - - [Tutorial 2: Convert Between Formats](#tutorial-2-convert-between-formats) - - [Tutorial 3: Semantic Search](#tutorial-3-semantic-search-with-vectors) - - [Tutorial 4: Knowledge Graph](#tutorial-4-building-a-knowledge-graph) -- [API Reference](#api-reference) -- [Troubleshooting](#troubleshooting) + - [Docker Compose](#docker-compose) + - [Kubernetes](#kubernetes) + - [systemd / PM2](#systemd--pm2) +- [Postgres / pgvector Backend](#postgres--pgvector-backend) +- [Backup, Restore & Encryption](#backup-restore--encryption) +- [AgentVault Integration](#agentvault-integration) +- [Health Checks & Troubleshooting](#health-checks--troubleshooting) - [Development](#development) - [Contributing](#contributing) - [License](#license) --- -## Quick Start - -Get up and running in 2 minutes: - -```bash -# Clone and install Node.js dependencies -git clone https://github.com/yourusername/politician.git -cd politician -npm install - -# Set up Python environment -cd python-sidecar -python3 -m venv venv -source venv/bin/activate # Windows: venv\Scripts\activate -pip install -r requirements.txt -python -m spacy download en_core_web_sm -cd .. - -# Build and run -npm run build -npm start -``` - -The server will start on stdio, ready for MCP client connections. - ---- - ## Requirements -### System Requirements - -| Component | Minimum | Recommended | -|-----------|---------|-------------| -| **OS** | macOS 12+, Ubuntu 20.04+, Windows 10+ (WSL2) | macOS 14+, Ubuntu 22.04+ | -| **RAM** | 4 GB | 8 GB+ | -| **Disk** | 2 GB free | 5 GB+ free | -| **CPU** | 2 cores | 4+ cores | - -### Software Requirements - -| Software | Version | Purpose | -|----------|---------|---------| -| **Node.js** | >= 20.0.0 | MCP server runtime | -| **Python** | >= 3.10 | ML sidecar service | -| **npm** | >= 10.0.0 | Package management | -| **pip** | >= 23.0 | Python package management | - -### Verify Requirements - -```bash -# Check Node.js version -node --version # Should be v20.x.x or higher - -# Check Python version -python3 --version # Should be 3.10.x or higher +| Software | Version | +|----------|---------| +| **Node.js** | >= 20.0.0 | +| **npm** | >= 10.0.0 | -# Check npm version -npm --version # Should be 10.x.x or higher -``` +Python 3.10+ is only needed if you run the optional [Python sidecar](#docker-compose) (FAISS rebuild / PolyVault bundle ops). It is not required for normal operation. --- -## Installation - -### Step 1: Clone the Repository - -```bash -git clone https://github.com/yourusername/politician.git -cd politician -``` - -### Step 2: Install Node.js Dependencies +## Quick Start ```bash +git clone https://github.com/johnnyclem/polytician.git +cd polytician npm install -``` - -This installs: -- `@modelcontextprotocol/sdk` - MCP server framework -- `better-sqlite3` - SQLite database driver -- `drizzle-orm` - Type-safe ORM -- `zod` - Schema validation -- `uuid` - UUID generation - -### Step 3: Set Up Python Environment - -Create and activate a virtual environment: - -```bash -cd python-sidecar - -# Create virtual environment -python3 -m venv venv - -# Activate it -# macOS/Linux: -source venv/bin/activate -# Windows (PowerShell): -.\venv\Scripts\Activate.ps1 -# Windows (CMD): -venv\Scripts\activate.bat -``` - -### Step 4: Install Python Dependencies - -```bash -pip install -r requirements.txt -``` - -This installs: -- `fastapi` + `uvicorn` - HTTP server -- `sentence-transformers` - Text embeddings (~400 MB model download) -- `spacy` - NER pipeline -- `faiss-cpu` - Vector similarity search -- `numpy` - Numerical operations - -### Step 5: Download spaCy Model - -```bash -python -m spacy download en_core_web_sm -``` - -This downloads the English NER model (~12 MB). - -### Step 6: Return to Project Root - -```bash -cd .. -``` - -### Step 7: Verify Installation - -```bash -# Type-check TypeScript -npm run typecheck - -# Build the project -npm run build - -# Verify Python sidecar can start -cd python-sidecar -python -c "from embeddings import get_model; print('Embeddings OK')" -python -c "from ner import get_nlp; print('NER OK')" -cd .. -``` - -If all commands succeed, the installation is complete. - ---- - -## Build Guide - -### Development Mode - -Run with hot-reload using `tsx`: - -```bash -npm run dev -``` - -This: -- Runs TypeScript directly without compilation -- Automatically restarts on file changes -- Provides better error messages with source maps - -### Production Build - -Compile TypeScript to JavaScript: - -```bash npm run build -``` - -Output is written to the `dist/` directory. - -Run the production build: - -```bash npm start ``` -### Type Checking - -Verify types without building: - -```bash -npm run typecheck -``` - -### Database Migrations - -If you modify `src/db/schema.ts`, generate migrations: - -```bash -npm run db:generate # Generate migration files -npm run db:push # Apply changes directly (dev only) -npm run db:studio # Open Drizzle Studio GUI -``` - -### Build Artifacts - -After building, you'll have: - -``` -dist/ -β”œβ”€β”€ index.js # Entry point -β”œβ”€β”€ server.js # MCP server setup -β”œβ”€β”€ db/ -β”‚ β”œβ”€β”€ schema.js # Database schema -β”‚ └── client.js # Database connection -β”œβ”€β”€ types/ -β”‚ β”œβ”€β”€ concept.js # Concept types -β”‚ └── thoughtform.js # ThoughtForm types -β”œβ”€β”€ services/ -β”‚ β”œβ”€β”€ python-bridge.js # Python HTTP client -β”‚ β”œβ”€β”€ concept.service.js # CRUD operations -β”‚ └── conversion.service.js # Conversions -└── commands/ - β”œβ”€β”€ save.js # Save handlers - β”œβ”€β”€ read.js # Read handlers - └── convert.js # Convert handlers -``` - ---- - -## Configuration - -### Environment Variables - -Create a `.env` file in the project root (copy from `.env.example`): - -```bash -cp .env.example .env -``` - -| Variable | Default | Description | -|----------|---------|-------------| -| `SIDECAR_HOST` | `127.0.0.1` | Python sidecar bind address | -| `SIDECAR_PORT` | `8787` | Python sidecar port | -| `DB_PATH` | `./data/concepts.db` | SQLite database path | -| `LOG_LEVEL` | `info` | Logging level (debug, info, warn, error) | -| `EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | Sentence transformer model | -| `SPACY_MODEL` | `en_core_web_sm` | spaCy NER model | - -### Database Settings - -The SQLite database is created automatically at `./data/concepts.db` with: - -- **WAL mode** enabled for better concurrency -- **Foreign keys** enabled for referential integrity -- Automatic indexes on `created_at` and `updated_at` +The server speaks MCP over **stdio** and starts an HTTP health endpoint on `:8787`. On first use it downloads the `all-MiniLM-L6-v2` embedding model (~30 MB) into `~/.polytician/models`; after that it runs fully offline. -To use a different location: +For local development with hot-reload: ```bash -export DB_PATH=/path/to/your/database.db +npm run dev # runs src/index.ts directly via tsx, restarts on change ``` -### Python Sidecar Settings - -The Python sidecar runs on `localhost:8787` by default. To change: - -1. Set `SIDECAR_PORT` environment variable -2. Update `python-sidecar/main.py` if running standalone - -### Model Configuration - -**Embedding Model** (sentence-transformers): -- Default: `all-MiniLM-L6-v2` (768 dimensions, ~80 MB) -- Alternative: `all-mpnet-base-v2` (768 dimensions, ~420 MB, higher quality) - -**NER Model** (spaCy): -- Default: `en_core_web_sm` (12 MB, fast) -- Alternative: `en_core_web_md` (40 MB, better accuracy) -- Alternative: `en_core_web_lg` (560 MB, best accuracy) - -To change models, update `python-sidecar/embeddings.py` and `python-sidecar/ner.py`. - --- -## Deployment - -### Local Development - -The simplest way to run for development: - -```bash -# Terminal 1: Run the MCP server -npm run dev -``` - -The Python sidecar is automatically spawned by the TypeScript server. +## Claude Desktop Integration -### Claude Desktop Integration - -Add to your Claude Desktop configuration file: +Add to your Claude Desktop config: **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` @@ -374,1225 +121,297 @@ Add to your Claude Desktop configuration file: ```json { "mcpServers": { - "politician": { + "polytician": { "command": "node", - "args": ["/absolute/path/to/politician/dist/index.js"], - "env": { - "NODE_ENV": "production" - } + "args": ["/absolute/path/to/polytician/dist/index.js"] } } } ``` -**Important**: Use the absolute path to `dist/index.js`. - -After saving, restart Claude Desktop. You should see "politician" in the MCP servers list. - -### Docker Deployment - -#### Using Docker Compose (Recommended) - -```bash -# Build and start -docker-compose up -d - -# View logs -docker-compose logs -f - -# Stop -docker-compose down -``` +Restart Claude Desktop and look for "polytician" in the MCP servers list. -#### Using Docker Directly +--- -```bash -# Build the image -docker build -t politician:latest . +## Configuration -# Run the container -docker run -d \ - --name politician \ - -v politician-data:/app/data \ - -p 8787:8787 \ - politician:latest +Polytician is configured entirely through environment variables (or a `.polytician.json` file in the project root or home directory β€” env vars win). Everything has a sensible default; you don't need to set anything to get started. -# View logs -docker logs -f politician +| Variable | Default | Description | +|----------|---------|--------------| +| `POLYTICIAN_DATA_DIR` | `~/.polytician` | Root directory for the SQLite DB and cached embedding model | +| `POLYTICIAN_HEALTH_PORT` | `8787` | Port for the `GET /health` HTTP endpoint | +| `POLYTICIAN_DB_BACKEND` | `sqlite` | `sqlite` or `postgres` β€” see [Postgres backend](#postgres--pgvector-backend) | +| `POLYTICIAN_POSTGRES_URL` | β€” | Connection string, required when `POLYTICIAN_DB_BACKEND=postgres` | +| `POLYTICIAN_EMBEDDING_MODEL` | `Xenova/all-MiniLM-L6-v2` | Any `@xenova/transformers`-compatible feature-extraction model | +| `POLYTICIAN_SIDECAR_URL` | β€” | Base URL of the [Python sidecar](#docker-compose), if running one | +| `POLYTICIAN_LLM_PROVIDER` | `none` | `anthropic`, `openai`, `sampling`, `agentvault`, or `none` β€” required for LLM-assisted conversions | +| `POLYTICIAN_LLM_MODEL` / `POLYTICIAN_LLM_API_KEY` | β€” | Provider-specific model name / key | +| `POLYTICIAN_NLP_PIPELINE` | `none` | `rule-based`, `llm`, or `none` β€” used by `markdownβ†’thoughtform` | +| `POLYTICIAN_NODE_ID` | random | Identifies this node in a distributed/multi-node deployment | +| `POLYTICIAN_ASYNC_INDEX_SYNC` | `false` | Enable background vector-index sync across nodes | +| `POLYTICIAN_ENCRYPT` | `false` | Encrypt PolyVault backup bundles (AES-256-GCM) β€” see [Backup, Restore & Encryption](#backup-restore--encryption) | +| `POLYTICIAN_BACKUP_THRESHOLD` | `50` | Auto-trigger a backup after this many saves (`0` disables) | +| `POLYTICIAN_AV_API_URL` / `POLYTICIAN_AV_API_TOKEN` | β€” | Enable the [AgentVault integration](#agentvault-integration) and its `vault_*` tools | -# Stop -docker stop politician && docker rm politician -``` +--- -#### Docker with Claude Desktop +## Tools Reference -For Claude Desktop integration with Docker, use a wrapper script: +All tools return `{ "content": [{ "type": "text", "text": "" }] }`; the examples below show the decoded JSON payload for brevity. Authoritative schemas live in `src/server.ts`. -```bash -#!/bin/bash -# save as: /usr/local/bin/politician-mcp -docker exec -i politician node /app/dist/index.js -``` +### `save_concept` -Then in `claude_desktop_config.json`: +Create or update a concept with one or more representations. Tags merge on update; namespace defaults to `"default"`. ```json +// Request { - "mcpServers": { - "politician": { - "command": "/usr/local/bin/politician-mcp" - } - } + "markdown": "Albert Einstein developed the theory of relativity.", + "tags": ["physics", "history"] } +// Response +{ "id": "...", "namespace": "default", "version": 1, "tags": ["physics", "history"], ... } ``` -### systemd Service (Linux) - -Create a service file at `/etc/systemd/system/politician.service`: - -```ini -[Unit] -Description=Politician MCP Server -After=network.target - -[Service] -Type=simple -User=politician -Group=politician -WorkingDirectory=/opt/politician -Environment=NODE_ENV=production -Environment=PATH=/opt/politician/python-sidecar/venv/bin:/usr/bin -ExecStart=/usr/bin/node /opt/politician/dist/index.js -Restart=always -RestartSec=10 - -# Logging -StandardOutput=journal -StandardError=journal -SyslogIdentifier=politician - -# Security -NoNewPrivileges=true -ProtectSystem=strict -ProtectHome=true -ReadWritePaths=/opt/politician/data - -[Install] -WantedBy=multi-user.target -``` - -Install and start: - -```bash -# Create service user -sudo useradd -r -s /bin/false politician +Pass `expectedVersion` to guard against concurrent writes β€” a mismatch throws `VERSION_CONFLICT` with the current version attached. -# Copy files -sudo cp -r . /opt/politician -sudo chown -R politician:politician /opt/politician +### `read_concept` -# Enable and start -sudo systemctl daemon-reload -sudo systemctl enable politician -sudo systemctl start politician +`{ "id": "...", "representations"?: ["vector"|"markdown"|"thoughtform"] }` β†’ the concept, optionally filtered to the requested representations. -# Check status -sudo systemctl status politician -sudo journalctl -u politician -f -``` +### `delete_concept` -### PM2 Process Manager - -Create `ecosystem.config.js`: - -```javascript -module.exports = { - apps: [{ - name: 'politician', - script: 'dist/index.js', - cwd: '/path/to/politician', - instances: 1, - autorestart: true, - watch: false, - max_memory_restart: '1G', - env: { - NODE_ENV: 'production', - SIDECAR_PORT: 8787 - }, - error_file: 'logs/err.log', - out_file: 'logs/out.log', - log_file: 'logs/combined.log', - time: true - }] -}; -``` +`{ "id": "..." }` β†’ `{ "deleted": "..." }` -Start with PM2: +### `list_concepts` -```bash -# Install PM2 globally -npm install -g pm2 +`{ "namespace"?, "limit"? (≀100, default 50), "offset"?, "tags"? }` β†’ paginated concepts scoped to the namespace. -# Start the application -pm2 start ecosystem.config.js +### `batch_save_concepts` -# Save process list -pm2 save +`{ "concepts": [{ "id"?, "markdown"?, "thoughtform"?, "embedding"?, "tags"? }, ...], "autoEmbed"?, "batchSize"? }` β†’ `{ "count", "ids": [...] }`. When `autoEmbed` is true, any entry with markdown but no embedding is embedded in batches of `batchSize` (default 50). -# Enable startup script -pm2 startup +### `search_concepts` -# View logs -pm2 logs politician +Semantic similarity search β€” provide `query` (auto-embedded) or a raw `vector`. -# Monitor -pm2 monit +```json +{ "query": "famous physicists", "k": 5, "namespace": "default" } ``` -### Cloud Deployment +Results are namespace-scoped by default; pass `crossNamespace: true` to search globally. -#### AWS EC2 / GCP Compute Engine +### `convert_concept` -1. Launch an instance (t3.medium / e2-medium minimum) -2. Install Node.js 20+ and Python 3.10+ -3. Clone the repository -4. Follow the [Installation](#installation) steps -5. Set up systemd or PM2 for process management -6. Configure security groups to block port 8787 from external access +`{ "id": "...", "from": "vector"|"markdown"|"thoughtform", "to": "vector"|"markdown"|"thoughtform" }` β†’ `{ "converted": { "from", "to" }, "concept": {...} }` -#### Docker on Cloud +| Conversion | Requires an LLM? | +|---|---| +| `thoughtform β†’ vector`, `thoughtform β†’ markdown`, `markdown β†’ vector` | No β€” deterministic | +| `markdown β†’ thoughtform`, `vector β†’ markdown`, `vector β†’ thoughtform` | Yes β€” set `POLYTICIAN_LLM_PROVIDER` (or `POLYTICIAN_NLP_PIPELINE=rule-based` for `markdown β†’ thoughtform`) | -For AWS ECS, GCP Cloud Run, or Azure Container Instances: +### `embed_text` -1. Push image to container registry: - ```bash - docker tag politician:latest gcr.io/your-project/politician:latest - docker push gcr.io/your-project/politician:latest - ``` +`{ "text": "..." }` β†’ `{ "dimension": 384, "embedding": [...] }`. Embeds arbitrary text without persisting a concept. -2. Deploy with persistent volume for `/app/data` +### `health_check` / `get_stats` -3. Ensure the container has at least 2 GB RAM for ML models +`{ "namespace"? }` β†’ server + embedding model + LLM provider status, and concept/representation counts for that namespace. -#### Important Security Notes +### `agentvault_backup` -- The Python sidecar should **never** be exposed to the public internet -- Always bind to `127.0.0.1` (localhost) only -- Use a reverse proxy (nginx, Caddy) if external access is needed -- The MCP server communicates via stdio, not network ports +`{ "namespace"? }` β†’ serializes every concept in the namespace into a signed JSON bundle: `{ "success", "conceptCount", "sizeBytes", "sha256", "namespace", "lastSynced", "createdAt" }`. See [Backup, Restore & Encryption](#backup-restore--encryption). ---- +### AgentVault-only tools (`vault_*`) -## Usage Manual +Registered only when `POLYTICIAN_AV_API_URL` / `POLYTICIAN_AV_API_TOKEN` are set: -### Core Concepts +| Tool | Purpose | +|---|---| +| `vault_infer` | Run a prompt through AgentVault's inference fallback chain (Bittensor β†’ Venice β†’ local), optionally saving the result as a concept | +| `vault_memory_push` | Push a concept's markdown/thoughtform to AgentVault's `memory_repo` canister | +| `vault_memory_pull` | Pull `concepts/*/markdown` entries from a `memory_repo` branch into local concepts | +| `vault_archive_concept` | Permanently archive a concept to Arweave, returning a transaction ID/URL | +| `vault_get_secret` | Fetch secret **metadata** (name, provider, rotation date, length) β€” never the raw value | +| `vault_memory_repo_log` | Inspect the `memory_repo` branch head and entry state | +| `vault_restore` | Restore concepts and rebuild the FAISS index from an inline bundle or file path | -#### The Three Representations +--- -Politician stores concepts in three interchangeable formats: +## Concepts -| Representation | Type | Best For | -|----------------|------|----------| -| **Vector** | `float[768]` | Semantic search, similarity matching | -| **Markdown** | `string` | Human-readable display, documentation | -| **ThoughtForm** | `JSON object` | Structured data, entity relationships | +A **concept** is the unit of memory. Any subset of its three representations can exist at once: -#### ThoughtForm Structure +| Representation | Type | Best for | +|---|---|---| +| **Vector** | `float[384]` | Semantic search, similarity matching | +| **Markdown** | `string` | Human-readable display | +| **ThoughtForm** | JSON object | Structured entities, relationships, a context graph | ```json { "id": "550e8400-e29b-41d4-a716-446655440000", - "rawText": "Original text content", - "language": "en", - "metadata": { - "timestamp": "2024-01-15T10:30:00Z", - "author": "user", - "tags": ["example", "demo"], - "source": "user_input" - }, - "entities": [ - { - "id": "ent_0", - "text": "John Smith", - "type": "PERSON", - "confidence": 1.0, - "offset": {"start": 0, "end": 10} - } - ], - "relationships": [ - { - "subjectId": "ent_0", - "predicate": "works_at", - "objectId": "ent_1" - } - ], - "contextGraph": { - "ent_0": ["ent_1"], - "ent_1": ["ent_0"] + "namespace": "default", + "version": 3, + "tags": ["physics", "history"], + "markdown": "Albert Einstein developed the theory of relativity...", + "thoughtform": { + "rawText": "Albert Einstein developed the theory of relativity...", + "entities": [{ "id": "ent_0", "text": "Albert Einstein", "type": "PERSON" }], + "relationships": [], + "contextGraph": {} }, - "embeddings": [0.123, -0.456, ...] + "embedding": [0.0234, -0.0891, "...384 floats total"] } ``` -#### Conversion Matrix - -``` - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ ThoughtForm β”‚ - β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ β”‚ β”‚ - β–Ό β”‚ β–Ό - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ Vector │◄───┴───│ Markdown β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -All 6 conversions are supported: -- `thoughtForm β†’ vector`: Embed raw_text -- `thoughtForm β†’ markdown`: Format as readable text -- `vector β†’ thoughtForm`: Find similar concepts, reconstruct -- `vector β†’ markdown`: Find similar concepts, summarize -- `markdown β†’ vector`: Embed markdown text -- `markdown β†’ thoughtForm`: Extract entities via NER +`convert_concept` moves between these on demand β€” deterministically for the vector/markdown/thoughtform-derived paths, and via your configured LLM or NLP pipeline for the paths that need to *generate* rather than *derive* content. --- -### Tutorial 1: Store Your First Concept +## Deployment -Let's store a simple concept and explore its representations. +### Docker Compose -#### Step 1: Generate a Concept ID +```bash +docker-compose up -d +``` -Call the `generate_id` tool: +`docker-compose.yml` provisions a full distributed reference setup: a `pgvector/pgvector` Postgres instance, two Polytician nodes (`polytician-1` / `polytician-2`, each with a distinct `POLYTICIAN_NODE_ID`) sharing that database, and a `sidecar` service built from `python-sidecar/` for FAISS rebuilds. For a single-node SQLite setup, run just the `polytician-1` service with `POLYTICIAN_DB_BACKEND=sqlite`. -```json -// Request -{} +### Kubernetes -// Response -{ - "success": true, - "data": { - "id": "550e8400-e29b-41d4-a716-446655440000" - } -} -``` +Manifests live in [`k8s/`](k8s/): -#### Step 2: Save as ThoughtForm +```bash +kubectl apply -f k8s/namespace.yml +kubectl apply -f k8s/postgres.yml +kubectl apply -f k8s/polytician.yml +kubectl apply -f k8s/sidecar.yml +``` -Call `save_concept_as_thoughtForm`: +`k8s/polytician.yml` deploys 3 replicas behind a `ClusterIP` service on port 8787 β€” pair with `POLYTICIAN_DB_BACKEND=postgres` and `POLYTICIAN_ASYNC_INDEX_SYNC=true` for a real multi-replica deployment. -```json -// Request -{ - "id": "550e8400-e29b-41d4-a716-446655440000", - "thoughtForm": { - "rawText": "Albert Einstein developed the theory of relativity while working at the Swiss Patent Office in Bern. His famous equation E=mcΒ² revolutionized physics.", - "language": "en" - }, - "tags": ["physics", "history", "science"] -} +### systemd / PM2 -// Response -{ - "success": true, - "data": { - "id": "550e8400-e29b-41d4-a716-446655440000" - } -} -``` +The build is a single `node dist/index.js` process reading stdio, so any standard Node process manager works β€” see `Dockerfile` for the exact runtime invocation to mirror in a unit file or `ecosystem.config.js`. -#### Step 3: Read It Back +--- -Call `read_concept_from_thoughtForm`: +## Postgres / pgvector Backend -```json -// Request -{ - "id": "550e8400-e29b-41d4-a716-446655440000" -} +Set `POLYTICIAN_DB_BACKEND=postgres` and `POLYTICIAN_POSTGRES_URL` to point at a Postgres instance with the `vector` extension available (the `pgvector/pgvector` Docker image is the easiest path). The adapter (`src/db/postgres-adapter.ts`) runs `CREATE EXTENSION IF NOT EXISTS vector` and creates its tables on startup β€” there's no separate migration step to run first. This backend is what backs the multi-node / distributed deployments in `docker-compose.yml` and `k8s/`. -// Response -{ - "success": true, - "data": { - "thoughtForm": { - "id": "550e8400-e29b-41d4-a716-446655440000", - "rawText": "Albert Einstein developed the theory of relativity...", - "language": "en", - "metadata": { - "timestamp": "2024-01-15T10:30:00Z", - "author": null, - "tags": ["physics", "history", "science"], - "source": "user_input" - }, - "entities": [], - "relationships": [], - "contextGraph": {} - } - } -} -``` +--- -#### Step 4: Check Available Representations +## Backup, Restore & Encryption -Call `get_representations`: +`agentvault_backup` (and the `agentvault-sync` CLI, see below) serialize a namespace's concepts into a single JSON bundle with a `sha256` integrity hash. Set `POLYTICIAN_ENCRYPT=true` (or pass `--encrypt`) to encrypt the bundle with AES-256-GCM behind a VetKeys-shaped crypto interface (`src/polyvault/crypto.ts`) β€” today that's a local AES-GCM adapter, laid out so a future IC/VetKeys threshold-key backend can drop in without changing the bundle format. -```json -// Request -{ - "id": "550e8400-e29b-41d4-a716-446655440000" -} +A standalone CLI is also available for scripted backup/restore/sync outside of an MCP client: -// Response -{ - "success": true, - "data": { - "vectors": false, - "md": false, - "thoughtForm": true - } -} +```bash +npx tsx bin/agentvault-sync.ts backup --out backup.json --namespace default +npx tsx bin/agentvault-sync.ts restore --file backup.json +npx tsx bin/agentvault-sync.ts sync --direction bidirectional --namespace default ``` -Only ThoughtForm is stored so far. Let's convert to other formats! +See [`docs/polyvault/spec-v1.md`](docs/polyvault/spec-v1.md) for the bundle format and [`docs/polyvault/runbook.md`](docs/polyvault/runbook.md) for operational guidance. --- -### Tutorial 2: Convert Between Formats +## AgentVault Integration -Starting from the concept created in Tutorial 1, let's generate all representations. +Polytician doubles as a semantic-memory source, on-chain backup target, and inference/secrets provider for [AgentVault](https://github.com/johnnyclem/agentvault)'s orchestrator, via the `vault_*` tools and `agentvault_backup` above. See: -#### Step 1: Convert ThoughtForm to Vector +- [`AGENTVAULT_COMPATIBILITY_PRD.md`](AGENTVAULT_COMPATIBILITY_PRD.md) β€” the spec for AgentVault's side of this integration +- [`docs/polyvault/spec-v1.md`](docs/polyvault/spec-v1.md) β€” the encrypted backup/restore bridge (PolyVault) +- [`docs/ecosystem/executive-summary.md`](docs/ecosystem/executive-summary.md) and [`docs/ecosystem/engineering-guide.md`](docs/ecosystem/engineering-guide.md) β€” cross-repo ecosystem evaluation, including what's actually shipped vs. still aspirational on AgentVault's side -Call `convert_concept_from_thoughtForm_to_vectors`: +--- -```json -// Request -{ - "id": "550e8400-e29b-41d4-a716-446655440000" -} +## Health Checks & Troubleshooting -// Response -{ - "success": true, - "data": { - "vector": [0.0234, -0.0891, 0.0456, ...] // 768 floats - } -} +```bash +curl http://localhost:8787/health ``` -This: -1. Reads the ThoughtForm's `rawText` -2. Sends it to the Python sidecar for embedding -3. Stores the vector in the database -4. Updates the FAISS index for similarity search - -#### Step 2: Convert ThoughtForm to Markdown - -Call `convert_concept_from_thoughtForm_to_md`: - ```json -// Request -{ - "id": "550e8400-e29b-41d4-a716-446655440000" -} - -// Response { - "success": true, - "data": { - "md": "# Albert Einstein developed the...\n\n## Metadata\n\n- **Language:** en\n- **Created:** 2024-01-15T10:30:00Z\n- **Tags:** physics, history, science\n\n## Content\n\nAlbert Einstein developed the theory of relativity..." - } + "status": "ok", + "checks": { + "database": { "status": "ok" }, + "vector_index": { "status": "ok" }, + "sidecar": { "status": "not_configured" } + }, + "timestamp": "2026-01-01T00:00:00.000Z" } ``` -#### Step 3: Verify All Representations +`sidecar: "not_configured"` is expected and harmless if you haven't set `POLYTICIAN_SIDECAR_URL` β€” it only affects FAISS rebuilds and PolyVault restore. -Call `get_representations`: +**Common issues:** -```json -// Response -{ - "success": true, - "data": { - "vectors": true, - "md": true, - "thoughtForm": true - } -} -``` +| Symptom | Fix | +|---|---| +| Slow first request | The embedding model is downloading (~30 MB) into `POLYTICIAN_DATA_DIR/models`; subsequent runs are instant | +| `Embedding dimension mismatch` | You've pointed `POLYTICIAN_EMBEDDING_MODEL` at a model that doesn't output 384-dim vectors | +| `markdown β†’ thoughtform` / `vector β†’ *` conversions fail | Set `POLYTICIAN_LLM_PROVIDER` (or `POLYTICIAN_NLP_PIPELINE=rule-based`) | +| Postgres backend won't start | Confirm `POLYTICIAN_POSTGRES_URL` is reachable and the role can `CREATE EXTENSION vector` | +| `VERSION_CONFLICT` on save | Another writer updated the concept first β€” re-read it and retry with the new version | -Now all three formats are stored! +Set `LOG_LEVEL=debug` for verbose logging. -#### Step 4: Read Each Format - -You can read any representation: +--- -```json -// read_concept_from_vectors -{"id": "550e8400-..."} -// Returns: {"vector": [0.0234, ...]} +## Development -// read_concept_from_md -{"id": "550e8400-..."} -// Returns: {"md": "# Albert Einstein..."} +``` +polytician/ +β”œβ”€β”€ src/ +β”‚ β”œβ”€β”€ index.ts # Entry point (stdio MCP server + HTTP health server) +β”‚ β”œβ”€β”€ server.ts # Tool registration +β”‚ β”œβ”€β”€ config.ts # Env-var / .polytician.json configuration +β”‚ β”œβ”€β”€ db/ # SQLite + Postgres adapters +β”‚ β”œβ”€β”€ services/ # concept, conversion, embedding, backup, index-sync +β”‚ β”œβ”€β”€ polyvault/ & lib/polyvault/ # Encrypted backup bundle format + FAISS client +β”‚ β”œβ”€β”€ integrations/agent-vault/ # AgentVault config, providers, vault_* tools +β”‚ └── sidecar/ # HTTP client for the optional Python sidecar +β”œβ”€β”€ python-sidecar/ # Optional Flask helper: FAISS rebuild, PolyVault bundles +β”œβ”€β”€ bin/agentvault-sync.ts # Standalone backup/restore/sync CLI +β”œβ”€β”€ tests/ # vitest suite (~35 files: tools, storage, polyvault, concurrency) +β”œβ”€β”€ docker-compose.yml, k8s/ # Distributed, Postgres-backed reference deployment +└── docs/ # PolyVault spec/runbook, ecosystem evaluation +``` -// read_concept_from_thoughtForm -{"id": "550e8400-..."} -// Returns: {"thoughtForm": {...}} +```bash +npm run dev # hot-reload dev server (tsx) +npm run build # compile to dist/ +npm run typecheck # tsc --noEmit +npm test # vitest run β€” tools, storage, concurrency, PolyVault, encryption +npm run test:watch +npm run quality # lint + typecheck + format:check +npm run quality:fix # lint:fix + format ``` ---- +### Adding a New Tool -### Tutorial 3: Semantic Search with Vectors +Register it in `src/server.ts` with `server.tool(name, description, zodInputShape, handler)` β€” see any existing tool for the pattern of validating input, calling into a service in `src/services/`, and returning `jsonResult(...)` / `errorResult(...)`. -Let's explore how vector similarity search works. +--- -#### Step 1: Store Multiple Related Concepts +## Contributing -Store several physics-related concepts: +1. Fork the repository and create a feature branch +2. Make your changes, adding or updating tests in `tests/` +3. Run `npm run quality && npm test` before pushing +4. Open a pull request describing the change and its motivation -```json -// Concept 1: Einstein -{ - "id": "aaaa-...", - "thoughtForm": {"rawText": "Albert Einstein developed the theory of relativity..."} -} +Please include Node.js version, OS, and reproduction steps when reporting issues. -// Concept 2: Newton -{ - "id": "bbbb-...", - "thoughtForm": {"rawText": "Isaac Newton formulated the laws of motion and universal gravitation..."} -} - -// Concept 3: Curie -{ - "id": "cccc-...", - "thoughtForm": {"rawText": "Marie Curie pioneered research on radioactivity and won two Nobel Prizes..."} -} - -// Concept 4: Shakespeare (unrelated) -{ - "id": "dddd-...", - "thoughtForm": {"rawText": "William Shakespeare wrote plays like Hamlet and Romeo and Juliet..."} -} -``` - -#### Step 2: Convert All to Vectors - -For each concept, call `convert_concept_from_thoughtForm_to_vectors`. - -#### Step 3: Store a Query Vector Only - -Create a new concept with just a vector (no text): - -```json -// save_concept_as_vectors -{ - "id": "query-...", - "vector": [/* 768 floats representing "famous physicists" */] -} -``` - -#### Step 4: Convert Vector to Markdown - -Call `convert_concept_from_vectors_to_md`: - -```json -// Request -{ - "id": "query-..." -} - -// Response -{ - "success": true, - "data": { - "md": "## Summary\n\nBased on 3 related concepts:\n\n- Concept `aaaa-...`\n- Concept `bbbb-...`\n- Concept `cccc-...`\n\n..." - } -} -``` - -The system found Einstein, Newton, and Curie as the most similar concepts (Shakespeare was excluded as unrelated). - ---- - -### Tutorial 4: Building a Knowledge Graph - -Extract entities and relationships to build a knowledge graph. - -#### Step 1: Store a Rich Text Concept - -```json -// save_concept_as_thoughtForm -{ - "id": "graph-1", - "thoughtForm": { - "rawText": "Elon Musk founded SpaceX in 2002 and Tesla in 2003. SpaceX is headquartered in Hawthorne, California. Tesla's main factory is in Fremont, California." - } -} -``` - -#### Step 2: Convert Markdown to ThoughtForm (NER Extraction) - -First, save as markdown, then convert to extract entities: - -```json -// save_concept_as_md -{ - "id": "graph-2", - "md": "Apple Inc. was founded by Steve Jobs and Steve Wozniak in Cupertino. Tim Cook is the current CEO." -} - -// convert_concept_from_md_to_thoughtForm -{ - "id": "graph-2" -} - -// Response -{ - "success": true, - "data": { - "thoughtForm": { - "id": "graph-2", - "rawText": "Apple Inc. was founded by Steve Jobs...", - "entities": [ - {"id": "ent_0", "text": "Apple Inc.", "type": "ORG", "confidence": 1.0, ...}, - {"id": "ent_1", "text": "Steve Jobs", "type": "PERSON", "confidence": 1.0, ...}, - {"id": "ent_2", "text": "Steve Wozniak", "type": "PERSON", "confidence": 1.0, ...}, - {"id": "ent_3", "text": "Cupertino", "type": "GPE", "confidence": 1.0, ...}, - {"id": "ent_4", "text": "Tim Cook", "type": "PERSON", "confidence": 1.0, ...} - ], - "relationships": [ - {"subjectId": "ent_1", "predicate": "founded", "objectId": "ent_0"}, - {"subjectId": "ent_2", "predicate": "founded", "objectId": "ent_0"} - ], - "contextGraph": { - "ent_0": ["ent_1", "ent_2", "ent_3", "ent_4"], - "ent_1": ["ent_0"], - "ent_2": ["ent_0"], - "ent_3": ["ent_0"], - "ent_4": ["ent_0"] - } - } - } -} -``` - -#### Step 3: Query the Graph - -The `contextGraph` is an adjacency list. You can: -- Find all entities connected to "Apple Inc." (`ent_0`) -- Trace relationships between people and organizations -- Build visualizations using the graph structure - ---- - -## API Reference - -> **Note:** The tool-by-tool reference below describes an earlier API surface -> and is pending a rewrite. The tools actually registered by the current -> server are: `save_concept`, `read_concept`, `delete_concept`, -> `list_concepts`, `batch_save_concepts`, `search_concepts`, -> `convert_concept`, `embed_text`, `health_check`, `get_stats`, -> `agentvault_backup`, and (when AgentVault is configured) `vault_infer`, -> `vault_memory_push`, `vault_memory_pull`, `vault_archive_concept`, -> `vault_get_secret`, `vault_memory_repo_log`, `vault_restore`. See -> `src/server.ts` for authoritative schemas. - -### Save Commands - -#### `save_concept_as_vectors` - -Save a 768-dimensional vector representation. - -**Input:** -```json -{ - "id": "string (UUID, required)", - "vector": "number[] (768 floats, required)", - "tags": "string[] (optional)" -} -``` - -**Output:** -```json -{ - "success": true, - "data": {"id": "..."} -} -``` - ---- - -#### `save_concept_as_md` - -Save a markdown representation. - -**Input:** -```json -{ - "id": "string (UUID, required)", - "md": "string (non-empty, required)", - "tags": "string[] (optional)" -} -``` - -**Output:** -```json -{ - "success": true, - "data": {"id": "..."} -} -``` - ---- - -#### `save_concept_as_thoughtForm` - -Save a ThoughtForm (structured JSON) representation. - -**Input:** -```json -{ - "id": "string (UUID, required)", - "thoughtForm": { - "rawText": "string (required)", - "language": "string (default: 'en')", - "metadata": { - "timestamp": "ISO 8601 datetime", - "author": "string | null", - "tags": "string[]", - "source": "string" - }, - "entities": "Entity[]", - "relationships": "Relationship[]", - "contextGraph": "Record", - "embeddings": "number[]" - }, - "tags": "string[] (optional)" -} -``` - -**Output:** -```json -{ - "success": true, - "data": {"id": "..."} -} -``` - ---- - -### Read Commands - -#### `read_concept_from_vectors` - -Read the vector representation. - -**Input:** -```json -{ - "id": "string (UUID, required)" -} -``` - -**Output:** -```json -{ - "success": true, - "data": { - "vector": [0.123, -0.456, ...] - } -} -``` - ---- - -#### `read_concept_from_md` - -Read the markdown representation. - -**Input:** -```json -{ - "id": "string (UUID, required)" -} -``` - -**Output:** -```json -{ - "success": true, - "data": { - "md": "# Title\n\nContent..." - } -} -``` - ---- - -#### `read_concept_from_thoughtForm` - -Read the ThoughtForm representation. - -**Input:** -```json -{ - "id": "string (UUID, required)" -} -``` - -**Output:** -```json -{ - "success": true, - "data": { - "thoughtForm": { - "id": "...", - "rawText": "...", - "entities": [...], - ... - } - } -} -``` - ---- - -### Convert Commands - -#### `convert_concept_from_thoughtForm_to_vectors` - -Embed ThoughtForm's raw_text into a 768-dim vector. - -**Input:** `{"id": "UUID"}` -**Output:** `{"vector": [...]}` - ---- - -#### `convert_concept_from_thoughtForm_to_md` - -Format ThoughtForm as human-readable markdown. - -**Input:** `{"id": "UUID"}` -**Output:** `{"md": "..."}` - ---- - -#### `convert_concept_from_vectors_to_thoughtForm` - -Find similar concepts via FAISS and reconstruct a ThoughtForm. - -**Input:** `{"id": "UUID"}` -**Output:** `{"thoughtForm": {...}}` - ---- - -#### `convert_concept_from_vectors_to_md` - -Find similar concepts and generate a summary. - -**Input:** `{"id": "UUID"}` -**Output:** `{"md": "..."}` - ---- - -#### `convert_concept_from_md_to_vectors` - -Embed markdown text into a 768-dim vector. - -**Input:** `{"id": "UUID"}` -**Output:** `{"vector": [...]}` - ---- - -#### `convert_concept_from_md_to_thoughtForm` - -Parse markdown and extract entities via NER. - -**Input:** `{"id": "UUID"}` -**Output:** `{"thoughtForm": {...}}` - ---- - -### Utility Commands - -#### `list_concepts` - -List all stored concepts with their available representations. - -**Input:** `{}` - -**Output:** -```json -{ - "success": true, - "data": { - "concepts": [ - { - "id": "...", - "createdAt": "2024-01-15T10:30:00Z", - "updatedAt": "2024-01-15T10:30:00Z", - "tags": ["..."], - "representations": { - "vectors": true, - "md": false, - "thoughtForm": true - } - } - ], - "count": 1 - } -} -``` - ---- - -#### `get_representations` - -Check which representations exist for a concept. - -**Input:** `{"id": "UUID"}` - -**Output:** -```json -{ - "success": true, - "data": { - "vectors": true, - "md": false, - "thoughtForm": true - } -} -``` - ---- - -#### `delete_concept` - -Delete a concept and all its representations. - -**Input:** `{"id": "UUID"}` - -**Output:** -```json -{ - "success": true, - "data": {"deleted": "..."} -} -``` - ---- - -#### `generate_id` - -Generate a new UUID for a concept. - -**Input:** `{}` - -**Output:** -```json -{ - "success": true, - "data": {"id": "550e8400-e29b-41d4-a716-446655440000"} -} -``` - ---- - -#### `health_check` - -Check the health of the server and Python sidecar. - -**Input:** `{}` - -**Output:** -```json -{ - "success": true, - "data": { - "server": "ok", - "python_sidecar": { - "status": "ok", - "services": { - "embeddings": true, - "ner": true, - "vector_index": true - }, - "index_stats": { - "total_vectors": 42, - "dimension": 768 - } - } - } -} -``` - ---- - -## Troubleshooting - -### Common Issues - -#### "Python sidecar failed to start" - -**Symptoms:** Server hangs or errors on startup. - -**Solutions:** -1. Check Python is installed: `python3 --version` -2. Verify virtual environment is activated -3. Check port 8787 is free: `lsof -i :8787` -4. Check Python dependencies: `pip list | grep fastapi` - -```bash -# Manual test -cd python-sidecar -source venv/bin/activate -python main.py -# Should output: "Uvicorn running on http://127.0.0.1:8787" -``` - ---- - -#### "Vector dimension mismatch" - -**Symptoms:** Error when saving vectors. - -**Solutions:** -- Ensure vectors have exactly 768 dimensions -- Check you're using `all-MiniLM-L6-v2` model -- Verify: `len(vector) == 768` - ---- - -#### "spaCy model not found" - -**Symptoms:** NER operations fail. - -**Solutions:** -```bash -cd python-sidecar -source venv/bin/activate -python -m spacy download en_core_web_sm -``` - ---- - -#### "FAISS index corrupted" - -**Symptoms:** Search operations fail or return wrong results. - -**Solutions:** -```bash -# Delete and regenerate the index -rm data/faiss.index data/id_map.json -# Restart the server - index will rebuild from database -``` - ---- - -#### "Database locked" - -**Symptoms:** Operations timeout or fail with lock errors. - -**Solutions:** -1. Ensure only one server instance is running -2. Check for zombie processes: `ps aux | grep politician` -3. Delete WAL files if corrupted: - ```bash - rm data/concepts.db-wal data/concepts.db-shm - ``` - ---- - -### Debug Mode - -Enable verbose logging: - -```bash -export LOG_LEVEL=debug -npm run dev -``` - ---- - -### Health Check - -Test the Python sidecar directly: - -```bash -curl http://localhost:8787/health -``` - -Expected response: -```json -{ - "status": "ok", - "services": { - "embeddings": true, - "ner": true, - "vector_index": true - }, - "index_stats": { - "total_vectors": 0, - "dimension": 768, - "id_count": 0 - } -} -``` - ---- - -## Development - -### Project Structure - -``` -politician/ -β”œβ”€β”€ src/ # TypeScript source -β”‚ β”œβ”€β”€ index.ts # Entry point -β”‚ β”œβ”€β”€ server.ts # MCP server setup -β”‚ β”œβ”€β”€ db/ -β”‚ β”‚ β”œβ”€β”€ schema.ts # Drizzle ORM schema -β”‚ β”‚ └── client.ts # Database connection -β”‚ β”œβ”€β”€ types/ -β”‚ β”‚ β”œβ”€β”€ concept.ts # Concept types & Zod schemas -β”‚ β”‚ └── thoughtform.ts # ThoughtForm types & schemas -β”‚ β”œβ”€β”€ services/ -β”‚ β”‚ β”œβ”€β”€ python-bridge.ts # HTTP client for sidecar -β”‚ β”‚ β”œβ”€β”€ concept.service.ts # CRUD operations -β”‚ β”‚ └── conversion.service.ts # Conversion logic -β”‚ └── commands/ -β”‚ β”œβ”€β”€ save.ts # Save command handlers -β”‚ β”œβ”€β”€ read.ts # Read command handlers -β”‚ └── convert.ts # Convert command handlers -β”œβ”€β”€ python-sidecar/ # Python ML services -β”‚ β”œβ”€β”€ main.py # FastAPI server -β”‚ β”œβ”€β”€ embeddings.py # sentence-transformers -β”‚ β”œβ”€β”€ ner.py # spaCy NER -β”‚ β”œβ”€β”€ vector_index.py # FAISS index -β”‚ └── requirements.txt # Python dependencies -β”œβ”€β”€ data/ # Runtime data (gitignored) -β”‚ β”œβ”€β”€ concepts.db # SQLite database -β”‚ β”œβ”€β”€ faiss.index # FAISS vector index -β”‚ └── id_map.json # FAISS ID mapping -β”œβ”€β”€ dist/ # Compiled JavaScript (gitignored) -β”œβ”€β”€ package.json -β”œβ”€β”€ tsconfig.json -└── drizzle.config.ts -``` - -### Adding a New Tool - -1. Define the handler in `src/commands/`: - -```typescript -export async function myNewTool(input: unknown): Promise> { - // Validate input with Zod - // Perform operation - // Return result -} - -export const myCommands = { - my_new_tool: { - description: "Description for Claude", - inputSchema: { /* JSON Schema */ }, - handler: myNewTool, - }, -}; -``` - -2. Register in `src/server.ts`: - -```typescript -import { myCommands } from "./commands/my-commands.js"; - -const commands = { ...allCommands, ...myCommands }; -``` - -3. Rebuild: `npm run build` - -### Running Tests - -```bash -# Type checking -npm run typecheck - -# Full test suite (vitest) -npm test - -# Watch mode -npm run test:watch - -# Lint + typecheck + formatting -npm run quality -``` - ---- - -## Ecosystem - -Polytician integrates with [AgentVault](https://github.com/johnnyclem/agentvault) as an -on-chain backup/sync target, inference/secrets provider, and MCP-based semantic memory -source for AgentVault's orchestrator. See: - -- [`AGENTVAULT_COMPATIBILITY_PRD.md`](AGENTVAULT_COMPATIBILITY_PRD.md) β€” the spec for AgentVault's side of this integration -- [`docs/polyvault/spec-v1.md`](docs/polyvault/spec-v1.md) β€” the encrypted backup/restore bridge (PolyVault) -- [`docs/ecosystem/executive-summary.md`](docs/ecosystem/executive-summary.md) and [`docs/ecosystem/engineering-guide.md`](docs/ecosystem/engineering-guide.md) β€” cross-repo ecosystem evaluation, including what's actually shipped vs. still aspirational on AgentVault's side - ---- - -## Contributing - -Contributions are welcome! Please follow these steps: - -1. Fork the repository -2. Create a feature branch: `git checkout -b feature/my-feature` -3. Make your changes -4. Run type checking: `npm run typecheck` -5. Commit with a descriptive message -6. Push to your fork: `git push origin feature/my-feature` -7. Open a Pull Request - -### Code Style - -- TypeScript: Use strict mode, prefer explicit types -- Python: Follow PEP 8, use type hints -- Commits: Use conventional commit messages - -### Reporting Issues - -Please include: -- Node.js and Python versions -- Operating system -- Error messages and logs -- Steps to reproduce - ---- +--- ## License -MIT License - -Copyright (c) 2024 - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. +MIT β€” see [LICENSE](LICENSE).