Cortex Local Agent

Cortex is a Windows-native local assistant built on the Matbot runtime. It combines local chat, provider selection, workspace isolation, durable memory, controlled file access, markdown hybrid retriever with exact and lexical reference lanes.

It also provides a tool-based expert panel for orchestrating multi-expertese work.

Matbot remains the underlying runtime and plugin system. Cortex is the product shell configured in this repository: the WebUI branding, PowerShell launch scripts, workspace registry, local services, and default plugin set.

PROJECTMEM is intentionally not integrated.

Conceptual Overview

Cortex is a local-agent workbench. It is meant to feel like one assistant in the browser, but internally it is a composed runtime:

• a WebUI for conversations, files, providers, experts, workspaces, and settings;
• a Matbot process that loads providers, plugins, tools, stores, sessions, and hooks from the active workspace configuration;
• local services that expose safe file indexing, safe file access, and Mem0;
• per-workspace persistent data, so different projects can carry different providers, plugins, memories, skills, files, and RAG indexes;
• retrieval layers that can bring back remembered facts, skills, indexed files, workspace markdown, and expert-specific files;
• an expert-panel tool that runs selected domain experts independently and can synthesize their opinions.

The important idea is that Cortex is not one hard-coded chatbot. It is a runtime whose behavior is assembled from configuration and plugins. The WebUI is just the visible control surface over that runtime.

The default installation gives one Default workspace. That workspace is the current repository setup: its matbot.yaml defines the OpenAI-compatible providers, the active plugins, the Cortex WebUI, workspace RAG, memory, skills, the expert panel, and file-management tools. New workspaces copy this shape and then diverge independently.

Documentation

This README covers product and repository orientation. Start with the user guide for installation and feature walkthroughs, then use the reference documents for configuration, architecture, and development details.

Document	Contents
Cortex User Guide	Installation, WebUI tour, conversations, workspaces, files, RAG, memory, skills, experts, sources, SQL, workflows, graph, reviews, plugins, scheduling, safety, and troubleshooting.
Architecture And Core Systems	How requests move through the system; the plugin, memory, inner-voice, scheduling, workspace, RAG, and expert systems.
Commands	Purpose of every PowerShell script, launcher/service startup paths, run.ps1 switches, and npm scripts.
Configuration Reference	Configuration files, secrets/environment, Mem0 Docker, host file access, Matbot runtime, providers, workspaces, workspace RAG, and expert panel config.
Plugins And Tools	Active and bundled plugins, adding plugins at runtime, and governed retrieval, workflow, evaluation, ROI, host-file, expert-panel, and cognition tools.
Memory And Retrieval	The "remember my name" flow, remembered_facts, KnowledgeIndex, workspace RAG, contextual_search, and memory-policy.json.
Expert Panel WebUI User Manual	Using the expert panel from the WebUI, modes, synthesis, citations, and troubleshooting.
WebUI	WebUI capabilities and behavior.
Testing	Node and Playwright test layers and coverage.
Reliability findings	Failure scenarios, implemented fixes, and regression validation.
Troubleshooting	Common failures and their fixes.

A good reading order is this overview, the Cortex User Guide, and then Architecture And Core Systems or the Configuration Reference when deeper operational or implementation detail is needed. The reference documents intentionally repeat some terms so each can be read on its own.

Repository Map

Path	Purpose
scripts\	PowerShell setup, run, stop, and health-check commands.
local-agent\file-index	JSON-backed file index with keyword search, metadata, hashing, exclusions, and likely-secret skipping.
local-agent\file-broker	Policy-aware file access service for listing, reading, and approved writes with diffs and backups.
local-agent\docker\mem0	Docker Compose stack for Mem0, Postgres/pgvector, Neo4j, and optional CUDA embeddings.
local-agent\config	Host-side configuration for file roots, security policy, path mapping, memory policy, and expert definitions.
local-agent\knowledge	Minimal file-backed knowledge samples for the default experts.
local-agent\matbot	Matbot runtime checkout, Cortex WebUI, providers, plugins, workspace registry, and per-workspace data.
docs	Detailed reference documentation extracted from this README.
tests	Node test suite plus Playwright WebUI coverage.

Technology

Cortex is predominantly a TypeScript/Node.js system with a Docker-hosted Python memory service and Windows PowerShell operational tooling. The table below lists the significant languages, runtimes, and libraries used by each module.

Module	Languages / runtime	Significant technologies and libraries
scripts\	Windows PowerShell	.ps1 setup/run/health scripts; WinSW wraps Matbot as a Windows service; Windows service control (services.msc, Get-Service).
local-agent\file-index	TypeScript, Node.js	Built-in node:http server (no web framework); minimatch for glob exclusions; node:crypto hashing; JSON-file persistence.
local-agent\file-broker	TypeScript, Node.js	Built-in node:http server; no runtime dependencies; unified diffs and file backups; JSON policy files.
local-agent\docker\mem0	YAML, Python (upstream image), Dockerfile	Docker Compose stack; Mem0 API server (patched to add psycopg, langchain-neo4j, rank-bm25); Postgres 16 with pgvector for Mem0 and workspace RAG; optional workspace-rag-cuda embedding service under the cuda profile; Neo4j 5; Mem0 API runs linux/arm64 under QEMU on amd64.
local-agent\matbot (runtime)	TypeScript (strict), Node.js >= 24	pnpm monorepo; agentic runner, plugin loader, hooks, stores; provider communication via raw fetch + SSE (no provider SDKs); Store compare-and-swap persistence.
Matbot WebUI (packages\plugins\frontend\web)	TypeScript (server), vanilla JavaScript (client)	Node HTTP + Server-Sent Events server; framework-free app.js/index.html/CSS; localStorage and Web Crypto in the browser.
Provider adapter (packages\plugins\providers\openai-compat)	TypeScript	OpenAI-compatible chat-completions adapter; streaming and tool calls over fetch + SSE; works with OpenAI and any compatible endpoint.
workspace-rag plugin	TypeScript, Node.js	V2 hierarchical Markdown parsing and SHA-256 content hashing; atomic Postgres/pgvector hybrid generations for document, section, and passage retrieval; CPU hash vectorizer fallback; optional CUDA embeddings through the workspace-rag-cuda HTTP service when launch-time CUDA probing succeeds.
Retrieval plugins (hybrid-knowledge-index, persist-ki-bge, rumsfeld)	TypeScript	KnowledgeIndex implementations querying Mem0 and file-index; optional BGE reranking in persist-ki-bge; contextual_search tool.
expert-panel plugin	TypeScript, Node.js	Tool-based multi-expert orchestration over the Matbot single-turn API; per-expert file retrieval and optional synthesis.
evaluation-observability plugin	TypeScript, Node.js	Store-backed end-to-end spans, redacted trace replay, deterministic and model-scored regression suites, operational metrics, cost accounting, and verified ROI evidence.
tests	JavaScript (.mjs), TypeScript	Node built-in test runner (node --test); Playwright for WebUI tests with a fake Matbot harness.
Root workspace	JSON, TypeScript	npm workspaces; TypeScript 5.x build (tsc); @playwright/test; shared @types/node.

Cross-cutting choices worth noting:

• Provider access never uses vendor SDKs; all model traffic is fetch plus Server-Sent Events parsing.
• Shared Matbot packages avoid Node-only primitives so they run in both Node and the browser; Node-specific behavior lives in -node packages and the host apps.
• The local HTTP services deliberately use Node's built-in http module instead of a web framework to keep dependencies minimal.

Using Cortex

The Cortex User Guide is the main task-oriented manual. It owns the installation requirements, first-run commands, WebUI walkthroughs, feature instructions, safety guidance, and common troubleshooting steps that previously lived in this README or were spread across technical reference files.

Running Tests

Before committing changes, run the complete test suite:

npm run test:all

This runs:

• Node tests for backend/runtime behavior (npm test)
• Matbot CLI tests for ephemeral-store and workspace-storage isolation (npm run test:cli)
• Playwright WebUI tests for browser interactions (npm run test:webui)

For more details on test layers, variants, and troubleshooting, see Testing.

Use workspace files as task inputs and outputs

The Files section is a workspace file shelf with explicit per-message attachments. Uploading a document adds it to the shelf and selects it for the next message; use the paperclip action to attach or detach an existing file. This is useful when Cortex needs source material for a task, for example to summarize a report, analyze a CSV, compare documents, transform data, or create another artifact from it. Cortex can also write results such as reports, charts, and exports back to the same area for you to open or download.

Uploaded files remain available to conversations in the active Cortex workspace. They are isolated from other Cortex workspaces and are not host filesystem files. Attaching does not place the file's full contents in every prompt. It adds a scoped reference to the next message so Cortex reads that workspace copy on demand, even when Workspace RAG contains a same-named host path. State the task, for example:

Read sales.csv, identify unusual changes, and create analysis.md.

The attachment selection clears after a successful send, keeping unrelated files out of later turns and model context. Workspace files are also separate from Workspace RAG: use the file shelf for explicit task inputs, temporary working material, and generated outputs; use Workspace RAG when Markdown folders should become persistent searchable knowledge. See Use Files And Workspace RAG for upload, open, and delete instructions.

Filesystem-backed workspace files keep their logical names and IDs. New writes store content versions under .data/files/.cortex-versions and atomically switch the corresponding metadata manifest. Use the Files panel or file tools to read the current version; a legacy payload at the original name is not authoritative after that entry is overwritten. Existing layouts remain readable. Unreferenced version blobs are reclaimed in batches of at most 100, after a 24-hour grace period, during periodic store maintenance triggered by writes or listing.

Provider request reliability

OpenAI-compatible, Anthropic, and OpenRouter providers enforce three deadlines through their parameters settings: requestTimeoutMs (60,000 by default, covering headers and retries), streamIdleTimeoutMs (120,000), and completionTimeoutMs (600,000, covering the entire completion). Values must be positive integer milliseconds, at most 3,600,000. Increase these for slow local models when needed. SSE comments do not reset the idle deadline, and activity never extends the overall deadline. Completed tool rounds and partial responses are saved when a completion fails; the session records an interruption marker, including for timeouts.

OpenRouter

Cortex includes a Node-hosted OpenRouter profile adapter for explicit https://openrouter.ai/api/v1 Chat Completions profiles. Configure an exact model ID and ${OPENROUTER_API_KEY} reference in the active workspace; keys are resolved by the vault and are never stored in profile responses or the WebUI. Streaming text, supported image inputs, Cortex local tools, cancellation, and usage accounting follow the normal conversation runner. See OpenRouter profiles for the safe configuration template, capability/routing controls, diagnostics, and the separately gated live smoke test.

Provider streams must confirm completion. OpenAI-compatible streamed errors, malformed events, and premature EOF are reported as failures; buffered tool calls from an unconfirmed response are not executed. Trailing usage is collected before the single terminal completion event. Anthropic streams must reach their message_stop event.

Background schedule recovery

Recurring schedules retry transient storage failures with backoff while retaining the same occurrence ID. every_action list reports schedulerState, an active schedulerError during recovery, and the last persisted scheduler error when available. Resume ensures that a stopped schedule has an armed loop. Startup respects the saved next-run time. If a previous process left an occurrence marked running, Cortex records that its effects are uncertain and schedules the next interval instead of immediately replaying it. Suspend/cancel changes are retained while completion records are retried.

Switch the Workspace RAG embedding model

Workspace RAG can use either the default sentence-transformers/all-MiniLM-L6-v2 model or intfloat/multilingual-e5-base. MiniLM produces 384-dimensional vectors and is smaller and faster. Multilingual E5 produces 768-dimensional vectors and is intended for multilingual and cross-language retrieval. Cortex automatically uses E5's query: and passage: prefixes when WORKSPACE_RAG_EMBEDDING_PROFILE=auto.

Changing models invalidates the current workspace's derivatives and starts a new V2 generation when Matbot starts. Stop Matbot before switching, particularly when a workspace contains many files. Use ingestion_cancel if an active V2 job must be cancelled; the prior complete generation remains published. Do not pass -v to Docker Compose when stopping services because the declared volumes contain Postgres data and the downloaded model cache.

Embeddings run in float16 by default on the CUDA sidecar, which roughly doubles throughput and halves GPU memory on tensor-core GPUs (every GeForce RTX card, including the RTX 3060) and enables TF32 matmuls alongside it. Because reduced precision changes vector values, the dtype is part of the embedding signature: switching WORKSPACE_RAG_EMBEDDING_DTYPE between float16, bfloat16, and float32 invalidates existing vectors and Cortex rebuilds them automatically during each affected workspace's next scan. Set WORKSPACE_RAG_EMBEDDING_DTYPE=float32 in local-agent\docker\mem0\.env to restore full precision for every workspace.

For large workspaces, Cortex streams each Markdown file within a bounded parser budget, keeps embedding batches in flight while Postgres writes and parsing continue (CORTEX_RAG_V2_EMBED_PIPELINE_DEPTH), ingests bulk backlogs several files at once with adaptive file concurrency that drops back to sequential for small incremental scans (CORTEX_RAG_V2_FILE_CONCURRENCY), rate-limits derivative and source-registry work, and stores high-cardinality source/graph metadata in WAL-mode SQLite. See docs\configuration.md for the full tuning surface.

1. Stop Cortex before changing the model:

   .\scripts\stop-local-agent.ps1

2. Edit the gitignored local-agent\docker\mem0\.env file. To select multilingual E5, use:

   WORKSPACE_RAG_EMBEDDING_MODEL=intfloat/multilingual-e5-base
   WORKSPACE_RAG_EMBEDDING_MODEL_REVISION=d13f1b27baf31030b7fd040960d60d909913633f
   WORKSPACE_RAG_EMBEDDING_PROFILE=auto
   WORKSPACE_RAG_EMBEDDING_BATCH_SIZE=128
   WORKSPACE_RAG_EMBEDDING_DTYPE=float16

   To switch back to MiniLM, use:

   WORKSPACE_RAG_EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
   WORKSPACE_RAG_EMBEDDING_MODEL_REVISION=46605decb5369335a3847c9f41bb0b896c07dd1a
   WORKSPACE_RAG_EMBEDDING_PROFILE=auto
   WORKSPACE_RAG_EMBEDDING_BATCH_SIZE=128
   WORKSPACE_RAG_EMBEDDING_DTYPE=float16

   Reduce the batch size if the CUDA worker runs out of GPU memory.

3. Rebuild and recreate the CUDA embedding sidecar:

   docker compose `
     -f .\local-agent\docker\mem0\docker-compose.yml `
     --profile cuda `
     up -d --build --force-recreate workspace-rag-cuda

   The first E5 start downloads the model into the persistent workspace-rag-models Docker volume and can take longer than subsequent starts.

4. Wait for the sidecar and verify its model contract:

   Invoke-RestMethod http://127.0.0.1:8890/health |
     Select-Object model, profile, dimensions, maxTokens, normalized,
       queryPrefix, documentPrefix, dtype

   E5 should report model intfloat/multilingual-e5-base, profile e5-asymmetric-v1, 768 dimensions, normalized output, and the query: / passage: prefixes. MiniLM should report sentence-transformers/all-MiniLM-L6-v2, profile plain-v1, 384 dimensions, normalized output, and empty prefixes.

5. Start Cortex and Matbot:

   .\scripts\run.ps1 -NoBrowser

   Matbot probes the sidecar at startup. The changed model, dimensions, or preprocessing signature causes Workspace RAG to re-embed existing documents automatically. Postgres keeps dimension-specific derivative tables: E5 uses unit_embeddings_768, while MiniLM uses unit_embeddings_384. Switching models does not delete the other model's table.

6. Check reindex progress and confirm that Matbot adopted the expected model:

   Invoke-RestMethod `
     -Method Post `
     -Uri http://127.0.0.1:19778/tools/workspace_rag `
     -ContentType "application/json" `
     -Body '{"action":"status"}' |
      Select-Object mode, available, activeState, activeGenerationId,
        lastSuccessfulReconcileAt, embeddingBackend,
        embeddingModel, embeddingDimensions, embeddingProfile,
        embeddingMaxTokens, backend, job, watcher

   Wait for job.state and activeState to report an active_* publication before treating the new generation as complete. Overlapping requests are coalesced and watcher.reconcileQueued makes a pending follow-up visible. If startup did not schedule reconciliation, request a full rebuild explicitly:

   Invoke-RestMethod `
     -Method Post `
     -Uri http://127.0.0.1:19778/tools/workspace_rag `
     -ContentType "application/json" `
     -Body '{"action":"reindex_now"}'

Operate hierarchical Workspace RAG V2

Workspace RAG uses V2 progressive collection → document → section → passage retrieval as its only index. It starts in primary mode, watches configured folders, and performs a periodic safety reconciliation. Search remains unavailable until the first validated generation is published.

Ingestion checkpoints publish independent snapshots. The running job keeps a separate staging generation for new work and deletion reconciliation. If a later step fails or is cancelled, search retains the last successfully published snapshot; restarting can resume completed work in staging. A checkpoint alone does not mean the whole reconciliation job has completed.

Setup failures and cancellation leave terminal job states with error details. If storage is unavailable, status reports that outage and retains the latest local job outcome; document counts in that response are unavailable and should not be interpreted as an empty index. The next ingestion marks an interrupted prior job as failed before attempting to resume its staging work.

Semantic summaries have a two-minute deadline covering summary generation and its embedding. Set CORTEX_RAG_V2_SUMMARY_TIMEOUT_MS to adjust it (10 to 3,600,000 milliseconds). Cancelling ingestion releases producers waiting for summary queue space; shutdown cancels all summary workers before closing storage.

1. Keep the default CORTEX_RAG_V2_MODE=primary. Set it to off only to disable Workspace RAG completely. Configure the V2 Postgres/object-store variables in the active workspace .env. For production, use a non-owner application database role, a separate CORTEX_RAG_V2_MIGRATION_POSTGRES_URL, and CORTEX_RAG_V2_REQUIRE_SEPARATE_DB_ROLES=1.

2. Start Cortex, then run a read-only resumable census:

   { "action": "corpus_census", "deep": true }

   An interrupted result supplies checkpoint; pass it back as resumeAfter. Use the measured percentiles and storage forecast to set the 20 MB/250 MB prototype tier thresholds appropriately for the corpus.

3. Start an explicit atomic generation when needed:

   { "action": "ingestion_start" }

   Use ingestion_status, ingestion_pause, ingestion_resume, ingestion_cancel, and ingestion_wait to control it. Starting again while a job is active returns that job instead of queuing a duplicate scan. Cancellation preserves the active generation, and publication is one atomic transaction after validation.

4. Exercise retrieval explicitly with v2_search and run evaluation_run. Keep authorization, citation, memory, relevance, latency, and degradation gates in release validation. Repeat evaluation_run with evaluationVariant set to flat_dense_baseline, lexical_only, dense_only, hybrid_rrf, hybrid_translated, hybrid_reranked, hierarchical, and hierarchical_lazy; each run persists its variant and metrics.

5. To enable the optional multilingual cross-encoder, start docker compose --profile reranker up -d --build workspace-rag-reranker and set CORTEX_RAG_V2_RERANKER_URL=http://127.0.0.1:8891. Retrieval falls back to fused lexical/dense results if this service times out or is unavailable. The Compose default pins the reranker model to an immutable Hugging Face revision and pins its referenced repository code separately; when changing models, set matching immutable WORKSPACE_RAG_RERANKER_MODEL_REVISION and WORKSPACE_RAG_RERANKER_CODE_REVISION values.

6. For semantic routing summaries, set CORTEX_RAG_V2_SUMMARY_PROVIDER to the name of a configured Matbot provider. Section, document, and collection summaries are generated asynchronously, versioned by source content and provider/model signature, and embedded only for routing. They can never be emitted as citation evidence. Split books can declare book_id and book_title (or collection_id and collection_title) in front matter to add collection routing above their chapter files.

Reclaim orphaned Workspace RAG data

Workspace RAG automatically reclaims database records that become unreachable after a folder is removed or a later successful reconciliation clears a previously deferred deletion. Cleanup also runs on an independent jittered timer, every six hours by default. A document version is orphaned only when no active or staging publication references it, so an in-progress generation and the currently queryable generation remain protected.

Each sweep removes eligible document versions and their sections, passages, embeddings, routing summaries, and stale ingestion-job records in bounded transactions. Empty retired publication generations are removed after their separate retention period. Cleanup is serialized per workspace context, skips deletion while ingestion is non-terminal, and applies a one-hour grace period by default to avoid racing newly created jobs or content. If a sweep is skipped, it is retried with bounded exponential backoff.

Run cleanup for the active context on demand with:

{ "action": "gc" }

Add "contextId" to target another context in the current workspace. The result reports deletion counts, and subsequent Workspace RAG status responses include the latest counts, timestamp, and duration under lastGc. Manual cleanup remains available when automatic cleanup is disabled.

Content-addressed source-object cleanup is separately controlled by CORTEX_RAG_V2_BLOB_GC_ENABLED and is off by default. When enabled, it deletes only stale, globally unreferenced objects from a managed store; objects still referenced by any workspace are retained, and external_immutable and manifest_only stores are never modified. Deleting an entire context purges its non-audit index state while retaining retrieval and evaluation audit records for their normal TTL-managed lifecycle.

Configure the sweep interval, grace period, transaction batch size, retired generation retention, and managed-blob cleanup with the Workspace RAG configuration settings.

For an operational rollback, set CORTEX_RAG_V2_MODE=off; this disables Workspace RAG rather than routing to an older index. Published V2 generations and immutable source versions are retained for diagnosis and later recovery. See Hybrid Retrieval Architecture for the data contracts, adoption gates, and completed implementation evidence.

For additional storage, CUDA, and troubleshooting options, see Workspace RAG configuration.

Use Commands for the complete operational command reference and Configuration Reference for provider, secret, workspace, RAG, file-access, and expert configuration.