diff --git a/docs/designs/external_memory/external-memory-integration.md b/docs/designs/external_memory/external-memory-integration.md new file mode 100644 index 00000000..a14823fd --- /dev/null +++ b/docs/designs/external_memory/external-memory-integration.md @@ -0,0 +1,558 @@ +# Design: External Memory Provider Integration + +- **Status:** Draft +- **Dependencies:** + - [Config-Driven Hooks](../config_driven_hooks.md) + - [Memory Architecture](memory_architecture.md) *(feat/memory_desighn_doc)* + +--- + +## Problem Statement + +QuickApps has its own memory implementation (LanceDB + DIAL file storage). Operators currently have no +way to substitute a different memory backend — they're locked into the QuickApps-native implementation. + +The goal is to let operators choose any open-source or self-hosted memory provider (mem0, basic-memory, +Graphiti, etc.) and wire it into a QuickApp without modifying application code. + +--- + +## Design Goals + +- An operator can configure an external memory backend by editing the app's JSON manifest only. +- User isolation is preserved: each DIAL user's memories are separate regardless of the backend. +- No skill is required for basic integration — the LLM understands provider tools from their native MCP descriptions, as in Claude Code and Cursor. Skills remain available as an optional layer for operators who need to enforce specific memory behaviors. +- Adding support for a new provider in the future requires minimal effort. + +--- + +## Background: How QuickApps Memory Works Today + +QuickApps memory uses three mechanisms: + +**1. MCP toolset** — the memory server is declared in `ApplicationConfig.tool_sets` as a `dial-mcp` entry. +The agent calls `store_memory` and `search_archive` tools during conversation. + +**2. Hook-based context injection** *(optional pattern)* — the `on_request_start` hook calls a retrieval +tool before the first LLM turn and injects the result as a synthetic `(ASSISTANT tool_call / TOOL +response)` pair into the message history via `_ConfigDrivenToolCallHook`. TTL caching prevents redundant +calls. + +**3. Skill** *(optional)* — a predefined Markdown skill tells the LLM which tools exist and when to +use them. For QuickApps' own memory server this is predefined. For external providers it is not +required — the LLM infers tool purpose from native MCP descriptions, the same way Claude Code and +Cursor work with MCP memory servers without any skill. + +The hook-based injection is **one retrieval pattern, not a requirement**. The alternative is reactive +retrieval: the LLM calls memory tools (`search`, `list`) when it decides past context is relevant. This +is how every standard MCP client (Claude Code, Cursor) works. Both patterns can be used with external +providers; hooks add value but are not a prerequisite for integration. + +**User isolation** is implicit: the memory MCP server authenticates via the user's DIAL API key, which +scopes all reads and writes to that user's DIAL file storage bucket. No `user_id` appears in tool +arguments — isolation is handled entirely by the auth layer. + +``` +Request arrives (DIAL API key = user A) + → MCPToolInitializer connects to memory-server via DIAL (DIAL forwards user's key) + → [Optional] Hook fires → calls memory-server_get_memories → injects top-N memories into history + → Agent runs, calls store_memory / search_archive as needed + → Memory server reads/writes user A's DIAL bucket (enforced by auth) +``` + +--- + +## Background: The External Memory Ecosystem + +### These are single-user tools, not multi-tenant services + +Every open-source memory provider surveyed — mem0-mcp, mcp-obsidian, basic-memory, Official MCP Memory, +Graphiti — was designed for **one person using one IDE**. In a typical Claude Code or Cursor setup, one +instance of the memory server equals one person's memory. The server runs as a local process or a personal +Docker container. + +``` +Typical usage (Claude Code / Cursor): + + Developer's machine + ┌──────────────────────────────────────────┐ + │ Cursor IDE │ + │ └── MCP client │ + │ └── connects to mem0-mcp │ ← one instance + │ └── stores to ~/mem0.db │ ← one user's data + └──────────────────────────────────────────┘ +``` + +The `user_id` / `group_id` parameters that some providers expose were added as an afterthought for managed +cloud tiers (mem0 cloud serves thousands of customers), not for the typical local use case. Several +providers (Official MCP Memory, basic-memory) have no user scoping at all. + +**Multi-user isolation is not a built-in property of these providers.** For providers that run as a +shared server (mem0, Graphiti), isolation must be layered on top — through an adapter (Approach A) or +injected via config variables (Approach B). For file-based providers (basic-memory, Official MCP Memory), +isolation is already available today: DIAL already provides per-user file storage, and pointing the +provider's data path at the user's DIAL bucket is sufficient. Containers make this even simpler (the +bucket is mounted as a volume) but are not a prerequisite. + +### MCP has no pre-turn injection + +The MCP specification defines no mechanism for a server to push context before each user turn. All memory +retrieval in Claude Code and Cursor is **reactive** — the LLM calls memory tools when it judges past +context is relevant. There is no automatic injection. + +**QuickApps' hooks system already does more than the MCP ecosystem standard.** The `on_request_start` hook +that calls a retrieval tool and synthetic-injects the result is a capability no standard MCP client has. + +### Tool interface across providers + +There is no naming standard. Three semantic roles appear consistently across all providers: + +| Role | What it does | Example names | +|---|---|---| +| **Write** | Store a new fact or observation | `add_memory`, `add_memories`, `write_note`, `add_observations` | +| **Search** | Retrieve by semantic query | `search_memories`, `search_nodes`, `search` | +| **List** | Get all/top memories without a query | `get_memories`, `list_memories`, `read_graph`, `get_user_context` | + +The **list** tool is the correct hook point if proactive `on_request_start` injection is configured — it +requires no query and returns the most relevant memories for the current user. It is not required for +reactive retrieval, where the LLM calls **search** directly when it needs context. + +### Provider storage backends + +Storage type determines whether a provider can use DIAL's per-user file storage for isolation. File-based +providers can write to a path inside the user's DIAL bucket — this works both today (current shared-service +model) and in the future container model. Providers backed by external databases cannot use DIAL file +storage and require a separate isolation mechanism. + +| Provider | Storage | File-based? | Notes | +|---|---|---|---| +| **Official MCP Memory** | Single JSON file | ✅ Yes | Path via `MCP_MEMORY_FILE_PATH` | +| **basic-memory** | Markdown files | ✅ Yes | Vault path configurable | +| **mem0** (Chroma + SQLite config) | Files on disk | ✅ Yes | Non-default; requires config change | +| **mem0** (default Docker) | PostgreSQL + pgvector | ❌ No | Needs external persistent DB | +| **Graphiti** | Neo4j or FalkorDB | ❌ No | Requires external graph DB service | +| **Zep** | Proprietary DB | ❌ No | Long-running server with own storage | +| **mcp-obsidian** | Obsidian vault files | ❌ No† | †Requires Obsidian desktop app running | + +--- + +## Model 1: Current Shared-Service Deployment + +Today, QuickApps runs as a single shared deployment: one process serves all users. + +The integration approach depends on the provider's storage model: + +**File-based providers** (basic-memory, Official MCP Memory, mem0 with Chroma) — user isolation is +already solved by DIAL's per-user file storage. QuickApps already writes `memory.lance/` into each +user's DIAL bucket for its own memory implementation; the same mechanism applies to any external +file-based provider. No `user_id`, no adapter. The provider is configured with a data path inside the +user's DIAL bucket, and the DIAL auth layer enforces isolation. + +``` +DIAL + ├── User A ──► QuickApps ──► file-based memory server (path = user A's DIAL bucket) + ├── User B ──► QuickApps ──► file-based memory server (path = user B's DIAL bucket) + └── ... ← isolation is structural; no user_id in tool calls +``` + +**Server-based providers** (mem0 with pgvector, Graphiti) — a single shared service instance handles +requests from many different DIAL users simultaneously. The memory server must never let one user's +memories bleed into another's context. This requires explicit `user_id` scoping at every tool call. + +``` +DIAL + ├── User A ──► QuickApps backend (shared) ──► Memory server (shared) + ├── User B ──► │ + └── User C ──► must isolate A/B/C +``` + +Two approaches address server-based isolation. + +--- + +### Approach A: Memory Adapter (recommended for Model 1) + +Build a thin MCP server for each provider that wraps the provider's native API and handles user identity +propagation internally. QuickApps always connects to the adapter using the same standard tool names, +regardless of which backend is underneath. + +``` +QuickApp config (identical for all backends) + └── tool_sets: [{ type: "dial-mcp", dial_id: "memory-adapter" }] + └── hooks: [{ tool: "memory-adapter_memory_get_context" }] ← optional + +DIAL routes the call → adapter receives X-DIAL-API-Key header + └── Adapter resolves user_id from header + └── Calls backend (mem0 / basic-memory / …) with user_id + └── Returns standardised response +``` + +**Confirmed:** `DialMCPToolSet` forwards the user's DIAL API key in request headers to the upstream MCP +server. The adapter can extract user identity from these headers with no changes to QuickApps core. + +#### Standard tool contract + +The adapter exposes a normalized interface so every Class 2 backend looks identical to QuickApps. +No skill is needed — the LLM understands these tools from their MCP descriptions. + +| Tool | Parameters | Description | +|---|---|---| +| `memory_store` | `content: str` | Store a new fact. | +| `memory_search` | `query: str`, `limit?: int` | Semantic search. | +| `memory_get_context` | *(none)* | *(optional)* Top-N memories without a query. Only needed if proactive hook injection is configured. | +| `memory_delete` | `memory_id: str` | *(optional)* Delete a specific memory. | +| `memory_clear` | *(none)* | *(optional)* Wipe all memories for the current user. | + +Normalization here serves a different goal than skills: it means every Class 2 adapter is wired by +the same one-line toolset config, and a single optional skill covers all of them if the operator +wants to enforce specific behavior. + +#### Example: mem0 adapter + +``` +[ QuickApp agent ] + │ memory_get_context / memory_store / memory_search + ▼ +[ mem0 Adapter ] ← DIAL deployment + │ reads X-DIAL-API-Key → resolves user_id + │ calls mem0 REST API with user_id + ▼ +[ mem0 server ] ← Docker: FastAPI + pgvector + │ data partitioned per user_id in pgvector + ▼ +[ PostgreSQL + pgvector ] +``` + +Adapter sketch (~100 lines Python): + +```python +from mcp.server.fastmcp import FastMCP +import httpx, os + +mcp = FastMCP("mem0-adapter") +MEM0_URL = os.environ["MEM0_URL"] + +def get_user_id(ctx) -> str: + return ctx.request_context.headers.get("x-dial-api-key", "anonymous") + +@mcp.tool() +async def memory_get_context(ctx) -> str: + uid = get_user_id(ctx) + r = await httpx.get(f"{MEM0_URL}/memories", params={"user_id": uid, "limit": 20}) + return format_memories(r.json()["results"]) + +@mcp.tool() +async def memory_store(content: str, ctx) -> str: + uid = get_user_id(ctx) + await httpx.post(f"{MEM0_URL}/memories", json={ + "messages": [{"role": "user", "content": content}], + "user_id": uid + }) + return "Stored." + +@mcp.tool() +async def memory_search(query: str, ctx, limit: int = 5) -> str: + uid = get_user_id(ctx) + r = await httpx.post(f"{MEM0_URL}/search", json={ + "query": query, "filters": {"user_id": uid}, "limit": limit + }) + return format_memories(r.json()["results"]) +``` + +QuickApp config (same for every operator using any compliant adapter): + +```json +{ + "tool_sets": [{ "type": "dial-mcp", "dial_id": "memory-adapter-mem0" }] +} +``` + +Optionally, add a hook for proactive injection and/or a skill for enforced behaviors: + +```json +{ + "tool_sets": [{ "type": "dial-mcp", "dial_id": "memory-adapter-mem0" }], + "hooks": [{ + "kind": "tool_call", + "event": "on_request_start", + "toolset_name": "memory-adapter-mem0", + "tool_name": "memory_get_context", + "arguments": {}, + "frequency": "append_if_changed", + "refresh_condition": { "kind": "ttl", "ttl_minutes": 5 } + }] +} +``` + +> **mem0's LLM dependency:** mem0 calls an LLM on every `add_memory` to extract structured facts from raw +> text. The adapter deployment must include an LLM endpoint (OpenAI, Anthropic, or a local Ollama/vLLM). +> This cannot be eliminated without forking mem0. + +#### Open problems in Approach A + +1. **Adapter maintenance** — each new provider needs a new adapter (~150 lines each). Adapters must be + built, tested, and deployed. Organisational question: separate repo vs. bundled with QuickApps. +2. **mem0's LLM cost** — every write triggers an LLM inference pass (~1–2 s, API cost). Operators must + supply an LLM endpoint. +3. **Graphiti/Zep** — require Neo4j or FalkorDB as the graph DB. High operational complexity. + +--- + +### Approach B: Dynamic Variable Injection in Hooks + +No adapter layer. QuickApps connects directly to a provider's MCP server. The hooks configuration is +extended with template variables resolved from the DIAL request context at runtime. + +``` +QuickApp config (provider-specific) + └── hooks: [{ tool: "mem0_get_memories", + arguments: {"filters": {"user_id": "{{dial_user_id}}"}} }] + +Request arrives + → Hook resolves {{dial_user_id}} → "user-abc-123" + → Calls mem0_get_memories(filters={user_id: "user-abc-123"}) +``` + +Proposed variables: + +| Variable | Value | +|---|---| +| `{{dial_user_id}}` | Authenticated DIAL user identifier | +| `{{app_id}}` | The QuickApp's DIAL deployment ID | +| `{{session_id}}` | Current conversation session ID | + +Code change in `_ConfigDrivenToolCallHook` — backwards-compatible, existing static configs unaffected: + +```python +def _resolve_arguments(self, arguments: dict, context: RequestContext) -> dict: + variables = { + "dial_user_id": context.user_id, + "app_id": context.app_id, + "session_id": context.session_id, + } + return json.loads(Template(json.dumps(arguments)).substitute(variables)) +``` + +#### Example: mem0 direct + +```json +{ + "tool_sets": [{ + "type": "mcp_http", + "url": "http://mem0-server:8888/mcp", + "name": "mem0", + "allowed_tools": ["add_memory", "get_memories", "search_memories"] + }], + "hooks": [{ + "kind": "tool_call", + "event": "on_request_start", + "toolset_name": "mem0", + "tool_name": "get_memories", + "arguments": { "filters": { "user_id": "{{dial_user_id}}" }, "limit": 20 }, + "frequency": "append_if_changed", + "refresh_condition": { "kind": "ttl", "ttl_minutes": 5 } + }] +} +``` + +No skill required — the LLM understands mem0's tools from their MCP descriptions. The LLM must pass +`"user_id": "{{dial_user_id}}"` correctly in write and search calls; this relies on the tool +descriptions making the argument mandatory and obvious, not on a skill. + +#### Open problems in Approach B + +1. **Template variable support doesn't exist yet** — code change required in QuickApps core. +2. **No skill portability** — without a normalized adapter, each provider has different tool names + and argument shapes. If an operator wants a skill, they must write one per provider. +3. **Nested JSON injection** — mem0's `user_id` lives inside `{"filters": {"user_id": "..."}}`. Simple + string substitution risks JSON injection if the user ID contains special characters. +4. **Weak isolation** — the memory server trusts whatever `user_id` the caller sends. A misconfigured + hook silently accesses the wrong user's data. Approach A validates against the DIAL auth header. +5. **Providers without `user_id` argument** — OpenMemory bakes `user_id` in the SSE URL path. Supporting + it would also require URL-level templating in the toolset config. + +--- + +### Comparison: Approach A vs B in Model 1 + +| Dimension | Approach A: Adapter | Approach B: Dynamic Variables | +|---|---|---| +| Operator config | Identical for all backends | Different per provider | +| Skill required | No (optional) | No (optional, but non-portable across providers) | +| User isolation | Strong — enforced in adapter | Weak — trusts config | +| QuickApps code changes | None | Yes — template expansion in hooks | +| New provider support | Build an adapter (~150 lines) | Document config per provider | +| Risk of user data leak | Low | Higher — wrong template = data leak | + +--- + +## Model 2: Future Container Deployment + +In the planned container model, each user session runs in its own isolated container: **one container = +one user × one QuickApp**. The user's DIAL storage bucket is mounted as a volume. + +This changes everything. + +### How the container model works for file-based providers + +A memory server running inside a per-user container is, by definition, serving exactly one user. It is +back to being the single-user tool it was designed to be. There is no shared state, no `user_id` needed +anywhere, no isolation machinery. + +Note: the isolation principle is not new — DIAL's per-user file storage already enforces it in the +current shared-service model. The container model makes it structurally impossible to misconfigure +(a wrong data path can't accidentally reach another user's bucket), and removes the need for any process +lifecycle management to keep per-user server instances separated. + +``` +User starts a QuickApp session + │ + ▼ +Container spins up (this user × this QuickApp only) +┌─────────────────────────────────────────────────────┐ +│ quickapps-backend memory-sidecar │ +│ │ (basic-memory, Official MCP, │ +│ └─────────────── or mem0 with Chroma) │ +│ │ │ +│ /data/memory/ ◄── mounted │ +│ ├── graph.json from this │ +│ ├── notes/ user's │ +│ └── ... DIAL bucket│ +└─────────────────────────────────────────────────────┘ + │ +Session ends → container stops → data in DIAL bucket +Next session → new container → same mount → memory intact +``` + +The memory sidecar is **stateless between sessions** — it writes to the mounted volume, not its own +filesystem. Spinning it up and down has zero data loss. + +**QuickApps' own memory implementation already follows this exact pattern** — LanceDB stores +`memory.lance/` in the user's DIAL bucket. The container model makes this the universal convention for +any provider. + +### Which providers fit the file-based / DIAL-bucket pattern + +File-based providers that accept a configurable data path can use DIAL's per-user file storage for +isolation. This applies to both the current shared-service model (path resolves to the user's DIAL +bucket) and the container model (path is a mounted volume from the same bucket). Providers backed by +external database services cannot participate. + +| Provider | File-based? | Data path config | Notes | +|---|---|---|---| +| **Official MCP Memory** | ✅ Yes | `MCP_MEMORY_FILE_PATH=/memory.jsonl` | Simplest option; no LLM needed; keyword search only | +| **basic-memory** | ✅ Yes | Vault path env var | Markdown notes; hybrid search; no LLM needed | +| **mem0** (Chroma + SQLite) | ✅ Yes (non-default config) | `path: /chroma` + `path: /mem0.db` | Semantic search; LLM needed for fact extraction | +| **mem0** (default pgvector) | ❌ No | — | PostgreSQL lives outside QuickApps' storage model | +| **Graphiti** | ❌ No | — | Neo4j / FalkorDB lives outside QuickApps' storage model | +| **Zep** | ❌ No | — | Zep server has its own persistent storage | +| **mcp-obsidian** | ❌ No | — | Requires Obsidian desktop app — incompatible with any server deployment | + +### mem0 with file-based backends + +mem0 can be made container-compatible by swapping its default PostgreSQL backend for local file-based +alternatives: + +```python +config = { + "vector_store": { + "provider": "chroma", + "config": { + "collection_name": "memories", + "path": "/data/memory/chroma" # ← mounted from DIAL bucket + } + }, + "db": { + "provider": "sqlite", + "config": { + "path": "/data/memory/mem0.db" # ← same mount + } + } +} +``` + +This preserves mem0's semantic search and LLM-based fact extraction. Chroma's local mode is less +battle-tested than pgvector under concurrent writes, but in a single-user sidecar this is not a concern. +The LLM dependency remains: every `memory_store` call triggers an LLM inference pass for fact extraction. +The natural choice is to route this through DIAL's own LLM deployment. + +### What the container model still needs + +Even in the container model, there are open design questions: + +1. **How does the operator declare a memory sidecar?** Options: + - A dedicated `memory` field in `ApplicationConfig` (`"memory": {"enabled": true, "provider": "basic-memory"}`) + - Extending the existing toolset config with a `"type": "sidecar-mcp"` variant + - The orchestration layer (whatever manages containers) handles sidecar injection separately from the app config + +2. **Standard tool names help for Class 2 adapters, not required for sidecars** — in the container + model, the sidecar's native tool names are sufficient: the LLM reads their MCP descriptions and + uses them without a skill. Normalized names only matter if the operator wants a single optional + skill that works across multiple providers. + +3. **Startup latency** — if proactive hook injection is configured, the memory sidecar must be ready + before the first `on_request_start` hook fires. With reactive retrieval only, startup latency is + less critical — the first tool call simply fails gracefully if the sidecar is still initialising. + +--- + +## Recommendation: Sequencing + +Three paths exist, with meaningfully different cost profiles: + +| Option | Build now | Works today? | Becomes dead weight? | +|---|---|---|---| +| **File-based provider via DIAL bucket** | Per-user process lifecycle mechanism; provider config | ✅ Yes | No — same mechanism works in container model | +| **Server-based adapter (Approach A)** | Adapter per provider (~150 lines each) | ✅ Yes | Partially — adapter becomes unnecessary once containers ship; but low code volume | +| **Wait for containers, then sidecar** | Spec the config format + tool contract | ❌ Not yet | No — cleanest long-term path | + +**Suggested path:** +- If external memory is needed today: start with a **file-based provider** (basic-memory or Official + MCP Memory) wired to the user's DIAL bucket. No `user_id`, no adapter, no QuickApps core changes. + This investment carries forward cleanly to the container model. +- If a server-based provider (mem0 pgvector) is specifically required: build an adapter. It is a small + investment and the multi-tenancy isolation is straightforward. +- The Dynamic Variable Injection approach (Approach B) is the highest-risk option and adds a + QuickApps core change; avoid unless there is a strong reason to connect directly to a provider's + native API without an adapter. + +--- + +## Out of Scope + +- **Building memory backends** — mem0, Graphiti, etc. are external projects. This document covers the + integration layer only. +- **UI for memory management** — viewing and deleting memories from the DIAL UI is covered separately. +- **Storage-native approach** — wrapping providers at the storage layer requires forking. Ruled out. +- **Zep Cloud / managed SaaS** — does not fit the self-hosted operator model. +- **Letta / MemGPT** — an agent framework, not a drop-in memory provider. Its memory tools are internal + to its own agent runtime. +- **Obsidian** — requires a running desktop application. Not suitable for any server deployment model. + +--- + +## Open Questions + +1. **Which integration path first?** File-based providers (basic-memory, Official MCP Memory) are viable + today via DIAL bucket access, with no adapter and no QuickApps core changes. Server-based adapters + (Approach A) are still worth building if a provider with richer semantics (mem0 pgvector) is + specifically requested. Approach B (dynamic variables) is the highest-risk option and should be + deprioritised. + +2. **Container granularity:** confirmed as per-user (one container = one user × one QuickApp). This + completely eliminates the `user_id` problem for Model 2. + +3. ~~**Does `DialMCPToolSet` forward the DIAL API key to the upstream server?**~~ **Resolved — yes.** + Approach A's user isolation model is sound without changes to QuickApps core. + +4. **Sidecar config format:** should memory be a first-class field in `ApplicationConfig` + (`"memory": {"provider": "basic-memory"}`), or expressed through the existing toolset + hook + mechanism with a new `"type": "sidecar-mcp"` variant? + +5. **mem0's LLM cost:** every `memory_store` triggers an LLM inference for fact extraction (~1–2 s, + API cost). Should the adapter / sidecar expose an `infer: false` mode that stores verbatim text + and skips extraction? + +6. **Reference provider for v1:** mem0 (with Chroma config) offers the best balance of semantic + search quality and container compatibility. basic-memory is simpler (no LLM dependency) but + keyword-only. Which is the right default to ship first? diff --git a/docs/designs/external_memory/file-based-providers-implementation-draft.md b/docs/designs/external_memory/file-based-providers-implementation-draft.md new file mode 100644 index 00000000..d3c31f05 --- /dev/null +++ b/docs/designs/external_memory/file-based-providers-implementation-draft.md @@ -0,0 +1,397 @@ +# File-Based Providers — Implementation Draft + +- **Status:** Draft +- **Related:** [File-Based Providers](file-based-providers.md) | [Memory Provider Classification](memory-provider-classification.md) + +--- + +## Scope + +This document translates the conceptual design in `file-based-providers.md` into concrete code +changes. It describes every file that needs to be created or modified, with implementation sketches +for each. + +--- + +## Prerequisites + +`file-based-providers.md` has one open question that must be resolved first: + +> **DIAL bucket as local path:** Is the DIAL bucket accessible as a local filesystem path, or does +> QuickApps sync files to/from DIAL storage? + +The answer determines the value of `{{dial_bucket_path}}`: + +- If DIAL storage is **mounted as a local volume** (e.g. via DIAL Files or a sidecar), `bucket_path` + is a real filesystem path and all providers work as-is. +- If DIAL storage is **remote only** (HTTP API), a local scratch directory must be used and files + synced to/from DIAL around the subprocess session. This is a more involved design and is **out of + scope for this draft** — assume local-path access below. + +`bucket_path` is resolved at request time: + +```python +bucket_resp = await self.__dial_client.bucket.get_raw() +bucket_path = bucket_resp.appdata or bucket_resp.bucket +``` + +--- + +## Changes Required + +The table below lists every file affected, whether it is new or modified, and the nature of the +change. + +| File | New / Modified | Change | +|---|---|---| +| `config/toolsets/subprocess_mcp.py` | **New** | `SubprocessMCPToolSet` config model | +| `config/toolsets/__init__.py` or union type | **Modified** | Add `SubprocessMCPToolSet` to the toolset union | +| `mcp_tooling/_abstract_mcp_connection_manager.py` | **New** | Protocol / ABC shared by both connection manager types | +| `mcp_tooling/_stdio_mcp_connection_manager.py` | **New** | Long-lived stdio connection manager | +| `mcp_tooling/_subprocess_pool.py` | **New** | Per-user subprocess pool with TTL eviction | +| `mcp_tooling/_mcp_tool.py` | **Modified** | `connection_manager` type annotation → Protocol | +| `mcp_tooling/_mcp_tool_initializer.py` | **Modified** | New `SubprocessMCPToolSet` branch in `_process_toolset()` | +| `mcp_tooling/mcp_module.py` | **Modified** | Bind `SubprocessPool` as singleton in DI | + +No changes to `StagedBaseTool`, the orchestrator, or any HTTP connection code. + +--- + +## 1. Config Model — `SubprocessMCPToolSet` + +**File:** `src/quickapp/config/toolsets/subprocess_mcp.py` + +```python +from pydantic import BaseModel, ConfigDict + + +class SubprocessMCPToolSet(BaseModel): + model_config = ConfigDict(frozen=True) + + type: Literal["subprocess-mcp"] + name: str + command: str # CLI command installed in the image + args: list[str] = [] + env: dict[str, str] = {} # supports {{dial_bucket_path}} template + enabled: bool = True + pool_ttl_minutes: int = 10 # ignored in container model (single-entry pool) + allowed_tools: list[str] | None = None +``` + +The `{{dial_bucket_path}}` substitution is performed in the initializer before the subprocess is +spawned — it is a plain string replacement, not a Jinja template. + +--- + +## 2. Connection Manager Protocol + +**File:** `src/quickapp/mcp_tooling/_abstract_mcp_connection_manager.py` + +`_MCPTool` currently has a concrete `_MCPConnectionManager` type annotation. To support both +HTTP and stdio managers without changing `_MCPTool`'s logic, extract a minimal Protocol: + +```python +from typing import Protocol +from mcp import Tool +from mcp.types import CallToolResult + + +class MCPConnectionManagerProtocol(Protocol): + async def get_tools_list(self) -> list[Tool]: ... + async def call_mcp_tool(self, tool_name: str, **kwargs) -> CallToolResult: ... +``` + +Both `_MCPConnectionManager` (HTTP) and `_StdioMCPConnectionManager` (stdio) satisfy this protocol +structurally — no changes to the existing class are needed. + +**Modification in `_mcp_tool.py`:** change the `connection_manager` constructor parameter type +from `_MCPConnectionManager` to `MCPConnectionManagerProtocol`. + +--- + +## 3. Stdio Connection Manager + +**File:** `src/quickapp/mcp_tooling/_stdio_mcp_connection_manager.py` + +The key difference from `_MCPConnectionManager`: the session must stay open across calls. +HTTP transports are stateless; stdio is a live process — closing the session kills it. + +```python +import logging +from contextlib import AsyncExitStack +from datetime import timedelta + +from mcp import ClientSession, Tool +from mcp.client.stdio import StdioServerParameters, stdio_client +from mcp.types import CallToolResult + +logger = logging.getLogger(__name__) + + +class _StdioMCPConnectionManager: + """MCP connection manager for stdio subprocess providers (file-based memory servers). + + Holds a single long-lived ClientSession for the lifetime of this manager instance. + Call start() before use; call stop() to cleanly terminate the subprocess. + """ + + def __init__(self, params: StdioServerParameters, timeout_seconds: float = 30.0): + self._params = params + self._timeout = timeout_seconds + self._exit_stack: AsyncExitStack | None = None + self._session: ClientSession | None = None + + async def start(self) -> None: + """Spawn the subprocess and initialize the MCP session.""" + self._exit_stack = AsyncExitStack() + read, write = await self._exit_stack.enter_async_context(stdio_client(self._params)) + self._session = await self._exit_stack.enter_async_context(ClientSession(read, write)) + await self._session.initialize() + logger.debug("Stdio MCP session started: %s", self._params.command) + + async def stop(self) -> None: + """Close the session and terminate the subprocess.""" + if self._exit_stack: + await self._exit_stack.aclose() + self._exit_stack = None + self._session = None + logger.debug("Stdio MCP session stopped: %s", self._params.command) + + async def get_tools_list(self) -> list[Tool]: + assert self._session is not None, "start() must be called before get_tools_list()" + result = await self._session.list_tools() + return result.tools + + async def call_mcp_tool(self, tool_name: str, **kwargs) -> CallToolResult: + assert self._session is not None, "start() must be called before call_mcp_tool()" + timeout = timedelta(seconds=self._timeout) + return await self._session.call_tool( + tool_name, kwargs or None, read_timeout_seconds=timeout + ) +``` + +--- + +## 4. Subprocess Pool + +**File:** `src/quickapp/mcp_tooling/_subprocess_pool.py` + +One pool instance lives as a DI singleton for the QuickApps process. It maps +`(user_id, toolset_name)` → `_StdioMCPConnectionManager` with TTL eviction. + +```python +import asyncio +import logging +import time +from dataclasses import dataclass, field + +from mcp.client.stdio import StdioServerParameters + +from ._stdio_mcp_connection_manager import _StdioMCPConnectionManager + +logger = logging.getLogger(__name__) + +_PoolKey = tuple[str, str] # (user_id, toolset_name) + + +@dataclass +class _PoolEntry: + manager: _StdioMCPConnectionManager + expires_at: float # monotonic time + lock: asyncio.Lock = field(default_factory=asyncio.Lock) + + +class SubprocessPool: + """Per-user subprocess pool for stdio MCP memory providers. + + In the shared-service model, one pool instance is shared across all requests. + In the container model (one container = one user), the pool will always have + exactly one entry that never expires — pool_ttl_minutes can be set to a very + large value or the entry simply never evicts. + """ + + def __init__(self) -> None: + self._entries: dict[_PoolKey, _PoolEntry] = {} + self._global_lock = asyncio.Lock() + + async def get_or_create( + self, + user_id: str, + toolset_name: str, + params: StdioServerParameters, + ttl_seconds: float, + timeout_seconds: float = 30.0, + ) -> _StdioMCPConnectionManager: + key: _PoolKey = (user_id, toolset_name) + now = time.monotonic() + + async with self._global_lock: + entry = self._entries.get(key) + if entry and time.monotonic() < entry.expires_at: + entry.expires_at = now + ttl_seconds # reset TTL on access + return entry.manager + + # Evict stale entry if present + if entry: + await self._evict(key, entry) + + manager = _StdioMCPConnectionManager(params, timeout_seconds=timeout_seconds) + await manager.start() + self._entries[key] = _PoolEntry( + manager=manager, + expires_at=now + ttl_seconds, + ) + return manager + + async def _evict(self, key: _PoolKey, entry: _PoolEntry) -> None: + try: + await entry.manager.stop() + except Exception: + logger.warning("Error stopping subprocess for key %s", key, exc_info=True) + self._entries.pop(key, None) + + async def evict_expired(self) -> None: + """Evict all expired entries. Call periodically from a background task.""" + now = time.monotonic() + async with self._global_lock: + expired = [k for k, e in self._entries.items() if now >= e.expires_at] + for key in expired: + await self._evict(key, self._entries[key]) +``` + +> **Background eviction:** A periodic task should call `pool.evict_expired()` every few minutes. +> This can be wired in `app.py` as a background asyncio task or a FastAPI `lifespan` handler. + +--- + +## 5. Initializer Branch + +**File:** `src/quickapp/mcp_tooling/_mcp_tool_initializer.py` + +Add a new branch in `_process_toolset()` for `SubprocessMCPToolSet`. The branch sits alongside +the existing `DialMCPToolSet` check. All downstream logic (tool listing → `_MCPTool` creation → +`mcp_context.extend_tools`) is identical. + +```python +# New import at top of file +from quickapp.config.toolsets.subprocess_mcp import SubprocessMCPToolSet +from mcp.client.stdio import StdioServerParameters +from ._subprocess_pool import SubprocessPool + +# In __init__, inject the pool: +# subprocess_pool: SubprocessPool + +async def _process_toolset(self, toolset_info: MCPToolSet | DialMCPToolSet | SubprocessMCPToolSet) -> None: + if not toolset_info.enabled: + return + + # --- existing DialMCPToolSet branch (unchanged) --- + if isinstance(toolset_info, DialMCPToolSet): + ... # existing code + + # --- new SubprocessMCPToolSet branch --- + elif isinstance(toolset_info, SubprocessMCPToolSet): + bucket_path = await self._resolve_bucket_path() + env = { + k: v.replace("{{dial_bucket_path}}", bucket_path) + for k, v in toolset_info.env.items() + } + params = StdioServerParameters( + command=toolset_info.command, + args=toolset_info.args, + env=env, + ) + connection_manager = await self.__subprocess_pool.get_or_create( + user_id=self._current_user_id(), + toolset_name=toolset_info.name, + params=params, + ttl_seconds=toolset_info.pool_ttl_minutes * 60, + ) + tools = await connection_manager.get_tools_list() + + # --- existing MCPToolSet branch (unchanged) --- + else: + connection_manager = self.__connection_manager_builder.build(toolset_info=toolset_info) + tools = await connection_manager.get_tools_list() + + # --- common tail: filter + build _MCPTool instances (unchanged) --- + if toolset_info.allowed_tools: + tools = [t for t in tools if t.name in toolset_info.allowed_tools] + ... +``` + +`_resolve_bucket_path()` and `_current_user_id()` need access to the DIAL client and the +request-scoped user identity — both are already injectable via the existing DI graph. + +--- + +## 6. DI Wiring + +**File:** `src/quickapp/mcp_tooling/mcp_module.py` + +Bind `SubprocessPool` as a singleton so the pool survives across requests: + +```python +binder.bind(SubprocessPool, to=SubprocessPool, scope=singleton) +``` + +Add `SubprocessPool` as a constructor parameter of `_MCPToolInitializer`. + +--- + +## What Does NOT Change + +| Component | Why unchanged | +|---|---| +| `_MCPConnectionManager` (HTTP) | No modification — existing HTTP transports are unaffected | +| `_MCPTool` logic | Only the type annotation on `connection_manager` changes | +| `StagedBaseTool` | No modification — tool execution path is transport-agnostic | +| Orchestrator / hooks | No modification | +| Existing config models (`MCPToolSet`, `DialMCPToolSet`) | No modification | + +--- + +## Data Flow Summary + +``` +Request arrives (user A, toolset: basic-memory) + │ + ├─ _MCPToolInitializer._process_toolset(SubprocessMCPToolSet) + │ ├─ resolve bucket_path ← dial_client.bucket.get_raw() + │ ├─ substitute {{dial_bucket_path}} in env + │ ├─ SubprocessPool.get_or_create(user_id="A", toolset_name="memory", ...) + │ │ ├─ pool hit → return existing _StdioMCPConnectionManager (TTL reset) + │ │ └─ pool miss → spawn "basic-memory mcp", initialize session, store in pool + │ ├─ connection_manager.get_tools_list() → list[Tool] + │ └─ build _MCPTool per tool → mcp_context.extend_tools(...) + │ + ├─ Orchestrator loop: LLM calls e.g. "memory_search" + │ └─ _MCPTool._run_in_stage_async() + │ └─ connection_manager.call_mcp_tool("search", query="...") + │ └─ _StdioMCPConnectionManager: session.call_tool(...) ← reuses live session + │ + └─ Response returned; subprocess stays alive in pool until TTL expires +``` + +--- + +## Open Questions (Remaining) + +1. **DIAL bucket local path:** Confirmed assumption above — needs verification with the DIAL Files + team that `bucket_resp.appdata` / `bucket_resp.bucket` is a local filesystem path accessible + from within the QuickApps process/container. + +2. **`_current_user_id()` in initializer:** The initializer needs the request-scoped user identity + to key the pool. Confirm how user identity is currently surfaced in the DI graph at request scope + (likely via `DIAL_API_KEY` or a dedicated request-scoped binding). + +3. **Background eviction task:** How should `pool.evict_expired()` be wired? Options: + - FastAPI `lifespan` background task (periodic `asyncio.sleep` loop) + - Triggered on every `get_or_create` call (lazy eviction, simpler) + +4. **mem0 local mode:** Still needs a custom `~100-line` MCP wrapper. Defer until there is + operator demand — basic-memory covers most use cases without the LLM write dependency. + +5. **Obsidian vault:** CLI arg instead of env var for the vault path. The `SubprocessMCPToolSet` + `args` field supports this — `{{dial_bucket_path}}` substitution in `args` needs the same + template logic applied to `env`. Minor extension. diff --git a/docs/designs/external_memory/file-based-providers.md b/docs/designs/external_memory/file-based-providers.md new file mode 100644 index 00000000..92c47587 --- /dev/null +++ b/docs/designs/external_memory/file-based-providers.md @@ -0,0 +1,443 @@ +# File-Based Memory Providers + +- **Status:** Draft +- **Related:** [Memory Provider Classification](memory-provider-classification.md) + +--- + +## Scope + +This document covers memory providers that store data in local files and expose an MCP interface. +Their defining property: isolation is structural — point the provider at user A's directory and it is +physically incapable of reading user B's data. + +QuickApps already uses this model for its own memory (LanceDB writes `memory.lance/` into the user's +DIAL bucket). All file-based external providers follow the same pattern. + +--- + +## Providers + +### 1. Official MCP Memory + +**Maintained by:** Anthropic / MCP steering group +**Repo:** `modelcontextprotocol/servers` — `src/memory/` +**Runtime:** Node.js (`npx -y @modelcontextprotocol/server-memory`) + +**Storage:** Single JSONL file. Every entity, relation, and observation is one line. + +**Data path config:** + +``` +MEMORY_FILE_PATH=/path/to/memory.jsonl (default: ./memory.jsonl) +``` + +**MCP tools:** + +| Tool | Key parameters | Description | +|---|---|---| +| `create_entities` | `entities[]` | Add entities with name, type, observations | +| `create_relations` | `relations[]` | Add directed edges between entities | +| `add_observations` | `entityName`, `contents[]` | Append facts to an existing entity | +| `delete_entities` | `entityNames[]` | Remove entities and their relations | +| `delete_observations` | `entityName`, `observations[]` | Remove specific facts | +| `delete_relations` | `relations[]` | Remove specific edges | +| `read_graph` | — | Return the **entire** graph as JSON | +| `search_nodes` | `query: str` | Keyword search across entity names, types, observations | +| `open_nodes` | `names[]` | Retrieve specific entities by name | + +**Search type:** Keyword-only. `search_nodes` does substring matching on text fields — no embeddings, +no semantic similarity. + +**LLM dependency:** None. + +**Proactive injection:** `read_graph` returns the full graph. Suitable for `on_request_start` hook +only if the memory is small (entire graph is injected). No `get_context` / top-N tool exists. + +**Notes:** +- Simplest possible implementation; good for basic use cases and prototyping. +- No concept of memory importance or recency — everything is equally ranked. +- Duplicate relations are silently skipped. Deleting non-existent items silently succeeds. +- Single file means the entire memory is read/written on every operation — does not scale to + thousands of facts, but fine for personal conversation memory. + +--- + +### 2. basic-memory + +**Maintained by:** Basic Machines Co. +**Repo:** `basicmachines-co/basic-memory` +**Runtime:** Python (`uv tool install basic-memory`, then `basic-memory mcp`) + +**Storage:** Markdown files in a vault directory + SQLite index (`~/.basic-memory/` by default). +Files are human-readable and can be version-controlled. + +**Data path config:** + +``` +BASIC_MEMORY_HOME=/path/to/vault # overrides default ~/basic-memory +BASIC_MEMORY_CONFIG_DIR=/path/to/cfg # isolates config dir (useful per-user/per-process) +``` + +Projects can also be registered at startup via `basic-memory project add `. +The project is **selected at server startup** — it cannot be switched per-request in a running process. + +**MCP tools (selected):** + +| Tool | Description | +|---|---| +| `write_note` | Create or overwrite a Markdown note | +| `read_note` | Read a note by title or permalink | +| `edit_note` | Partial update to a note section | +| `delete_note` | Remove a note | +| `search` / `search_notes` | Hybrid search (semantic + full-text) | +| `recent_activity` | Most recently modified notes | +| `list_directory` | Browse vault structure | +| `build_context` | Follow `memory://` wikilinks recursively, building a context blob | +| `get_current_project` | Returns the active project name and path | + +**Search type:** Hybrid — FastEmbed semantic embeddings (local, no external API) + SQLite full-text. +FastEmbed model (`bge-small-en-v1.5` by default) runs on CPU; first run downloads the model (~50 MB). + +**LLM dependency:** None for storage. Embeddings are local (FastEmbed). + +**Proactive injection:** `recent_activity` returns the most recently modified notes — usable as a +no-query "what's relevant now" injection point. Alternatively, `search` with a broad query. + +**Notes:** +- The richest file-based option: semantic search + graph navigation + human-readable files. +- Notes structure aligns well with conversation memory (title = topic, body = observations). +- No built-in `user_id`; isolation is entirely via data path. +- FastEmbed model download adds ~1–3 s to first startup per installation. +- `build_context` is a powerful tool for multi-hop retrieval — the agent can traverse related notes. + +--- + +### 3. Obsidian Vault (filesystem-direct MCP) + +**Context:** An Obsidian vault is a plain directory of Markdown files with an optional `.obsidian/` +metadata folder. No Obsidian desktop app is required to read or write vault files — they are +ordinary files on disk. + +Multiple MCP servers access vault files directly: + +| Server | Repo | Notes | +|---|---|---| +| **obsidian-mcp** | `StevenStavrakis/obsidian-mcp` | 12 tools, vault path as CLI arg, no app needed | +| **seekstone** | — | "575× smaller payloads, no app required" | +| **vault-cortex** | — | Plugin-free, full vault access | + +**Data path config** (obsidian-mcp as reference): + +``` +# path passed as CLI argument: +npx -y obsidian-mcp /absolute/path/to/vault +``` + +**MCP tools (obsidian-mcp):** + +| Tool | Description | +|---|---| +| `read-note` | Read note contents | +| `create-note` | Create a new note | +| `edit-note` | Modify existing note | +| `delete-note` | Remove a note | +| `move-note` | Rename/relocate | +| `search-vault` | Full-text search across all notes | +| `add-tags` / `remove-tags` | Tag management | +| `list-available-vaults` | Multi-vault support | + +**Search type:** Full-text (keyword). No semantic search in standard servers; some third-party servers +add embeddings. + +**LLM dependency:** None. + +**Proactive injection:** No direct "get recent / get all" tool in standard servers; would need a +`search-vault` call with an empty or broad query. + +**Relationship to basic-memory:** Conceptually very similar — both are Markdown-based knowledge +graphs. Key differences: + +| | basic-memory | Obsidian vault | +|---|---|---| +| Linking style | `memory://` permalinks | Wikilinks `[[Note Name]]` | +| Search | Hybrid (semantic + full-text) | Full-text only (standard servers) | +| Index | SQLite | File system only | +| Primary audience | AI memory tool | Human note-taking tool | +| Use case in QuickApps | Fresh conversation memory | Access to user's existing knowledge base | + +**Notes:** +- Best fit when the user already maintains an Obsidian vault and wants the agent to read/write it. +- As a **from-scratch** conversation memory store, basic-memory is a better choice (richer search, + AI-native structure). +- Vault path is a positional CLI argument, not an env var — requires different startup wiring than + the other providers. + +--- + +### 4. mem0 (local / Chroma + SQLite) + +**Maintained by:** mem0ai +**Repo:** `mem0ai/mem0` +**Runtime:** Python library; no standalone MCP server for local mode (see notes) + +**Storage:** Chroma vector DB (local files) + SQLite for history. Paths configurable via Python config +or `MEM0_DIR` env var. + +**Data path config:** + +```python +config = { + "vector_store": { + "provider": "chroma", + "config": { + "collection_name": "memories", + "path": "/path/to/chroma" # local Chroma data dir + } + } +} +# SQLite history: controlled by MEM0_DIR env var (default ~/.mem0/) +# or set directly: memory = Memory(config=config, history_db_path="/path/to/mem0.db") +``` + +``` +MEM0_DIR=/path/to/mem0-data # controls history.db location +``` + +**MCP situation:** The official `mem0-mcp-server` package connects to the **mem0 cloud API** and +requires `MEM0_API_KEY`. There is no official standalone MCP server for the open-source local mode. +Using mem0 locally in QuickApps requires either: + +- A **custom MCP wrapper** (~100 lines Python) that imports the `mem0` library and exposes tools +- Or running mem0's **self-hosted server** (Docker Compose with REST API) and connecting via HTTP — + but that is the pgvector path (Class 2), not file-based + +**Effective tools (if custom wrapper is written):** + +| Tool | Parameters | Description | +|---|---|---| +| `add_memory` | `content: str` | Store; triggers LLM fact extraction | +| `search_memories` | `query: str`, `limit?: int` | Semantic search via Chroma | +| `get_memories` | — | List all memories | +| `delete_memory` | `memory_id: str` | Delete by ID | + +**Search type:** Semantic (Chroma + embedding model). Default embedding: OpenAI `text-embedding-3-small` +(can be swapped to local model). + +**LLM dependency:** Yes, **required on every write**. mem0 calls an LLM to extract structured facts +from raw text before storing. This cannot be disabled without forking the library. Adds ~1–2 s latency +and API cost per `add_memory`. + +**Proactive injection:** `get_memories` lists all memories — usable as a no-query injection point. + +**Notes:** +- Richest semantic memory of the four, but the LLM write dependency is a significant operational cost. +- No official local MCP server — requires custom wrapper to use file-based mode. +- For most QuickApps use cases, basic-memory provides comparable quality with no LLM dependency. +- Relevant if operators need mem0's specific fact-extraction behaviour (deduplication, contradiction + resolution, structured entity graph alongside vector store). + +--- + +## Summary Comparison + +| | Official MCP Memory | basic-memory | Obsidian vault | mem0 (local) | +|---|---|---|---|---| +| **Storage** | Single JSONL file | Markdown + SQLite | Markdown directory | Chroma + SQLite | +| **Path config** | `MEMORY_FILE_PATH` | `BASIC_MEMORY_HOME` | CLI arg (`/path`) | `MEM0_DIR` / Python config | +| **Search** | Keyword | Hybrid (semantic + FTS) | Full-text | Semantic | +| **LLM on write** | No | No | No | Yes (required) | +| **Local embeddings** | No | Yes (FastEmbed, ~50 MB) | No | Optional (or OpenAI) | +| **Native MCP** | ✅ Yes | ✅ Yes | ✅ Yes | ❌ Needs custom wrapper | +| **`list`/`get_context` tool** | `read_graph` (full graph) | `recent_activity` | No standard tool | `get_memories` | +| **Runtime** | Node.js | Python (uv) | Node.js | Python | +| **Complexity** | Minimal | Low | Low | Medium + LLM setup | +| **Best for** | Prototyping, simple memory | General-purpose AI memory | Existing Obsidian users | Rich semantic memory | + +--- + +## How providers are distributed and run + +File-based providers are ordinary CLI tools published as packages — there is no source code to pull, +no build step, no custom deployment. They are installed into the QuickApps Docker image alongside +QuickApps itself, and the subprocess just calls the installed command by name. + +```dockerfile +# Dockerfile — QuickApps image +FROM python:3.13 + +# basic-memory — Python package, installs `basic-memory` CLI +RUN uv tool install basic-memory + +# Official MCP Memory — npm package, installs command in PATH +RUN npm install -g @modelcontextprotocol/server-memory + +# obsidian-mcp +RUN npm install -g obsidian-mcp +``` + +After that, starting a provider is just: + +```python +# MCP Python SDK — stdio_client spawns the process and speaks MCP over stdin/stdout +from mcp.client.stdio import stdio_client, StdioServerParameters + +params = StdioServerParameters( + command="basic-memory", # CLI command installed in the image + args=["mcp"], + env={"BASIC_MEMORY_HOME": bucket_path} # user's DIAL bucket path +) +async with stdio_client(params) as (read, write): + async with ClientSession(read, write) as session: + await session.initialize() + # agent calls tools through session normally +``` + +`stdio_client` spawns the subprocess, pipes stdin/stdout, and wraps everything in the MCP protocol. +This is exactly how Claude Desktop connects to MCP servers. No HTTP server, no DIAL deployment, +no wrapper needed — the provider runs as a local process. + +**The user's bucket path** is resolved at request time via the existing DIAL client API: + +```python +bucket_resp = await self.__dial_client.bucket.get_raw() +bucket_path = bucket_resp.appdata or bucket_resp.bucket +``` + +Isolation is structural: each subprocess gets a different `bucket_path` in its environment, so it +is physically incapable of reading another user's data. + +**Constraint:** the set of supported providers is determined by what is installed in the Docker image. +An operator cannot add an arbitrary provider without a new image build. This is an intentional +tradeoff — providers are vetted and pre-packaged, not pulled from the internet at runtime. + +--- + +## The Generic Integration Challenge + +### Only one real problem: subprocess lifecycle + +Given the above, the only remaining design question is how QuickApps manages the subprocess lifetime. + +**Shared-service model (current):** QuickApps serves many users from one process. A per-user +subprocess pool is needed — keyed by `(user_id, toolset_name)`, with TTL eviction. + +``` +User A — first request: + bucket_path = await dial_client.bucket.get_raw() + pool.get("user-A", "basic-memory") → miss + → spawn: basic-memory mcp (env: BASIC_MEMORY_HOME=) + → ClientSession.initialize() + → store in pool with TTL=10min + +User A — next request (within TTL): + pool.get("user-A", "basic-memory") → hit + → reuse session, reset TTL + +User A — idle > TTL: + → close session → subprocess exits + → evict from pool +``` + +**Container model (future):** one container = one user. The subprocess starts once when the +container starts and lives for its entire lifetime. No pool, no TTL, no keying by user_id — +just `stdio_client` started at toolset initialization and kept alive. + +The same `SubprocessMCPToolSet` implementation works in both models — the difference is only +whether a pool is needed. In the container model the pool has exactly one entry that never expires. + +--- + +## Generic Config Format (Sketch) + +```json +{ + "tool_sets": [{ + "type": "subprocess-mcp", + "name": "memory", + "command": "basic-memory", + "args": ["mcp"], + "env": { + "BASIC_MEMORY_HOME": "{{dial_bucket_path}}/memory/basic-memory" + }, + "pool_ttl_minutes": 10 + }] +} +``` + +For Official MCP Memory: + +```json +{ + "tool_sets": [{ + "type": "subprocess-mcp", + "name": "memory", + "command": "npx", + "args": ["-y", "@modelcontextprotocol/server-memory"], + "env": { + "MEMORY_FILE_PATH": "{{dial_bucket_path}}/memory/memory.jsonl" + }, + "pool_ttl_minutes": 10 + }] +} +``` + +Key points: +- `"type": "subprocess-mcp"` — new toolset type; spawns a local process via MCP stdio transport + instead of connecting to a remote HTTP endpoint. +- `{{dial_bucket_path}}` — template variable resolved at request time from + `dial_client.bucket.get_raw()`. +- `pool_ttl_minutes` — how long to keep the subprocess alive after the last request. In the + container model this field is irrelevant (process lives for the container lifetime). +- No skill, no adapter, no tool renaming — provider runs as-is with its native tool names. + +--- + +## Skills: are they needed at all? + +In Claude Code and Cursor, these providers work without any skill. The LLM sees the tool names and +their MCP descriptions and decides autonomously when to call `write_note`, `search`, `read_graph`, etc. +This is the standard MCP memory experience — no external instruction file, no normalized contract. + +QuickApps can work the same way: connect the provider as a toolset and let the LLM use the tools +based on their native descriptions. No skill required for the integration to function. + +A skill adds value only when a specific memory workflow must be **enforced**, not just suggested: + +| Behavior | Without skill | With skill | +|---|---|---| +| LLM calls search when it thinks it's needed | ✅ Works via LLM judgment | Same | +| LLM saves facts at its own discretion | ✅ Works via LLM judgment | Same | +| Always save facts at end of every turn | ❌ LLM may omit | ✅ Enforced by instruction | +| Always search memory before answering about user preferences | ❌ LLM may skip | ✅ Enforced | + +**Recommendation:** ship without a skill by default. Operators who want tighter control can add a +skill variant themselves via the existing `skills` config field. No normalized tool contract is +required — each provider's native tool names and descriptions are sufficient. + +This also eliminates the need for tool renaming, adapter layers, or any normalization infrastructure +for the file-based provider class. + +--- + +## Open Questions + +1. **DIAL bucket as local path:** How does `{{dial_bucket_path}}` resolve? Is the DIAL bucket always + accessible as a local filesystem path, or does QuickApps sync files to/from DIAL storage? The + answer determines whether subprocess-mcp is straightforward or requires a file-sync step. + +2. **Subprocess pool implementation:** Should this be a new core feature in QuickApps, or a + lightweight standalone module? What eviction policy (TTL, LRU, max size)? + +3. **basic-memory startup model:** `basic-memory mcp` starts a stdio MCP server. Does QuickApps' + existing MCP wiring support stdio transport, or only HTTP/SSE? If only HTTP, a shim is needed. + +4. **mem0 local wrapper:** Is it worth writing the ~100-line custom MCP wrapper for mem0 local mode, + or should mem0 integration always go through the Class 2 adapter (self-hosted server with pgvector)? + +5. **Obsidian vault vs. basic-memory:** Obsidian vault MCP is the right tool when users bring their + own vault. basic-memory is better for fresh-start AI memory. Should QuickApps support both, or + treat Obsidian as a knowledge-base toolset (not a memory provider)? + +6. **Optional skill shape:** If an operator wants to enforce specific memory behaviors via a skill, + should QuickApps ship a default per-provider skill template, or leave it entirely to the operator? diff --git a/docs/designs/external_memory/memory-provider-classification.md b/docs/designs/external_memory/memory-provider-classification.md new file mode 100644 index 00000000..7785cc56 --- /dev/null +++ b/docs/designs/external_memory/memory-provider-classification.md @@ -0,0 +1,315 @@ +# Memory Provider Classification + +- **Status:** Draft +- **Related:** [External Memory Integration](external-memory-integration.md) + +--- + +## Purpose + +The external memory integration document established that there is no single generic solution for all +memory providers — isolation model, storage topology, and deployment assumptions differ too much across +providers. This document classifies known open-source memory providers into integration classes, with a +proposed generic solution for each class. + +--- + +## Note on hooks and retrieval patterns + +QuickApps' current memory implementation uses an `on_request_start` hook to proactively inject memories +into the conversation context before the first LLM turn. **This is not a requirement for external memory +integration.** Hooks are one optional pattern; the standard alternative is reactive retrieval — the LLM +calls memory tools (search, list) when it decides it needs past context, exactly as every other MCP +memory client (Claude Code, Cursor, etc.) works. + +Both patterns are valid and can be mixed: + +| Pattern | How it works | When to use | +|---|---|---| +| **Reactive** | LLM calls `memory_search` / `memory_get_context` when it needs context | Simpler integration; works with every provider out of the box | +| **Proactive (hook)** | `on_request_start` hook calls a retrieval tool and injects result before first turn | Guarantees relevant context is always present; requires a `list`/`get_context` tool on the provider | + +The classification below does not assume proactive injection. Where relevant it notes whether the +provider's tool set supports it. + +--- + +## Classification Dimensions + +The classification uses four dimensions that determine the integration effort in QuickApps: + +| Dimension | Why it matters | +|---|---| +| **Isolation model** | Determines what user-isolation work QuickApps must do | +| **Storage paradigm** | Determines how user data is persisted and whether DIAL bucket storage is usable | +| **Deployment topology** | Determines how the service is positioned relative to QuickApps | +| **LLM dependency** | Determines operator cost and operational complexity | + +--- + +## Class 1 — File-Based Personal Providers + +### Characteristics + +- Store all data in local files or an embedded SQLite/file-based vector store +- Designed for **one user, one instance** — no multi-tenancy concept +- Zero external service dependencies — start/stop like any process +- Data path is configurable via an environment variable + +### Providers + +| Provider | Storage | Search | LLM needed? | MCP native? | +|---|---|---|---|---| +| **Official MCP Memory** | Single JSONL file (`memory.jsonl`) | Keyword (entity/observation text) | No | ✅ Yes | +| **basic-memory** | Markdown files + SQLite / Postgres | Hybrid (FastEmbed semantic + full-text) | No | ✅ Yes | +| **mem0** (Chroma + SQLite config) | Chroma vector DB + SQLite (both on local disk) | Semantic | Yes — LLM call on `add_memory` for fact extraction | ✅ Yes (via mem0 MCP server) | + +### Fit with deployment models + +| Model | Fit | Notes | +|---|---|---| +| **Model 1 — shared service (current)** | ✅ Viable today | QuickApps already has per-user access to DIAL file storage. The MCP server is configured to use a per-user DIAL bucket path — isolation is structural, not credential-based | +| **Model 2 — containers** | ✅ Natural fit | Same principle; the DIAL bucket is mounted as a volume. No `user_id` needed in either case | + +The critical point: **DIAL already provides per-user isolated storage.** Any file-based MCP server that +accepts a configurable data path gets user isolation for free by pointing at the user's DIAL bucket. This +is exactly how QuickApps' own memory implementation (LanceDB) already works — it writes `memory.lance/` +into the user's DIAL bucket today, without containers. + +### Generic solution + +The MCP server is started (or addressed) with a data path that resolves to this user's DIAL file storage. +No `user_id` parameter, no adapter, no credential forwarding. The DIAL storage layer enforces isolation. + +``` +Request arrives (user A) + ├── QuickApps resolves user A's DIAL bucket path + ├── Starts / connects to MCP memory server with data_path = + │ (basic-memory vault, Official MCP Memory JSONL, mem0 Chroma dir, …) + └── Agent calls memory tools reactively — or hook pre-injects if configured +``` + +Config sketch (same shape for any file-based provider): + +```json +{ + "tool_sets": [{ + "type": "dial-mcp", + "dial_id": "basic-memory", + "env": { "BASIC_MEMORY_VAULT": "{{dial_bucket_path}}/memory/basic-memory" } + }] +} +``` + +No adapter needed. No normalized tool contract required unless proactive hook injection is desired +(in which case the provider must expose a `list` / `get_context` tool). + +### Open questions for this class + +- How does QuickApps launch a per-user MCP server process (or route to a per-user instance)? Today, + `dial-mcp` connects to a pre-deployed DIAL service — per-user process lifecycle is not yet modelled. +- mem0's LLM fact extraction: route through DIAL's LLM gateway or require separate endpoint config? +- If proactive hook injection is used: provider must expose a no-argument `list`/`get_context` tool; + not all Class 1 providers have this (Official MCP Memory uses `read_graph` which returns the full graph). + +--- + +## Class 2 — Multi-Tenant Server Services + +### Characteristics + +- A **single service instance** serves all users simultaneously +- User isolation is explicit: each API call carries a `user_id` / `namespace` parameter +- Backed by a persistent vector store (pgvector, Qdrant, proprietary) that partitions data per user +- Designed for server deployment — Docker Compose, Kubernetes, or a managed cloud + +### Providers + +| Provider | Storage | Search | LLM needed? | MCP native? | Multi-tenancy | +|---|---|---|---|---|---| +| **mem0** (default — pgvector / Qdrant) | PostgreSQL + pgvector **or** Qdrant | Semantic + BM25 hybrid | Yes — LLM call on `add_memory` for fact extraction | ✅ Yes | `user_id` / `agent_id` / `run_id` | +| **Supermemory** (self-hosted) | Proprietary (Ollama local or external LLM) | Hybrid (RAG + personalized memory) | Optional — local Ollama or external LLM | ✅ Yes (`https://mcp.supermemory.ai/mcp`) | `containerTag` isolation | + +### Fit with deployment models + +| Model | Fit | Notes | +|---|---|---| +| **Model 1 — shared service** | ✅ Natural fit | Designed for this. One deployed instance, `user_id` scopes each user's memories | +| **Model 2 — containers** | ❌ Wasteful | Running a fresh server-class database per container is over-engineered; use Class 1 instead | + +### Generic solution (Model 1 — adapter pattern) + +QuickApps connects through a thin **memory adapter** (one per provider), which reads the user identity +from the `X-DIAL-API-Key` header and injects it as `user_id` into every API call (see Approach A in the +integration doc). The adapter exposes a normalized tool contract identical across all providers: + +| Tool | Parameters | Notes | +|---|---|---| +| `memory_store` | `content: str` | Core write tool — always required | +| `memory_search` | `query: str`, `limit?: int` | Core search tool — always required | +| `memory_get_context` | *(none)* | Optional — top-N memories without a query; needed only if proactive hook injection is configured | + +No skill is required — the LLM understands the adapter's tools from their MCP descriptions, exactly as +in Claude Code and Cursor. A skill can be added optionally by operators who need to enforce specific +memory behaviors (e.g. "always save facts at end of every turn"). + +``` +QuickApp agent + └── memory_store / memory_search / memory_get_context + ↓ + Class 2 Adapter ← reads X-DIAL-API-Key → resolves user_id + ↓ + mem0 / Supermemory backend ← calls API with user_id + ↓ + pgvector / Qdrant / proprietary storage ← partitioned per user_id +``` + +### Open questions for this class + +- mem0's LLM cost: every `add_memory` triggers ~1–2 s LLM inference + API charges. Should the adapter + expose a `verbatim: true` mode that bypasses fact extraction? +- Supermemory self-hosted: requires either Ollama (local LLM) or external API. Operator must supply. +- Adapter maintenance: build one adapter per provider (~150 lines Python each). Org question: same repo + as QuickApps, or separate adapters repo? + +--- + +## Class 3 — Graph-Database Services + +### Characteristics + +- Memory is modelled as a **temporal knowledge graph** (entities + relations + time-validity windows) +- Requires a dedicated graph database (**Neo4j** or **FalkorDB**) as the storage engine +- Provides the richest retrieval semantics (graph traversal + semantic + BM25) but at high operational cost +- Multi-tenancy is possible but requires developer-managed namespace/group scoping in the graph + +### Providers + +| Provider | Storage | Search | LLM needed? | MCP native? | Multi-tenancy | +|---|---|---|---|---|---| +| **Graphiti** | Neo4j or FalkorDB (no file-based option) | Semantic + BM25 + graph traversal | Yes — LLM call for entity/relation extraction | ✅ Yes | Group-level scoping (developer-managed) | +| **Zep** (commercial) | Proprietary (built on Graphiti) | Same as Graphiti | Yes | ✅ Yes | Built-in (managed SaaS) | + +### Fit with deployment models + +| Model | Fit | Notes | +|---|---|---| +| **Model 1 — shared service** | ⚠️ Possible but complex | Shared Neo4j with group-scoped namespaces per user; developer must implement isolation | +| **Model 2 — containers** | ❌ No fit | Neo4j / FalkorDB cannot run per-user as a lightweight file-based sidecar | + +### Generic solution + +Class 3 is **not a candidate for a generic solution** in the same sense as Classes 1 and 2: + +- The graph DB infrastructure cost is prohibitive for a lightweight integration. +- User isolation requires custom developer work at the graph level — no simple `user_id` parameter. +- The LLM dependency (entity extraction) adds latency and cost on every write. + +**Recommendation:** Defer Class 3 integration until there is explicit operator demand. The adapter +pattern (Class 2) could technically wrap Graphiti, but the operational overhead undermines the value. + +If Graphiti is ever needed, the integration path is: shared Graphiti instance → group/namespace scoping +per DIAL user → adapter layer exposing a consistent tool interface. + +--- + +## Class 4 — Agent-Framework-Embedded Memory + +### Characteristics + +- Memory is an internal subsystem of a **full agent framework** — it cannot be detached and run as a + standalone service +- The framework controls the LLM loop, tool calling, and memory management as a unified system +- Exposing memory as an external API is non-trivial or explicitly unsupported + +### Providers + +| Provider | Storage | Notes | +|---|---|---| +| **Letta / MemGPT** | Relational DB (PostgreSQL via alembic) | Memory blocks are part of the Letta agent runtime; no standalone memory MCP server | +| **LangMem** | Pluggable (InMemory / AsyncPostgresStore) | Library for LangGraph — memory is wired into the graph; no standalone service | +| **Agno** | Operator-configured | Built-in memory within the Agno agent platform; not extractable | + +### Fit with deployment models + +Neither model applies — these are **competing agent platforms**, not composable memory providers. + +### Generic solution + +**Out of scope.** Integrating these would mean replacing QuickApps' own orchestration with a third-party +framework, which is the inverse of the QuickApps design goal (operator chooses the memory backend, not +the entire agent stack). + +--- + +## Class 5 — Document Retrieval / RAG Systems + +### Characteristics + +- Designed to **index and retrieve documents**, not to accumulate conversation-level memories +- Typically batch-ingested content (files, web pages, emails) rather than agent observations +- May overlap with memory conceptually but serve a different interaction pattern + +### Providers + +| Provider | Storage | Notes | +|---|---|---| +| **Kernel Memory** (.NET) | Azure AI Search, Postgres, Elasticsearch, Qdrant, Redis, local | Multi-tenancy via security filters; no MCP; ChatGPT plugin + Semantic Kernel integration | +| **OpenSearch ml-commons** | OpenSearch cluster | Full text + vector search over indexed documents; requires OpenSearch infrastructure | +| **mcp-apple-notes** | LanceDB (local) | Semantic search over a personal Apple Notes vault — personal RAG, not memory | + +### Fit with deployment models + +These providers can be wired into QuickApps as **knowledge base tool sets** (already supported via +`dial-mcp` or `mcp_http`), but they serve a different purpose from the memory hooks discussed in the +integration doc. + +### Generic solution + +**Out of scope for the memory integration pattern.** Use existing toolset configuration to connect +document retrieval systems. The `on_request_start` hook pattern does not apply here — retrieval should +be reactive (agent calls search when needed), not proactively injected at turn start. + +--- + +## Summary + +| Class | Examples | Model 1 fit | Model 2 fit | Generic solution | +|---|---|---|---|---| +| **1 — File-based personal** | Official MCP Memory, basic-memory, mem0 (Chroma) | ✅ Viable today | ✅ Natural | MCP server pointed at user's DIAL bucket path; isolation is structural | +| **2 — Multi-tenant server** | mem0 (pgvector), Supermemory | ✅ Natural | ❌ Wasteful | Adapter propagates user_id from DIAL auth header; no skill required | +| **3 — Graph-database** | Graphiti, Zep | ⚠️ Complex | ❌ | No generic solution; per-operator custom integration if ever demanded | +| **4 — Framework-embedded** | Letta, LangMem, Agno | ❌ | ❌ | Out of scope — competing agent platforms | +| **5 — Document RAG** | Kernel Memory, OpenSearch | n/a | n/a | Out of scope — use existing toolset config; not conversation memory | + +--- + +## Implications for the Integration Roadmap + +### Short term (current shared-service model) + +**Both Class 1 and Class 2 providers are viable today.** + +- **Class 1** (file-based): viable because QuickApps already has per-user DIAL bucket access, which is + exactly the isolation mechanism these providers need. The main open question is how QuickApps starts or + addresses a per-user MCP server process. This needs design work but no external infrastructure. +- **Class 2** (multi-tenant server): viable as a shared service with an adapter that propagates `user_id` + from the DIAL auth header. Requires deploying the provider (e.g. mem0 with pgvector) separately. + +For Class 1, no proactive hook injection is required — the LLM can call memory tools reactively. This +makes the initial integration even simpler: just expose the provider's MCP tools as a toolset. No skill +needed — the LLM reads the tools' native MCP descriptions, exactly as in Claude Code and Cursor. + +### Long term (container model) + +Class 1 providers become the natural default in the container model — the DIAL bucket is mounted as a +volume and the sidecar writes to it directly. Class 2 remains relevant for operators who run +shared-service deployments or require richer semantic search. + +### Which provider to ship first + +| Priority | File-based (Class 1) | Multi-tenant server (Class 2) | +|---|---|---| +| **First** | basic-memory — no LLM, Markdown files + SQLite, native MCP, hybrid search | mem0 (pgvector) — richest semantics, good self-hosting story | +| **Second** | Official MCP Memory — even simpler, keyword-only, zero dependencies | Supermemory — simpler API, optional local LLM | +| **Defer** | mem0 (Chroma) — LLM dependency adds operational complexity | Graphiti — high infra cost | diff --git a/docs/designs/external_memory/subprocess-mcp-implementation.md b/docs/designs/external_memory/subprocess-mcp-implementation.md new file mode 100644 index 00000000..f6e6b96c --- /dev/null +++ b/docs/designs/external_memory/subprocess-mcp-implementation.md @@ -0,0 +1,392 @@ +# subprocess-mcp: Implementation and Container Model Considerations + +- **Status:** Implemented (hypothesis validated) +- **Related:** [File-Based Providers](file-based-providers.md) | [Implementation Draft](file-based-providers-implementation-draft.md) + +--- + +## Scope + +This document covers: + +1. What was actually built — the concrete code implementing the `subprocess-mcp` toolset type. +2. What was validated — two providers tested end-to-end, confirming the generic approach. +3. What must be addressed when QuickApps adopts a per-user container deployment model. + +The implementation draft in `file-based-providers-implementation-draft.md` describes the full +production design (per-user subprocess pool with TTL eviction). This document describes the +**simplified version built to validate the hypothesis**, and flags what changes when containers ship. + +--- + +## What Was Built + +### Goal + +Validate the hypothesis: *running a file-based MCP memory server as a subprocess is a viable generic +mechanism for plugging in different memory providers — no HTTP server, no DIAL deployment, no adapter.* + +### Design Simplification (vs. Draft) + +The draft proposed a `SubprocessPool` keyed by `(user_id, toolset_name)` with TTL eviction, designed +for the shared-service model where many users share one QuickApps process. The implementation instead +uses a simpler `_StdioConnectionRegistry` keyed by `toolset_name` only: + +| | Draft | Implemented | +|---|---|---| +| Pool key | `(user_id, toolset_name)` | `toolset_name` | +| TTL eviction | Yes — idle subprocesses cleaned up | No — subprocess lives for the process lifetime | +| Multi-user isolation | Per-user subprocess instances | Container-level (one container = one user) | + +This simplification is **correct for the container model** where one QuickApps process serves exactly +one user. It would break in the shared-service model — that path requires the full pool design from +the draft. + +### New Files + +#### `src/quickapp/config/toolsets/subprocess_mcp.py` + +Config model for the `"type": "subprocess-mcp"` toolset: + +```python +class SubprocessMCPToolSet(BaseToolSet): + type: Literal["subprocess-mcp"] = Field(default="subprocess-mcp", ...) + name: str + command: str # CLI command installed in the Docker image + args: list[str] = [] + env: dict[str, str] = {} # supports {{dial_bucket_path}} template variable + allowed_tools: list[str] | None = None + attachment: AttachmentConfig + fallback_configuration: ToolFallbackConfig +``` + +Both `env` values and `args` entries support `{{dial_bucket_path}}` substitution, performed at +request time before the subprocess is spawned. + +#### `src/quickapp/mcp_tooling/_abstract_mcp_connection_manager.py` + +Structural Protocol shared by HTTP (`_MCPConnectionManager`) and stdio (`_StdioMCPConnectionManager`): + +```python +class MCPConnectionManagerProtocol(Protocol): + async def get_tools_list(self) -> list[Tool]: ... + async def call_mcp_tool(self, tool_name: str, **kwargs) -> CallToolResult: ... +``` + +#### `src/quickapp/mcp_tooling/_stdio_mcp_connection_manager.py` + +Long-lived stdio session manager. Unlike the HTTP manager (stateless, reconnects per call), this +holds a single open `ClientSession` for the lifetime of the manager — closing it terminates the +subprocess. + +Key implementation detail: `call_mcp_tool` passes `kwargs` directly (not `kwargs or None`). An +empty dict `{}` is falsy in Python, so the `or None` pattern would send `None` to the subprocess for +zero-argument tools, which is rejected by Node.js/Zod as `undefined`. See regression test +`test_call_mcp_tool_no_args_passes_empty_dict`. + +#### `src/quickapp/mcp_tooling/_stdio_connection_registry.py` + +Process-level singleton registry: `toolset_name` → `_StdioMCPConnectionManager`. On first access +for a given toolset name, spawns the subprocess and stores the manager. Subsequent requests reuse +the live session. + +### Modified Files + +**`mcp_tooling/_mcp_tool.py`** — `connection_manager` type annotation changed from +`_MCPConnectionManager` to `MCPConnectionManagerProtocol`. No logic changes. + +**`mcp_tooling/_mcp_tool_initializer.py`** — new `_process_subprocess_toolset()` method: + +```python +async def _process_subprocess_toolset(self, toolset_info: SubprocessMCPToolSet) -> None: + bucket_path = os.environ.get("DIAL_BUCKET_PATH", "") + def _sub(value: str) -> str: + return value.replace("{{dial_bucket_path}}", bucket_path) + env = {k: _sub(v) for k, v in toolset_info.env.items()} + args = [_sub(a) for a in toolset_info.args] + params = StdioServerParameters(command=toolset_info.command, args=args, env=env) + connection_manager = await self.__stdio_registry.get_or_create(label, params) + tools = await connection_manager.get_tools_list() + # filter by allowed_tools, build _MCPTool instances, extend context +``` + +`DIAL_BUCKET_PATH` is resolved from the process environment, not from the DIAL API. This means the +bucket path is the same for all requests in the process — correct for per-user containers, wrong for +shared-service. In the production shared-service implementation the draft describes resolving +bucket_path via `dial_client.bucket.get_raw()` per request. + +**`mcp_tooling/mcp_tooling_module.py`** — `_StdioConnectionRegistry` bound as singleton. + +**`Dockerfile`** — Node.js, npm, and MCP servers installed in the runtime stage: + +```dockerfile +RUN apk add --no-cache nodejs npm && \ + npm install -g @modelcontextprotocol/server-memory@2026.1.26 @modelcontextprotocol/server-filesystem && \ + npm cache clean --force && \ + rm -rf /root/.npm && \ + which mcp-server-memory && which mcp-server-filesystem +``` + +### `{{dial_bucket_path}}` Substitution + +The template variable is substituted in both `env` and `args` before the subprocess is spawned: + +| Config location | Example | +|---|---| +| `env` | `"MEMORY_FILE_PATH": "{{dial_bucket_path}}/memory.jsonl"` | +| `args` | `["{{dial_bucket_path}}/files"]` | + +--- + +## Hypothesis Validation + +Two structurally different providers were tested end-to-end in the local docker-compose environment. + +### Provider 1: `mcp-server-memory` + +```json +{ + "type": "subprocess-mcp", + "name": "memory", + "command": "mcp-server-memory", + "args": [], + "env": { "MEMORY_FILE_PATH": "{{dial_bucket_path}}/memory.jsonl" } +} +``` + +Result: subprocess spawned, session initialized, knowledge graph tools (`create_entities`, +`search_nodes`, `read_graph`, etc.) registered and callable. `memory.jsonl` written to the user's +bucket volume at `/data/bucket/memory.jsonl`. + +Bug encountered: `read_graph` (a zero-argument tool) returned `isError=true` with `expected object, +received undefined`. Root cause: `kwargs or None` in `call_mcp_tool` — `{}` is falsy, so the server +received `null` instead of `{}`. Fixed by removing `or None`. + +### Provider 2: `mcp-server-filesystem` + +```json +{ + "type": "subprocess-mcp", + "name": "filesystem", + "command": "mcp-server-filesystem", + "args": ["{{dial_bucket_path}}/files"], + "env": {} +} +``` + +Result: subprocess spawned, session initialized, filesystem tools (`list_directory`, `read_file`, +`write_file`, etc.) registered and callable. `write_file` successfully wrote +`/data/bucket/files/user_profile.json`. + +Issue encountered: `mcp-server-filesystem` validates all declared paths at startup and exits if any +do not exist. The `/data/bucket/files` directory must be created before the first request. See +[Pre-existing Directory Requirement](#pre-existing-directory-requirement) below. + +### Conclusion + +The same `subprocess-mcp` mechanism works for any stdio MCP server regardless of: +- What language it is implemented in (both tested providers are Node.js) +- How the data path is passed — `env` variable or positional CLI argument +- What tools it exposes — the tool list is fetched via `list_tools()` at initialization + +--- + +## Container Model: What Must Be Addressed + +### Current Assumption + +The implementation assumes one QuickApps container per user. The `_StdioConnectionRegistry` is a +process-level singleton keyed by toolset name only. This is correct in the container model and wrong +in the shared-service model. + +### Mounting the User's Bucket + +The DIAL bucket must be mounted as a volume into the QuickApps container. The container sees it as a +local filesystem path, which is passed to the MCP server via `DIAL_BUCKET_PATH`. + +Example docker-compose fragment (used in validation): + +```yaml +quick-apps: + environment: + DIAL_BUCKET_PATH: /data/bucket + volumes: + - ./docker_compose_files/quick-apps/bucket:/data/bucket +``` + +In production container orchestration, each user's container gets their own DIAL bucket mounted at +the same well-known path. The MCP server writes to subdirectories inside the bucket; data persists +across container restarts because the backing storage is the user's DIAL file storage. + +### Pre-existing Directory Requirement + +Some MCP servers validate their data paths at startup and refuse to start if the path does not exist. +`mcp-server-filesystem` exhibits this behaviour — passing a non-existent directory as a CLI argument +causes it to exit immediately, which results in `McpError: Connection closed` in QuickApps. + +Two options to handle this: + +**Option A — Pre-create directories in the image entrypoint.** +The container entrypoint script creates standard subdirectories under `$DIAL_BUCKET_PATH` before +starting uvicorn: + +```sh +# docker_entrypoint.sh (addition) +mkdir -p "${DIAL_BUCKET_PATH}/files" +``` + +This is the simplest fix. The downside is that the Dockerfile must know which subdirectories each +provider requires. + +**Option B — Catch `McpError: Connection closed` and retry after mkdir.** +In `_StdioConnectionRegistry.get_or_create`, if `start()` raises `McpError` whose message contains +`Connection closed`, inspect the `args` for paths and attempt to create them: + +```python +try: + await manager.start() +except McpError: + _ensure_arg_paths_exist(params.args) + await manager.start() +``` + +This is self-healing but requires the registry to know which arguments are filesystem paths — fragile +for the general case. + +**Recommendation:** Option A. The set of providers baked into the image is fixed at build time; the +required directories are known. Document the convention: providers that accept a data path via args +must declare the default subdirectory name in their Dockerfile comment. + +### Adding New Providers to the Image + +The set of available MCP servers is determined at image build time. To add a new provider: + +1. Install it in the `runtime` stage of the Dockerfile: + +```dockerfile +# Node.js package +npm install -g @scope/mcp-server-name + +# Python package (uv) +uv tool install basic-memory +``` + +2. Add a `which ` check to verify it landed in `PATH`. +3. If the provider requires a pre-existing directory, add `mkdir -p` to the entrypoint. +4. Add a sample app config entry to `docker_compose_files/core/configuration/applications.json`. + +Providers are never fetched at runtime — the image is the contract. An operator cannot configure +a provider that is not installed in the image without a new image build. This is intentional: +providers are vetted and pinned, not pulled from the internet on demand. + +### Alternative: Installing Providers Without Modifying the Dockerfile + +If modifying the Dockerfile is not possible (e.g. the image is owned by another team or released +separately), there are three approaches for delivering MCP server binaries to the container. + +#### Option A — Init container (Kubernetes) + +A dedicated init container installs providers into a shared `emptyDir` volume before the main +container starts. The main container adds the volume's `bin/` directory to `PATH` via an env var. + +```yaml +initContainers: + - name: install-mcp-providers + image: node:20-alpine + command: ["npm", "install", "-g", "--prefix", "/mcp-tools", + "@modelcontextprotocol/server-memory@2026.1.26"] + volumeMounts: + - name: mcp-tools + mountPath: /mcp-tools +containers: + - name: quick-apps + env: + - name: PATH + value: "/mcp-tools/bin:/usr/local/bin:/usr/bin:/bin" + volumeMounts: + - name: mcp-tools + mountPath: /mcp-tools +``` + +**Pros:** QuickApps image stays clean; provider versions are declared in the deployment manifest, +not baked into the image; no Dockerfile changes required. +**Cons:** Requires Node.js (or uv) in the init container image; `PATH` must be explicitly forwarded +to the main container. + +#### Option B — `npx` / `uvx` on first spawn + +Instead of pre-installing, run providers through `npx -y` (Node.js) or `uvx` (Python), which +download and cache the package on first invocation: + +```json +{ + "type": "subprocess-mcp", + "command": "npx", + "args": ["-y", "@modelcontextprotocol/server-memory@2026.1.26"] +} +``` + +**Pros:** Zero changes to the image (only Node.js or uv must be present); the provider version is +declared directly in the app config. +**Cons:** First start incurs a network request and download latency; does not work in air-gapped +environments; the npm cache lives inside the container and is lost on restart. + +#### Option C — Pre-populated volume mount + +The operator maintains a host path or persistent volume with pre-installed npm packages and mounts +it into the container. The container adds the volume's `bin/` to `PATH`. + +**Pros:** Fully offline after initial setup; providers can be updated without rebuilding or +redeploying the image. +**Cons:** Requires an out-of-band process to populate and update the volume; adds operational +complexity for the platform team. + +#### Recommendation + +| Scenario | Recommended option | +|---|---| +| Kubernetes, image not owned by this team | **A** — init container | +| Development / quick validation | **B** — `npx -y` / `uvx` | +| Air-gapped production, no Dockerfile access | **C** — pre-populated volume | +| Full control over image | Dockerfile (original approach) | + +### Graceful Subprocess Shutdown + +Currently, subprocesses spawned by `_StdioConnectionRegistry` are not explicitly stopped when the +QuickApps process shuts down. The OS terminates child processes when the parent exits, so there is +no data loss risk — but subprocess cleanup is not controlled. + +For production, add a FastAPI `lifespan` shutdown hook: + +```python +@asynccontextmanager +async def lifespan(app: FastAPI): + yield + await stdio_registry.stop_all() +``` + +### Subprocess Crash Recovery + +If a spawned subprocess crashes after initialization, the registry holds a dead +`_StdioMCPConnectionManager`. The next tool call will fail with a `McpError`. There is currently no +reconnect logic. + +For production, detect `McpError` in `_MCPTool._run_in_stage_async` and evict + restart the manager +via the registry on the next request. In the container model a crashed subprocess is likely a fatal +error (provider misconfiguration or corrupt data); crashing the container and letting the orchestrator +restart it may be the correct response rather than silent retry. + +--- + +## Summary: What Works Today vs. What Needs Work + +| Concern | Status | Notes | +|---|---|---| +| Generic subprocess-mcp toolset type | ✅ Done | `env` and `args` substitution both work | +| Provider installed via npm in Dockerfile | ✅ Done | `mcp-server-memory`, `mcp-server-filesystem` | +| Bucket volume mount | ✅ Done (in docker-compose) | Convention: `DIAL_BUCKET_PATH=/data/bucket` | +| Per-user isolation | ✅ By container boundary | Works because one container = one user | +| Pre-existing directory creation | ⚠️ Manual today | See Option A — add to entrypoint | +| Graceful subprocess shutdown | ⚠️ Missing | Add lifespan hook | +| Subprocess crash recovery | ⚠️ Missing | No reconnect; crash = dead registry entry | +| Shared-service model (multiple users, one process) | ❌ Not implemented | Requires full `SubprocessPool` from the draft |