roadmap.md

Roadmap — milestones, v1 cut-line, deferred items, licenses

Milestones

The build proceeds inside-out: start at the storage layer, port the indexer to match the Python prototype's behaviour exactly, then wire transports, then live mode. Each milestone has a concrete acceptance check.

M1 — Storage + indexer (foundation)

Crates: escurel-storage, escurel-md, escurel-index, escurel-embed (without runtime — stub embeddings first).

• LaneStore trait + FsStore implementation (dev-only).
• S3Store implementation — required for the substrate deployment target (S3 is the production default per locked decision 7). The substrate-target cut-line ships S3 as GA, not as a feature flag.
• Markdown parser (regex-based wikilink parser).
• Frontmatter parser (YAML; tolerant of the prototype's conventions).
• DuckDB schema migrations (pages, links, blocks, crdt_ops, crdt_snapshots, frontmatter_index).
• vss and fts extension load + HNSW + BM25 index creation on blocks.
• Pre-deployment retrieval-quality spike per ../adr/0001-duckdb-only-storage.md: build the 460-block evaluation harness against DuckDB vss + fts + RRF fusion; run the 10,120-block synonym-mutant stress corpus; meet the acceptance table in the ADR's Pre-deployment gate section. Outcome gates M2 work.
• update_page, rebuild, audit running end-to-end.

Acceptance. Port the 28-assertion e2e test from the Python prototype to Rust. All 28 assertions PASS with timings within 2× of the prototype's.

M2 — Embedding + retrieval

Crates: escurel-embed proper, plus search / resolve / expand / neighbours / list_skills / list_instances / run_stored_query over a direct Rust API (no transport yet).

• EmbeddingGemma loading via candle (CPU first; CUDA/Metal as feature flags).
• Gemini API adapter behind embedding.provider = gemini.
• Embed worker pool with the per-tenant queue.
• All seven read tools functional from a unit-test harness.

Acceptance. Agent-reproduction harness ports to Rust and produces the same 3/3 task success at ≤ 1.5 k tokens. EmbeddingGemma vs. the prototype's BGE-large quality difference logged but not gating.

M3 — Transports + auth + quotas

Crates: escurel-server gateway, escurel-auth, escurel-quota.

• axum HTTP server with MCP/JSON-RPC framing.
• WebSocket endpoint (without live mode yet — just presence and search subscriptions stubbed out).
• OIDC verification with JWKS caching.
• Token-bucket quotas wired in.
• escurel CLI as a thin MCP-over-HTTP client.

Acceptance. Run the cold-start verification through the CLI against a real running server. 8/8 queries correct under both policies. Validate OIDC against a Keycloak test instance.

M4 — Live CRDT mode + admin surface

Crates: escurel-crdt, plus the admin half of escurel-server.

• Loro adapter, LiveDoc actor per page, op log + snapshot persistence.
• WebSocket op streaming.
• open_session / apply_op / close_session over HTTP and the WebSocket op stream.
• Admin endpoints: tenant CRUD, export/import, attach_external, audit/rebuild streaming.
• Two-stage reconciler for external markdown edits.

Acceptance. Two CLI processes concurrently edit the same page over WS; merged state is consistent on both sides. Kill the server mid-session; on restart the op log replays cleanly and the next apply_op succeeds.

M5 — Observability + substrate deployment readiness + hardening

Crates: escurel-obs. Plus substrate-binding artefacts ship in this milestone (S3Store has moved to M1).

• OTel traces + metrics + JSON logs wired everywhere.
• /metrics Prometheus endpoint.
• Failure injection tests covering the recovery matrix from storage.md, including the cattle-node-loss → auto-rebuild-from-markdown path.
• License audit re-run; deps frozen.
• End-to-end deploy doc.
• Substrate-target artefacts (per ../deploy/substrate.md): the repo container image (Dockerfile → ghcr); the substrate-repo Kamal deploy contract (kamal/dz-escurel/deploy.yml — host-1 pin, STOP-FIRST, /data Volume) + apps/registry.yml row (external build, exposure, Secret Manager secrets); inclusion in the substrate Volume backup.

Acceptance. End-to-end smoke against three deploys: single-binary on a laptop (FS, no OTLP); systemd unit on a VM (FS, OTLP to local Tempo/Prometheus); substrate nonprod (S3 LaneStore = Hetzner Object Storage, OTLP to substrate collector, cattle-node loss → auto-rebuild). All three pass the prototype's e2e verification suite.

M6 — v1 ship

Cut a release. Tag v1.0.0. Publish operator docs.

M7 — Event-sourcing surface (post-v1)

escurel models memory as a triad — Events · Skills · Instances — where the current state of an instance is the projection of its event sequence, mediated by the skills that describe how to process each event. v1 ships the Skills + Instances legs and the event log via existing primitives; M7 makes the Event leg and the projection first-class, deliberately extending the v1 contract (each change ships with its spec/contract/ADR update):

• Events / inbox store. A dedicated events table (a real inbox queue) tightly bound to the page model: each event's label_skill links to the skill that knows how to process it, and instance_page_id links to the instance it belongs to once processed. Tools: capture_event, list_inbox, list_events, assign_event. (This is an intentional break of the v1 "events are ordinary instances; no new storage shape" invariant.)
• Frontmatter-link indexing so an instance's frontmatter relations (about:/derived_from:) are real backlinks.
• Historical state via CRDT snapshots: expand(as_of=T | version) re-materializes an instance's frontmatter+body at a past instant (extends the v1 rule that markdown instances ignore @version).
• Outbound webhook on new inbox items (the v1 transports are all inbound; this is a new platform surface).
• External-agent projection. The fold event→state is performed by an external agent (using the event's label_skill as context); the server stays automation-free, consistent with the v1 contract. The reference implementation is the escurel-demo-agent crate — it reads the inbox (notified by the capture webhook, or polling), routes each event to its instance (pre-flag, else a label_skill → instance table), and folds it in via assign_event. v1.5's in-server projection rules-engine remains out (see below).

The reference consumer is the escurel-explore event/instance workspace; the reference processor is the escurel-demo-agent crate.

M8 — Project-memory: the provenance graph (post-v1)

escurel evolves from a generic agent knowledge base into a persistent project-memory for data scientists/analysts, per ADR-0010. A long-running project holds two evolving networks — a knowledge graph (data → analysis → results → hypotheses) and an expectation graph (goals → priorities → constraints → success-criteria) — and most lost context comes from the expectation side. The contribution is a hypothesis-centric, expectation-aware memory that records why decisions were made, why paths were abandoned, and how expectations drifted.

Shipped as a non-breaking, opt-in layer:

• The project-memory skill pack — eleven first-class entity skills (Stakeholder, Goal, Expectation, Constraint, Priority, Success-Criterion, Hypothesis, Dataset, Analysis, Result, Decision) + an overview skill, distributed via the existing signed base-layer pack mechanism. Entities are ordinary skill pages (the PageType enum is untouched); provenance relations are typed frontmatter wikilinks (derived_from, motivated_by, supersedes, …) whose kind is carried by links.src_field.
• The resolved_links view — one derived DuckDB VIEW that resolves the slug-valued links.dst_page to a real page_id (dropping danglers) and projects the relation kind. Rebuilt on every open; fully derivable from markdown per ADR-0001.
• Four bounded read tools — provenance_ancestry (multi-hop "everything this rests on / derives from it"), provenance_path (shortest path / reachability), expectation_drift (the cross-graph "decisions resting on a since-superseded expectation" query), and abandoned_paths (dead-ended branches). Parameterized (no agent SQL), depth-bounded, cycle-guarded, and fail-closed on ACL.
• Automation-free (v1 contract preserved). escurel serves these queries; synthesising "the most promising next steps" is an external agent's job, not a server-side rules engine.

Backend. The engine runs on stock DuckDB recursive CTEs — zero new dependency, the full query surface. A DuckPGQ (MATCH) backend sits behind a reserved GraphBackend seam, gated on a go/no-go spike: DuckPGQ is a per-DuckDB-version community extension, and no build exists for the pinned DuckDB v1.5.3 (the spike 404s), so the MATCH backend is not built and the feature is complete on CTEs. The #[ignore] spike re-checks on each DuckDB bump; see ../notes/discovered/2026-07-31-duckpgq-unavailable-on-1.5.3.md.

v1 cut-line — what is in vs. out

In:

• All 12 agent tools from ../contract/agent-interface.md
• Live CRDT mode + whole-page fallback on all three transports
• S3 LaneStore is the production default (Hetzner Object Storage as the reference substrate target); local FS retained as a dev-only convenience
• Gemini (gemini-embedding-001) is the default embedder (binary ships the gemini feature; keyless → zero-vector fallback so dev/CI/air-gapped boots stay clean)
• EmbeddingGemma in candle (CPU; CUDA/Metal flags) as the air-gapped/local provider (provider = embeddinggemma) — what the substrate jobspec pins
• Three quota dimensions (queries, writes+embeds, concurrent sessions)
• Admin API: tenant CRUD, export/import, rebuild, audit, attach_external, embedding_reload, compact_db, quota_get, health
• OTel + JSON logs + /metrics
• escurel CLI (operator + agent-style)
• Mandatory escurel meta-skill shipped with every new tenant
• Event-typed skills supported via existing primitives; at: denormalised to an indexed column on both the DuckDB pages and blocks tables; list_instances accepts order_by and the operator-wrapped FilterClause syntax
• External instance backends — the InstanceBackend trait with three impls (markdown | sql_view | document). sql_view: read-only DuckDB views over postgres/mysql/sqlite/erpl/json_dir/parquet_dir, server-side credential registry, validate_bindings, ACL-before-fusion search. document: PDF/DOCX/PPTX/XLSX + text uploaded via /ingest / /ingest/upload, extracted in-process by kreuzberg (default-on), chunked + embedded. Every external instance keeps a markdown overlay page (see protocol.md). Deferred to a future change request: row-grain SQL instances + write-back CRDT, multi-document instances, LLM-driven document processing, the LanceDB retrieval hatch
• Remote (proxy) instance backends — openapi + mcp: an instance is a live window onto a remote object, fetched live on expand (no DuckDB copy) with optional write-back via write_instance. openapi proxies a REST/OpenAPI endpoint; mcp proxies an upstream MCP server (escurel is the client). Base URL + auth held in an admin external_endpoints registry referenced by name (SSRF / secrets-in-markdown guard); remote data feeds no search lane (capabilities.search: "none"). See protocol.md

Out (deferred):

• Federation across tenants
• Live cursors; v1 has presence badges only
• Sidecar embedding adapter (Ollama / TEI / vLLM) — the trait exists; concrete impl is post-v1
• Web admin UI — CLI only in v1
• Reranker beyond the small CE head bundled with EmbeddingGemma; bge-reranker-large as a --features rerank option only
• Auto-provision-on-first-request tenant flow
• Live search subscriptions (search_subscribe over WS) — schema reserved, off behind a feature flag
• Event-derived state projection (rules engine that maps events to state mutations automatically) — see below
• Direct measurement at 1 M instances; v1 extrapolates from 100 k measured
• Multi-model variance run of agent harness benchmarks across GPT and Gemini

Notes on deferred items

Event-derived state projection. v1 records events and state side-by-side without deriving one from the other. An author or agent who creates a meeting instance with a follow_ups: [[decision-record::expand-phoenix-scope]] link is expected to also create the decision-record. Automatic projection (a rule: "when meeting commits with follow_ups X, upsert decision-record X with caused_by: meeting::Y") is on the roadmap as v1.5. The design implication today: events and state are both recorded, not derived. A future projection rules layer would sit between the indexer and the live mode and emit synthetic writes.

FTS retrieval quality. FTS is the most stress-sensitive retrieval path. A synonym-mutant stress corpus is part of the pre-deployment gate in ../adr/0001-duckdb-only-storage.md, with a declared 0.60 nDCG target. If tokenizer tuning (stemmer / k1 / b) cannot close the gap, the fallback is not to revert the consolidation; it is to keep DuckDB for vector + relational + CRDT and attach a separate external FTS engine (e.g. Tantivy) for the lexical column only. v1 ships DuckDB FTS as the default with a config flag (retrieval.fts_backend = "duckdb" | "external") that switches to the external-engine path.

FTS tokenizer tuning. Stemming + k1 + b tuning on a more realistic distractor distribution is the prerequisite to making fts_backend = "duckdb" viable for all tenants. Tracked as a research item under the consolidation gate.

Licenses

The v1 dep set (Rust crates):

crate	license	notes
object_store	Apache-2.0 / MIT	from the Apache Arrow project
duckdb (Rust bindings)	MIT	bindings; DuckDB itself MIT; the vss and fts extensions are part of the DuckDB extension ecosystem under MIT
loro	MIT	CRDT
candle-core, candle-nn, candle-transformers	MIT/Apache-2.0	HF inference runtime
axum, hyper, tokio	MIT	HTTP + runtime
tower, tower-http	MIT	middleware
tracing, tracing-opentelemetry	MIT	logs + traces
opentelemetry, opentelemetry-otlp	Apache-2.0	OTel SDK
prometheus	Apache-2.0	scrape endpoint
jsonwebtoken	MIT	JWT verification
reqwest	MIT/Apache-2.0	HTTP client (Gemini adapter)
serde, serde_json, serde_yaml, toml	MIT/Apache-2.0	(de)serialisation
clap	MIT/Apache-2.0	CLI
dashmap, lru	MIT	data structures
ulid	MIT	page id generation
regex, aho-corasick	MIT/Apache-2.0	wikilink parsing

Permissive across the board. No GPL surface. Closed-source SaaS operators can ship this without surfacing source. Embedding model weights (google/embeddinggemma-300m) are governed by the Gemma license, which permits commercial use including hosted services subject to the prohibited-use policy — operators should review independently if their use case is unusual.

After v1

The shape of the v1.5 / v2 directions:

1. Event-derived state projection — a rules-engine layer that materialises supersession / chain links from event follow_ups fields. Optional per-tenant; off by default (the absence of automation is part of the v1 contract).
2. Live cursors and OT-grade collaboration — once the dependent libraries stabilise.
3. Federation — query routing across tenants for an admin asking cross-tenant questions; behind a permission layer.
4. Sidecar embedding adapter — TEI / vLLM / Ollama; gives GPU-resource-constrained operators a way to pool inference capacity.
5. Reranker upgrade — bge-reranker-large by default; the CE head from EmbeddingGemma stays as the fast-path.
6. Web admin UI — replaces the CLI for everyday operator tasks. The CLI stays for scripting.
7. Multi-region S3 backends — read-replica datasets for geographic distribution; the writer remains single-region.
8. Substrate-target package — a published container image + reusable Kamal deploy/registry fragments, so substrate operators consume escurel-server as a turnkey workload. Per-target binding docs grow as new substrate targets are adopted (managed-K8s, single-VM, etc.).

None of these break the v1 contract. The Rust crate layout was chosen with these extensions in mind: each lives in a new crate that depends on the existing ones, not in a refactor.