Skip to content

protocol.md

Latest commit

 

History

History
1542 lines (1329 loc) · 80 KB

protocol.md

File metadata and controls

1542 lines (1329 loc) · 80 KB
Raw

Protocol — MCP/HTTP, WebSocket

HTTP is the sole transport surface. Both transports expose the same logical surface — the agent tools and the admin endpoints. They differ only in framing and streaming model. The tool semantics are the contract in ../contract/agent-interface.md; this doc specifies the wire shapes.

Transport summary

transport mount what it carries streaming default for
MCP-over-HTTP /mcp JSON-RPC 2.0 framed as MCP method calls; one HTTP request per call; long-running calls block until done and return the final result none (blocking) agents, MCP clients, CLI/TUI, admin/operator tools
WebSocket /ws Bidirectional. Used for live CRDT op streams, presence pings, and search-result streaming full-duplex live mode, web client

Alongside the two tool transports the server also mounts a set of plain HTTP endpoints (no JSON-RPC framing): GET /openapi.json (an OpenAPI 3.1 document generated from the same tools/list payload, for non-MCP HTTP clients), the liveness/readiness probes GET /healthz + GET /readyz, GET /version, GET /metrics (a Prometheus scrape, optionally on a dedicated listener), and the two document-ingest routes POST /ingest + POST /ingest/upload (see Instance backends).

Execution labels (WI-8 / REQ-LABEL-01). Every tools/list entry carries an additive execution: "deterministic" | "orchestration" label. deterministic = the result is a pure function of KB state + arguments (reads, queries, validation, bundle builds); orchestration = the call advances loop state (writes, events, sessions, lifecycle). The default for a new tool is orchestration (fail-closed: nothing masquerades as deterministic compute by omission). This makes the interlocked-loops "deterministic-first" invariant machine-visible: a per-phase tool surface can hand a compute step deterministic tools only.

Auth is the same on both (OIDC Bearer in Authorization header; see platform.md). Tenant resolution is the same (one tenant per token claim). Quotas apply uniformly.

Default exposure. MCP/HTTP and WebSocket are designed for ingress behind a reverse proxy + authentication terminator (the proxy may also terminate TLS). The choice of which transports a particular deployment exposes is per-target; see ../deploy/substrate.md §5 for the substrate-target binding (typically internal/tailnet-only via kamal-proxy, declared in apps/registry.yml).

Shared types

These are referenced from every tool, expressed as JSON Schema.

PageRef

{
  page_id: string,        // ULID, canonical id
  slug:    string | null, // mutable human-friendly slug
  skill:   string,        // the skill id this page declares or is an instance of
  page_type: "skill" | "instance"
}

Hit

{
  page_id: string,
  slug:    string | null,
  skill:   string,
  page_type: "skill" | "instance",
  anchor?: string,        // only for granularity=block
  snippet: string,
  score:   number,        // RRF-fused (or rerank score when reranking is on)
  similarity: number,     // raw vector cosine similarity of the hit
  frontmatter_excerpt: { [key: string]: any }   // includes description and at if present
}

WikilinkParsed

{
  skill:   string | null,  // null for bare [[id]]
  id:      string | null,  // null for bare [[skill]]
  anchor:  string | null,
  version: string | null,
  alias:   string | null
}

Issue (the validate / apply_op / update_page shared shape)

{
  severity: "error" | "warning",
  code:     string,        // e.g. "unknown_skill", "anchor_missing", "frontmatter_required_key_missing"
  location: string,        // e.g. "line:32 col:8" or "frontmatter.at"
  message:  string,
  suggestion?: string
}

validate reports every finding it has. update_page refuses a write only for a deliberately narrow subset — link integrity and page identity — so that seeding and forward references keep working; see the filter in tool_update_page.

frontmatter_autonomy_unknown (error, frontmatter.autonomy, skill pages only) is the one code whose blocking is operator-controlled, via ESCUREL_AUTONOMY_LINT = off (default) | log | enforce. autonomy: was unvalidated free-form frontmatter before it was recognised, so blocking on it by default would make a page that already carries a junk value unwritable — for any edit, not only an edit to that field. validate reports it in all three modes; log additionally warns to the operator log while writing, and leaves the response identical to off.

FilterClause (used by search)

type FilterValue =
  | string | number | boolean | null
  | { ">=" : any }   // operator-wrapped value
  | { "<=" : any }
  | { ">"  : any }
  | { "<"  : any }
  | { "in" : any[] }
  | { "not": any }   // value-level negation

type Filter = { [frontmatter_key: string]: FilterValue }

This filter syntax is intentionally minimal — agents express date-range queries ({at: {">=" : "2026-04-01"}}), enumerations ({status: {in: ["open", "in_review"]}}), and null-checks ({prev_review: null}) without learning SQL. The dispatcher translates the filter to a DuckDB prepared statement; unknown keys, unknown operators, or type-mismatched values return Issue rows rather than dispatching.

Agent surface

The agent tools, grouped by axis. Inputs and outputs given as JSON Schema. (The full non-admin agent surface is ~22 tools — the read / write / event / session tools below plus fetch_blob, query_instance, write_instance, list_snapshots, append_message, list_messages; admin/operator tools are in Admin surface.)

Read tools

Several read tools share two optional overlay/time-travel params:

  • scenario (string) — a what-if overlay. Absent/null reads the base corpus only; a named scenario reads base ∪ overlay, the overlay winning per slug. Accepted by expand, resolve, neighbours, search, and list_instances.
  • as_of (RFC 3339 string) — a time-travel cut; state/edges/ blocks born after it are excluded (see the M7 note under expand). Accepted by expand, neighbours, search, and list_instances.

search

// request
{
  "q": "Acme renewal risk",          // single query; provide this OR `queries`
  "queries": ["Acme renewal", "Acme churn risk"],  // optional; 2–8 phrasings
                                     // fused (RRF) into one ranking
  "k": 10,
  "granularity": "block",            // "block" | "page", default "block"
  "page_type": "any",                // "skill" | "instance" | "any"
  "skill": "customer",               // optional filter; pushes link_skill predicate to DuckDB
  "filter": { "at": { ">=": "2026-04-01" } },  // optional frontmatter filter (FilterClause)
                                               // (events use this to time-window)
  "page_id": null,                   // optional; restrict search to one page's blocks
  "as_of": null,                     // optional RFC 3339 time-travel cut
  "scenario": null                   // optional what-if overlay
}
// response
{ "hits": [Hit, ...], "granularity": "block" }

Provide exactly one of q / queries (at least one non-empty string). filter is the FilterClause shape. It is applied after vector + FTS retrieval as a metadata post-filter, so it doesn't degrade recall; only the response is narrowed. Useful for "find recent meetings about Acme": search(q='Acme', skill='meeting', filter={at: {">=": "2026-04-01"}}).

resolve

// request
{ "wikilink": "[[customer::acme-corp]]", "scenario": null }
// response
{
  "parsed": WikilinkParsed,
  "page": PageRef,         // or null if not found (or ACL-denied)
  "exists": true
}

Returns the parsed link plus the resolved page metadata. resolve does NOT fetch the body — that's expand.

expand

// request
{ "page_id": "01HXMQ...", "as_of": null, "scenario": null, "full": false }
// response
{
  "page": PageRef,
  "frontmatter": { ... },         // full frontmatter
  "body":   "...markdown body...",
  "blocks": [ { "anchor": "blk-acme-signals", "content": "..." }, ... ],
  "wikilinks_out": [ WikilinkParsed, ... ],
  // external-backend instances only (see Instance backends):
  "backend_projection": null,      // sql_view: { view, rows, source, truncated, issue? }
  "chunks_total": null,            // document: total chunk count
  "chunks_truncated": false        // document: blocks are a bounded lead of chunks_total
}

scenario is the what-if overlay and as_of the time-travel cut (shared read-tool params, above). full (bool, default false) is a document-instance knob: when true expand returns every chunk block (the single-document detail / heatmap view) instead of the bounded lead — chunks_truncated is then always false.

Not yet implemented. An earlier draft returned a snapshot_version field (the replayed CRDT snapshot marker); the current builder never emits it. Use list_snapshots to enumerate an instance's replayable taken_at points.

Historical state (M7). as_of = T (RFC 3339) reconstructs the instance as it was at T: when the page has a CRDT snapshot taken at-or-before T, that snapshot is materialized + re-parsed (so the returned frontmatter/body are the historical values — the projection of its events up to T). A page with no snapshot history at- or-before T falls through to the at_ts birth filter (returns the page when born, null when not). This extends the v1 rule that markdown instances ignore @version silently — markdown instances with a seeded snapshot history now honour the time cut.

  • list_snapshots (read) — {page_id} → {snapshots: [taken_at, …]}, the RFC-3339 timestamps of an instance's snapshot history, oldest first. These are the discrete points expand(as_of=T) can replay — the "state over time" version markers in the instance view. Follows the page's own read ACL, exactly like list_op_authors: denial is absence, not error — a page you may not read reports the same empty history as a page that does not exist (no existence oracle).

  • list_op_authors (read) — {page_id} → {page_id, ops: [{op_id, hlc, applied_at, principal}, …]}, oldest first: who wrote each live-editing op on the page (escurel#357 / CR-6). principal is the subject the gateway verified for the apply_op (or WS op frame) that carried it — not the Loro peer id in the op payload, which identifies a device rather than a person. null for ops applied before the gateway recorded a principal. Ops already subsumed by a snapshot and swept by compact_lanes are gone, so this is the retained tail of the history. Returns no op bytes.

    Gated on the page's own read ACL (may_read_instance, the predicate expand/search/list_instances share): authorship is metadata about the page, and in a shared tenant "who touched this instance" is exactly what an instance acl: block exists to keep inside its engagement. Denial is absence, not error — a denied caller gets the empty history a page with no ops returns, so the refusal is not an existence oracle.

    The page-level counterpart is expand's page.last_written_by: the verified principal behind the page's most recent whole-page write. null on an as_of read — a CRDT snapshot stores document bytes, not an author, and reporting the current writer against a past state would be a plausible-looking lie.

For events: expand returns the full body of an event instance including any narrative text and follow-up links. Anchor support is the same as any other instance.

fetch_blob

// request
{ "page_id": "01HXMQ..." }
// response
{
  "blob": {                        // or null (see below)
    "page_id":      "01HXMQ...",
    "content_type": "application/pdf",   // sniffed (pdf / docx / pptx / xlsx / text)
    "size":         104857,
    "bytes_base64": "JVBERi0x..."        // the original retained file bytes
  }
}

Returns the original retained file behind a document-backed instance (the blob named by backend_ref.blob_id), base64-encoded with a sniffed content type, for a faithful client-side preview of the source document. blob is null for a non-document page, a missing page, or an instance the caller may not read (ACL-mirrored on expand; existence is not leaked). The transfer is capped at 25 MiB.

neighbours

// request
{
  "page_id":  "01HXMQ...",
  "direction": "both",             // "in" | "out" | "both"
  "link_skill": null,              // optional single skill filter
  "as_of": null,                   // optional RFC 3339 time-travel cut
  "scenario": null                 // optional what-if overlay
}
// response
{
  "edges": [
    {
      "src_page": "01HX...",       // source page id
      "dst_page": "01HY...",       // destination page id
      "link_skill": "meeting",
      "link_version": null,
      "dst_anchor": null           // anchor on the destination, if the link targets one
    },
    ...
  ]
}

Edges whose other endpoint is an owner-private instance the caller cannot read are dropped (fail-closed ACL).

Not yet implemented. The request keys link_skill_in (a multi-skill array) and order_by (a <field> <asc|desc> sort on the target's frontmatter), and the edge fields anchor / src_anchor / target_frontmatter_excerpt, are not accepted/emitted by the current dispatcher — a caller that sends the extra request keys has them silently ignored.

Provenance graph (provenance_ancestry / provenance_report)

Bounded, read-only traversals over the provenance graph (ADR-0010). Where neighbours is one-hop, these walk multi-hop paths over the derived resolved_links view (which resolves each wikilink's slug to a real page_id and exposes the relation kind — the frontmatter field name, e.g. derived_from, motivated_by). They are parameterized and bounded — the agent supplies named scalars and an allow-listed relations filter, never SQL — and every returned instance is ACL-filtered fail-closed (a hop/row that crosses an owner-private page the caller can't read is dropped; a path query through a private node reports reachable: false with no path, never leaking existence). max_hops is clamped server-side (≤ 12). The engine runs on stock DuckDB recursive CTEs; a DuckPGQ MATCH backend sits behind a reserved seam, gated on the extension becoming available for the pinned DuckDB (currently unavailable — see the discovered note).

// provenance_ancestry — "everything this rests on" (up) / "…derived from it" (down)
// request
{ "page_id": "markdown/instances/result/gbm-auc.md",
                                     // aliases: from_page / from_page_id
  "direction": "up",                 // "up" (default) | "down"
  "relations": ["produced_by","uses"], // optional allow-list; empty = all kinds
  "max_hops": 5,                     // optional, default 5, capped at 12
  "as_of": null,                     // optional RFC 3339 source-birth cut
  "to_page": null }                  // optional destination (alias: to_page_id)
// response (no to_page — the classic walk)
{ "hops": [ { "page_id": "…/analysis/gbm-run.md", "skill": "analysis",
             "relation": "produced_by", "depth": 1 }, ... ] }

// provenance_ancestry with to_page — shortest path / reachability between two pages
// request
{ "page_id": "…/result/r.md", "to_page": "…/dataset/d.md",
  "direction": "up", "relations": ["derived_from"], "max_hops": 5 }
// response
{ "reachable": true, "path": ["…/result/r.md", "…/analysis/a.md", "…/dataset/d.md"], "depth": 2 }

// provenance_report — cross-graph reports; kind selects the report
// request:  { "kind": "drift", "skill": null }  // kind: "drift" | "abandoned"
//                                               // (unknown kind → -32602);
//                                               // skill optionally restricts
// response (kind = "drift") — decisions resting on a since-superseded expectation
{ "kind": "drift",
  "rows": [ { "decision_page_id": "…/decision/ship.md", "decision_skill": "decision",
             "expectation_page_id": "…/expectation/churn-v1.md",
             "superseding_page_id": "…/expectation/churn-v2.md",
             "decided_at": "2026-02-06T09:00:00Z",
             "superseded_at": "2026-03-02T10:00:00Z" }, ... ] }

// response (kind = "abandoned") — nodes retired by supersession/abandonment
// { "kind": "abandoned",
//   "rows": [ { "page_id": "…/expectation/churn-v1.md",
//               "skill": "expectation", "via": "supersedes" }, ... ] }

(Before the 2026-08-14 surface consolidation these were four tools: provenance_path is now provenance_ancestry + to_page, and expectation_drift / abandoned_paths are provenance_report kinds — note the abandoned rows moved from a nodes key to rows.)

provenance_report(kind: "drift") is the cross-graph "lost context" query: a decision whose motivated_by/addresses expectation was later replaced by a supersedes revision authored after the decision (superseded_at > decided_at). These tools are available for any tenant; they read whatever provenance relations exist and are most meaningful with the project-memory skill pack subscribed.

list_skills

// request: {}
// response
{
  "skills": [
    {
      "id": "customer",
      "description": "...",
      "required_frontmatter": ["tier", "opened", "status"],
      "optional_frontmatter": ["mrr_band", "owner", ...],
      "is_event_typed": false,      // true iff `at` is in required_frontmatter
      "autonomy": "review",         // omitted when undeclared OR unrecognised
      "params": [                   // omitted entirely when none are declared
        { "name": "window", "kind": "string", "required": true,
          "label": "Window", "description": "e.g. 30d" },
        { "name": "depth",  "kind": "integer", "required": false }
      ],
      "backend": { "kind": "markdown" },
      "capabilities": { "writable": true, "granularity": "block",
                        "search": "hybrid", "supports_crdt": true }
    },
    {
      "id": "erp_customer",
      "description": "...",
      "required_frontmatter": [...],
      "optional_frontmatter": [...],
      "is_event_typed": false,
      "backend": { "kind": "sql_view" },        // or "document"
      "capabilities": { "writable": false, "granularity": "page",
                        "search": "late_materialized", "supports_crdt": false }
    },
    ...
  ]
}

is_event_typed is a derived convenience flag (true iff at is in required_frontmatter); the agent does not need to compute it from the field list.

The catalogue is caller-scoped, and the acl block is admin-only (#374). Like every other read verb, list_skills filters: a skill whose declared acl.read does not intersect the caller's effective groups is absent from the response — denial as absence, never an error, so a client can trust the catalogue instead of re-filtering it. Two narrowings keep this from making skills into access-control containers: a skill with no acl: block falls through to the tenant default (read: [public] as shipped) and stays visible to everyone, and the structural owner group is treated as satisfied, so the legacy visibility: owner mapping (acl.read: [owner]) never hides a type whose instances are private. The admin role bypasses, as everywhere.

The acl object itself is projected only to an admin caller. Group names are authorisation metadata, not schema: in a shared tenant they are named per engagement, so handing every token holder the grant list would disclose the customer roster and the authorisation topology. A non-admin row omits acl entirely — indistinguishable from a skill that declares no block. visibility and owner_field are retained for every caller: they describe how instances behave and name no group.

autonomy reports the human-in-the-loop policy the skill page declares via its autonomy: frontmatter key — auto (a write derived from this skill commits directly), review (held for human approval), or confirm (as review, plus an out-of-band notification). Escurel does not enforce the policy; the gateway stays automation-free. It recognises the key so a consumer has one declared place to read it from, and so validate can object to a typo at authoring time.

The field is omitted when the key is absent and when its value is not one of the three. Those two cases collapse on purpose: an unrecognised value must never be reported as auto, because a consumer reading auto switches a human gate off — autonmy: review silently meaning "commit ungated" is a data-governance incident rather than a bug. A client treats an omitted autonomy as "hold for review" and calls validate to learn which of the two cases it is. Case and surrounding whitespace are normalised, so Auto and " auto " are both auto.

Invocation parameters (params)

required_frontmatter is the shape of the instances a skill produces. params is the shape of what one run of the skill takes. For an instance-creating skill the two nearly coincide, which is why the distinction went unremarked for so long; they part company as soon as the skill's job is not "make one instance shaped like this" — a report skill parameterised by window and grouping, a workflow skill taking a target and a depth budget, an analysis skill taking two instances to compare.

Escurel does not execute skills and never binds these values. It reports what the page declares, so a client can build an input form from the catalogue alone without expanding every page.

A skill page declares them either as a sequence (the params: idiom query pages already use, and the only form that preserves the author's field order):

params:
  - {name: window,   kind: string,  required: true, label: Window,
     description: 'e.g. 30d'}
  - {name: grouping, kind: string}
  - {name: depth,    kind: integer}

or as a mapping of name to attributes, whose order is the frontmatter's key order rather than the author's:

params:
  window: {kind: string, required: true, description: 'e.g. 30d'}

kind is reported as one of string | integer | boolean — deliberately exactly the A2UI form field kinds, so the catalogue renders with no mapping layer at the consumer. text / int / bool are accepted as synonyms in frontmatter (that is the spelling authors arrive with from the query-page params: block, where type: names a richer, SQL-bound vocabulary), as is type: in place of kind:. Case and surrounding whitespace are normalised. required: defaults to false; label and description are omitted rather than emptied when undeclared, so a client falls back to name.

A kind: the server cannot read degrades to string; the parameter is still reported. This is the opposite of the autonomy: rule above, and deliberately so. There, dropping the value is the safe direction — only an explicit auto may switch a human gate off. Here, dropping the parameter would delete a possibly-required field from a generated form; the run would then be invoked without it and fail with nothing on the page to explain why. An over-permissive text box under-validates, a missing box loses data. validate reports the mis-declaration as a warning (frontmatter_param_kind_unknown) so the author still learns, without failing a write for a key that has never been validated. A params: block that is neither shape, or an entry with no name:, is an error (frontmatter_params_malformed) — there is nothing to degrade to when a parameter has no name to be passed under.

params is omitted from the response entirely when a skill declares none, so every skill page written before this key existed has a byte-identical row. The key is read on SKILL pages only: on an instance page params: is already taken by [[query::*]] pages and is untouched.

backend.kind (markdown | sql_view | document) and the capabilities object tell the agent where a skill's instances live and what may be done with them — see Instance backends. A skill that declares no backend: block is markdown with the writable, block-grain, CRDT capabilities above (the historical default, so existing clients are unaffected).

Per-instance access control (visibility / owner_field)

A skill page MAY declare a read policy for its instances:

visibility: owner        # public | owner   (default: public)
owner_field: credential  # frontmatter field naming the owning principal
  • visibility: public (the default, and the only behaviour before this field existed) — any authenticated caller in the tenant may read an instance of this skill.
  • visibility: owner — an instance is readable only by its owning principal or the admin role. The owner is the verified token sub that equals the instance's owner_field value: either a direct value (e.g. credential carries the platform sub) or a [[skill::id]] wikilink, resolved to the linked instance's credential.

Enforcement is deterministic and applied on every read path: expand of a non-owned owner-instance returns {"page": null} (absence, not an error — existence is not leaked); list_instances and search filter out non-owned owner-instances; the admin role bypasses. The decision is a pure comparison on the request path — never an LLM, agent, or classifier. Owner-visibility is reported back on list_skills as "visibility" + "owner_field".

Group ACL (acl: block, group ACL v1)

A skill MAY instead declare a per-CRUD group ACL — a superset of visibility/owner_field (see ADR-0004):

owner_field: author       # still drives the `owner` group
acl:
  read:   [public]
  create: [owner]
  update: [owner, moderator]
  delete: [admin]

Each verb lists group names. public / owner / admin are reserved special groups (always-present / structural-owner / verified-admin respectively, and never grantable via a token claim or a membership row); any other name is a custom group, satisfied when it is present in the caller's groups_claim JWT array or in the DuckDB-canonical group_members table. An action is allowed iff the caller's effective group set intersects the verb's list (admin always bypasses); empty intersection → deny (fail-closed, no deny rules in v1).

A verb omitted from the block, or a skill with neither acl: nor visibility:, falls through to the tenant default — the acl_defaults: block on the escurel meta-skill page, or, when unset, the shipped default (read:[public], writes [admin]) that reproduces the pre-RBAC behaviour. A legacy visibility: field with no acl: block maps deterministically (public → open read + admin writes; owner → owner-all), so existing pages are unchanged. delete is enforced as update in v1 (there is no distinct delete operation at the write boundary). Membership is mutated by the admin-only add_group_member / remove_group_member / list_group_members tools. list_skills reports the resolved block as "acl" to an admin caller only, and filters the catalogue by the skill's acl.read for everyone else (#374 — see list_skills); visibility/owner_field are retained for every caller. (Capability-tool RBAC is phase 2.)

Instance-level acl: overrides

An instance page MAY carry its own acl: block, in exactly the same shape. It is resolved per verb, most specific first: the instance's block → the skill's block → the tenant default → deny.

# markdown/instances/customer_note/hoffmann-1.md
type: instance
skill: customer_note
id: hoffmann-1
acl:
  read:   [engagement-hoffmann]
  update: [engagement-hoffmann]

This is what lets two instances of one shared skill be readable by two different groups — the account/engagement shape, where the unit of access is the record and not its author (which is all owner_field could express). Enforcement is the same fail-closed predicate on the same paths: expand, search, list_instances, resolve, neighbours, query_instance, the WebSocket attach, and the event bus (a filed event is exactly as visible as its instance). Denial is absence, never a distinguishable error.

Two deliberate limits:

  • An instance block is consulted per verb. Declaring only read: narrows reads and leaves update/create falling through to the skill. Narrowing a write means declaring the write verb too.
  • On a write, the block that decides is the one on the stored page, not the one in the incoming content — otherwise a caller could authorise its own write by shipping a block that grants itself. A create has no stored page and therefore stays skill-grained: the create grant is "may add instances of this type", a skill-level claim. (A caller who may update a page may, as everywhere, rewrite its acl: block.)

An instance with no acl: block resolves exactly as before, so every page authored before this existed is unaffected and the change needs no rollout flag of its own. The write half remains gated by ESCUREL_WRITE_ACL.

list_instances

// request
{
  "skill_id": "meeting",
  "frontmatter_key":   "source",    // optional single-field equality filter…
  "frontmatter_value": "gmail",     // …both must be present to apply
  "order_by": "at desc",            // optional; "at asc" | "at desc" only
  "limit":    50,
  "as_of":    null,                 // optional RFC 3339 time-travel cut
  "scenario": null                  // optional what-if overlay
}
// response
{
  "instances": [
    { "page_id": "01HX...", "skill": "meeting",
      "frontmatter": { "at": "...", "with": "...", ... },
      "at": "2026-04-12T10:00:00+02:00" },   // the typed `at`, or null
    ...
  ],
  "next_cursor": null               // string = more rows; null = done
}

next_cursor is an opaque resume cursor: a string when rows lie past the page — pass it back as cursor to continue — and null on the final page. Only null means done; the ACL filter runs after limit and can shorten any page. An undecodable cursor is -32602.

This is the event-log primitive. Unlike search, list_instances does not accept the operator-wrapped FilterClause object — its only filter is the single frontmatter_key = frontmatter_value string-equality pair. order_by is restricted to at asc / at desc. Owner-private instances the caller cannot read are filtered out.

query_instance

The one query surface. (The legacy admin-gated run_stored_query tool — pre-declared arbitrary SQL over the whole corpus, with no per-row owner to ACL against — was removed in the 2026-08-14 surface consolidation; query_instance accepts query_id as an alias for ref, so old callers keep their argument spelling. For event-volume queries that exceed the markdown-friendly scale (~1 M), operators move the event records to an external DuckLake table and the agent reaches them through a [[query::*]] page via query_instance instead of list_instances — see storage.md.)

A parameterised, full-result-set read over one sql_view instance's view. The query page (a [[query::*]] instance) declares a target: [[skill::id]] naming the sql_view instance and references its view via the {{target}} placeholder:

# markdown/instances/query/sales-by-category.md
type: instance
skill: query
id: sales-by-category
target: "[[sales::eu-2026]]"        # the sql_view instance to read
params:
  - {name: min, type: number, required: true}
sql: "SELECT category, SUM(amount)::BIGINT AS total
      FROM {{target}} WHERE amount >= :min GROUP BY category"
// request                          // response
{                                   {
  "ref":    "sales-by-category",      "rows":      [ { "category": "hw", "total": 50 } ],
  "params": { "min": 10 }             "schema":    [ { "name": "total", "type": "BIGINT" } ],
}                                     "truncated": false
                                    }

ref is the query id or its [[query::id]] wikilink. Two trust boundaries are kept separate by construction:

  • Value position — every :param runtime value is bound as a positional DuckDB prepared-statement parameter, so injection through a param value is impossible and it never flows through the sql_view filter-interpolation path.
  • Identifier position — {{target}} resolves to the target's managed vw_… view name, allow-listed through the same vw_-prefix guard the projection path uses (never a bound value).

Access: query_instance is an agent tool gated by the per-instance read ACL on the target instance (may_read_instance, fail-closed): the caller must be allowed to read the underlying data, not merely the query template. Admin bypasses; a denied caller gets an authorisation error. The result set is capped at MAX_RESULT_ROWS (10 000) with truncated set when the cap clipped the tail.

Write tools

validate

// request
{ "content": "---\nskill: meeting\n...\n---\n# ...", "as_page_id": null }
// response
{ "ok": true, "issues": [Issue, ...] }

ok is false iff any Issue is error-severity (warnings do not fail a draft); the full issues list is always returned.

open_session / apply_op / close_session (live CRDT)

// open_session request:  { "page_id": "01HX..." }
// open_session response: { "session": "sess_...", "head_version": "v42",
//                          "ws_url": "/ws" }
//
// apply_op request:  { "session": "sess_...", "op": "<base64 Loro op bytes>" }
// apply_op response: { "ok": true, "merged_version": "v43" }
//
// close_session request:  { "session": "sess_...", "commit": true }
// close_session response: { "ok": true, "final_version": "v50", "issues": [] }

op is base64-encoded Loro op bytes. ws_url is the relative /ws path (the gateway does not know its public origin, so it never emits a full wss:// URL). Not yet implemented: open_session does not return the page content, and apply_op returns neither content, conflicts, nor issues — a client reads the merged document over the WS channel or via expand.

The ws_url returned by open_session is the recommended channel for apply_op — the WS path delivers ops with lower overhead than HTTP. MCP-over-HTTP clients without WS may continue calling apply_op over HTTP.

Write ACL. The session surface enforces the same write policy as update_page (ESCUREL_WRITE_ACL): open_session refuses a caller who may not write the page (JSON-RPC -32000, data code forbidden — a session's op stream would edit the page byte by byte), and close_session re-checks that policy at commit time, since the ACL can change while a session is open. A refused commit returns update_page's denial shape ({ok: false, issues: [{code: "forbidden", …}]}) and leaves the session open, so the caller can still discard with commit: false. apply_op itself stays keyed by session possession (session ids are unguessable); the open and close gates are what hold.

update_page (whole-page fallback)

// request
{
  "page_id":     "01HX...",
  "content":     "---\n...\n---\n# ...",
  "base_version": "v42"            // optional; required only if the client knows it
}
// response
{
  "ok":          true,
  "new_version": "v43",
  "auto_merged": true,             // true iff a stale base_version was three-way-merged
  "issues":      [Issue, ...]
}

If base_version is supplied and the head has advanced, the server attempts a CRDT-aware three-way merge (Loro): it reconstructs the base snapshot the client branched from, forks it into the head and incoming edits as concurrent Loro branches, and unions them. A clean merge is persisted and the response carries auto_merged: true. The server refuses to persist a merge that no longer parses or whose frontmatter matches neither side (both sides changed the same key) — that is an unresolvable conflict: {ok: false, issues: [{code: "conflict", ...}], head_content: "..."}, and the client re-drafts against head_content. (Auto-merge needs the base snapshot; a base_version older than the first update_page snapshot, or a bare session op-count with no snapshot, always conflicts.)

update_page (and apply_op) against an instance whose skill is a non-writable backend (sql_view, document) is rejected with {ok: false, issues: [{code: "backend_read_only", ...}]} — the external source/blob is canonical and is never written back through the page API. See Instance backends.

update_page against a base-layer page — one whose stored frontmatter carries layer: base@<pack>@<version>, i.e. it was imported from a subscribed skill pack — is rejected with {ok: false, issues: [{code: "layer_read_only", location: "frontmatter.layer", ...}]} (REQ-LAYER-02). The guard keys off the stored page's layer, so stripping the layer: field from the draft is not an unlock; a draft declaring layer: base@… is rejected the same way (base pages are created by pack import only, never by update_page). open_session on a base-layer page fails with a JSON-RPC -32000 error whose message starts layer_read_only: — live CRDT co-authoring must not bypass the guard. Pages without a layer: field (every pre-layer page) and pages declaring layer: overlay are unaffected. list_skills reports each skill's layer ("overlay" default, or the base@<pack>@<version> pin) so agents and operators can tell stable from editable.

Shadowing (REQ-LAYER-03). A tenant overlay skill page MAY declare the same skill id as an imported base page — that is how a tenant specialises pack content without forking it. Page-level precedence with drift visibility: resolve prefers the overlay; list_skills reports ONE entry per skill id (the overlay) with an additive shadows: "base@<pack>@<version>" pin; expand of the shadowing overlay carries an additive shadow object — {base_page_id, pack, base: {…the base page's frontmatter…}} — so the base values stay visible, never silently masked (the same namespacing discipline as the sql_view source object). The base page itself is untouched (INV-SHADOW): expanding it directly returns the pack's pristine content, and a future pack upgrade rebases against it. import_pack therefore lands a base skill beneath an existing tenant skill of the same id (the overlay direction of pack_skill_collision no longer refuses; two BASE pages with one id still do — no precedence exists between packs).

Events / inbox (M7 — Event-sourcing surface)

Events are the dynamic input of the memory triad (Events · Skills · Instances). They live in a dedicated events store (not pages); each event's label_skill links to the skill that knows how to process it, and instance_page_id links to the instance it belongs to once processed. The inbox is the status = 'inbox' view. All four tools are MCP tools over POST /mcp with the usual quota debits (capture_event/assign_event = Writes; list_inbox/list_events = Queries).

  • capture_event (write) — append an event to the inbox. Input: {event_id?, at?, source?, mime?, label_skill?, instance_page_id?, title?, body?, provenance?} (event_id is a server ULID when absent; instance_page_id only pre-flags a candidate — the event stays in the inbox until assign_event). Returns the stored event ({event_id, at, status: "inbox", …}).
  • list_inbox (read) — {limit?, cursor?} → {events: [Event, …], next_cursor?}, unprocessed events, newest first.
  • list_events (read) — {instance_page_id, limit?, cursor?} → {events: [Event, …], next_cursor?}, that instance's processed event history, oldest first (the sequence whose projection is its state).

Both listings paginate with an opaque cursor: next_cursor is present iff rows lie past the page — pass it back as cursor to continue. Only its absence means the listing is complete; a short page never does (the per-event ACL filter runs after limit and legitimately shortens pages). An undecodable cursor is -32602 invalid_params.

  • assign_event (write) — {event_id, instance_page_id} → marks the event processed and bound to the instance. This is the (external) agent's act of folding the event into state.

An Event is {event_id, at, source, mime, label_skill, instance_page_id, status, title, body, provenance}.

Per-event ACL (ESCUREL_EVENT_ACL: off | log | enforce, default off). An inbox event is unreviewed third-party text, so on a shared tenant the event surface is scoped per caller, not just per tenant. The rule: an event's visibility follows the record it belongs to; until it belongs to one, it follows the person who captured it.

  • capture_event stamps the verified caller subject into the stored event as provenance.captured_by, overwriting any caller-supplied value under that key. This happens in every mode, off included — the stamp is data, not a decision, and it is what makes a later switch to enforce meaningful.
  • An event whose instance_page_id is set is exactly as visible as that instance (the same per-instance read ACL expand/list_instances apply). An un-triaged event is visible only to the subject that captured it. Admin bypasses. An event captured before the stamp existed carries none and stays ungated, the same compat fallback the chat ACL takes for an unresolvable owner.
  • capture_event is idempotent on event_id and returns the STORED first-writer row, which makes a guessed id a read. When the caller may not see that row it gets its OWN submission back instead, wearing the stored event_id — indistinguishable from a first capture, disclosing neither the stored content nor that the id was taken. The rightful owner's retry still reads back the authoritative stored event, so retry convergence is unchanged. The capture webhook always carries the stored event, never the echo.
  • list_inbox / list_events drop the events the caller may not see, post-query, so a page may be shorter than limit. assign_event refuses a claim of an event the caller may not see as not found, byte-identical to a claim of an event that does not exist — an error that distinguished them would be an existence oracle. The compare-and-set is unchanged behind it.

The knob is separate from ESCUREL_WRITE_ACL on purpose. The inbox is a work queue: a drain loop under a non-admin agent token legitimately reads events it did not capture, and enabling this would blind it. Run log first to find those callers (they are warned, not denied), then enforce. On a gateway with no verifier wired every caller is admin and this gate is inert, exactly like the instance ACL.

Capture webhook (opt-in). When ESCUREL_WEBHOOK_URL is set, each capture_event fires a fire-and-forget HTTP POST of the stored event's JSON to that URL — the notification an external processing agent subscribes to. Delivery never blocks or fails the capture (a down sink is logged and dropped); the agent may also poll list_inbox, so a missed POST self-heals. The fold event→state remains the external agent's job (via assign_event + update_page); the server stays automation-free.

The delivered payload always carries an additional tenant_id field — the gateway's authoritative tenant (single-tenant per indexer) — so the receiver knows which tenant the event belongs to without a side channel.

When ESCUREL_WEBHOOK_SECRET is also set, the gateway authenticates the POST: it serializes the body once, computes HMAC-SHA256 over those exact body bytes under the secret, and sends it as the header X-Escurel-Webhook-Signature: sha256=<hex> (lowercase hex of the 32-byte digest), POSTing the same bytes with content-type: application/json. The receiver recomputes the HMAC over the raw request body and rejects a missing/mismatched signature (a constant-time compare). With no secret configured the POST is unsigned (dev). This is the only ingress trust anchor between the gateway and the external runner.

Instance backends

By default an instance's data is native markdown (writable, block-grain, CRDT-backed). External backends let an instance's data live elsewhere while every escurel invariant holds — single referent space, markdown-canonical, derivable index, fail-closed ACL, single-writer:

  • sql_view — a read-only DuckDB VIEW over an external relational source (postgres / mysql / sqlite / erpl / json_dir / parquet_dir).
  • document — an uploaded file (PDF / DOCX / PPTX / XLSX, or text) extracted, chunked, and embedded into one page-with-blocks.
  • openapi / mcp — live remote (proxy) instances: the body/data is fetched live on expand from a REST/OpenAPI endpoint (openapi) or an upstream MCP server (mcp), with optional write-back. Nothing is materialised in DuckDB. See Remote backends.

The unifying idea: every external instance keeps a markdown overlay page — the page is the instance in the referent space (identity, links, ACL, history all reuse the existing machinery), and a backend_ref frontmatter block binds it to the external data. All novelty is confined to where the body/data comes from, so resolve / expand / neighbours / list_* / search route through the backend transparently and no dispatcher or wire change is needed to add one. A skill selects its backend in frontmatter:

backend:
  kind: sql_view            # markdown (default) | sql_view | document | openapi | mcp
  # …kind-specific config (see below)…

list_skills reports each skill's backend.kind + a capabilities object (writable, granularity, search, supports_crdt); sql_view, document, openapi, and mcp are all writable: false, so update_page / apply_op against them return backend_read_only (the overlay/source is not editable through the page API — remote backends accept write-back only through the explicit write_instance tool). Remote backends additionally report search: "none" — their live data is never indexed, so it feeds no search lane (the overlay page itself is still indexed and searchable like any page).

sql_view

Skill frontmatter declares the source, a projection, and the columns that feed search:

backend:
  kind: sql_view
  source: { connector: postgres, attach: crm_pg, relation: public.customers,
            filter: "region = 'EU'" }      # filter is optional, injection-guarded
  project: { customer_id: id, display_name: name }   # source col → overlay field
  search_text: [name, notes]               # columns that enter late FTS
  projection_limit: 50                     # optional; rows expand renders (default 50)
  • Secrets never live in markdown. source.attach names a credential registered out-of-band via register_credential (admin), realised as a DuckDB CREATE SECRET. list_credentials returns names only.
  • create_sql_instance {skill, id, overlay_body?} materialises the instance under the write lock: it ATTACHes the source READ_ONLY, creates a managed vw_<…> view, captures a source_schema_fingerprint, and writes the overlay page with backend_ref { kind: "sql_view", view, binding_hash, source_schema_fingerprint }.
  • Reads (expand) merge the overlay (which wins) with a bounded projection of the view (expand.backend_projection = {view, rows, source, truncated, issue?}); a colliding source field is exposed under source.<field>. Never an unbounded dump.
  • validate_bindings (admin) re-probes each view and compares the stored fingerprint; on drift the binding is marked binding_degraded and that view's reads fail closed (an Issue, not wrong rows).
  • Search contributes candidates only (late-materialised FTS over search_text); the dispatcher applies the fail-closed ACL predicate to every lane before RRF fusion (INV-ACL-FUSION), and a view whose owner can't be resolved denies non-admins. See storage.md.

document

Skill frontmatter declares accepted MIME types and chunking:

backend:
  kind: document
  accepts: [application/pdf, text/plain]
  chunk: { max_chars: 800, overlap: 80 }
  lead_chunks: 8                           # optional; chunk lead expand returns (default 8)

An accepts: entry is an exact MIME or a type wildcard (audio/*), claiming every subtype of that type. */* is not a wildcard — an unhandled MIME must still park rather than be swallowed. An exact claim always beats a wildcard one, so a broad collection cannot divert an upload from a skill that named the MIME; within a tier the first skill by id order wins.

Ingestion is event-driven, deposited-before-processed (an upload is never lost), and runs the extractor off the per-tenant write lock:

  1. An external client deposits a blob and notifies escurel via one of two authenticated HTTP endpoints (see below).
  2. The MIME is routed to a document skill whose accepts: lists it; an unmatched MIME is parked with Issue(no_handler_skill) and the inbox blob is retained.
  3. escurel records an immutable ingest Event (auditable; same event log as capture_event).
  4. A deterministic worker extracts (kreuzberg for PDF/DOCX/PPTX/XLSX — on by default; plain-text for text/*) → chunks → embeds → materialises one instance = one page with N chunk blocks, under a brief write lock. The blob is canonical (content-addressed, retained); chunks are derivable (rebuild re-extracts).

audio/* takes the retain-only path (CR-4): escurel does not transcribe, so the recording materialises as status: ok with chunk_count: 0 — an instance with ordinary identity, links, ACL and history whose value is its retained bytes, with the transcript supplied separately by the caller as its own content. backend_ref.extracted then carries bytes, codec and (where the container states it exactly, i.e. WAV) duration_ms. Like every other document instance, a recording is managed by the ingest pipeline: update_page against it is refused with backend_read_only.

expand returns the overlay + the top-k relevant chunks (chunks_total, chunks_truncated), never the full text. backend_ref carries { kind: "document", blob_id, content_type, extract_engine, chunk_count, status } (content_type is the MIME the upload declared; fetch_blob prefers it over sniffing the bytes). Document chunks are ordinary blocks, so search rides the same ACL-before-fusion path as markdown.

Ingest endpoints

Two authenticated HTTP routes (not MCP tools), rate-limited per tenant as Writes:

route body purpose
POST /ingest { blob_id, content_type, title?, skill?, event_id? } ingest a blob already deposited in the tenant's blobs/inbox/ area
POST /ingest/upload { content_type, bytes_b64, title?, skill?, event_id? } deposit (base64) and ingest in one call

skill explicitly targets a document-backend skill that accepts the MIME (422 otherwise, create-ACL enforced); absent, the skill is resolved from the MIME (REQ-DOC-06). event_id is the caller's idempotency key, fed to capture_event's dedup: a redelivery with the same key is acknowledged { status: "duplicate", event_id, blob_id } without a second inbox event and without re-running the extraction worker (escurel#382). Absent, the server mints a ULID per request.

The download twin is GET /blob/{page_id} (bearer-authed): the retained original bytes verbatim, with the declared (or sniffed) Content-Type and an honest Content-Length — no base64 detour and no 25 MiB cap, unlike the fetch_blob tool it shares its ACL with. An absent page, a hidden page and a page with no retained blob are one indistinguishable 404 (no existence oracle).

Both ingest routes sit behind the same cross-cutting gates as MCP dispatch: a suspended tenant refuses agent tokens (403 tenant_suspended), a ducklake reader refuses outright (503 read_only_replica — retry against the writer), and the per-tenant Writes budget applies (429).

Both return the pipeline outcome: { status, event_id, blob_id, page_id?, handler_skill?, chunk_count?, issue? } where status ∈ materialised | extraction_failed | no_handler | duplicate. On extraction failure the inbox blob is retained and the instance is marked extraction_failed (the upload is never lost).

Remote backends (openapi / mcp)

Unlike sql_view / document (materialised, read-only), the two remote (proxy) backends keep no local copy: an instance is a live window onto a remote object. Its identity, links, ACL, and history are the ordinary overlay page, but its body/data is fetched live on expand, and — because these backends declare a write op — edits are forwarded upstream via the explicit write_instance tool (never update_page; the remote source is canonical). openapi proxies a REST/HTTP endpoint; mcp proxies an upstream MCP server (escurel is the MCP client, calling a tool or reading a resource).

Four invariants that these backends deliberately revise vs. the materialised external backends, and how each is kept safe:

  1. Read-only → write-back. Remote instances are writable: false w.r.t. update_page/CRDT (the overlay body is a live projection, not co-authored), but accept write-back through write_instance, gated by the target instance's acl.update. The remote op is value-bound (payload + id map), never string-spliced.
  2. No search lane. Live data is never indexed → capabilities.search: "none". The overlay page's own metadata/body stays indexed and searchable.
  3. SSRF / secrets-in-markdown. A skill's backend.endpoint names an admin-registered endpoint (base URL + auth held server-side in the external_endpoints registry), never a raw URL — so tenant markdown can never make the server fetch an arbitrary host, and no secret enters the corpus.
  4. Live-read failure. A read that times out / errors returns the overlay page + backend_projection.issue — never a partial or fabricated body (the binding_degraded policy).

Skill frontmatter declares the endpoint, the read/write ops, and a project map (response JSON $.a.b path or bare key → overlay field):

# openapi — read + write
backend:
  kind: openapi
  endpoint: crm_rest                 # admin-registered (URL + auth server-side)
  read:  { operationId: getCustomer, path: /customers/{id} }   # method defaults GET
  write:                             # omit ⇒ read-only
    method: POST
    path: /customers/{id}/orders/{order_id}   # {order_id} from the payload
    body: { sku: "{sku}", qty: "{qty}", via: "escurel" }   # optional template
  project: { display_name: $.name, tier: $.account_tier }
# mcp — read-only resource
backend:
  kind: mcp
  endpoint: upstream_kb              # points at the upstream server's /mcp
  read:  { resource: "kb://article/{id}" }   # or { tool: getArticle }
  project: { title: $.title }

{name} placeholders in a path / resource / body template are filled from the overlay instance id ({id}) and, on a write, the payload's scalar fields — flattened to dotted keys, so {order_id} and {customer.tier} both resolve. A placeholder that cannot be resolved fails the call closed (unfilled path/body placeholders), never sending a literal {x}. For an OpenAPI write, an optional body: template reshapes the payload: an exact "{name}" leaf keeps its JSON type (a number stays a number, an object stays an object), while embedded {name} interpolates as a string; omit body: to send the payload verbatim. For an MCP write, the payload's fields are merged into the tool-call arguments. A read/write is also refused (fail-closed) when the skill's backend kind does not match the kind its endpoint was registered under. expand returns the overlay merged with the live projection under backend_projection = { source, fields, issue? }; backend_ref carries just { kind, endpoint } — the read/write ops are re-derived from the skill's backend: block on each call, never persisted per-instance. In a read:/write: op only path + method (operationId is accepted but ignored) drive the OpenAPI call.

New MCP tools:

  • write_instance (write) — { ref, payload } → forwards a write to the target remote instance's upstream write op and returns the re-projected instance. ref is the instance id or [[skill::id]]. Gated by the target's acl.update (fail-closed; admin bypasses). A skill whose binding declares no write op is refused (backend_read_only).
  • create_remote_instance (admin) — { skill, id, overlay_body? } materialises the overlay page + backend_ref for an openapi/mcp skill (the binding comes from the skill's backend: block, never the caller — the create_sql_instance pattern).

New admin endpoint-registry tools (mirror register_credential &c.): register_endpoint { name, kind, base_url, auth, secret? }, list_endpoints {} (names/URLs only, secret never echoed), delete_endpoint { name }, validate_endpoints {} (probe each registered endpoint's reachability; unreachable ⇒ that skill's reads fail closed).

Admin surface

The admin/operator capabilities are exposed as admin-role-gated MCP tools over POST /mcp — there is no separate admin service. Each requires the admin role on the OIDC token (configurable; see platform.md); a call from a token without the required role yields JSON-RPC error code -32001. The noun-first tenant_* family and embedding_reload also accept verb-first dispatch ALIASES (create_tenant, list_tenants, get_tenant, update_tenant, delete_tenant, export_tenant, import_tenant, reload_embedding) — resolved to the canonical name before quota, metrics, gating and dispatch, and never advertised in tools/list. Admin-scope tools are exempt from the tenant's agent rate budget, keyed on the same scope label discovery filters by.

The gate is enforced at dispatch (-32001), and since the scope label landed, discovery matches it: every tools/list entry carries scope: "agent" | "admin", and an agent-role token receives only the scope: "agent" subset — the tools it can actually call. Admin tokens (and verifier-less dev mode) receive the whole surface (see MCP-over-HTTP framing). Tenant resolution rules are different on admin tools — the tenant is named explicitly (tenant_id) rather than taken from the token's claim — but because a gateway is single-tenant, a tenant_id that names a tenant other than the one this gateway serves is refused (-32002); an empty value means "this gateway's tenant".

tool inputs outputs purpose
tenant_create {tenant_id, display_name?} {spec: {tenant_id, display_name}} provision a new tenant (no quotas input)
tenant_list {} {tenants: [{tenant_id, display_name}]} enumerate tenants
tenant_get {tenant_id} {spec: {tenant_id, display_name}} fetch one
tenant_update {tenant_id, display_name?, status?, quotas?, embedding_provider?} {spec, rebuild_required} partial update: rename, suspend/resume (status: active|suspended — a suspended tenant rejects non-admin calls), per-tenant quotas, and embedding_provider (zero|gemini|embeddinggemma; changing it moves the vector space → rebuild_required: true, run rebuild). Live suspend/quota apply to the served tenant; embedding takes effect on next boot/rebuild (#247)
tenant_delete {tenant_id, confirm} (confirm must equal tenant_id) {deleted} hard-delete a tenant + its on-disk state
tenant_export {tenant_id} {format_version, tarball_b64, bytes, sha256} (tarball: canonical markdown only, gzip'd; sha256 = hex of the tarball body) export (blocking)
tenant_import {tenant_id, tarball_b64} {bytes_imported} restore markdown into an existing tenant (blocking)
export_pack {tenant_id, id, version, vertical, publisher, skills, include_instances?} {manifest, tarball_b64, bytes} build a skill pack: a deterministic tar+gz of the named skills' pages (+ instances when include_instances) with an HMAC-signed manifest ({format_version, id, version, vertical, publisher, page_count, content_hash, signature}). Requires ESCUREL_PACK_SECRET (refuses unsigned, pack_secret_not_configured); fails closed on credential-shaped page content (pack_secret_detected). See ADR-0006
import_pack {tenant_id, manifest, tarball_b64, allow_vertical_mismatch?} {pack, version, vertical, pages_imported, layer} import a signed pack as the tenant's pinned, read-only base layer: signature + content_hash verify fail-closed before unpacking (pack_signature_invalid); unsafe pack ids refuse (pack_id_invalid); unsafe entry paths / malformed pages refuse (pack_malformed) with the WHOLE pack validated before the first page lands (a bad page ⇒ zero landed pages); pages land under the reserved markdown/base/<pack>/ namespace stamped layer: base@<id>@v<version>; the pin is recorded in pack_subscriptions (a canonical input, like the credential registry). A version change on a subscribed pack refuses (pack_version_pinned), a same-version re-publish with different bytes refuses (pack_content_mismatch) — upgrades are an explicit future rebase; an unrelated vertical refuses (vertical_mismatch) unless allow_vertical_mismatch; a skill id another indexed skill page already declares refuses (pack_skill_collision — explicit shadowing is a future feature). Transport-neutral: an air-gapped tarball and a live pull are the same call. See ADR-0007
unsubscribe_pack {tenant_id, pack_id} {pack, pages_removed} drop a subscription cleanly: every base page the pack landed is removed (so rebuild cannot resurrect orphaned base content), then the pin; tenant overlays survive (a shadow simply stops shadowing); a later import_pack starts from zero. Refuses unknown packs (pack_not_subscribed)
list_packs {} {packs: [{pack_id, version, vertical, publisher, content_hash}]} the subscribed packs and their pins
rebase_pack {tenant_id, manifest, tarball_b64, acknowledge_conflicts?, dry_run?} {ok, issues, pack, from_version, to_version, pages_imported, pages_removed, conflicts_acknowledged}; with dry_run: {ok, dry_run: true, issues, pack, from_version, to_version, would_import, would_remove} the reviewed upgrade of a subscribed pack (REQ-REBASE-01/02) — the only operation that moves a version pin. Validates like import_pack (verify before unpack, whole pack before the first write); a field the tenant's shadow overrides AND the new version changes surfaces as a rebase_conflict Issue (skill <id> · <field>, body included) and blocks until acknowledge_conflicts=true — never auto-resolved; orphaned base pages the new version no longer ships are removed; the pin moves last. dry_run=true runs the full validation + conflict scan, applies nothing, and reports the plan (ok = a real run with the same arguments would apply: no conflicts, or conflicts with acknowledge_conflicts=true; issues stay listed either way). Refuses non-subscribed packs (pack_not_subscribed) and non-upgrades (pack_rebase_not_an_upgrade)
submit_promotion {tenant_id, candidate_id, vertical, skills} {manifest, tarball_b64, bytes, event_id} the L2→L3 harvest: propose a scrubbed, signed pack candidate from this node's own skills. Default-deny (REQ-PROMO-01): every id must be a tenant-authored SKILL page carrying the curator-set promotable: true marker — instances never promote, base-layer pages are the hub's; one ineligible id refuses the whole request (promotion_not_eligible). The deterministic scrubber (the export deny set) fails the submission closed on credential-shaped content (pack_secret_detected). Setting promotable: true via update_page is itself curator-gated (promotable_requires_curator for non-admin callers). Every submission emits an immutable audit event (source: "promotion", what/when/by-whom). Maker/checker: the candidate carries version: 0; a hub curator reviews and publishes deliberately. See ADR-0008
admin_audit {tenant_id} {markdown_not_in_duckdb: [...], indexed_but_no_markdown: [...]} drift detection (two-way diff)
rebuild {tenant_id} {done, total, current_page} recover the index from canonical markdown (blocking)
attach_external {tenant_id, source_url} {source_id} (derived catalog alias) attach an external read-only DuckDB source
register_credential {name, connector, secret} {ok} register a named external-source credential (server-side; secret never echoed) — see Instance backends
list_credentials {} {credentials: [{name, connector, created_at, created_by}]} enumerate registered credentials (names only)
delete_credential {name} {ok} remove a credential
create_sql_instance {skill, id, overlay_body?} {page_id, view} materialise a read-only sql_view instance from a sql_view skill
validate_bindings {} {bindings: [{page_id, view, status, detail?}]} re-probe every sql_view; binding_degraded ⇒ that view reads fail closed
register_endpoint / list_endpoints / delete_endpoint / validate_endpoints see Remote backends — remote-backend endpoint registry
create_remote_instance {skill, id, overlay_body?} {page_id, kind, endpoint} materialise an openapi/mcp overlay instance
admin_index_query {table, limit?} {rows, schema} read up to limit rows from an allow-listed index table (pages/blocks/links/crdt_ops/crdt_snapshots/chat_messages)
admin_list_lanes {} {lanes: [{name, backend, tenants_present}]} enumerate configured LaneStores
admin_lane_keys {lane?, prefix?, limit?} {keys: [{key, size_bytes}]} list lane keys under a prefix
admin_lane_blob {lane?, key} {bytes_base64, content_type} fetch one lane blob (≤ 1 MiB)
admin_webhook_deliveries {limit?} {configured, deliveries: [...]} recent outbound capture-webhook delivery outcomes
add_group_member {group_id, subject} {ok} add a principal to a custom RBAC group
remove_group_member {group_id, subject} {ok} remove a principal from a group
list_group_members {group_id} {members: [{group_id, subject, added_at, added_by}]} list a group's members (audit)
embedding_reload {} {model_revision} hot-reload the embedding model after a degraded start
compact_lanes {tenant_id} {ops_compacted, bytes_reclaimed} compact CRDT op lanes (CHECKPOINT + VACUUM + PRAGMA hnsw_compact_index, blocking)
admin_quota {tenant_id} {queries_remaining, writes_remaining, embeds_remaining, concurrent_sessions} inspect the per-tenant quota snapshot
admin_delete_chat_history {chat_group_id?, before_ts?, author?} {deleted} destructive purge of the conversation log

write_instance ({ref, payload} → the re-projected instance) is an agent tool, not admin-gated — it is authorised by the target instance's acl.update; see Remote backends.

There is no health MCP tool. Liveness/version are the plain HTTP endpoints GET /healthz + GET /readyz + GET /version (see the Transport summary).

The admin_audit and rebuild tools are the operational recovery path. The cost is ~32 ms/page; a 1000-page tenant rebuilds in ~32 s.

Long-running operations

rebuild, compact_lanes, tenant_export, and tenant_import take a while. They are ordinary blocking tools/call requests over POST /mcp: the call holds the connection open until the operation finishes and returns the final result in the JSON-RPC result (shapes in the table above). There is no streaming or SSE. SSE/streaming for progress is a possible future enhancement, not current behaviour.

tenant_export as the backup-contract producer

tenant_export is the only backup hook exposed by escurel-server. The server never writes to a backup bucket itself; external backup orchestrators (e.g. the substrate's tenant-export shipper named in ../deploy/substrate.md §4) call this endpoint on a schedule and ship the bytes to a durable target.

Contract relied on by orchestrators:

  • Read-only. Does not hold the tenant write lock; concurrent exports are allowed and foreground writes proceed normally during export.
  • Read-snapshots. Exports may include a small lag (one write transaction worst case) relative to the latest committed write.
  • Deterministic tarball. Canonical markdown only — the gzip'd tar contains the tenant's markdown/ tree and nothing else (the derivable DuckDB index, cache/, and spool/ are excluded; rebuild reconstructs the index from this markdown). See storage.md.
  • Blocking result. The call returns {format_version, tarball_b64, bytes, sha256} once the whole export is assembled: format_version is the export-format version (int) and sha256 is the hex digest of the tarball body, so consumers verify before treating the tarball as durable.
  • Failures. Surface as a JSON-RPC error object whose data carries retryable: bool (see §Error data). Retryable errors invite the consumer to re-issue the call; non-retryable errors indicate corruption and require operator intervention.
  • Idempotency is the consumer's responsibility. Re-running an export produces a new tarball with potentially different bytes (if writes happened in between); consumers key snapshots by {tenant_id, started_at}, not by content hash.

MCP-over-HTTP framing

Standard JSON-RPC 2.0 envelope wrapping each tool call. Tool-name mapping is:

search           → method = "tools/call", name = "search"
resolve          → method = "tools/call", name = "resolve"
...
update_page      → method = "tools/call", name = "update_page"

Tool discovery is the usual MCP tools/list response; every entry carries its JSON Schema input definition plus two additive labels: execution: "deterministic" | "orchestration" and scope: "agent" | "admin". tools/list is role-scoped: an agent-role token receives only the scope: "agent" subset (28 tools it can actually call); an admin token — and verifier-less dev mode — receives the whole surface. The admin role is still enforced at tools/call dispatch (-32001); the scope label is ratcheted against the dispatch arms by tool_registry_conformance, so the two cannot drift. The full (unfiltered) payload is also published as an OpenAPI 3.1 document at GET /openapi.json for non-MCP HTTP clients.

Long-running tools (rebuild, compact_lanes, tenant_export, tenant_import) block until done and return their final result in the JSON-RPC result. There is no SSE; live op streams use the WebSocket transport (/ws).

Error data (machine-readable refusals)

JSON-RPC refusals carry an additive error.data: { code, retryable, … } object. error.code / error.message are the frozen wire contract (never branch on message wording); data.code is the stable string a client branches on, and data.retryable tells it whether re-issuing the same call can ever succeed. Codes in use:

data.code numeric retryable meaning
admin_required -32001 no admin-gated tool, agent token
failed_precondition -32002 no e.g. foreign tenant_id on a single-tenant gateway
forbidden -32003 no authenticated but not permitted (ACL)
tenant_suspended -32003 no suspend gate; admin can resume
read_only_replica -32004 yes retry the same call against the writer
unsupported_on_replica -32005 no surface not wired on this replica
publish_unavailable -32006 no not a ducklake writer
quota_exhausted -32000 yes plus dimension, retry_after_ms
layer_read_only -32000 no pack-managed base page
session_cap_reached -32000 yes concurrent-sessions cap; retry later
unknown_session -32603 no session never opened / already closed — reopen
event_not_found -32602 no absent OR hidden by the event ACL (no existence oracle)
already_assigned -32602 no another worker won the assign_event CAS

Errors without data are unclassified internal faults; treat as non-retryable server errors. New codes are additive.

WebSocket framing

Single endpoint, /ws. Connection auth in the upgrade request (Authorization: Bearer ...). The upgrade applies the same tenant suspend gate as POST /mcp (#247): a suspended tenant refuses non-admin bearers with HTTP 403 (tenant_suspended) before the socket opens; an admin still connects (to resume). Once connected, the client sends a hello frame:

{ "type": "hello", "session": "sess_xyz" }    // attaches to an open CRDT session
// or
{ "type": "hello", "presence_only": true }    // presence + search subscriptions only

Message types:

type direction payload
op C→S { session, op: <Loro op> }
op_ack S→C { session, merged_version, content, conflicts, issues } — to the originator
peer_op S→C { session, merged_version, content, op } — to the other attached peers
presence bidi { session, user, anchor } (heartbeat every 10 s)
resync_required S→C { session, skipped, message } — this peer fell behind
search_subscribe C→S { subscription_id, q, k?, filter? } — live search: runs the real ACL-fused search now (the initial search_event is the ack) and re-runs it on every index mutation, pushing updated hits; a missing/empty q answers {type:"error", code:"invalid_subscription"}; presence-only connections
search_event S→C { subscription_id, hits: [...] } — the current results for the subscribed query, never a delta. A reconnecting client therefore needs no replay: it re-sends search_subscribe and the initial frame already carries whatever changed while it was away (asserted by a_resubscribe_carries_what_changed_while_the_client_was_gone). Subscriptions do not survive the socket, and are ACL-fused on every re-run, not only at subscribe time (a_subscription_is_not_a_side_channel_around_the_acl)
event_subscribe C→S { subscription_id, since_event_id? } — push freshly captured bus events (#333); presence-only connections
event_subscribe_ack S→C { subscription_id } — subscription is live
event S→C { subscription_id, event: <Event>, replayed? } — one captured event this caller may read (ESCUREL_EVENT_ACL filtered, same rule as list_inbox); replayed: true marks catch-up frames
event_lagged S→C { subscription_id, skipped, message } — push stream has gaps; poll list_inbox to catch up
close C→S { session, commit: bool }
error S→C { code, message }

The two connection modes (a state machine, not a flat table)

The hello frame chooses one of two modes irrevocably for the socket's lifetime:

  • session mode ({hello, session}) — the live-CRDT surface: op / op_ack / peer_op / presence (server-rebuilt, broadcast) / resync_required / close. event_subscribe is NOT available here and answers unknown_frame.
  • presence-only mode ({hello, presence_only: true}) — the subscription surface: event_subscribe (+ ack/event/ event_lagged), search_subscribe, and presence (echoed to the sender only, not broadcast).

A client that wants live co-editing AND event push holds two sockets.

Resume (since_event_id)

A reconnecting subscriber passes the last event id it processed as since_event_id: after the ack, the still-inbox events captured after that id are replayed oldest-first, marked replayed: true, before the live stream. This is a best-effort, inbox-only resume, not a gap-free one: the replay reads list_inbox, which is a queue, not an event log — an event that was assigned or processed while the subscriber was disconnected has left the inbox and is not replayed. A consumer that must not miss terminal transitions reconciles those via list_events on the instances it cares about. The server subscribes to the live bus before the replay query, so an event landing in between may arrive twice; dedupe by event_id (duplicates are recoverable). Ordering rides the event-id sort — exact for server-minted ULIDs; a caller-supplied id scheme must be monotonic to resume on. The replay window is the most recent 10 000 inbox rows; further behind than that, rebuild via list_inbox pagination.

Token lifetime

Auth happens once, at the upgrade. The server does NOT re-verify the bearer mid-connection: a socket outlives its token's exp, and an ACL revocation bites at the next attach (documented above), not immediately. Deployments that need a hard bound put an idle/lifetime limit on the proxy in front (kamal-proxy), or the consumer reconnects periodically — since_event_id replays what is still in the inbox across the reconnect (best-effort; see Resume above).

Multi-peer sessions

A session fans out. An op applied by one attached client is delivered to every other client attached to the same session as a peer_op, and a presence frame reaches the other peers (this is what makes live cursors possible). The originator receives op_ack and not peer_op — it has already been told its write landed, and applying its own edit twice is the bug a naive echo introduces.

peer_op carries the merged content as well as the raw op, so a peer can either apply the op to its local replica or render the merged text directly.

Attaching is a read, and is ACL-gated. A principal who may not read the page — the same may_read_instance decision expand applies — is refused the attach with { "type": "error", "code": "forbidden" }. Without this a session would be a side channel around the instance ACL, since a session's live content is exactly the material being edited.

Two properties a client must design for:

  • ACL is evaluated at attach, not per frame. Evaluating per frame would put a database round trip in front of every keystroke. The consequence is that an ACL revoked mid-session takes effect when that peer next attaches, not immediately. Disconnect a peer explicitly if you need revocation to bite at once.
  • Presence fields are server-rebuilt. The server emits the session it knows the sender is attached to and copies through only cursor, anchor and user. A peer cannot make another peer see a frame that claims a different session, and unrecognised keys are dropped rather than relayed.
  • There is no replay of missed frames. A peer that disconnects, or that falls far enough behind the per-session broadcast buffer to receive resync_required, must re-read the page (expand) and re-attach. The session's CRDT state is authoritative and lossless — what is not retained is the delivery history, so reconciliation is a re-read rather than a catch-up stream.

Live search subscriptions are the WS-only feature where the server pushes new hits as new pages are indexed (useful for agents watching for new events in a stream). Not v1; placeholder in the schema, off behind a feature flag.

Versioning

The MCP initialize handshake echoes the client's requested protocolVersion when present (default 2025-06-18) and reports serverInfo = { name: "escurel", version }, where version is the escurel-server crate semver (CARGO_PKG_VERSION) — not a bare 1.

There is no separate WebSocket protocol-version string (escurel-ws/1 is not implemented — the /ws upgrade performs no version negotiation).

The MCP tool JSON Schemas are served via the tools/list handshake so client implementations can pin to a version.