feat(sdk): mutable-catalog loader seam (upsert/remove/onChange)

.github/workflows/python.yml

@@ -78,3 +78,35 @@ jobs:
       - name: Run the telemetry example
         working-directory: examples/telemetry-python
         run: uv run --python 3.11 main.py
+
+  # Reference SKILL.md loader (ADR-0003/0005): pure Python, but depends on the SDK
+  # for the CatalogLoader seam, so it builds the local ratel-ai (maturin/native)
+  # before installing itself. Separate job because that build is maturin-specific.
+  local-skills:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: src/sdk/local-skills-py
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: src/sdk/python/native
+      - uses: astral-sh/setup-uv@v5
+      - name: Create venv + install (local ratel-ai via maturin, then this loader)
+        run: |
+          uv venv --python 3.11 .venv
+          # Install ratel-ai from the local source so the CatalogLoader seam this
+          # loader targets is present (the published wheel predates it). Then install
+          # this loader and its dev tools with --no-deps, so the local ratel-ai build
+          # is never replaced by the same-versioned published wheel.
+          uv pip install --python .venv ../python
+          uv pip install --python .venv --no-deps -e .
+          uv pip install --python .venv pyyaml types-PyYAML pytest pytest-asyncio ruff mypy
+      - name: Lint
+        run: .venv/bin/ruff check .
+      - name: Typecheck
+        run: .venv/bin/mypy ratel_ai_local_skills
+      - name: Test
+        run: .venv/bin/pytest

docs/adr/0003-catalog-source-interface.md

@@ -38,10 +38,27 @@ implementation of the already-published contract, not a new design.
   in-process registration (the floor) or from a **source loader** that pulls a published
   catalog and hydrates the local registries. Retrieval (`search_capabilities` /
   `invoke_tool` / `get_skill_content`) always runs locally over those registries.
-- `RATEL_URL` names a remote source and selects its loader; unset is the embedded floor.
-  Application code does not change ([ADR-0002](0002-product-split-engine-local-cloud.md)).
-- Additional sources (a local file/dir loader, git, a self-hosted endpoint) are added as
-  loaders when a use case appears.
+- The seam the SDK ships has two layers, not a loader framework. The floor is the
+  **mutable-catalog surface**: a loader pushes skills with `SkillCatalog.upsert` (returns the
+  added-vs-replaced signal), drops them with `remove`, reads current state with `get`/`has`,
+  and the host observes churn via `onChange` — the single staleness hook to re-emit
+  `tools/list_changed` and re-read a cached `search_capabilities` description on an
+  empty↔non-empty transition. On top of it the SDK formalizes a **loader lifecycle contract**,
+  `CatalogLoader` (`start(catalog)` / `stop` / `refresh`, each sync-or-async), attached with the
+  free function `attachLoader(catalog, loader)` (mirrors `registerMcpServer`) which starts the
+  loader and returns a `detach`/`refresh` handle; the catalog itself stays loader-blind. The
+  loader still **owns its own sync loop** and drives the catalog through `upsert`/`remove` — the
+  contract is lifecycle-only, with no SDK-owned snapshot diffing. A loader is any separate
+  package that holds a catalog and mirrors its source (the managed cloud, a DB, a local
+  file/dir, git, a self-hosted endpoint); loaders ship as separate packages,
+  `@ratel-ai/local-skills` / `ratel-ai-local-skills` (a directory of SKILL.md files,
+  [ADR-0005](0005-first-class-skills.md)) being the first, reference implementation.
+  Loader-scoped telemetry is deferred to the Cloud loader: an attach-scoped span would
+  misrepresent loop-based hydration, and a new local-trace event family is core work.
+- Loader-specific source selection and config — e.g. the Cloud loader's `RATEL_URL` and
+  bearer key — live in the loader package, not the SDK; the SDK stays source-agnostic.
+  Application code still does not change
+  ([ADR-0002](0002-product-split-engine-local-cloud.md)).
 
 ### The wire contract
 

docs/adr/0005-first-class-skills.md

@@ -23,9 +23,12 @@ sole loader of the skills it manages: the host must not auto-scan them.
 ### Source
 
 Skills load from a Ratel-managed folder (default `~/.ratel/skills/`) that the host does not
-auto-scan. With a remote source configured, the same catalog hydrates from the pull-sync
-contract instead ([ADR-0003](0003-catalog-source-interface.md)); either way the gateway is
-the only loader.
+auto-scan, served by the reference **local-skills loader** package (`@ratel-ai/local-skills` /
+`ratel-ai-local-skills`) that implements the [ADR-0003](0003-catalog-source-interface.md)
+`CatalogLoader` contract. With a remote source configured, the same catalog hydrates from the
+pull-sync contract instead ([ADR-0003](0003-catalog-source-interface.md)); either way the
+gateway is the only loader. The catalog is mutable at runtime (`upsert`/`remove`) and emits
+change notifications (`onChange`), so a loader can hydrate and evolve it after construction.
 
 ### Surface: one search call, two reserved buckets
 

pnpm-workspace.yaml

@@ -1,5 +1,6 @@
 packages:
   - 'src/sdk/ts'
+  - 'src/sdk/local-skills'
   - 'src/telemetry/ts'
   - 'src/telemetry/ts-otlp'
   - 'examples/*'

src/core/src/skill_registry.rs

@@ -120,6 +120,20 @@ impl SkillRegistry {
         });
     }
 
+    /// Remove a skill by id — see [`crate::ToolRegistry::remove`]. Returns
+    /// whether the id was present; an unknown id is a silent no-op.
+    pub fn remove(&mut self, skill_id: &str) -> bool {
+        if self.skills.shift_remove(skill_id).is_none() {
+            return false;
+        }
+        self.dense.invalidate(skill_id);
+        self.sink.record(TraceEvent::SkillChurn {
+            kind: ChurnKind::Remove,
+            skill_id: skill_id.to_string(),
+        });
+        true
+    }
+
     /// Number of registered skills (distinct ids).
     pub fn len(&self) -> usize {
         self.skills.len()
@@ -610,6 +624,99 @@ mod tests {
         )));
     }
 
+    #[test]
+    fn remove_drops_the_skill_and_returns_true() {
+        let mut reg = catalog();
+        assert!(reg.remove("api-design"));
+        assert_eq!(reg.len(), 1);
+        let hits = reg.search("design a REST endpoint with pagination", 5);
+        assert!(
+            !hits.iter().any(|h| h.skill_id == "api-design"),
+            "a removed skill never ranks"
+        );
+    }
+
+    #[test]
+    fn remove_unknown_id_is_a_silent_no_op() {
+        let sink = Arc::new(MemorySink::new("s"));
+        let mut reg = SkillRegistry::with_trace_sink(sink.clone());
+        reg.register(skill("s", "s", "REST API design", &["api"]));
+        sink.drain();
+        assert!(!reg.remove("missing"));
+        assert_eq!(reg.len(), 1);
+        assert!(sink.drain().is_empty(), "no churn event for an unknown id");
+    }
+
+    #[test]
+    fn remove_emits_remove_churn_once() {
+        let sink = Arc::new(MemorySink::new("s"));
+        let mut reg = SkillRegistry::with_trace_sink(sink.clone());
+        reg.register(skill("s", "s", "REST API design", &["api"]));
+        sink.drain();
+        reg.remove("s");
+        let events = sink.drain();
+        let removes: Vec<_> = events
+            .iter()
+            .filter(|e| {
+                matches!(
+                    &e.event,
+                    TraceEvent::SkillChurn {
+                        kind: ChurnKind::Remove,
+                        skill_id,
+                    } if skill_id == "s"
+                )
+            })
+            .collect();
+        assert_eq!(removes.len(), 1, "exactly one Remove churn event");
+    }
+
+    #[test]
+    fn semantic_search_succeeds_after_remove_without_rebuild() {
+        // Removing a skill drops its corpus entry and its cached vector together,
+        // so the cache still covers the corpus — no manual rebuild needed.
+        let mut reg = with_embedder(Arc::new(StubEmbedder));
+        reg.register(skill(
+            "api-design",
+            "api-design",
+            "REST API design",
+            &["api"],
+        ));
+        reg.register(skill(
+            "slides",
+            "slides",
+            "HTML slides frontend",
+            &["frontend"],
+        ));
+        reg.build_embeddings().unwrap();
+        assert!(reg.remove("api-design"));
+        let hits = reg
+            .search_with_method("frontend slides", 5, Origin::Direct, SearchMethod::Semantic)
+            .expect("no EmbeddingsNotBuilt after remove");
+        assert_eq!(hits.first().map(|h| h.skill_id.as_str()), Some("slides"));
+        assert!(!hits.iter().any(|h| h.skill_id == "api-design"));
+    }
+
+    #[test]
+    fn removed_then_re_registered_id_re_embeds_fresh() {
+        // Remove must invalidate the cached vector: a later register of the same
+        // id (a fresh insert, so register's replace path never fires) would
+        // otherwise reuse the stale embedding.
+        let mut reg = with_embedder(Arc::new(StubEmbedder));
+        reg.register(skill("s", "s", "REST API design", &["api"])); // dense: api bucket
+        reg.build_embeddings().unwrap();
+        reg.remove("s");
+        reg.register(skill("s", "s", "HTML slides frontend", &["frontend"])); // → frontend bucket
+        reg.build_embeddings().unwrap();
+        let hits = reg
+            .search_with_method("frontend slides", 5, Origin::Direct, SearchMethod::Semantic)
+            .unwrap();
+        assert_eq!(hits.first().map(|h| h.skill_id.as_str()), Some("s"));
+        assert!(
+            hits[0].score > 0.9,
+            "ranks with the freshly-embedded vector"
+        );
+    }
+
     #[test]
     fn register_and_search_emit_trace_events() {
         let sink = Arc::new(MemorySink::new("test-session"));

src/core/src/tool_registry.rs

@@ -165,6 +165,43 @@ impl ToolRegistry {
         });
     }
 
+    /// Remove a tool by id, dropping its corpus entry and its cached embedding
+    /// together — the cache keeps covering the corpus, so semantic/hybrid
+    /// searches keep working with no rebuild. `shift_remove` preserves the
+    /// insertion order of the survivors (deterministic ranking ties). Emits a
+    /// [`ChurnKind::Remove`] churn event on a hit; an unknown id is a silent
+    /// no-op (returns `false`, no event).
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use ratel_ai_core::{Tool, ToolRegistry};
+    ///
+    /// let mut registry = ToolRegistry::new();
+    /// registry.register(Tool {
+    ///     id: "read_file".into(),
+    ///     name: "read_file".into(),
+    ///     description: "Read a file".into(),
+    ///     input_schema: serde_json::json!({}),
+    ///     output_schema: serde_json::json!({}),
+    /// });
+    ///
+    /// assert!(registry.remove("read_file"));
+    /// assert!(!registry.remove("read_file")); // already gone
+    /// assert!(registry.is_empty());
+    /// ```
+    pub fn remove(&mut self, tool_id: &str) -> bool {
+        if self.tools.shift_remove(tool_id).is_none() {
+            return false;
+        }
+        self.dense.invalidate(tool_id);
+        self.sink.record(TraceEvent::IndexChurn {
+            kind: ChurnKind::Remove,
+            tool_id: tool_id.to_string(),
+        });
+        true
+    }
+
     /// Number of registered tools (distinct ids).
     pub fn len(&self) -> usize {
         self.tools.len()
@@ -963,6 +1000,97 @@ mod tests {
         assert!(hits.is_empty());
     }
 
+    #[test]
+    fn remove_drops_the_tool_and_returns_true() {
+        let mut reg = ToolRegistry::new();
+        reg.register(tool("read_file", "read a file"));
+        reg.register(tool("delete_file", "delete a file"));
+        assert!(reg.remove("read_file"));
+        assert_eq!(reg.len(), 1);
+        let hits = reg.search("read a file", 5);
+        assert!(
+            !hits.iter().any(|h| h.tool_id == "read_file"),
+            "a removed tool never ranks"
+        );
+    }
+
+    #[test]
+    fn remove_unknown_id_is_a_silent_no_op() {
+        let sink = Arc::new(MemorySink::new("s"));
+        let mut reg = ToolRegistry::with_trace_sink(sink.clone());
+        reg.register(tool("read_file", "read a file"));
+        sink.drain();
+        assert!(!reg.remove("missing"));
+        assert_eq!(reg.len(), 1);
+        assert!(sink.drain().is_empty(), "no churn event for an unknown id");
+    }
+
+    #[test]
+    fn remove_emits_remove_churn_once() {
+        let sink = Arc::new(MemorySink::new("s"));
+        let mut reg = ToolRegistry::with_trace_sink(sink.clone());
+        reg.register(tool("read_file", "read a file"));
+        sink.drain();
+        reg.remove("read_file");
+        let events = sink.drain();
+        let removes: Vec<_> = events
+            .iter()
+            .filter(|e| {
+                matches!(
+                    &e.event,
+                    TraceEvent::IndexChurn {
+                        kind: ChurnKind::Remove,
+                        tool_id,
+                    } if tool_id == "read_file"
+                )
+            })
+            .collect();
+        assert_eq!(removes.len(), 1, "exactly one Remove churn event");
+    }
+
+    #[test]
+    fn semantic_search_succeeds_after_remove_without_rebuild() {
+        // Removing a tool drops its corpus entry and its cached vector together,
+        // so the cache still covers the corpus — no manual rebuild needed.
+        let mut reg = catalog(Arc::new(StubEmbedder));
+        reg.build_embeddings().unwrap();
+        assert!(reg.remove("read_file"));
+        let hits = reg
+            .search_with_method(
+                "delete something",
+                5,
+                Origin::Direct,
+                SearchMethod::Semantic,
+            )
+            .expect("no EmbeddingsNotBuilt after remove");
+        assert_eq!(
+            hits.first().map(|h| h.tool_id.as_str()),
+            Some("delete_file")
+        );
+        assert!(!hits.iter().any(|h| h.tool_id == "read_file"));
+    }
+
+    #[test]
+    fn removed_then_re_registered_id_re_embeds_fresh() {
+        // Remove must invalidate the cached vector: a later register of the same
+        // id (a fresh insert, so register's replace path never fires) would
+        // otherwise reuse the stale embedding.
+        let mut reg = with_embedder(Arc::new(StubEmbedder));
+        reg.register(tool("t", "read a file")); // dense vec keyed on "read"
+        reg.build_embeddings().unwrap();
+        reg.remove("t");
+        reg.register(tool("t", "delete a file")); // fresh insert, keyed on "delete"
+        reg.build_embeddings().unwrap();
+        let hits = reg
+            .search_with_method("delete", 5, Origin::Direct, SearchMethod::Semantic)
+            .unwrap();
+        assert_eq!(hits.first().map(|h| h.tool_id.as_str()), Some("t"));
+        assert!(
+            hits[0].score > 0.9,
+            "ranks with the freshly-embedded vector"
+        );
+    }
+
     #[test]
     fn register_and_search_emit_trace_events() {
         let sink = Arc::new(MemorySink::new("test-session"));