feat(sdk): mutable-catalog loader seam (upsert/remove/onChange) - #112
Draft
rstagi wants to merge 12 commits into
Draft
feat(sdk): mutable-catalog loader seam (upsert/remove/onChange)#112rstagi wants to merge 12 commits into
rstagi wants to merge 12 commits into
Conversation
shift_remove keeps survivor order; dense.invalidate on hit so the cache keeps covering the corpus (semantic search needs no rebuild) and a removed-then-re-registered id re-embeds fresh. Unknown id: false, no event.
Native SkillRegistry.remove pass-through; catalog upsert returns the added-vs-replaced signal, remove drops entry+embedding together (no rebuild), onChange is the single staleness hook (Set-backed, snapshot iterate, throwing listeners isolated).
Native SkillRegistry.remove pass-through + .pyi stub; catalog mirror of the TS seam (set-backed listeners, snapshot iterate, raising listeners isolated).
Seam is SkillCatalog.upsert/remove + onChange; loader config (RATEL_URL, keys) lives in the loader package, SDK stays source-agnostic; catalog is runtime-mutable with change notifications.
… semantics register commits the mutation before the eager buildEmbeddings; a model failure must not swallow the staleness signal (notify in finally, both SDKs). New locks: listener unsubscribing/subscribing mid-notify, settled post-mutation state visible from listeners.
Formal lifecycle seam over the mutable-catalog surface (ADR-0003): start/stop/refresh, each sync-or-async. attachLoader free fn (mirrors registerMcpServer) starts the loader and returns a detach/refresh handle; double-attach guarded by a module-private WeakSet, detach idempotent and re-attachable. No SDK-owned diffing; no telemetry in v1.
Python mirror of the TS loader seam. CatalogLoader as a typing.Protocol (start/stop/refresh returning Awaitable[None] | None, so plain def and async def both conform under mypy strict); attach_loader absorbs sync-or-async via inspect.isawaitable; CatalogLoaderHandle dataclass mirrors McpServerHandle. WeakSet double-attach guard, idempotent detach, re-attach on failure.
Reference CatalogLoader (ADR-0003/0005): hydrate a SkillCatalog from a directory of <name>/SKILL.md files (default ~/.ratel/skills). YAML frontmatter via the yaml package (SDK stays dependency-lean); non-recursive sorted scan; field defaults + type validation with per-scan diagnostics; refresh diffs by raw-text (no re-embed of untouched files) and only removes ids it loaded, never foreign skills. New pnpm-workspace package + folder README + src/sdk/README.
Python mirror of @ratel-ai/local-skills: LocalSkillsLoader subclasses the CatalogLoader Protocol (mypy strict enforces conformance) with plain sync lifecycle methods that attach_loader absorbs. pyyaml frontmatter parsing, same layout contract, defaults, diagnostics, and raw-text refresh diff. New hatchling package + folder README + src/sdk/README; CI job builds the local ratel-ai (maturin) then installs the loader with --no-deps.
0003: the seam now has two layers — the mutable-catalog surface plus a formal CatalogLoader lifecycle contract (start/stop/refresh) attached via attachLoader; loader owns its loop, loaders ship as separate packages, local-skills is the reference, loader telemetry deferred. 0005: the managed folder is served by the reference local-skills loader package.
…arity
The yaml package defaults to YAML 1.2 core (on/off/yes/no are strings, dup keys
throw) while pyyaml.safe_load is YAML 1.1 (booleans, dup keys last-wins), so the
two mirrored loaders parsed the same SKILL.md differently. Parse TS frontmatter
with { version: '1.1', uniqueKeys: false } to match. Lock the shared behavior
with tests in both packages; align the py README build steps to the CI flow.
Reconcile the loader-facing mutable-catalog seam with main's async batch register (embedding off-thread, typed errors, configurable models): - SkillCatalog.upsert now yields the added-vs-replaced signal after the awaitable settles; register/remove/onChange keep the seam, notifyChange fires synchronously on the metadata commit. - Public buildEmbeddings folded into register (buildDense) per main. - SkillRegistry facades (TS+Py) grow remove; TS native remove takes the RwLock write guard. - local-skills(-py) loaders await upsert; loader lifecycle is async.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ships the loader seam end-to-end: the mutable-catalog surface, the formal
CatalogLoaderlifecycle contract on top of it, and a reference loader that proves it on a real source.Mutable-catalog surface (SDK + core).
SkillCatalog.upsert(added-vs-replaced signal),remove, andonChange(the single staleness hook fortools/list_changedre-emits and capability-description re-reads), TS + Python parity. Core gains symmetricSkillRegistry::remove/ToolRegistry::remove(shift_remove+ dense-cache invalidate, first emitter ofChurnKind::Remove), so semantic search keeps working after a remove with no rebuild.registernotifies in afinally, so a failed eager embed can't swallow the staleness signal for an already-committed mutation.CatalogLoaderlifecycle contract (SDK).start(catalog)/stop/refresh, each sync-or-async; attached with the free functionattachLoader(catalog, loader)/attach_loader(...)(mirrorsregisterMcpServer) which starts the loader and returns adetach/refreshhandle. The loader owns its own sync loop and drives the catalog throughupsert/remove— lifecycle-only, no SDK-owned diffing. Double-attach guarded (WeakSet); detach idempotent + re-attachable; astartfailure keeps partial hydration and re-allows attach; astopfailure still detaches. TSinterface/ Pythontyping.Protocol(first in the SDK; mypy-strict conformance for both plaindefandasync def). No loader telemetry in v1 (deferred to the Cloud loader).Reference loader (two new packages).
@ratel-ai/local-skillsandratel-ai-local-skillsload a directory of<name>/SKILL.mdfiles (default~/.ratel/skills, ADR-0005) into a catalog — real YAML frontmatter (aligned to YAML 1.1 across both SDKs so they parse identically), non-recursive sorted scan, field defaults + type validation with per-scan diagnostics, and a raw-text refresh diff that skips untouched files and never removes foreign skills. Shipped separately so the SDK stays dependency-lean; both are the first implementations of the loader seam, mirroring the planned@ratel-ai/cloud/ratel-ai-cloudlayout.ADR-0003/0005 amended: the seam now has two layers (mutable surface + formal
CatalogLoader), and the managed folder is served by the reference local-skills loader package.Extracted from #100, rebuilt on latest main. Publishing the two new packages is a separate follow-up (RC-first, first publish manual); the concrete Cloud loader packages ship in a separate PR.