From 50721cab841573ee68f75950a88c2df3ac1e7cb0 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 25 Aug 2026 15:29:10 -0400 Subject: [PATCH 01/55] docs(client): document Agent Skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the feature now that all of it exists. The README section and the contributor notes both span every layer in the stack, so they land once rather than as six partial edits to the same two files. `README.md` gains the public API — the accessors, `writeSkills`, the store seam, the exported constants, and the types — and states two things plainly rather than leaving a reader to discover them. Content delivery is not wired up in this release: everything runs against `SkillStore`, the shipped default is absent, and the accessors throw an actionable error until one is configured. And the security posture, including the limitation: Node exposes no `renameat`/`unlinkat`, so the final rename cannot be performed relative to a pinned directory descriptor, and an attacker who already has write permission on the managed root can still win a race to redirect a write or a delete outside it. Keep the managed root writable only by the process running the SDK. Two behaviours a reader would otherwise be surprised by are stated where the conservatism itself is described. First, the one narrow exception to "a file you placed yourself is never overwritten": a colliding file whose bytes *already are* the resolved content is adopted rather than refused, which is what lets a reconcile killed between the content writes and the final manifest write heal itself instead of wedging those skills forever. Nothing is overwritten, because nothing needs to be. Differing bytes, and bytes that cannot be read at all, are still refused and left untouched. Second, the two key constraints this layer imposes that the data model does not — the 255-byte path-component bound and the 22 Windows reserved device names — including why a key like `aux` still parses everywhere else, and the residual the SDK cannot check: the bound is per component, so a deep root plus a long key can still exceed Windows' `MAX_PATH`. `agents.md` records what a future change has to preserve: the module ownership rows, the one-way dependency direction (`types` <- `skills-core` <- `skills`, and `types` + `safe-fs` + `skills-core` <- `skills-fs`) and why closing a cycle there would let the accessor and filesystem layers disagree about whether a store is configured, and a Dependencies table giving a reason for every runtime dependency and optional peer. Skill content is documented as what it now is: `Skill.content` is a `Uint8Array` of the verified verbatim bytes, opaque to the SDK — no frontmatter accessor, no YAML dependency, and consumers who want structure parse the bytes themselves. Four new pitfalls, each one a mistake that would pass review: interpreting skill content anywhere in the SDK, reading `writeSkills`'s `timeout` as milliseconds, relaxing a path or manifest check in `skills-fs.ts`, and bypassing `fsOps` for a destructive call — which makes the operation invisible to the assertions that prove no operation was attempted, so they then pass vacuously. The path-check pitfall carries the three specifics most likely to be widened by someone who has not read the reasoning: why the reserved-name rejection is unconditional rather than platform-gated and why it lives in `keyRejectionReason` rather than in the key grammar (a grammar-level rejection would invalidate the whole AI Config, and a dropped `skillRefs` entry lets prune delete the on-disk copy — "fails to write on Windows" becoming "gets deleted on Linux"); that adoption is a narrowing of the clobber refusal and requires an exact hash match; and that every flag in `readRegularFile` is load-bearing, `O_NONBLOCK` most of all, since opening a FIFO with no writer blocks forever. Also renames one pre-existing test to describe its behaviour directly instead of citing a specification section. Workspace 1148 passing (+2 skipped). typecheck, biome, and sherif clean. Co-Authored-By: Claude Fable 5 --- packages/client/README.md | 215 ++++++++++++++++++- packages/client/agents.md | 96 ++++++++- packages/client/src/__tests__/client.test.ts | 2 +- 3 files changed, 308 insertions(+), 5 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 628a24b5..81e69e88 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -80,7 +80,7 @@ await shutdown(); | Export | Description | |---|---| | `initClient(options?)` | Auto-discover and initialize `@launchdarkly/node-server-sdk`. Optional — the first AI API call triggers lazy init when `LD_SDK_KEY` is set. Returns `Promise`. | -| `initClient(client)` | **BYOC overload** — accept a pre-initialized `LDClientInterface` (e.g. from `@launchdarkly/vercel-server-sdk` or any edge runtime). Skips SDK auto-discovery. | +| `initClient(client, options?)` | **BYOC overload** — accept a pre-initialized `LDClientInterface` (e.g. from `@launchdarkly/vercel-server-sdk` or any edge runtime). Skips SDK auto-discovery. | | `getClient()` | Return the initialized `LDClientInterface`. Throws if `initClient` has not completed. | | `shutdown()` | Flush all events and telemetry, then close the client. Call before process exit. | | `waitForTelemetry()` | Wait for the OTel provider to be ready. Useful to avoid dropping early spans. | @@ -225,6 +225,208 @@ if (!result.enabled) { --- +### Agent Skills + +Agent Skills are versioned `SKILL.md` documents managed in LaunchDarkly, attachable to AI Config variations by reference. This package surfaces which skills a config references, retrieves their content, and materializes them onto disk where agent runtimes (the Claude Agent SDK and friends) can discover them. + +**Content delivery is not wired up in this release.** Everything below runs against the `SkillStore` seam; the shipped default is *absent*, so the accessors throw an actionable error until a store is configured. `InMemorySkillStore` is provided for local development, tests, and bring-your-own-content. The real transport ships in a follow-up. + +```ts +import { createHash } from 'node:crypto'; +import { + allSkills, + getSkill, + getSkillResult, + getSkills, + initClient, + InMemorySkillStore, + skillRefs, + writeSkills, +} from '@launchdarkly/ai-server'; + +// A store serves wire-shaped raw objects. `contentHash` is sha256, lowercase +// hex, over the verbatim UTF-8 bytes of `content` — content that does not hash +// to it is withheld, so this is not a field to hand-wave. +const content = '---\nname: PDF Extraction\n---\nExtract text from PDFs.\n'; +const store = new InMemorySkillStore(); +store.put({ + key: 'pdf-extraction', + version: 2, + content, + contentHash: createHash('sha256').update(Buffer.from(content, 'utf-8')).digest('hex'), +}); + +// Configure it as part of ordinary initialization... +await initClient({ skillStore: store }); +// ...or alongside a pre-initialized client on an edge runtime: +// await initClient(myEdgeClient, { skillStore: store }); + +// Which skills does a resolved config reference? A pure projection — no I/O, +// and it works before any client exists. +const refs = skillRefs({ + model: { name: 'claude-opus-4-5' }, + provider: { name: 'Anthropic' }, + instructions: 'Summarize the attached document.', + skills: [{ key: 'pdf-extraction', version: 2 }], +}); // [{ key: 'pdf-extraction', version: 2 }] + +// In real use the config comes from LaunchDarkly: +// const info = await inspectConfig('doc-agent', { kind: 'user', key: 'user-123' }); +// const refs = skillRefs(info.config); + +// Retrieve content. Every skill is hash-verified before you see it. +// `Skill.content` is a Uint8Array — the verified verbatim bytes, exactly what +// was hashed. The SDK never interprets them; decode or parse them yourself. +const newest = await getSkill('pdf-extraction'); // newest available +const pinned = await getSkill('pdf-extraction', { version: 2 }); // exact version, or null +const batch = await getSkills(refs); // input order; misses omitted +// `getSkill` returns null for four different reasons. When you need to tell them +// apart — to fail closed on suspected tampering — ask for the outcome instead. +const outcome = await getSkillResult('pdf-extraction', { version: 2 }); +if (outcome.reason === 'integrity_failure') throw new Error(outcome.detail ?? 'withheld'); +const everything = await allSkills(); +const text = new TextDecoder().decode(newest?.content); // if you want a string + +// Materialize onto disk at //SKILL.md. Only the leaf directory is +// ever created, so `.claude` must already exist — a typo in the path is an +// error, not an invitation to scatter a directory tree. +const report = await writeSkills(refs, '.claude/skills'); +if (!report.ok) { + for (const action of report.errors) { + console.error(`skill ${action.key}: ${action.error}`); + } +} +``` + +`writeSkills` is a **reconcile**, not a copy. It records what it wrote in `/.launchdarkly-skills.json` and only ever overwrites or deletes paths that manifest lists under a matching key — a file you placed yourself is reported as an error and left alone. Revocation is pruning: a skill absent from the resolved set is removed on the next run. + +One narrow exception keeps that conservatism from becoming a trap. The manifest is written atomically, once, last, so a process killed after a skill file lands but before that write would leave the file at a managed path with no manifest entry — which is exactly what an unmanaged collision looks like, so every later reconcile would refuse it and the skill would stay stuck. So a colliding file whose bytes **already are** the resolved content is adopted: recorded in the manifest and reported `skipped_current`, and a crashed reconcile heals itself on the next run. Nothing is overwritten, because nothing needs to be — the only file ever adopted is one that is already byte-identical to what LaunchDarkly resolved. A file whose bytes differ, or one that cannot be read at all, is still refused and left untouched. + +```ts +// Everything the store holds, materialized at boot. +const report = await writeSkills('*', '.claude/skills', { + prune: true, // default — remove formerly-managed skills no longer resolved + timeout: 10, // SECONDS, not milliseconds (cross-language contract) + onUnavailable: 'keep', // 'keep' reports a failed retrieval; 'raise' throws +}); +``` + +| Export | Description | +|---|---| +| `skillRefs(config)` | Project a config's `skills` array into typed `SkillReference[]`. Pure — no client, no store, no telemetry. `[]` when absent. | +| `getSkill(key, { version? })` | One verified skill. Omit `version` for the newest available. Resolves to `null` for not-found or a version mismatch; throws only when no store is configured. | +| `getSkillResult(key, { version? })` | The same retrieval as `getSkill`, reported instead of collapsed. Resolves to `{ skill, reason, detail }`, where `reason` is one of `ok` / `absent` / `integrity_failure` / `store_unavailable` / `wrong_version`. Throws only when no store is configured — same as `getSkill`. See [fail closed on tampering](#fail-closed-on-tampering-getskillresult). | +| `getSkills(refs)` | Batch form. Accepts `SkillReference` values and bare key strings (string = latest). Results follow input order; missing or unverifiable entries are omitted. | +| `allSkills()` | Every verified skill the store holds. | +| `writeSkills(skills, root, options?)` | Materialize to `//SKILL.md`. Accepts `Skill` / `SkillReference` / key strings, or the literal `'*'`. Returns a `ReconcileReport`. Throws for a caller error — an unusable `root`, a bare string other than `'*'` — as distinct from the per-skill `error` actions in the report. | +| `InMemorySkillStore` | A `SkillStore` backed by a plain object. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`. Holds one object per key, so it answers a version pin with what it has and lets the accessor refuse a mismatch. | +| `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | +| `createSkillOutcome(init)` | Build a frozen `SkillOutcome`. Exported for tests and for wrapping your own retrieval in the same shape. | +| `SKILL_OBJECT_KIND` | `'skill'` — the delivery object kind. | +| `SKILL_FILENAME` | `'SKILL.md'`. | +| `MANIFEST_FILENAME` | `'.launchdarkly-skills.json'` — add this to your `.gitignore` if you do not commit materialized skills. | +| `MANIFEST_VERSION` | `1`. | +| `MAX_SKILL_CONTENT_BYTES` | `10485760` (10 MiB) — the hard cap. Content above it is withheld. A backstop against absurd input, set well above the platform's own limit, which is enforced before delivery; do not read it as that limit. | + +`ReconcileReport` exposes `actions`, `ok` (true iff no action is an `error`), and `errors` (the error actions, in order), so callers never re-derive the filter. Each `ReconcileAction` carries `key`, `action` (`written` | `updated` | `skipped_current` | `removed` | `error`), and nullable `version` / `path` / `error`. A failure belonging to the whole run rather than one skill — a corrupt manifest, for instance — carries the **empty string** in `key`. + +**Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly: Node exposes no `renameat`/`unlinkat`, so the final rename cannot be performed relative to a pinned directory descriptor. An attacker who already has **write permission on the managed root** can therefore still win a race to redirect a write or a delete outside it. Keep the managed root writable only by the process running the SDK. + +**Two constraints on skill keys, imposed here rather than by the data model.** A key becomes a single directory name, so `writeSkills` rejects — as a reported `error` action, on every platform — a key over 255 bytes and the 22 Windows reserved device names (`con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9`). Both are checked in the filesystem layer only: a key like `aux` remains valid everywhere else, so an AI Config referencing it still parses and its other skills still materialize. One residual the SDK cannot check for you: the 255-byte bound is per *path component*, not on the total path, so `//SKILL.md` can still exceed Windows' 260-character `MAX_PATH` if the root is deep and the key is long. The root is yours, so budget for it there. + +**Skill content is opaque to the SDK.** A verified `Skill` carries `content` as a `Uint8Array` — the exact bytes that were hashed — and the SDK never parses, decodes, or interprets it anywhere. There is deliberately no frontmatter accessor and no YAML dependency: a consumer that wants the frontmatter decodes the bytes and parses them itself, with whatever parser and bounds it trusts. + +**No LaunchDarkly telemetry is emitted for skills.** Signals go through an internal no-op emitter; `client.track()` is never called and no LD context is involved. + +#### Observability: integrity failures are logged for your SIEM + +A skill that fails integrity verification is **withheld** — the accessor returns `null`, `writeSkills` reports an `error` action, and no unverified byte reaches your agent. Every failure additionally writes one line to `console.error` in a fixed, machine-parseable shape, so that withholding is *detectable* and not merely correct: + +```text +[LaunchDarkly] ld.skills.integrity_failure {"action":"withheld","event":"ld.skills.integrity_failure","expected_hash":"5f2b...","language":"typescript","observed_hash":"9c14...","reason":"content hash mismatch","reason_code":"hash_mismatch","skill_key":"pdf-extraction","version":2} +``` + +The line is a `[LaunchDarkly] ` prefix, the event name, a space, and a single JSON object. To ingest it: match `ld.skills.integrity_failure`, take everything from the first `{`, parse it. + +**The record is written regardless of how you configure telemetry.** It is not sampled, not batched, and not conditional on a LaunchDarkly connection — no LaunchDarkly telemetry is emitted for skills at all. If you send LaunchDarkly nothing, this log record is your complete detection surface for tampered or malformed skill content. + +**`ld.skills.integrity_failure` is a stability commitment.** The event name will not be renamed, and no field will be renamed or removed, outside a major release with a changelog entry. It is safe to build an alert on. + +| Field | Always present | Value | +|---|---|---| +| `event` | yes | `ld.skills.integrity_failure`. | +| `action` | yes | `withheld` — what the SDK did about it. Content that fails verification is never returned and never written to disk. | +| `skill_key` | yes | The skill key, or `` when the key itself failed validation. Keys arrive from the wire, so one is never echoed verbatim. | +| `reason_code` | yes | A stable token from the closed vocabulary below. Alert on this field, not on `reason`. | +| `reason` | yes | Human-readable detail, including byte counts. Wording may change between releases. | +| `language` | yes | `typescript`. Distinguishes SDKs in a polyglot fleet; the Python SDK emits the same record with `python`. | +| `version` | no | The skill version. Omitted when the delivered version was not an integer >= 1. | +| `expected_hash` | no | The `contentHash` delivered with the content, or `` when it was not 64 lowercase hex characters. Omitted when the failure happened before any hash was read. | +| `observed_hash` | no | The sha256 this SDK computed locally. Omitted when the failure happened before hashing. | + +Optional fields are **omitted, never null** — the absence of `observed_hash` means no hash was computed, which is itself information. Skill content, filesystem paths, and credentials never appear in the record; the key and the expected hash are shape-checked and replaced with the placeholders above when they do not look like what they claim to be, so a hostile store cannot use the log line to exfiltrate a skill body. + +| `reason_code` | What happened | +|---|---| +| `not_an_object` | The store served something that is not an object. | +| `invalid_key` | The key does not match `^[a-z0-9][a-z0-9-]*$` within 256 characters. | +| `invalid_version` | The version is not an integer >= 1. | +| `missing_content` | `content` is absent or not a string. | +| `missing_content_hash` | `contentHash` is absent or not a string. | +| `not_utf8` | The content has no UTF-8 encoding (a lone surrogate), so there are no bytes LaunchDarkly could have hashed. | +| `over_size_cap` | The content exceeds `MAX_SKILL_CONTENT_BYTES`, so it is inauthentic whatever it hashes to. | +| `hash_mismatch` | The content does not hash to the `contentHash` delivered alongside it. | + +These eight tokens are the whole vocabulary, and the Python SDK emits the same eight for the same conditions — including identical JSON key order — so one parser and one alert rule cover a polyglot fleet. + +#### Fail closed on tampering: `getSkillResult` + +The log record above is the *operator's* view. Your application gets the same distinction programmatically from `getSkillResult`, which returns the outcome instead of collapsing it: + +```ts +import { getSkillResult } from '@launchdarkly/ai-server'; + +const outcome = await getSkillResult('pdf-extraction', { version: 2 }); + +switch (outcome.reason) { + case 'ok': + return outcome.skill; // non-null exactly here + case 'integrity_failure': + // Content and its declared digest disagreed. Do not proceed on a fallback: + // the safe interpretation is that skill delivery is being tampered with. + console.error(`refusing to start: ${outcome.detail}`); + process.exit(1); + case 'absent': + // Nobody configured this skill, or it was revoked. Ordinary; carry on. + return null; + case 'wrong_version': + // The pinned version is not what the store holds — a rollout skew, usually. + return null; + case 'store_unavailable': + // The store could not answer. An outage, not an answer of "no" — retry or + // run degraded, but do not treat it as a revocation. + return null; +} +``` + +| `reason` | `skill` | What happened | +|---|---|---| +| `ok` | the skill | Retrieved and verified. | +| `absent` | `null` | The store holds nothing under that key. Not configured, not yet delivered, or revoked. | +| `integrity_failure` | `null` | Content failed verification and was withheld. The `ld.skills.integrity_failure` record above was written for the same failure. | +| `wrong_version` | `null` | A version was pinned and the store answered with a different one. | +| `store_unavailable` | `null` | The store threw. Nothing was retrieved, so nothing is known either way. | + +`detail` carries the human-readable reason for every non-`ok` outcome and is `null` for `ok`. It is safe to log or surface: it names the key, the requested and held versions, and the failure category, and never skill content or a filesystem path. + +**`getSkill` is unchanged.** It still resolves to `null` for all four failure outcomes and still rejects only when no store is configured — no existing caller has to change anything. The two accessors run the identical lookup and the identical verification, and differ *only* in what they report: `getSkill` gives you the skill or `null`, `getSkillResult` also gives you the reason. Nothing is retried, cached, or logged twice — by the time either returns, an integrity failure has already written its record — so reaching for `getSkillResult` costs nothing but the wider return type. + +These five tokens are the whole vocabulary, and the Python SDK publishes the same five for the same conditions. There is deliberately no batch equivalent: `getSkills` and `allSkills` still omit entries they could not return, so call `getSkillResult` per key where the outcome matters. + +**`hash_mismatch` deserves a page, not a dashboard.** The other codes are consistent with a malformed store, a bad deployment, or a truncated response. `hash_mismatch` means content and its declared digest disagree, which is the shape of active tampering with skill delivery — in transit, in a cache, or in whatever backs your `SkillStore`. If you serve skill content only from LaunchDarkly, `over_size_cap` and `not_utf8` warrant alerts on the same reasoning: neither should ever occur. + +--- + ### Utility Helpers ```ts @@ -257,3 +459,14 @@ All types are re-exported from this package. Handler packages import them from h | `GraphNode` / `GraphEdge` | A node (evaluated agent config + edges) and a directed edge (with handoff data) | | `ProviderGraphResponse` | The value returned by `graph(...).invoke()`: `{ response, usage, trackData, judgeResults? }` | | `GraphTopology` | The parsed graph flag shape (`root` + `edges`) | +| `Skill` | A verified skill document: `key`, `version`, `content` (`Uint8Array` — the verified verbatim bytes), `contentHash`, `name`, `description` | +| `SkillReference` | A version-pinned pointer to a skill: `{ key, version }` | +| `SkillOutcome` | What `getSkillResult()` resolves to: `{ skill, reason, detail }`. `skill` is non-null exactly when `reason` is `'ok'` | +| `SkillOutcomeReason` | The closed set of retrieval outcomes: `'absent' \| 'integrity_failure' \| 'ok' \| 'store_unavailable' \| 'wrong_version'` | +| `SkillStore` | The structural seam skill content is retrieved through: `getObject(kind, key, version?)`, `allObjects`, optional `addListener` | +| `RawSkillObject` | The wire shape a `SkillStore` serves, before verification. Every field is untrusted. | +| `ReconcileReport` | The result of `writeSkills()`: `{ actions, ok, errors }` | +| `ReconcileAction` | One outcome from a reconcile: `{ key, action, version, path, error }` | +| `ReconcileActionKind` | The closed set of reconcile outcomes: `'written' \| 'updated' \| 'skipped_current' \| 'removed' \| 'error'` | +| `OnUnavailable` | `'keep' \| 'raise'` — how `writeSkills` reacts to content it could not retrieve | +| `WriteSkillsOptions` | Options accepted by `writeSkills()` (`prune`, `timeout` in seconds, `onUnavailable`) | diff --git a/packages/client/agents.md b/packages/client/agents.md index bed95ad1..da08dcde 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -25,11 +25,17 @@ No other `@launchdarkly/ai-*` package may define or duplicate these. They import | `src/client.ts` | `config()` | | `src/tracking.ts` | `executeAndTrack`, `executeAndStream`, `wrapToolHandlers` | | `src/graph.ts` | `graph()`, `resolveGraph()` | -| `src/types.ts` | All shared TypeScript types — including owned `LDContext`, `LDClientInterface`, `LDClientInterface` | +| `src/types.ts` | All shared TypeScript types — including owned `LDContext`, `LDClientInterface`, `LDClientInterface` — plus the Agent Skills value types, their freezing factories, and `parseAiConfig`'s validators | | `src/utils.ts` | `parseTemplate`, `parseJSONWithPossibleFences`, `createHandler` | | `src/registry.ts` | `Registry`, `globalRegistry`, `compose` | +| `src/skills-core.ts` | Agent Skills internals shared by the two layers above it: the store and telemetry seams, module state, integrity verification, store resolution | +| `src/skills.ts` | `skillRefs`, the content accessors, `InMemorySkillStore`, and the documented test-injection hooks | +| `src/skills-fs.ts` | `writeSkills` — the manifest format, on-disk filenames, and reconcile semantics | +| `src/safe-fs.ts` | Symlink-refusing filesystem primitives. Knows nothing about skills; owns the single interceptable rename and unlink call sites | | `src/index.ts` | Public barrel — the only surface handler packages import from | +The Agent Skills modules are a deliberate split, and the dependencies run **one way only**: `types` ← `skills-core` ← `skills`, and `types` + `safe-fs` + `skills-core` ← `skills-fs`. `skills-core.ts` imports neither `skills.ts` nor `skills-fs.ts`. Do not add an edge that closes a cycle; the reason the store and the emitter live in `skills-core.ts` is so the accessor layer and the filesystem layer cannot disagree about whether one is configured. + --- ## Public Exports (`src/index.ts`) @@ -51,6 +57,17 @@ export { Registry, globalRegistry, compose } from './registry.js'; export { parseTemplate, parseJSONWithPossibleFences, createHandler } from './utils.js'; export { graph, resolveGraph } from './graph.js'; export { parseUsage, normalizeMode, parseAiConfig } from './tracking.js'; + +// Agent Skills +export { skillRefs, getSkill, getSkillResult, getSkills, allSkills, InMemorySkillStore } from './skills.js'; +export { SKILL_OBJECT_KIND, MAX_SKILL_CONTENT_BYTES } from './skills-core.js'; +export { writeSkills, SKILL_FILENAME, MANIFEST_FILENAME, MANIFEST_VERSION } from './skills-fs.js'; +export type { WriteSkillsOptions } from './skills-fs.js'; +export { createSkill, createSkillOutcome, createSkillReference } from './types.js'; +export type { + Skill, SkillOutcome, SkillOutcomeReason, SkillReference, SkillStore, RawSkillObject, + ReconcileAction, ReconcileActionKind, ReconcileReport, OnUnavailable, +} from './types.js'; ``` When adding a new export, add it here. Handler packages must never import from sub-paths (e.g. `@launchdarkly/ai-server/dist/client`). @@ -180,13 +197,42 @@ When `enabled` is `false`, `config` is always `null`. When `enabled` is `true` b - **Lazy initialization.** Importing the package does not initialize the LD client. The first API call that needs LaunchDarkly (`extractVariation`, graph resolution, etc.) calls `initClient()` internally, provided `LD_SDK_KEY` is set. - **Explicit initialization — Node SDK path.** `initClient(options?)` dynamically imports `@launchdarkly/node-server-sdk` at runtime (optional peer dep). If the package is not installed it throws with a clear message. -- **Explicit initialization — BYOC path.** `initClient(client)` accepts any pre-initialized object that satisfies `LDClientInterface` — this is the path for Vercel, Cloudflare, or other edge runtimes whose SDK has different init semantics. No `@launchdarkly/node-server-sdk` is required. +- **Explicit initialization — BYOC path.** `initClient(client, options?)` accepts any pre-initialized object that satisfies `LDClientInterface` — this is the path for Vercel, Cloudflare, or other edge runtimes whose SDK has different init semantics. No `@launchdarkly/node-server-sdk` is required. The optional second argument carries the same options bag as the other overload, which is how a BYOC caller configures `skillStore`. +- **`skillStore` is the one option applied on every call.** Every other option is ignored once the client singleton exists. `skillStore` is applied *before* the idempotency check, so a client that initialized lazily — or without a store — can be given one afterwards. A nullish `skillStore` never clears a configured store; only `shutdown()` does that, and it clears the skills state unconditionally, ahead of its own early return, because that state can exist without a client. - **Return value.** `initClient()` returns `Promise`. Callers that don't need the instance may discard the return value — this is a non-breaking change from the previous `Promise` signature. - `getClient()` throws if `initClient()` has not resolved — any code that calls `getClient()` directly must ensure initialization has occurred. - `shutdown()` must be called before process exit. It flushes OTel spans, flushes LD events, and closes the LD client. `client.close()` runs even when `flush()` throws. --- +## Dependencies + +Tier 0, so the runtime surface is deliberately tiny: two hard dependencies, and everything else either optional or dev-only. Nothing here may grow without a reason recorded in this table. + +### Runtime (`dependencies`) + +| Package | Why | +|---|---| +| `@opentelemetry/api` | The tracer/span API used on every instrumented path (`tracking.ts`, `graph.ts`, `content.ts`, `utils.ts`). API-only — the *SDK* half is an optional peer, so a consumer that never configures OTel still gets no-op spans rather than a crash. | +| `dotenv` | `.env` loading for `LD_SDK_KEY` and the OTel endpoint variables, imported as `dotenv/config` from `lifecycle.ts`. | + +### Optional peers (`peerDependencies`, every one `optional: true`) + +| Package | Why | +|---|---| +| `@launchdarkly/node-server-sdk` | The default Node client, imported dynamically by `initClient()`'s options overload. Optional because the BYOC overload (`initClient(client)`) targets edge runtimes that supply their own client, and requiring it would force an unused Node SDK into every Vercel/Cloudflare install. Absent ⇒ a clear throw, only on the path that needs it. | +| `@opentelemetry/sdk-trace-node`, `@opentelemetry/sdk-trace-base`, `@opentelemetry/resources`, `@opentelemetry/core`, `@opentelemetry/context-async-hooks` | Tracer provider, span processor, resource attributes, propagators, and async context — loaded dynamically by `setupTelemetry()`. Optional so telemetry is opt-in; see [OTel Setup](#otel-setup) for the install command. | +| `@opentelemetry/exporter-trace-otlp-http`, `@opentelemetry/otlp-exporter-base` | OTLP/HTTP export and its compression enum. Same optionality, same loader. | + +### Dev-only (`devDependencies`) — the ones with a contract attached + +| Package | Why | +|---|---| +| the optional peers, mirrored | Each optional peer is repeated here so the test suite can import it. A peer that is *only* a peer would not be installed in this workspace and its tests could not run. | +| `vitest`, `typescript`, `@types/node` | Test runner, compiler, and Node type definitions. | + +--- + ## Common Pitfalls ### 1. Calling `getClient()` before `initClient()` resolves @@ -197,6 +243,28 @@ When `enabled` is `false`, `config` is always `null`. When `enabled` is `true` b `shutdown()` is idempotent — calling it a second time is a no-op. However, if `client.flush()` throws during the first call, the singleton is still cleared so a second `shutdown()` call will not re-attempt the flush or throw. Ensure your process-exit handler does not assume `shutdown()` will re-try a failed flush. If you need guaranteed delivery, call `client.flush()` yourself and handle the error before calling `shutdown()`. +### 3. Interpreting skill content anywhere in the SDK + +`Skill.content` is a `Uint8Array` — the verified verbatim bytes, exactly what was hashed — and the SDK treats it as opaque. There is deliberately no frontmatter accessor, no YAML dependency, and no decode step outside integrity verification (which encodes the wire string to bytes exactly once, in `skills-core.ts`). Do not add a parser, a convenience accessor, or an encoding assumption; consumers who want structure parse the bytes themselves. This is a cross-language contract with the Python SDK. + +### 4. Assuming `writeSkills`'s `timeout` is in milliseconds + +It is in **seconds**, defaulting to `10`. The signature is a cross-language contract that must match the Python SDK exactly, so the usual TypeScript `timeoutMs` instinct is wrong here. + +### 5. Relaxing a path or manifest check in `skills-fs.ts` + +`keyRejectionReason` and `unsafePathReason` are shared by the write and prune paths precisely so the two cannot disagree about which paths this SDK may destroy, and both are **non-relaxable**. The same goes for the manifest rules: a destructive operation is permitted only on a path the manifest lists under a matching key, and a corrupt manifest suppresses every destructive action. Each of these has a dedicated abuse-case test in `src/__tests__/skills-fs.test.ts`; if one starts failing, the defense is what changed, not the test. + +Three specifics inside those two functions that a later contributor is most likely to widen: + +- **The 22 Windows reserved device names** — `con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9` — are rejected by `keyRejectionReason`, unconditionally on every platform. Do not add a `process.platform === 'win32'` gate: a managed root written by a Linux container and read from a Windows host is an ordinary deployment, so the on-disk result must not depend on which OS ran the reconcile, and with no Windows CI runner in either repo a platform branch would be untestable. Do not move the check into `isValidSkillKey` or `SKILL_KEY_PATTERN` either. `parseAiConfig` fails closed on a bad `skills` entry, so a grammar-level rejection would invalidate the *whole* AI Config — model, provider, instructions, tools — for a customer who never touches Windows, and `skillRefs` would silently drop the reference, which lets `writeSkills` prune the skill's on-disk copy. That turns "fails to write on Windows" into "gets deleted on Linux". The 255-byte path-component bound lives in this layer for the same reason. The set is exactly the reserved names: `com0` and `lpt0` are not reserved, and no case folding or suffix stripping is needed because the key grammar admits no uppercase, no `.`, and no `$`. +- **The adoption rule is a narrowing of the clobber refusal, not a hole in it.** A file at a managed path with no manifest entry is adopted — recorded, and reported `skipped_current` — only when its on-disk sha256 equals the resolved content hash, which is what lets a reconcile killed between the content writes and the final manifest write heal itself instead of wedging those skills forever. Adopt on anything weaker than an exact hash match and the guarantee is gone. `skipped_current` is reused deliberately: adding a member to `ReconcileActionKind` would break every consumer with an exhaustive `switch`. +- **`readRegularFile` is what makes that read safe**, and every part of it is load-bearing: `O_NONBLOCK`, because opening a FIFO with no writer blocks forever and would hang the reconcile along with the event loop; `O_NOFOLLOW`; and an `fstat` on the *handle* rather than a `stat` on the path, refusing anything that is not a regular file. A failed read is a refusal and never a fall-through to the write. There is no `O_BINARY` — Node does no CRLF translation — which is the one place this deliberately differs from the Python twin. + +### 6. Bypassing `fsOps` for a destructive filesystem call + +`safe-fs.ts` routes the final rename and the managed-file unlink through the `fsOps` record so tests can intercept exactly those two operations. The orphaned-temp sweep goes through `unlinkNoFollow` for the same reason, and derives its filename pattern from `tempNamePattern` in `safe-fs.ts` rather than carrying a copy: that sweep is only entitled to unlink a file because the *name* identifies it as one this SDK created, so two spellings of the naming rule would eventually let it either miss orphans or remove something it did not write. Calling `fs.rename`/`fs.unlink` directly makes the operation invisible to the atomicity and "no operation was attempted" assertions, which then pass vacuously. Note also the limitation those tests document: Node exposes no `renameat`/`unlinkat`, so `SUPPORTS_DIR_FD` is `false` and the TOCTOU swap-race tests are skipped — the residual exposure is real and recorded, not fixed. + --- ## Adding a New Export @@ -211,5 +279,27 @@ When `enabled` is `false`, `config` is always `null`. When `enabled` is `true` b - Do not add dependencies on any `@launchdarkly/ai-*` handler package. This package has no upward dependencies. - Do not add a hard dependency on `@launchdarkly/node-server-sdk` or any other LD SDK. The node SDK must remain an optional peer, discovered via dynamic `import()`. - Handler packages and consumer code must import `LDContext` from `@launchdarkly/ai-server` — not directly from any LD SDK. The owned definition in `src/types.ts` is structurally compatible with all LD SDK versions. -- Do not weaken the `parseAiConfig` validation — handler packages rely on `config` being valid when they receive it. +- Do not weaken the `parseAiConfig` validation — handler packages rely on `config` being valid when they receive it. The `skills` array in particular fails **closed**: one malformed reference fails the whole parse, because silently dropping it would materialize a partial skill set without telling anyone. +- Do not interpret skill content anywhere — no frontmatter parsing, no YAML dependency, no encoding assumption beyond the one wire-string encode inside integrity verification. See pitfall 3. +- Skills telemetry goes through the private emitter seam in `skills-core.ts` and nowhere else. **Never** call `client.track()` for a skills operation: these are LaunchDarkly product-analytics signals, not customer analytics, and `track()` would require an LD context, spend the customer's event volume, and land in their data export. Exactly three signal names exist (`AgentControl Skill Integrity Failure`, `AgentControl Skill Materialized`, `AgentControl Skill Revoked Received`) and they are an allowlist, not a floor — in particular, `AgentControl Skill SDK Reference Returned` and `AgentControl Skill Content Retrieved` must **never** be emitted. Every signal is constructed by a `record*` function in `skills-core.ts`, so the allowlist is enforceable by reading one section of one file. +- No filesystem paths and no skill content in telemetry — hashes and byte counts only. Paths belong in the `ReconcileReport`, which is user-facing API. +- The `ld.skills.integrity_failure` log record is a **documented customer-facing contract**, not a debug line. It is constructed in exactly one place — `recordIntegrityFailure` — and written to `console.error` on every failure, unconditionally, because it is the only detection surface a customer with telemetry off has and the only one that can exist where no telemetry destination is reachable. Its keys are inserted in **alphabetical order** so `JSON.stringify` is byte-identical to the Python SDK's `json.dumps(record, sort_keys=True, separators=(",", ":"))` for the same failure, modulo `language`; do not reorder them. Absent optional fields are omitted, never emitted as `null`. Renaming the event, renaming or dropping a field, or emitting a null is a breaking change to a security control — the README documents this record for customers to alert on. +- `reason_code` is a **closed eight-token vocabulary**, one token per call site of `recordIntegrityFailure`: `hash_mismatch`, `invalid_key`, `invalid_version`, `missing_content`, `missing_content_hash`, `not_an_object`, `not_utf8`, `over_size_cap`. A ninth token means adding it in the Python SDK, in the README's `reason_code` table, and in the vocabulary test in `src/__tests__/skills.test.ts`, in the same change — a token that exists on one side only silently breaks a customer's cross-language alert rule. `reason_code` deliberately stays **out** of the telemetry properties: the three signals' property keys are the allowlist above, and the vocabulary belongs to the customer-owned detection path rather than to LaunchDarkly's counter. +- `SkillOutcomeReason` is a **closed five-token vocabulary** and the public half of what `resolveFromStore` decides: `absent`, `integrity_failure`, `ok`, `store_unavailable`, `wrong_version`. It is published by `getSkillResult` as API, customers branch on it, and the Python SDK publishes the same five for the same conditions — so a sixth token is a cross-language change, and it means adding it in the Python SDK, in the README's `reason` table, and in the union-exhaustiveness test in `src/__tests__/skills.test.ts`, in the same change. Note this is a **different and coarser** vocabulary from `IntegrityReasonCode`: the eight integrity tokens say *which check failed* and go to the operator through the log record; the five outcome tokens say *what the caller got* and are the actionable programmatic surface. `verifyRawSkill` returns `null` and does not surface which integrity token fired, deliberately — plumbing the finer token into `SkillOutcome` would change that function's return type for a detail the operator already has. +- `Resolution.reason` is set **explicitly at every construction site**, never derived from `Resolution.error`. Pattern-matching a prose string to decide what a customer's fail-closed branch sees is the fragility the typed outcome exists to remove, and the field is required precisely so a new internal outcome has to pick a public token rather than inherit `'absent'` by omission. The mapping today: + + | Where the resolution is built | `reason` | + |---|---| + | `resolveFromStore` — the store threw (also sets `unavailable`) | `store_unavailable` | + | `resolveFromStore` — `raw` is not an object (null, an array, a scalar) | `absent` | + | `resolveFromStore` — `verifyRawSkill` returned `null` | `integrity_failure` | + | `resolveFromStore` — `skill.version !== wantedVersion` | `wrong_version` | + | `resolveFromStore` — success | `ok` | + | `skills-fs.ts` `resolveReference` — deadline exhausted, or no store configured (both set `unavailable`) | `store_unavailable` | + + Adding a seventh row means answering "which of the five does a caller see?" before writing the code. `unavailable` stays a separate field rather than folding into `reason`: it is narrower, it is what suppresses pruning, and the two bottom rows above never reach an accessor at all. +- `wantedVersion` is passed **into** `SkillStore.getObject(kind, key, version)`, and the post-hoc `skill.version !== wantedVersion` check is kept anyway. The parameter is there because a store may hold several versions of one key and only the store can pick between them; the equality check is a **defense**, not the selection mechanism, because the store is untrusted. Removing either one is wrong: without the parameter a satisfiable pin gets reported as `wrong_version`, and without the check a lying store gets its answer through. `InMemorySkillStore` holds one object per key and so ignores the parameter by design — it answers with what it has rather than with `null`, so a pin it cannot satisfy reports `wrong_version` rather than `absent`. Giving it real multi-version semantics is a separate change that pulls in `allObjects` and `allSkills`. +- `getSkill`'s contract — "resolves to `null`, never rejects; throws only when no store is configured" — is **frozen**. It is documented in its JSDoc and in the README, and every existing caller treats that `null` as "no skill", so a reason must be added *alongside* it (as `getSkillResult` was) and never by changing what `getSkill` returns. `getSkillResult` is a projection of the same `Resolution` and shares the single throw; it records no telemetry and writes no log record of its own, because the integrity record already fired inside verification before the resolution returned and reporting it again would double-count one failure. `getSkills` and `allSkills` deliberately have no reporting equivalents yet. +- This package has no logger abstraction — every `console.*` call site carries a `biome-ignore` saying so. Introducing a `logger` option is a package-wide API decision affecting unrelated call sites, not a skills change; until one exists, the integrity record has to be self-describing in the string it logs, which is why the event name appears both in the prefix and in the JSON. +- Signal names, property keys, wire field names (`contentHash`), the manifest filename and format, and the exported constants are **identical strings** to the Python SDK. A polyglot fleet has to reconcile the same directory identically, so changing one of these is a cross-language breaking change. - `parseUsage` must continue to accept `input_tokens/output_tokens`, `inputTokens/outputTokens`, and `input/output` as all existing handlers return one of these variants. diff --git a/packages/client/src/__tests__/client.test.ts b/packages/client/src/__tests__/client.test.ts index 21f9b2c2..b859c7d4 100644 --- a/packages/client/src/__tests__/client.test.ts +++ b/packages/client/src/__tests__/client.test.ts @@ -276,7 +276,7 @@ describe('config() — single handler', () => { expect(result.response).toBe('{"name":"Alice"}'); }); - it('returns raw string when outputFormat is set but output cannot be parsed as JSON (§3.10 best-effort)', async () => { + it('returns raw string when outputFormat is set but output cannot be parsed as JSON (best-effort)', async () => { const outputFormat = { type: 'object', properties: { name: { type: 'string' } } }; (extractVariation as ReturnType).mockResolvedValue({ config: { ...mockConfig, outputFormat }, From c8545548a4680e3fec6ae6a4a9d68c7882d51c75 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 4 Sep 2026 10:20:03 -0400 Subject: [PATCH 02/55] =?UTF-8?q?docs(client):=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20close=20out=20the=20security=20review's=20SDK-side?= =?UTF-8?q?=20remainder?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Node half of the four items left open in the response to the Agent Skills security design review. Three are documentation, one is tests; no behavior changes, and `safe-fs.ts`'s implementation is deliberately untouched. **Privilege separation** (row 9 docs half, and the agreed counter-proposal for row 26). The recommended deployment runs the reconcile as a different identity than the agent, which is the whole reason the `0644`/`0755` modes deny anything: the agent reads its instructions and cannot rewrite them, or the manifest. That is the mitigation for AZ-1, a prompt-injected agent editing its own skills. Write access to the manifest is the worse half — it is what tells the *next* reconcile which paths the SDK may delete — which is why the prune path re-validates every entry from scratch rather than trusting it. It matters more here than in Python, because this runtime's residual race *requires* write permission on the managed root, and privilege separation is what denies it. The README's security-posture note now points at the new section instead of ending on a bare "keep the root writable only by the SDK". The README also hands the operator the check to run, because the SDK cannot run it: it knows only its own identity, which trivially has write access, having just written there. So `ReconcileReport` grows no writability field — the review asked for one and we declined, since any check the SDK could make would answer a different question than the one asked and manufacture false confidence exactly where caution is wanted. `agents.md` records that reasoning so the field is not added later by someone reading its absence as an oversight. **Three hostile-manifest prune tests** (row 12 remainder): a well-formed manifest listing `/etc/passwd`, `../../../etc/passwd`, and a path under a parent that has since become a symlink. The prune path already refuses all three, so these turn asserted into verified, and they mirror the Python suite case for case. Two things make them worth more than their line count. They are deliberately *well-formed* — the corrupt-manifest block above them proves nothing here, because a corrupt manifest suppresses every destructive action wholesale, whereas these manifests give the implementation everything it needs to prune. And "deleted nothing" is asserted through an `fsOps.unlink` spy rather than by checking that `/etc/passwd` still exists: the test process cannot delete that file anyway, so the obvious assertion would pass against an implementation with no path check at all. **One sentence on** `'*'` (row 16 remainder). It materializes the whole project library, so every skill's `description` enters the agent's context — including skills no AI Config references and skills belonging to other teams. **The platform bound is now explicit** (row 2 residual), in `safe-fs.ts`, `agents.md` and the README — and it is a sharper bound here than in Python. Node exposes no `*at()` family on *any* platform, so the racy per-component `lstat` floor is not the Windows fallback it is in Python; it is the only implementation, Linux included. The README previously said this of the rename alone; it is true of every destructive step. Windows reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) are not implemented, by decision: Windows is not a supported or tested platform for this release, neither repository has a Windows CI runner, and Node offers no primitive that would make such checks meaningful here anyway. This retroactively lowers the priority of the row 25 reserved-device-name work, noted where that code lives. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 33 +++++- packages/client/agents.md | 7 ++ .../client/src/__tests__/skills-fs.test.ts | 107 ++++++++++++++++++ packages/client/src/safe-fs.ts | 20 ++++ 4 files changed, 166 insertions(+), 1 deletion(-) diff --git a/packages/client/README.md b/packages/client/README.md index 81e69e88..90ced584 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -311,6 +311,12 @@ const report = await writeSkills('*', '.claude/skills', { }); ``` +**Know what `'*'` asks for.** It materializes the **whole project library**, which puts every +skill's `description` into the agent's context — including skills no AI Config references and +skills belonging to other teams. `writeSkills(skillRefs(config), root)`, the form used in the +example above, materializes only what the resolved variation actually asked for. Reach for +`'*'` when you genuinely want the entire library on disk. + | Export | Description | |---|---| | `skillRefs(config)` | Project a config's `skills` array into typed `SkillReference[]`. Pure — no client, no store, no telemetry. `[]` when absent. | @@ -330,7 +336,7 @@ const report = await writeSkills('*', '.claude/skills', { `ReconcileReport` exposes `actions`, `ok` (true iff no action is an `error`), and `errors` (the error actions, in order), so callers never re-derive the filter. Each `ReconcileAction` carries `key`, `action` (`written` | `updated` | `skipped_current` | `removed` | `error`), and nullable `version` / `path` / `error`. A failure belonging to the whole run rather than one skill — a corrupt manifest, for instance — carries the **empty string** in `key`. -**Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly: Node exposes no `renameat`/`unlinkat`, so the final rename cannot be performed relative to a pinned directory descriptor. An attacker who already has **write permission on the managed root** can therefore still win a race to redirect a write or a delete outside it. Keep the managed root writable only by the process running the SDK. +**Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly, and it is a platform bound rather than a detail: Node exposes no `renameat`/`unlinkat`/`openat`, so no destructive step can be performed relative to a pinned directory descriptor. Every check here is therefore a per-component `lstat` taken immediately before the path-based operation — a check-then-use race, not a closed window — and unlike the Python SDK, which closes that window on POSIX by holding a descriptor opened `O_RDONLY|O_DIRECTORY|O_NOFOLLOW` for the whole reconcile, **this floor is what runs on every platform Node supports, Linux included.** An attacker who already has **write permission on the managed root** can therefore win a race to redirect a write or a delete outside it. Windows adds nothing worse and nothing better: reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) are deliberately not implemented in this release, and Windows is not a tested platform for it — neither SDK repository has a Windows CI runner. So write permission on the managed root is *the* security boundary here. Keep it writable only by the identity running the reconcile — see [privilege separation](#privilege-separation-the-agent-must-not-be-able-to-rewrite-its-own-skills). **Two constraints on skill keys, imposed here rather than by the data model.** A key becomes a single directory name, so `writeSkills` rejects — as a reported `error` action, on every platform — a key over 255 bytes and the 22 Windows reserved device names (`con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9`). Both are checked in the filesystem layer only: a key like `aux` remains valid everywhere else, so an AI Config referencing it still parses and its other skills still materialize. One residual the SDK cannot check for you: the 255-byte bound is per *path component*, not on the total path, so `//SKILL.md` can still exceed Windows' 260-character `MAX_PATH` if the root is deep and the key is long. The root is yours, so budget for it there. @@ -425,6 +431,31 @@ These five tokens are the whole vocabulary, and the Python SDK publishes the sam **`hash_mismatch` deserves a page, not a dashboard.** The other codes are consistent with a malformed store, a bad deployment, or a truncated response. `hash_mismatch` means content and its declared digest disagree, which is the shape of active tampering with skill delivery — in transit, in a cache, or in whatever backs your `SkillStore`. If you serve skill content only from LaunchDarkly, `over_size_cap` and `not_utf8` warrant alerts on the same reasoning: neither should ever occur. +#### Privilege separation: the agent must not be able to rewrite its own skills + +**The recommended deployment runs `writeSkills` as a different identity than the agent.** Reconcile as one user, run the agent as another. Everything the reconcile puts on disk is owner-write-only, and set explicitly rather than inherited from the process umask: skill files and the manifest at `0644` (applied to the open file handle, so it cannot be redirected), the per-skill `//` directories at `0755`, and the execute bit never set on anything. Those modes are only a defense if the two identities actually differ — under a single identity they describe a directory the agent can freely rewrite. + +**What to verify, as the identity that will run the agent.** The SDK cannot check this for you (see below), so make it a deployment step: confirm the agent's identity has no write access to + +- the managed root itself, +- the per-skill directories `//` and the files `//SKILL.md`, +- the manifest at `/.launchdarkly-skills.json`. + +```bash +# Run as the agent's user. Every line should print DENIED. +root=.claude/skills +for target in "$root" "$root/.launchdarkly-skills.json" "$root"/*/ "$root"/*/SKILL.md; do + [ -e "$target" ] || continue + if [ -w "$target" ]; then echo "WRITABLE — fix this: $target"; else echo "DENIED: $target"; fi +done +``` + +The managed root's own mode is **yours, not the SDK's**: `writeSkills` creates only that one leaf directory, and does so with the process umask, precisely because the root is a path you chose. Own it — `chown reconcile-user:agent-group` plus `chmod 0755` on the root is the shape that makes the rest of the tree's modes mean something. It is also the mitigation for the race described under *Security posture* above: that race requires write permission on the managed root, which privilege separation is what denies. + +**Why this is the mitigation that matters.** A `SKILL.md` is agent *instructions*. An agent that can write its own skills directory can rewrite its own instructions, and an agent processing untrusted input is exactly the thing that might be induced to do so. Write access to the manifest is worse than write access to a skill, because the manifest is what tells the *next* reconcile which paths the SDK owns and may delete: an agent that can edit it can keep a skill LaunchDarkly has revoked, or aim the SDK's own delete path at something it should not touch. `writeSkills` re-validates every manifest entry from scratch for exactly that reason — it treats that file as untrusted input, never as authorization — but an agent that cannot edit it at all is the stronger position, and only your deployment can provide that. + +**The SDK deliberately does not report whether the root is writable.** There is no such field on `ReconcileReport`, and its absence is a decision rather than an oversight. The SDK knows only its own identity, which trivially has write access — it just wrote there. It cannot know which identity will later run the agent, so any check it could make would answer a different question than the one that matters, and would read as reassurance exactly where caution is wanted. You know both identities; the SDK knows one. + --- ### Utility Helpers diff --git a/packages/client/agents.md b/packages/client/agents.md index da08dcde..0e874551 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -265,6 +265,13 @@ Three specifics inside those two functions that a later contributor is most like `safe-fs.ts` routes the final rename and the managed-file unlink through the `fsOps` record so tests can intercept exactly those two operations. The orphaned-temp sweep goes through `unlinkNoFollow` for the same reason, and derives its filename pattern from `tempNamePattern` in `safe-fs.ts` rather than carrying a copy: that sweep is only entitled to unlink a file because the *name* identifies it as one this SDK created, so two spellings of the naming rule would eventually let it either miss orphans or remove something it did not write. Calling `fs.rename`/`fs.unlink` directly makes the operation invisible to the atomicity and "no operation was attempted" assertions, which then pass vacuously. Note also the limitation those tests document: Node exposes no `renameat`/`unlinkat`, so `SUPPORTS_DIR_FD` is `false` and the TOCTOU swap-race tests are skipped — the residual exposure is real and recorded, not fixed. +### 7. "Fixing" the platform bound, or adding a writability field to `ReconcileReport` + +Two decisions here look like unfinished work and are not. Do not reverse either without reopening the security review. + +- **Windows reparse-point checks (`GetFileAttributesW`, `FILE_FLAG_OPEN_REPARSE_POINT`) are deliberately not implemented.** Windows is not a supported or tested platform for this release, neither repository has a Windows CI runner, and Node gives this module no `*at()` primitive that would make such checks meaningful anyway — the racy per-component `lstat` floor is universal here, Linux included, not a Windows-only fallback as it is in Python. The bound is documented in `safe-fs.ts` and the README instead. It also retroactively lowers the priority of the reserved-device-name work in `skills-fs.ts`: keep that code, since it keeps a root written on Linux usable when read from Windows, but do not read it as evidence that Windows is hardened. If Windows becomes supported, add the CI runner first and revisit both together. +- **`ReconcileReport` must not grow a "managed root is writable" field.** The security review asked for one; we declined, and the reasoning is load-bearing. The SDK knows only its *own* identity, which trivially has write access — it just wrote there — and cannot know which identity will later run the agent. Any check it could perform would answer a different question than the one asked and would manufacture false confidence exactly where caution is wanted. The real mitigation is deployment-side: run the reconcile as a different identity than the agent, so the `0644`/`0755` modes deny something and a prompt-injected agent cannot rewrite its own instructions or the manifest. The operator's verification steps live in the README. + --- ## Adding a New Export diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index a21a33cf..92d375ed 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -1673,6 +1673,113 @@ describe('writeSkills corrupt manifest', () => { }); }); +// ─── A well-formed manifest naming a path the SDK could not have written ───── + +/** + * The three literal cases the security review names for the prune path. + * + * The distinction from the corrupt-manifest block above is the whole point: a + * corrupt manifest suppresses every destructive action wholesale, so those tests + * say nothing about these. Each manifest here is *well-formed* — parseable, a + * `manifestVersion` this release understands, a real `entries` map, and an entry + * whose `key` is a perfectly valid skill key genuinely absent from the requested + * set. The implementation has every input it needs to prune, and must refuse + * anyway, because the recorded *path* is not one this SDK could have written. + * + * The manifest is untrusted input: a plain file on the customer's disk that + * anything with write access to the managed root can edit, and `prune` is the + * one code path in the SDK that deletes. So a recorded path never authorizes its + * own removal — it must match `/SKILL.md` for a re-validated key, and the + * target is recomputed from the *current* managed root rather than read back out + * of the entry. + */ +const HOSTILE_RECORDED_PATHS: string[] = [ + // Absolute: the classic. A recorded path read back and unlinked as-is is a + // delete of an attacker-chosen file with the reconcile's own privileges. + '/etc/passwd', + // Traversing: the same attack against an implementation that rejects a + // leading slash and then joins the rest onto the managed root. + '../../../etc/passwd', +]; + +/** + * Records every prune unlink without performing it. + * + * Asserting only that `/etc/passwd` still exists proves nothing: the test + * process cannot delete it anyway, so that assertion passes against an + * implementation with no path check at all — permissions would be doing the + * work. What has teeth is that the removal is never *attempted*: the refusal + * happens above the syscall, on a path the SDK recomputes rather than trusts. + * `fsOps.unlink` is the single call site the prune deletes through. + */ +function recordUnlinks(): string[] { + const targets: string[] = []; + vi.spyOn(fsOps, 'unlink').mockImplementation(async (target: string) => { + targets.push(target); + }); + return targets; +} + +describe('writeSkills hostile manifest prune', () => { + it.each(HOSTILE_RECORDED_PATHS)('refuses a recorded path outside the root: %s', async (recorded) => { + const unlinked = recordUnlinks(); + await writeManifest(root, { + manifestVersion: 1, + entries: { [recorded]: manifestEntry('a', 1, SKILL_BODY) }, + }); + + const report = await writeSkills([], root); + + expect(report.ok).toBe(false); + const action = actionsByKey(report).a; + expect(action.action).toBe('error'); + // The refusal is about ownership of the path, not about the file's state. + expect(action.error).toContain('could own'); + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + // Nothing was even attempted, let alone completed. + expect(unlinked).toEqual([]); + expect(await exists('/etc/passwd')).toBe(true); + // Left in place rather than tidied away: dropping the entry would let a + // single hostile edit erase the SDK's own record of what it manages. + expect(await readManifest(root)).toHaveProperty(['entries', recorded]); + }); + + it('refuses an entry under a parent that has since become a symlink', async () => { + // The recorded path is the SDK's own, and is still not enough. The entry is + // exactly what a legitimate reconcile writes — `a/SKILL.md` under key `a` — + // so the shape check that catches the two cases above passes here. What + // changed is the disk underneath it. This is the case a validate-then-act + // implementation fails: the manifest and the entry are both entirely + // legitimate, and only the current state of the parent is not. + const elsewhere = path.join(scratch, 'elsewhere'); + await mkdir(elsewhere); + const victim = path.join(elsewhere, SKILL_MD); + await writeFile(victim, 'victim content\n', 'utf-8'); + + // Managed legitimately first, so the manifest entry is one this SDK really + // did write... + const managed = await placeManaged(root, 'a', SKILL_BODY); + // ...then the parent directory is swapped for a link out of the root. + await rm(managed); + await rm(path.join(root, 'a'), { recursive: true }); + await symlink(elsewhere, path.join(root, 'a'), 'dir'); + + const unlinked = recordUnlinks(); + const report = await writeSkills([], root); + + expect(report.ok).toBe(false); + const action = actionsByKey(report).a; + expect(action.action).toBe('error'); + expect(action.error).toContain('symlink'); + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + expect(unlinked).toEqual([]); + // The file the symlink pointed at is untouched, and so is the link. + expect(await readFile(victim, 'utf-8')).toBe('victim content\n'); + expect((await lstat(path.join(root, 'a'))).isSymbolicLink()).toBe(true); + expect(await readManifest(root)).toHaveProperty(['entries', `a/${SKILL_MD}`]); + }); +}); + // ─── Telemetry seam (write half) ─────────────────────────────────────── describe('writeSkills telemetry', () => { diff --git a/packages/client/src/safe-fs.ts b/packages/client/src/safe-fs.ts index cdefe074..6e73707b 100644 --- a/packages/client/src/safe-fs.ts +++ b/packages/client/src/safe-fs.ts @@ -8,6 +8,26 @@ * opened with `O_NOFOLLOW`, every temp file is created exclusively in the target's * own directory, and the pinned directory's identity is re-checked immediately * before each destructive step. + * + * **Platform bound — the floor is what runs everywhere, deliberately.** The + * Python SDK closes the swap window on POSIX with a descriptor walk; this module + * cannot, on any platform, because Node exposes no `*at()` family at all (see + * {@link SUPPORTS_DIR_FD}). So unlike Python, where the racy floor is the + * Windows-only fallback, here it is the *only* implementation — Linux included. + * Windows is additionally not a supported or tested platform for this release: + * reparse-point checks (`GetFileAttributesW`, or opening with + * `FILE_FLAG_OPEN_REPARSE_POINT`) are **not implemented, by decision rather than + * oversight**, since neither SDK repository has a Windows CI runner and Node + * gives this module no primitive that would make them meaningful. + * + * The consequence is a single sentence, and it belongs in every deployment + * review: write permission on the managed root is *the* security boundary for + * skills materialization, so the privilege-separated deployment the README + * documents — reconcile identity separate from agent identity — is not advice but + * the mitigation. Relatedly, this bound retroactively lowers the priority of the + * Windows reserved-device-name work in `skills-fs.ts`: that code stays, because + * it keeps a managed root written on Linux usable when read from Windows, but it + * is not evidence that Windows is a hardened target. It is not. */ import { randomBytes } from 'node:crypto'; From 6e2ea64943dcf53c59fb34200e2cb1abe68831b6 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 11 Sep 2026 16:30:40 -0400 Subject: [PATCH 03/55] fix(client): a broken store answer is not an absent skill Two untrusted-store answers were read as data rather than as failures. `allRawObjects` collapsed a non-object listing to `{}` with no error, so a store that served nothing usable was indistinguishable from one holding no skills. `resolveFromStore` read identity off the object without checking it against the key that was asked for, so an answer served under a different key came back under the caller's key while carrying its own. Both are now withheld and reported, alongside the version check that already guarded the same way. `allRawObjects` now returns the listing together with the reason it could not answer, with a throwing store caught and worded in the same place, so `allSkills` and the `'*'` reconcile path no longer each re-derive the log line and the message. Ports launchdarkly/python-ai-sdk 5817d89. Co-Authored-By: Claude Fable 5.1 --- packages/client/src/__tests__/skills.test.ts | 74 +++++++++++++++++++- packages/client/src/skills-core.ts | 64 ++++++++++++++--- packages/client/src/skills-fs.ts | 9 +-- packages/client/src/skills.ts | 14 +--- 4 files changed, 132 insertions(+), 29 deletions(-) diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 7c8af037..c27489c5 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -46,7 +46,7 @@ import { InMemorySkillStore, skillRefs, } from '../skills.js'; -import { MAX_SKILL_CONTENT_BYTES } from '../skills-core.js'; +import { allRawObjects, MAX_SKILL_CONTENT_BYTES, requireStore } from '../skills-core.js'; import type { RawSkillObject, Skill, SkillOutcomeReason, SkillStore } from '../types.js'; import { createReconcileAction, @@ -603,6 +603,29 @@ describe('getSkill', () => { it('returns null for a missing key, never raising', async () => { expect(await getSkill('nope')).toBeNull(); }); + + it('withholds a store answering under a different key', async () => { + // The key needs the same post-fetch defense the version already has. + // Identity is read off the object itself, and the store is untrusted. An + // answer served under a different key would otherwise be handed back under + // the key the caller asked for while carrying its own. + const aliasing: SkillStore = { + getObject() { + return rawSkill({ key: 'other-key' }); + }, + allObjects() { + return {}; + }, + }; + _setStore(aliasing); + + expect(await getSkill('asked-for')).toBeNull(); + + const outcome = await getSkillResult('asked-for'); + expect(outcome.skill).toBeNull(); + expect(outcome.reason).toBe('wrong_version'); + expect(outcome.detail).toBe("skill 'asked-for' is not available: the store answered under key 'other-key'"); + }); }); // ─── getSkills ───────────────────────────────────────────────────────── @@ -680,6 +703,55 @@ describe('allSkills', () => { _setStore(new InMemorySkillStore()); expect(await allSkills()).toEqual([]); }); + + it('reports a non-object listing as a broken store, not an empty one', async () => { + // `allSkills` has no way to report the difference, so it returns an empty + // list either way — but the reason has to reach the caller that does act on + // it. Collapsing the answer to "no skills" reads downstream as "every skill + // was revoked". + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + try { + const brokenListing: SkillStore = { + getObject() { + return null; + }, + allObjects() { + return null as unknown as Record; + }, + }; + _setStore(brokenListing); + + expect(await allSkills()).toEqual([]); + + const { objects, error } = allRawObjects(requireStore()); + expect(objects).toEqual({}); + expect(error).toBe('the skill store listed skills as null rather than an object'); + expect(spy.mock.calls.map(([line]) => String(line))).toEqual([ + '[LaunchDarkly] Skill store listed skills as null rather than an object', + '[LaunchDarkly] Skill store listed skills as null rather than an object', + ]); + } finally { + spy.mockRestore(); + } + }); + + it('names the type a broken listing came back as', () => { + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + try { + const listingAs = (value: unknown): SkillStore => ({ + getObject: () => null, + allObjects: () => value as Record, + }); + expect(allRawObjects(listingAs([])).error).toBe('the skill store listed skills as array rather than an object'); + expect(allRawObjects(listingAs('x')).error).toBe('the skill store listed skills as string rather than an object'); + expect(allRawObjects(listingAs(undefined)).error).toBe( + 'the skill store listed skills as undefined rather than an object', + ); + expect(allRawObjects(new InMemorySkillStore())).toEqual({ objects: {}, error: null }); + } finally { + spy.mockRestore(); + } + }); }); // ─── Integrity verification ──────────────────────────────────────────── diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 91aa5451..ff76df75 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -424,10 +424,49 @@ export function verifyRawSkill(raw: unknown): Skill | null { }); } -/** Lists every raw object the store holds. Propagates whatever it throws. */ -export function allRawObjects(store: SkillStore): Record { - const objects = store.allObjects(SKILL_OBJECT_KIND); - return typeof objects === 'object' && objects !== null && !Array.isArray(objects) ? objects : {}; +/** The one wording for "the store could not answer", used by every path. */ +export function storeThrew(error: unknown): string { + const name = error instanceof Error ? error.constructor.name : 'unknown error'; + const message = error instanceof Error ? error.message : String(error); + return `the skill store threw ${name}: ${message}`; +} + +/** Every raw object the store holds, or the reason it could not answer. */ +export type RawListing = { + readonly objects: Record; + /** `null` when the store answered; otherwise why it could not. */ + readonly error: string | null; +}; + +/** + * Lists every raw object the store holds, or reports why it could not. + * + * A throwing store is caught here rather than propagated so that `allSkills` and + * the `'*'` reconcile path log and word the failure identically. Letting the + * exception out instead would make each of them re-derive the log line and the + * message, which is the drift this module exists to prevent. + * + * An answer that is not an object is a broken store, on the same footing as one + * that threw — **not** an empty one. Collapsing it to `{}` would make a store + * that served nothing usable indistinguishable from a store that holds no + * skills, which reads downstream as "every skill was revoked". + */ +export function allRawObjects(store: SkillStore): RawListing { + let objects: unknown; + try { + objects = store.allObjects(SKILL_OBJECT_KIND); + } catch (error) { + // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; a failing store must be visible + console.error(`[LaunchDarkly] Skill store threw while listing skills: ${storeThrew(error)}`); + return { objects: {}, error: storeThrew(error) }; + } + if (typeof objects !== 'object' || objects === null || Array.isArray(objects)) { + const typeName = Array.isArray(objects) ? 'array' : objects === null ? 'null' : typeof objects; + // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; a failing store must be visible + console.error(`[LaunchDarkly] Skill store listed skills as ${typeName} rather than an object`); + return { objects: {}, error: `the skill store listed skills as ${typeName} rather than an object` }; + } + return { objects: objects as Record, error: null }; } // --------------------------------------------------------------------------- @@ -475,18 +514,19 @@ export type Resolution = { * versions of one key and only it can pick between them; `null` asks for the * newest. The equality check afterwards is kept as a **defense**, not as the * selection mechanism: the store is untrusted, so an answer that is not the - * version that was asked for is withheld rather than returned. + * version that was asked for is withheld rather than returned. The key is + * checked the same way and for the same reason: identity is read off the object + * itself, so an answer served under a different key would otherwise be returned + * under the caller's key while carrying its own. */ export function resolveFromStore(store: SkillStore, key: string, wantedVersion: number | null): Resolution { let raw: RawSkillObject | null | undefined; try { raw = store.getObject(SKILL_OBJECT_KIND, key, wantedVersion); } catch (error) { - const name = error instanceof Error ? error.constructor.name : 'unknown error'; - const message = error instanceof Error ? error.message : String(error); // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; a failing store must be visible - console.error(`[LaunchDarkly] Skill store threw while retrieving '${key}': ${message}`); - return { error: `the skill store threw ${name}: ${message}`, reason: 'store_unavailable', unavailable: true }; + console.error(`[LaunchDarkly] Skill store threw while retrieving '${key}': ${storeThrew(error)}`); + return { error: storeThrew(error), reason: 'store_unavailable', unavailable: true }; } if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) { @@ -497,6 +537,12 @@ export function resolveFromStore(store: SkillStore, key: string, wantedVersion: if (skill === null) { return { error: `skill '${key}' failed integrity verification and was withheld`, reason: 'integrity_failure' }; } + if (skill.key !== key) { + return { + error: `skill '${key}' is not available: the store answered under key '${skill.key}'`, + reason: 'wrong_version', + }; + } if (wantedVersion !== null && skill.version !== wantedVersion) { return { error: `skill '${key}' version ${wantedVersion} is not available (the store holds version ${skill.version})`, diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index b05634ee..0af5ddaf 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -506,13 +506,8 @@ function resolveAll(deadline: number, onUnavailable: OnUnavailable): { requests: // Deliberately not via allSkills(), which reports a throwing store as an empty // result — that would look like "every skill was revoked" and let prune delete // the lot. - let objects: Record; - try { - objects = allRawObjects(store); - } catch (error) { - const name = error instanceof Error ? error.constructor.name : 'unknown error'; - return unavailableRun(unavailable(`the skill store threw ${name}: ${messageOf(error)}`), onUnavailable); - } + const { objects, error } = allRawObjects(store); + if (error !== null) return unavailableRun(unavailable(error), onUnavailable); return { requests: Object.entries(objects).map(([key, raw]) => pendingForRaw(key, raw)), diff --git a/packages/client/src/skills.ts b/packages/client/src/skills.ts index 8cd88062..2d090918 100644 --- a/packages/client/src/skills.ts +++ b/packages/client/src/skills.ts @@ -253,18 +253,8 @@ export async function getSkills(refs: ReadonlyArray): P * configured. */ export async function allSkills(): Promise { - const store = requireStore(); - - let objects: Record; - try { - objects = allRawObjects(store); - } catch (error) { - // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; a failing store must be visible - console.error( - `[LaunchDarkly] Skill store threw while listing skills: ${error instanceof Error ? error.message : String(error)}`, - ); - return []; - } + const { objects, error } = allRawObjects(requireStore()); + if (error !== null) return []; const skills: Skill[] = []; for (const raw of Object.values(objects)) { From 67cda6fab0301afec0e7c7724f3019810c5a55e9 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 11 Sep 2026 16:30:41 -0400 Subject: [PATCH 04/55] test(client): a broken store answer never deletes managed files Covers the reconcile side of the withheld-answer fix: a listing that is not an object leaves every managed file alone rather than reading as a full revocation, and an answer served under a different key writes nothing, is reported against the key that was asked for, and does not reach that other key's file. Each one previously deleted a file and reported a clean run. Ports launchdarkly/python-ai-sdk 4c6d965. Co-Authored-By: Claude Fable 5.1 --- .../client/src/__tests__/skills-fs.test.ts | 81 +++++++++++++++++++ 1 file changed, 81 insertions(+) diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index 92d375ed..7eb380a8 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -718,6 +718,87 @@ describe('writeSkills resilience', () => { ); }); + it('never prunes on a non-object listing', async () => { + // A store that cannot list is not a store holding nothing. A listing + // collapsed to "no skills" is indistinguishable from every skill having + // been revoked, and prune would then delete every managed file and report a + // clean run. The listing failure has to reach the prune gate as an + // incomplete run. + const noListing: SkillStore = { + getObject() { + return null; + }, + allObjects() { + return null as unknown as Record; + }, + }; + const existing = await placeManaged(root, 'pdf-extraction', SKILL_BODY); + _setStore(noListing); + + const report = await writeSkills('*', root); + + expect(report.ok).toBe(false); + expect(await readFile(existing, 'utf-8')).toBe(SKILL_BODY); + expect(report.actions.map((a) => a.action)).toEqual(['error']); + expect(errorMessages(report).some((m) => m.includes('rather than an object'))).toBe(true); + // The entry survives, so the next reconcile picks it up. + expect(Object.keys((await readManifest(root)).entries as Record)).toContain( + `pdf-extraction/${SKILL_MD}`, + ); + }); + + it('writes nothing for an answer served under another key', async () => { + // The file is named after the key the object carries, so a store answering + // under a different key would write one path and prune another. Left + // unchecked, the run wrote the aliased key, then deleted it in the same + // pass because prune keys off the request — and reported ok. The requested + // key has to be the one the outcome is reported against. + const aliasing: SkillStore = { + getObject() { + return rawSkill('other-key'); + }, + allObjects() { + return {}; + }, + }; + _setStore(aliasing); + + const report = await writeSkills(['requested-key'], root); + + expect(report.ok).toBe(false); + expect(report.actions.map((a) => a.action)).toEqual(['error']); + // Reported against the key that was asked for, not the one served. + expect(actionsByKey(report)['requested-key'].action).toBe('error'); + expect(await exists(path.join(root, 'other-key'))).toBe(false); + expect((await readManifest(root)).entries).toEqual({}); + }); + + it('an answer served under another key never reaches that key’s file', async () => { + // Both keys are requested here, so nothing is prunable and the write itself + // is what is under test: unchecked, the object served under the alias is + // written to the *other* key's path, clobbering the content that key's own + // lookup resolved — and the run still reports ok. + const aliased = 'aliased\n'; + const aliasing: SkillStore = { + getObject(_kind, key) { + return key === 'other-key' ? rawSkill('other-key') : rawSkill('other-key', 2, aliased); + }, + allObjects() { + return {}; + }, + }; + const existing = await placeManaged(root, 'other-key', SKILL_BODY); + _setStore(aliasing); + + // The alias is resolved last, so an unchecked write would land on top. + const report = await writeSkills(['other-key', 'requested-key'], root); + + expect(report.ok).toBe(false); + expect(await readFile(existing, 'utf-8')).toBe(SKILL_BODY); + expect(actionsByKey(report)['requested-key'].action).toBe('error'); + expect(actionsByKey(report)['other-key'].action).toBe('skipped_current'); + }); + it('never corrupts the manifest on an unavailable run', async () => { await placeManaged(root, 'a', SKILL_BODY); const before = await readManifest(root); From 6279f4a351832dea1dd6d52791d6a2fa581c5446 Mon Sep 17 00:00:00 2001 From: Patrick Kaeding Date: Fri, 4 Sep 2026 12:12:23 -0400 Subject: [PATCH 05/55] =?UTF-8?q?test(client):=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20the=20root-swap=20races=20the=20pinned=20handle=20d?= =?UTF-8?q?oes=20not=20close?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three expected-failure tests (skills-fs-root-swap.test.ts) for SEC-8985 row 2. resolveRoot validates the root once and returns a path; nothing holds it open. Each write and prune re-opens / by path with O_NOFOLLOW, which guards only the final component, so a root swapped for a symlink after validation redirects the open into the attacker's directory. assertUnswapped does not notice, because it lstat's the same swapped path the handle came from. Precondition is write permission on the root's parent, which the README privilege-separation checklist does not mention, so the documented mitigation for Node's residual exposure does not cover this. The tests state the contract (nothing lands outside the root, no outside file is overwritten, no outside file is removed) under a knownGap() wrapper: pytest's strict xfail narrowed to assertion errors, which it.fails is not, so a harness breakage is a real failure and the wrapper fails the run the day the body passes. Separate file because the swap must land before the per-skill directory is opened, which means intercepting mkdir/open from node:fs/promises with a file-wide vi.mock. Co-Authored-By: Claude Fable 5.1 (cherry picked from commit 2b9025e75a64e927b3c64d1342f724e4b56795b2) --- .../src/__tests__/skills-fs-root-swap.test.ts | 242 ++++++++++++++++++ 1 file changed, 242 insertions(+) create mode 100644 packages/client/src/__tests__/skills-fs-root-swap.test.ts diff --git a/packages/client/src/__tests__/skills-fs-root-swap.test.ts b/packages/client/src/__tests__/skills-fs-root-swap.test.ts new file mode 100644 index 00000000..3abf5f8b --- /dev/null +++ b/packages/client/src/__tests__/skills-fs-root-swap.test.ts @@ -0,0 +1,242 @@ +/** + * `writeSkills` — the swap one level up: the managed *root* replaced by a + * symlink, not `/`. + * + * `resolveRoot` validates the root once and returns a path; nothing holds it + * open. Each write and each prune then opens `/` *by path* with + * `O_NOFOLLOW | O_DIRECTORY` and pins that. `O_NOFOLLOW` guards only the final + * component, so the root and every ancestor are re-resolved on every such open, + * and a root swapped for a symlink after validation redirects the open into the + * attacker's directory. `assertUnswapped` does not notice: it compares the + * handle against an `lstat` of the *same swapped path*, so the two agree. + * `atomicWriteIn` does refuse the manifest rewrite afterwards, but by then the + * skill file is already outside the root. + * + * The `/` swap races in `skills-fs.test.ts` are skipped off + * `SUPPORTS_DIR_FD`, because Node cannot close that window. These are not + * skipped: they show that the *documented* mitigation for the residual exposure + * is not sufficient either. The README's privilege-separation checklist denies + * the agent identity the root, the skill directories, the files and the + * manifest, and says nothing about the root's parent. In the documented layout + * (`/.claude/skills`) that parent is `.claude`, which the agent identity + * typically owns, and write permission there is all this takes. + * + * Every test states the contract — nothing lands outside the root, no outside + * file is overwritten, no outside file is removed — and runs under `knownGap`, + * which passes only while the contract is violated. On Linux the fix is to hold + * the root handle and address children through `/proc/self/fd//`, + * which the kernel resolves against the pinned inode rather than the name; + * everywhere else it is adding the root's parent to the README checklist. + * + * In its own file because the swap has to land *before* the per-skill directory + * is opened, which means intercepting `mkdir` and `open` from `node:fs/promises` + * rather than the `fsOps` rename/unlink hook, and `vi.mock` is file-wide. + */ + +import { createHash } from 'node:crypto'; +import { lstat, mkdir, mkdtemp, readdir, readFile, realpath, rename, rm, symlink, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { _clearState } from '../skills.js'; +import { writeSkills } from '../skills-fs.js'; +import type { Skill } from '../types.js'; +import { createSkill } from '../types.js'; + +const SKILL_MD = 'SKILL.md'; +const MANIFEST_NAME = '.launchdarkly-skills.json'; +const SKILL_BODY = '---\nname: Test Skill\n---\nDo the thing.\n'; +const NEVER_FIRED = 'the race never fired; the test proves nothing'; + +// ─── The mkdir/open interception hook ──────────────────────────────────────── +// +// Pass-throughs until a test arms one of them for one exact path; everything +// else `node:fs/promises` exports is the real thing. `nth` picks which matching +// call fires, because a run can open the same skill directory more than once +// and only the open *after* the last containment check is the one that counts. + +const race = vi.hoisted(() => ({ + on: null as 'mkdir' | 'open' | null, + trigger: '', + nth: 1, + seen: 0, + fire: null as (() => Promise) | null, + fired: false, +})); + +vi.mock('node:fs/promises', async (importOriginal) => { + const actual = await importOriginal(); + const call = (fn: unknown, args: unknown[]) => (fn as (...a: unknown[]) => Promise)(...args); + async function maybeFire(op: 'mkdir' | 'open', target: unknown): Promise { + if (race.on !== op || race.fired || target !== race.trigger) return; + race.seen += 1; + if (race.seen < race.nth) return; + race.fired = true; + await race.fire?.(); + } + return { + ...actual, + mkdir: async (...args: unknown[]) => { + await maybeFire('mkdir', args[0]); + return call(actual.mkdir, args); + }, + open: async (...args: unknown[]) => { + await maybeFire('open', args[0]); + return call(actual.open, args); + }, + }; +}); + +function arm(on: 'mkdir' | 'open', trigger: string, fire: () => Promise, nth = 1): void { + Object.assign(race, { on, trigger, nth, seen: 0, fire, fired: false }); +} + +function disarm(): void { + Object.assign(race, { on: null, trigger: '', nth: 1, seen: 0, fire: null, fired: false }); +} + +/** + * pytest's `xfail(strict=True, raises=AssertionError)`, which `it.fails` is not. + * + * `it.fails` accepts *any* throw as the expected failure, so a harness bug — a + * race that never fires, a `writeSkills` that rejects — would masquerade as the + * known gap. This accepts only a failed assertion, rethrows everything else, and + * fails loudly the day the body passes, so the wrapper cannot outlive the fix. + */ +async function knownGap(body: () => Promise): Promise { + try { + await body(); + } catch (error) { + if ((error as Error | undefined)?.name === 'AssertionError') return; + throw error; + } + throw new Error('the root-swap gap is closed: drop knownGap() and assert the contract directly'); +} + +// ─── Helpers ───────────────────────────────────────────────────────────────── + +function hash(content: string): string { + return createHash('sha256').update(Buffer.from(content, 'utf-8')).digest('hex'); +} + +function skill(key: string, version = 1, content = SKILL_BODY): Skill { + return createSkill({ key, version, content: new TextEncoder().encode(content), contentHash: hash(content) }); +} + +async function writeManifest(root: string, raw: unknown): Promise { + await writeFile(path.join(root, MANIFEST_NAME), JSON.stringify(raw), 'utf-8'); +} + +/** Pre-create a file AND its manifest entry — i.e. an SDK-managed path. */ +async function placeManaged(root: string, key: string, content: string, version = 1): Promise { + await mkdir(path.join(root, key), { recursive: true }); + await writeFile(path.join(root, key, SKILL_MD), content, 'utf-8'); + await writeManifest(root, { + manifestVersion: 1, + entries: { [`${key}/${SKILL_MD}`]: { key, version, sha256: hash(content), writtenAt: '2026-08-14T19:00:00Z' } }, + }); +} + +async function exists(target: string): Promise { + try { + await lstat(target); + return true; + } catch { + return false; + } +} + +// ─── Per-test scratch directory ────────────────────────────────────────────── + +let scratch: string; +let root: string; + +/** Where the real root ends up after the swap. */ +const movedTo = (): string => `${root}.real`; + +/** Renames the managed root aside and leaves a symlink to `outside` in its place. */ +async function swapRoot(outside: string): Promise { + await rename(root, movedTo()); + await symlink(outside, root, 'dir'); +} + +beforeEach(async () => { + _clearState(); + disarm(); + // realpath, as in skills-fs.test.ts: writeSkills resolves the root, and the + // trigger has to compare equal to the path the implementation actually opens. + scratch = await realpath(await mkdtemp(path.join(tmpdir(), 'ld-ai-skills-root-swap-'))); + root = path.join(scratch, 'skills'); + await mkdir(root); +}); + +afterEach(async () => { + disarm(); + _clearState(); + await rm(scratch, { recursive: true, force: true }); +}); + +// ─── The three root-swap races ─────────────────────────────────────────────── + +describe('writeSkills root swap races', () => { + it('a root swapped at the skill directory create cannot redirect the write', () => + knownGap(async () => { + // First reconcile against a fresh root: `/a` does not exist, so + // `openOrCreateDirectory` calls `mkdir(/a)`. The swap fires there; + // the mkdir, the O_NOFOLLOW open and assertUnswapped's lstat all resolve + // through the link, and SKILL.md is written into `/a/`. + const outside = path.join(scratch, 'outside'); + await mkdir(outside); + arm('mkdir', path.join(root, 'a'), () => swapRoot(outside)); + + const report = await writeSkills([skill('a')], root); + + if (!race.fired) throw new Error(NEVER_FIRED); + expect(await readdir(outside)).toEqual([]); + // Either the skill landed in the real root or the run says it did not. + if (report.ok) expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe(SKILL_BODY); + })); + + it('a root swapped at the skill directory open cannot clobber an outside file', () => + knownGap(async () => { + // Update of an already-managed skill: `/a` exists, so the mkdir + // fails EEXIST and the swap fires at the O_NOFOLLOW open that follows. + // `/a/SKILL.md` — a file the manifest never recorded — is + // replaced with LaunchDarkly-served content. + const outside = path.join(scratch, 'outside'); + await mkdir(path.join(outside, 'a'), { recursive: true }); + const victim = path.join(outside, 'a', SKILL_MD); + await writeFile(victim, 'precious\n', 'utf-8'); + await placeManaged(root, 'a', SKILL_BODY); + arm('open', path.join(root, 'a'), () => swapRoot(outside)); + + const report = await writeSkills([skill('a', 2, 'served update\n')], root); + + if (!race.fired) throw new Error(NEVER_FIRED); + expect(await readFile(victim, 'utf-8')).toBe('precious\n'); + if (report.ok) expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe('served update\n'); + })); + + it('a root swapped at the prune cannot redirect the unlink', () => + knownGap(async () => { + // The destructive side. The orphan-temp sweep opens `/a` first and + // `pruneOne` re-runs the realpath containment check after it, so a swap + // at that open would be caught; the second open — pruneOne's own, after + // the check — is not. The unlink and the rmdir then both resolve through + // the swapped root, and an outside file and its directory are removed. + const outside = path.join(scratch, 'outside'); + await mkdir(path.join(outside, 'a'), { recursive: true }); + const victim = path.join(outside, 'a', SKILL_MD); + await writeFile(victim, 'precious\n', 'utf-8'); + await placeManaged(root, 'a', SKILL_BODY); + arm('open', path.join(root, 'a'), () => swapRoot(outside), 2); + + const report = await writeSkills([], root); + + if (!race.fired) throw new Error(NEVER_FIRED); + expect(await exists(victim)).toBe(true); + expect(await readFile(victim, 'utf-8')).toBe('precious\n'); + if (report.ok) expect(await exists(path.join(movedTo(), 'a', SKILL_MD))).toBe(false); + })); +}); From 4c406036fc455d4542ff4e13008494d666a63e9e Mon Sep 17 00:00:00 2001 From: Patrick Kaeding Date: Fri, 4 Sep 2026 12:41:26 -0400 Subject: [PATCH 06/55] test(client): let the root-swap races fail outright The expected-failure marker made the suite green while the contract was violated. A red run is the demonstration: the tests assert the contract, the code does not meet it, and they go green when the root is pinned for the reconcile, with nothing to remove. Co-Authored-By: Claude Fable 5.1 (cherry picked from commit 31d47dbcf9380099ae03071856d0c876f6854fb3) --- .../src/__tests__/skills-fs-root-swap.test.ts | 137 ++++++++---------- 1 file changed, 58 insertions(+), 79 deletions(-) diff --git a/packages/client/src/__tests__/skills-fs-root-swap.test.ts b/packages/client/src/__tests__/skills-fs-root-swap.test.ts index 3abf5f8b..d566a912 100644 --- a/packages/client/src/__tests__/skills-fs-root-swap.test.ts +++ b/packages/client/src/__tests__/skills-fs-root-swap.test.ts @@ -22,8 +22,8 @@ * typically owns, and write permission there is all this takes. * * Every test states the contract — nothing lands outside the root, no outside - * file is overwritten, no outside file is removed — and runs under `knownGap`, - * which passes only while the contract is violated. On Linux the fix is to hold + * file is overwritten, no outside file is removed. The code does not meet it, so + * they fail, and the red run is the demonstration. On Linux the fix is to hold * the root handle and address children through `/proc/self/fd//`, * which the kernel resolves against the pinned inode rather than the name; * everywhere else it is adding the root's parent to the README checklist. @@ -96,24 +96,6 @@ function disarm(): void { Object.assign(race, { on: null, trigger: '', nth: 1, seen: 0, fire: null, fired: false }); } -/** - * pytest's `xfail(strict=True, raises=AssertionError)`, which `it.fails` is not. - * - * `it.fails` accepts *any* throw as the expected failure, so a harness bug — a - * race that never fires, a `writeSkills` that rejects — would masquerade as the - * known gap. This accepts only a failed assertion, rethrows everything else, and - * fails loudly the day the body passes, so the wrapper cannot outlive the fix. - */ -async function knownGap(body: () => Promise): Promise { - try { - await body(); - } catch (error) { - if ((error as Error | undefined)?.name === 'AssertionError') return; - throw error; - } - throw new Error('the root-swap gap is closed: drop knownGap() and assert the contract directly'); -} - // ─── Helpers ───────────────────────────────────────────────────────────────── function hash(content: string): string { @@ -180,63 +162,60 @@ afterEach(async () => { // ─── The three root-swap races ─────────────────────────────────────────────── describe('writeSkills root swap races', () => { - it('a root swapped at the skill directory create cannot redirect the write', () => - knownGap(async () => { - // First reconcile against a fresh root: `/a` does not exist, so - // `openOrCreateDirectory` calls `mkdir(/a)`. The swap fires there; - // the mkdir, the O_NOFOLLOW open and assertUnswapped's lstat all resolve - // through the link, and SKILL.md is written into `/a/`. - const outside = path.join(scratch, 'outside'); - await mkdir(outside); - arm('mkdir', path.join(root, 'a'), () => swapRoot(outside)); - - const report = await writeSkills([skill('a')], root); - - if (!race.fired) throw new Error(NEVER_FIRED); - expect(await readdir(outside)).toEqual([]); - // Either the skill landed in the real root or the run says it did not. - if (report.ok) expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe(SKILL_BODY); - })); - - it('a root swapped at the skill directory open cannot clobber an outside file', () => - knownGap(async () => { - // Update of an already-managed skill: `/a` exists, so the mkdir - // fails EEXIST and the swap fires at the O_NOFOLLOW open that follows. - // `/a/SKILL.md` — a file the manifest never recorded — is - // replaced with LaunchDarkly-served content. - const outside = path.join(scratch, 'outside'); - await mkdir(path.join(outside, 'a'), { recursive: true }); - const victim = path.join(outside, 'a', SKILL_MD); - await writeFile(victim, 'precious\n', 'utf-8'); - await placeManaged(root, 'a', SKILL_BODY); - arm('open', path.join(root, 'a'), () => swapRoot(outside)); - - const report = await writeSkills([skill('a', 2, 'served update\n')], root); - - if (!race.fired) throw new Error(NEVER_FIRED); - expect(await readFile(victim, 'utf-8')).toBe('precious\n'); - if (report.ok) expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe('served update\n'); - })); - - it('a root swapped at the prune cannot redirect the unlink', () => - knownGap(async () => { - // The destructive side. The orphan-temp sweep opens `/a` first and - // `pruneOne` re-runs the realpath containment check after it, so a swap - // at that open would be caught; the second open — pruneOne's own, after - // the check — is not. The unlink and the rmdir then both resolve through - // the swapped root, and an outside file and its directory are removed. - const outside = path.join(scratch, 'outside'); - await mkdir(path.join(outside, 'a'), { recursive: true }); - const victim = path.join(outside, 'a', SKILL_MD); - await writeFile(victim, 'precious\n', 'utf-8'); - await placeManaged(root, 'a', SKILL_BODY); - arm('open', path.join(root, 'a'), () => swapRoot(outside), 2); - - const report = await writeSkills([], root); - - if (!race.fired) throw new Error(NEVER_FIRED); - expect(await exists(victim)).toBe(true); - expect(await readFile(victim, 'utf-8')).toBe('precious\n'); - if (report.ok) expect(await exists(path.join(movedTo(), 'a', SKILL_MD))).toBe(false); - })); + it('a root swapped at the skill directory create cannot redirect the write', async () => { + // First reconcile against a fresh root: `/a` does not exist, so + // `openOrCreateDirectory` calls `mkdir(/a)`. The swap fires there; + // the mkdir, the O_NOFOLLOW open and assertUnswapped's lstat all resolve + // through the link, and SKILL.md is written into `/a/`. + const outside = path.join(scratch, 'outside'); + await mkdir(outside); + arm('mkdir', path.join(root, 'a'), () => swapRoot(outside)); + + const report = await writeSkills([skill('a')], root); + + if (!race.fired) throw new Error(NEVER_FIRED); + expect(await readdir(outside)).toEqual([]); + // Either the skill landed in the real root or the run says it did not. + if (report.ok) expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe(SKILL_BODY); + }); + + it('a root swapped at the skill directory open cannot clobber an outside file', async () => { + // Update of an already-managed skill: `/a` exists, so the mkdir + // fails EEXIST and the swap fires at the O_NOFOLLOW open that follows. + // `/a/SKILL.md` — a file the manifest never recorded — is + // replaced with LaunchDarkly-served content. + const outside = path.join(scratch, 'outside'); + await mkdir(path.join(outside, 'a'), { recursive: true }); + const victim = path.join(outside, 'a', SKILL_MD); + await writeFile(victim, 'precious\n', 'utf-8'); + await placeManaged(root, 'a', SKILL_BODY); + arm('open', path.join(root, 'a'), () => swapRoot(outside)); + + const report = await writeSkills([skill('a', 2, 'served update\n')], root); + + if (!race.fired) throw new Error(NEVER_FIRED); + expect(await readFile(victim, 'utf-8')).toBe('precious\n'); + if (report.ok) expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe('served update\n'); + }); + + it('a root swapped at the prune cannot redirect the unlink', async () => { + // The destructive side. The orphan-temp sweep opens `/a` first and + // `pruneOne` re-runs the realpath containment check after it, so a swap + // at that open would be caught; the second open — pruneOne's own, after + // the check — is not. The unlink and the rmdir then both resolve through + // the swapped root, and an outside file and its directory are removed. + const outside = path.join(scratch, 'outside'); + await mkdir(path.join(outside, 'a'), { recursive: true }); + const victim = path.join(outside, 'a', SKILL_MD); + await writeFile(victim, 'precious\n', 'utf-8'); + await placeManaged(root, 'a', SKILL_BODY); + arm('open', path.join(root, 'a'), () => swapRoot(outside), 2); + + const report = await writeSkills([], root); + + if (!race.fired) throw new Error(NEVER_FIRED); + expect(await exists(victim)).toBe(true); + expect(await readFile(victim, 'utf-8')).toBe('precious\n'); + if (report.ok) expect(await exists(path.join(movedTo(), 'a', SKILL_MD))).toBe(false); + }); }); From 0479a61a868dc3dbe4d9e60bb14aee30cb806047 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 4 Sep 2026 15:27:24 -0400 Subject: [PATCH 07/55] fix(client): pin the skills root through /proc/self/fd on Linux MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SEC-8985 row 2. `resolveRoot` validated the managed root and returned a path; nothing held it open. Every write and prune then opened `/` by path with `O_NOFOLLOW`, which guards only the final component, so the root was re-resolved on each one and a root swapped for a symlink after validation redirected the open into the attacker's directory. `assertUnswapped` agreed with itself, because it lstat'd the same swapped path the handle came from. Only the manifest write pinned the root, and by then the skill files were already outside it. The precondition is write access to the root's *parent* — `.claude` for a root of `.claude/skills`, which the agent identity typically owns. The root is now opened `O_RDONLY|O_DIRECTORY|O_NOFOLLOW` once, fstat'd, and held for the whole reconcile, and every child is addressed as `/proc/self/fd//`. The kernel resolves that prefix to the pinned inode rather than to the name, which gives it the property an `*at()` call would have; the per-skill directory is pinned in turn for the files inside it. Gated on a new `SUPPORTS_PROC_FD` probe (Linux plus a usable `/proc/self/fd`, verified at load by resolving a live descriptor back to its own inode), so macOS and Windows keep the per-component lstat floor unchanged — `assertUnswapped` is skipped only for a directory the caller addressed through a descriptor, which is why the floor's own tests still exercise it on Linux. Containment checks in `unsafePathReason` stay as defense in depth; they are no longer the boundary here. A swap landing before the pin fails the open and is reported as a run-level error rather than thrown, since the root was usable when the caller named it. `atomicWriteIn` is gone: the root is always pinned by the caller now, so a primitive whose job was to re-open a directory by path is exactly the footgun this bug was made of. Descriptor addresses are stripped out of `errno` messages, which leaves the offending file named relative to whichever directory was pinned. Tests: the two `/` races in skills-fs.test.ts now run wherever either capability probe is true, so they are live on Linux. The root-swap file from Security's PR #49 keeps its assertions; its trigger matches the final component plus the shape of its parent rather than one exact path, so the same bodies fire the swap against both the old and new addressing — pinning the trigger to `path.join(root, key)` would have made them pass vacuously the moment the fix landed. Verified in a Linux container: all four fail against the pre-fix code with real assertion failures (the write escapes, the outside victim is clobbered, the outside victim is unlinked) and pass against this commit. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 23 +- packages/client/src/__tests__/safe-fs.test.ts | 32 +-- .../src/__tests__/skills-fs-root-swap.test.ts | 135 ++++++++--- .../client/src/__tests__/skills-fs.test.ts | 76 ++++-- packages/client/src/safe-fs.ts | 155 ++++++++++--- packages/client/src/skills-fs.ts | 216 ++++++++++++++---- 6 files changed, 487 insertions(+), 150 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 90ced584..2d04a8d5 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -336,7 +336,12 @@ example above, materializes only what the resolved variation actually asked for. `ReconcileReport` exposes `actions`, `ok` (true iff no action is an `error`), and `errors` (the error actions, in order), so callers never re-derive the filter. Each `ReconcileAction` carries `key`, `action` (`written` | `updated` | `skipped_current` | `removed` | `error`), and nullable `version` / `path` / `error`. A failure belonging to the whole run rather than one skill — a corrupt manifest, for instance — carries the **empty string** in `key`. -**Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly, and it is a platform bound rather than a detail: Node exposes no `renameat`/`unlinkat`/`openat`, so no destructive step can be performed relative to a pinned directory descriptor. Every check here is therefore a per-component `lstat` taken immediately before the path-based operation — a check-then-use race, not a closed window — and unlike the Python SDK, which closes that window on POSIX by holding a descriptor opened `O_RDONLY|O_DIRECTORY|O_NOFOLLOW` for the whole reconcile, **this floor is what runs on every platform Node supports, Linux included.** An attacker who already has **write permission on the managed root** can therefore win a race to redirect a write or a delete outside it. Windows adds nothing worse and nothing better: reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) are deliberately not implemented in this release, and Windows is not a tested platform for it — neither SDK repository has a Windows CI runner. So write permission on the managed root is *the* security boundary here. Keep it writable only by the identity running the reconcile — see [privilege separation](#privilege-separation-the-agent-must-not-be-able-to-rewrite-its-own-skills). +**Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly, and it is a platform bound rather than a detail. Node exposes no `renameat`/`unlinkat`/`openat`, so no destructive step can be addressed relative to a pinned directory descriptor *directly* — and what the SDK can do about that differs by platform: + +- **On Linux, the swap window is closed.** The managed root is opened `O_RDONLY|O_DIRECTORY|O_NOFOLLOW` once and that descriptor is held for the whole reconcile, and every child — each `//`, each `SKILL.md`, each temp file, the manifest — is addressed as `/proc/self/fd//`. The kernel resolves that prefix to the inode the descriptor holds rather than to the name it was opened under, which gives it the same property an `*at()` call would: renaming the root, or replacing it with a symlink, after validation cannot redirect a write or a delete. +- **On macOS and Windows, the window is narrowed but not closed.** There is no traversable descriptor prefix (`/dev/fd//` does not resolve), so each check is a per-component `lstat` taken immediately before the path-based operation — a check-then-use race. An attacker who already has **write permission on the managed root, or on any of its ancestor directories**, can win that race and redirect a write or a delete outside the root. Windows additionally has no reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) in this release and is not a tested platform — neither SDK repository has a Windows CI runner. + +So on those two platforms, write permission on the managed root **and on its ancestors** is *the* security boundary here. Keep them writable only by the identity running the reconcile — see [privilege separation](#privilege-separation-the-agent-must-not-be-able-to-rewrite-its-own-skills). The ancestor requirement is not a technicality: for a root of `.claude/skills`, the parent is `.claude`, which an agent identity often owns, and that alone is enough. **Two constraints on skill keys, imposed here rather than by the data model.** A key becomes a single directory name, so `writeSkills` rejects — as a reported `error` action, on every platform — a key over 255 bytes and the 22 Windows reserved device names (`con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9`). Both are checked in the filesystem layer only: a key like `aux` remains valid everywhere else, so an AI Config referencing it still parses and its other skills still materialize. One residual the SDK cannot check for you: the 255-byte bound is per *path component*, not on the total path, so `//SKILL.md` can still exceed Windows' 260-character `MAX_PATH` if the root is deep and the key is long. The root is yours, so budget for it there. @@ -438,19 +443,31 @@ These five tokens are the whole vocabulary, and the Python SDK publishes the sam **What to verify, as the identity that will run the agent.** The SDK cannot check this for you (see below), so make it a deployment step: confirm the agent's identity has no write access to - the managed root itself, +- **the managed root's parent, and every ancestor directory above it** — write access to any of them allows the root's own name to be swapped for a symlink, which redirects the reconcile's writes and deletes outside the root on macOS and Windows (on Linux the descriptor pinning described above closes this), - the per-skill directories `//` and the files `//SKILL.md`, - the manifest at `/.launchdarkly-skills.json`. ```bash # Run as the agent's user. Every line should print DENIED. root=.claude/skills -for target in "$root" "$root/.launchdarkly-skills.json" "$root"/*/ "$root"/*/SKILL.md; do +targets="$root $root/.launchdarkly-skills.json $root/*/ $root/*/SKILL.md" + +# Every ancestor of the root, up to /: write access to any of them is enough to +# swap the root itself for a symlink. +ancestor=$(dirname "$(cd "$(dirname "$root")" && pwd)/$(basename "$root")") +while :; do + targets="$targets $ancestor" + [ "$ancestor" = / ] && break + ancestor=$(dirname "$ancestor") +done + +for target in $targets; do [ -e "$target" ] || continue if [ -w "$target" ]; then echo "WRITABLE — fix this: $target"; else echo "DENIED: $target"; fi done ``` -The managed root's own mode is **yours, not the SDK's**: `writeSkills` creates only that one leaf directory, and does so with the process umask, precisely because the root is a path you chose. Own it — `chown reconcile-user:agent-group` plus `chmod 0755` on the root is the shape that makes the rest of the tree's modes mean something. It is also the mitigation for the race described under *Security posture* above: that race requires write permission on the managed root, which privilege separation is what denies. +The managed root's own mode is **yours, not the SDK's**: `writeSkills` creates only that one leaf directory, and does so with the process umask, precisely because the root is a path you chose. Own it — `chown reconcile-user:agent-group` plus `chmod 0755` on the root is the shape that makes the rest of the tree's modes mean something. It is also the mitigation for the race described under *Security posture* above: on macOS and Windows that race requires write permission on the managed root or on one of its ancestors, which privilege separation is what denies. **Why this is the mitigation that matters.** A `SKILL.md` is agent *instructions*. An agent that can write its own skills directory can rewrite its own instructions, and an agent processing untrusted input is exactly the thing that might be induced to do so. Write access to the manifest is worse than write access to a skill, because the manifest is what tells the *next* reconcile which paths the SDK owns and may delete: an agent that can edit it can keep a skill LaunchDarkly has revoked, or aim the SDK's own delete path at something it should not touch. `writeSkills` re-validates every manifest entry from scratch for exactly that reason — it treats that file as untrusted input, never as authorization — but an agent that cannot edit it at all is the stronger position, and only your deployment can provide that. diff --git a/packages/client/src/__tests__/safe-fs.test.ts b/packages/client/src/__tests__/safe-fs.test.ts index 7b03916d..672c2a92 100644 --- a/packages/client/src/__tests__/safe-fs.test.ts +++ b/packages/client/src/__tests__/safe-fs.test.ts @@ -36,9 +36,7 @@ vi.mock('node:fs/promises', async () => { }; }); -const { atomicWrite, atomicWriteIn, openDirectoryNoFollow, openOrCreateDirectory, unlinkNoFollow } = await import( - '../safe-fs.js' -); +const { atomicWrite, openDirectoryNoFollow, openOrCreateDirectory, unlinkNoFollow } = await import('../safe-fs.js'); let scratch: string; @@ -158,16 +156,6 @@ describe('atomicWrite', () => { } expect(await readdir(dir)).toEqual(['SKILL.md']); }); - - it('atomicWriteIn refuses a symlinked directory', async () => { - const outside = path.join(scratch, 'outside'); - await mkdir(outside); - const link = path.join(scratch, 'link'); - await symlink(outside, link, 'dir'); - - await expect(atomicWriteIn(link, 'SKILL.md', Buffer.from('body\n', 'utf-8'))).rejects.toThrow(); - expect(await readdir(outside)).toEqual([]); - }); }); /** @@ -175,9 +163,21 @@ describe('atomicWrite', () => { * between validation and the destructive call: there is no `*at()` family, so the * rename and the unlink resolve their directory by path. * - * The swap is staged directly: pin the handle, replace the directory, then invoke - * the primitive. Both primitives re-check independently, so both halves are - * required. + * Off `SUPPORTS_PROC_FD` it is the *whole* defense against a directory swapped + * between validation and the destructive call: Node exposes no `*at()` family, so + * the rename and the unlink resolve their directory by path. The swap-race cases + * in `skills-fs.test.ts` cannot reach it — they fire the swap from the + * rename/unlink hook, which by construction runs *after* the check. Verified by + * mutation: with the check neutered, the entire suite stays green. + * + * So the swap is staged here instead: pin the handle, then replace the directory, + * then invoke the primitive. Both primitives re-check independently, so both + * halves are required. + * + * These call the primitives with a plain path, which is what keeps them + * meaningful on Linux too: the check is skipped only for a directory addressed + * through `directoryAddress`, so passing a path exercises the floor on every + * platform rather than testing nothing wherever the fast path exists. */ describe('pinned-directory identity re-check', () => { /** Moves `dir` aside and leaves a symlink to `outside` in its place. */ diff --git a/packages/client/src/__tests__/skills-fs-root-swap.test.ts b/packages/client/src/__tests__/skills-fs-root-swap.test.ts index d566a912..8565481a 100644 --- a/packages/client/src/__tests__/skills-fs-root-swap.test.ts +++ b/packages/client/src/__tests__/skills-fs-root-swap.test.ts @@ -2,31 +2,32 @@ * `writeSkills` — the swap one level up: the managed *root* replaced by a * symlink, not `/`. * - * `resolveRoot` validates the root once and returns a path; nothing holds it - * open. Each write and each prune then opens `/` *by path* with - * `O_NOFOLLOW | O_DIRECTORY` and pins that. `O_NOFOLLOW` guards only the final - * component, so the root and every ancestor are re-resolved on every such open, - * and a root swapped for a symlink after validation redirects the open into the - * attacker's directory. `assertUnswapped` does not notice: it compares the - * handle against an `lstat` of the *same swapped path*, so the two agree. - * `atomicWriteIn` does refuse the manifest rewrite afterwards, but by then the - * skill file is already outside the root. + * The defect these were written against (SEC-8985 row 2): `resolveRoot` + * validated the root once and returned a path, and nothing held it open. Each + * write and each prune then opened `/` *by path* with + * `O_NOFOLLOW | O_DIRECTORY` and pinned that. `O_NOFOLLOW` guards only the final + * component, so the root and every ancestor were re-resolved on every such open, + * and a root swapped for a symlink after validation redirected the open into the + * attacker's directory. `assertUnswapped` did not notice, because it compared the + * handle against an `lstat` of the *same swapped path*. Only the manifest write + * pinned the root, and by then the skill file was already outside it. The + * precondition is write permission on the root's *parent* — in the documented + * layout (`/.claude/skills`) that is `.claude`, which the agent identity + * typically owns. * - * The `/` swap races in `skills-fs.test.ts` are skipped off - * `SUPPORTS_DIR_FD`, because Node cannot close that window. These are not - * skipped: they show that the *documented* mitigation for the residual exposure - * is not sufficient either. The README's privilege-separation checklist denies - * the agent identity the root, the skill directories, the files and the - * manifest, and says nothing about the root's parent. In the documented layout - * (`/.claude/skills`) that parent is `.claude`, which the agent identity - * typically owns, and write permission there is all this takes. + * The fix pins the root to a descriptor for the whole reconcile and addresses + * every child as `/proc/self/fd//`, which the kernel resolves against + * the pinned inode rather than against the name. That exists on Linux only, so + * this file is gated on `SUPPORTS_PROC_FD`: it runs on CI and is skipped on macOS + * development machines, where the per-component `lstat` floor applies and the + * mitigation is the README's privilege-separation checklist — now including the + * root's ancestors. * - * Every test states the contract — nothing lands outside the root, no outside - * file is overwritten, no outside file is removed. The code does not meet it, so - * they fail, and the red run is the demonstration. On Linux the fix is to hold - * the root handle and address children through `/proc/self/fd//`, - * which the kernel resolves against the pinned inode rather than the name; - * everywhere else it is adding the root's parent to the README checklist. + * Every test states the contract rather than the mechanism: nothing lands outside + * the root, no outside file is overwritten, no outside file is removed. The + * trigger matches both the pre-fix and post-fix spellings of the child path (see + * the hook below), so these same bodies failed against the unfixed code and pass + * against the fixed one — which is the only thing that makes them evidence. * * In its own file because the swap has to land *before* the per-skill directory * is opened, which means intercepting `mkdir` and `open` from `node:fs/promises` @@ -39,6 +40,7 @@ import { tmpdir } from 'node:os'; import path from 'node:path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { SUPPORTS_PROC_FD } from '../safe-fs.js'; import { _clearState } from '../skills.js'; import { writeSkills } from '../skills-fs.js'; import type { Skill } from '../types.js'; @@ -51,14 +53,30 @@ const NEVER_FIRED = 'the race never fired; the test proves nothing'; // ─── The mkdir/open interception hook ──────────────────────────────────────── // -// Pass-throughs until a test arms one of them for one exact path; everything -// else `node:fs/promises` exports is the real thing. `nth` picks which matching -// call fires, because a run can open the same skill directory more than once -// and only the open *after* the last containment check is the one that counts. +// Pass-throughs until a test arms one of them; everything else +// `node:fs/promises` exports is the real thing. `nth` picks which matching call +// fires, because a run can open the same skill directory more than once and only +// the open *after* the last containment check is the one that counts. +// +// The trigger matches on the operation's **final component plus the shape of its +// parent**, not on one exact string. That is deliberate and it is what makes +// these tests worth anything: the fixed code addresses `/` as +// `/proc/self/fd//`, so a trigger pinned to `path.join(root, key)` +// would stop matching the moment the fix landed — the swap would never fire, the +// contract would never be tested, and the suite would go green for the wrong +// reason. Matching both spellings means one test body runs against either +// implementation, which is the only way the red-to-green transition means +// anything. const race = vi.hoisted(() => ({ + /** Which operation the swap rides on, or `null` when disarmed. */ on: null as 'mkdir' | 'open' | null, - trigger: '', + /** `'child'` matches an operation on a child of the root; `'root'` the root open itself. */ + mode: 'child' as 'child' | 'root', + /** For `'child'`: the final component the operation must target. */ + name: '', + /** The realpath'd managed root — the name a child carries before the fix. */ + root: '', nth: 1, seen: 0, fire: null as (() => Promise) | null, @@ -68,13 +86,26 @@ const race = vi.hoisted(() => ({ vi.mock('node:fs/promises', async (importOriginal) => { const actual = await importOriginal(); const call = (fn: unknown, args: unknown[]) => (fn as (...a: unknown[]) => Promise)(...args); + const PROC = '/proc/self/fd'; + + function matches(op: 'mkdir' | 'open', target: unknown): boolean { + if (race.on !== op || race.fired || typeof target !== 'string') return false; + if (race.mode === 'root') return target === race.root; + if (path.basename(target) !== race.name) return false; + // A child of the managed root, named either through the root's own path + // (unfixed) or through a held descriptor (fixed). + const parent = path.dirname(target); + return parent === race.root || parent.startsWith(`${PROC}/`); + } + async function maybeFire(op: 'mkdir' | 'open', target: unknown): Promise { - if (race.on !== op || race.fired || target !== race.trigger) return; + if (!matches(op, target)) return; race.seen += 1; if (race.seen < race.nth) return; race.fired = true; await race.fire?.(); } + return { ...actual, mkdir: async (...args: unknown[]) => { @@ -88,12 +119,18 @@ vi.mock('node:fs/promises', async (importOriginal) => { }; }); -function arm(on: 'mkdir' | 'open', trigger: string, fire: () => Promise, nth = 1): void { - Object.assign(race, { on, trigger, nth, seen: 0, fire, fired: false }); +/** Fires `fire` on the `nth` `op` targeting `/`. */ +function arm(on: 'mkdir' | 'open', name: string, fire: () => Promise, nth = 1): void { + Object.assign(race, { on, mode: 'child', name, nth, seen: 0, fire, fired: false }); +} + +/** Fires `fire` on the open of the managed root itself — i.e. just before it is pinned. */ +function armRootOpen(fire: () => Promise): void { + Object.assign(race, { on: 'open', mode: 'root', name: '', nth: 1, seen: 0, fire, fired: false }); } function disarm(): void { - Object.assign(race, { on: null, trigger: '', nth: 1, seen: 0, fire: null, fired: false }); + Object.assign(race, { on: null, mode: 'child', name: '', nth: 1, seen: 0, fire: null, fired: false }); } // ─── Helpers ───────────────────────────────────────────────────────────────── @@ -151,6 +188,7 @@ beforeEach(async () => { scratch = await realpath(await mkdtemp(path.join(tmpdir(), 'ld-ai-skills-root-swap-'))); root = path.join(scratch, 'skills'); await mkdir(root); + race.root = root; }); afterEach(async () => { @@ -161,7 +199,7 @@ afterEach(async () => { // ─── The three root-swap races ─────────────────────────────────────────────── -describe('writeSkills root swap races', () => { +describe.skipIf(!SUPPORTS_PROC_FD)('writeSkills root swap races', () => { it('a root swapped at the skill directory create cannot redirect the write', async () => { // First reconcile against a fresh root: `/a` does not exist, so // `openOrCreateDirectory` calls `mkdir(/a)`. The swap fires there; @@ -169,7 +207,7 @@ describe('writeSkills root swap races', () => { // through the link, and SKILL.md is written into `/a/`. const outside = path.join(scratch, 'outside'); await mkdir(outside); - arm('mkdir', path.join(root, 'a'), () => swapRoot(outside)); + arm('mkdir', 'a', () => swapRoot(outside)); const report = await writeSkills([skill('a')], root); @@ -189,7 +227,7 @@ describe('writeSkills root swap races', () => { const victim = path.join(outside, 'a', SKILL_MD); await writeFile(victim, 'precious\n', 'utf-8'); await placeManaged(root, 'a', SKILL_BODY); - arm('open', path.join(root, 'a'), () => swapRoot(outside)); + arm('open', 'a', () => swapRoot(outside)); const report = await writeSkills([skill('a', 2, 'served update\n')], root); @@ -209,7 +247,7 @@ describe('writeSkills root swap races', () => { const victim = path.join(outside, 'a', SKILL_MD); await writeFile(victim, 'precious\n', 'utf-8'); await placeManaged(root, 'a', SKILL_BODY); - arm('open', path.join(root, 'a'), () => swapRoot(outside), 2); + arm('open', 'a', () => swapRoot(outside), 2); const report = await writeSkills([], root); @@ -218,4 +256,29 @@ describe('writeSkills root swap races', () => { expect(await readFile(victim, 'utf-8')).toBe('precious\n'); if (report.ok) expect(await exists(path.join(movedTo(), 'a', SKILL_MD))).toBe(false); }); + + it('a root swapped before it is pinned is refused with a run-level error', async () => { + // The one window pinning cannot cover, and the reason the pin is not the + // whole story: `resolveRoot` validated the root's *name*, and the swap lands + // between that and the open that turns the name into a descriptor. The + // O_NOFOLLOW open then fails on the symlink. + // + // Refused as a run-level `error` action rather than thrown. `resolveRoot` + // throws because an unusable root is a caller mistake; a root that was a + // real directory an instant ago and is a symlink now is an attack in + // flight, and it belongs in the report next to every other refusal. + const outside = path.join(scratch, 'outside'); + await mkdir(outside); + await placeManaged(root, 'a', SKILL_BODY); + armRootOpen(() => swapRoot(outside)); + + const report = await writeSkills([skill('a', 2, 'served update\n')], root); + + if (!race.fired) throw new Error(NEVER_FIRED); + expect(report.ok).toBe(false); + expect(report.errors.some((action) => action.key === '')).toBe(true); + // Nothing was attempted through the link, and the real root is untouched. + expect(await readdir(outside)).toEqual([]); + expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe(SKILL_BODY); + }); }); diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index 7eb380a8..ef81a3be 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -8,7 +8,7 @@ import { execFileSync } from 'node:child_process'; import { createHash } from 'node:crypto'; -import { constants as fsConstants } from 'node:fs'; +import { constants as fsConstants, realpathSync } from 'node:fs'; import { chmod, lstat, @@ -26,7 +26,7 @@ import { tmpdir } from 'node:os'; import path from 'node:path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -import { fsOps, SUPPORTS_DIR_FD } from '../safe-fs.js'; +import { fsOps, SUPPORTS_DIR_FD, SUPPORTS_PROC_FD } from '../safe-fs.js'; import { _clearState, _setEmitterForTesting, _setStore, InMemorySkillStore, skillRefs } from '../skills.js'; import { MAX_SKILL_CONTENT_BYTES } from '../skills-core.js'; import { writeSkills } from '../skills-fs.js'; @@ -141,6 +141,24 @@ class RecordingEmitter { type RenameCall = { src: string; dst: string }; +/** + * A descriptor-addressed path, resolved back to the real one. + * + * On Linux the implementation issues its operations against + * `/proc/self/fd//`, so the kernel resolves them from a pinned inode + * rather than from a name — see `SUPPORTS_PROC_FD` in `safe-fs.ts`. These tests + * assert on *where* an operation landed, which means undoing that addressing, and + * it can only be undone while the descriptor is still open. Hence inside the + * interception rather than in the assertion. + * + * Only the parent is resolved: the final component of a rename destination does + * not exist yet. Off the fast path this is the identity function. + */ +function realize(target: string): string { + if (!target.startsWith('/proc/self/fd/')) return target; + return path.join(realpathSync(path.dirname(target)), path.basename(target)); +} + /** * Records — and optionally fails — every atomic rename of a `SKILL.md`. * @@ -159,7 +177,7 @@ function interceptRename(options: { fail?: boolean } = {}): RenameCall[] { const real = fsOps.rename.bind(fsOps); vi.spyOn(fsOps, 'rename').mockImplementation(async (src: string, dst: string) => { if (dst.endsWith(SKILL_MD)) { - calls.push({ src, dst }); + calls.push({ src: realize(src), dst: realize(dst) }); if (options.fail) throw new Error(INJECTED); } return real(src, dst); @@ -173,7 +191,7 @@ function interceptUnlink(options: { fail?: boolean } = {}): string[] { const real = fsOps.unlink.bind(fsOps); vi.spyOn(fsOps, 'unlink').mockImplementation(async (target: string) => { if (target.endsWith(SKILL_MD)) { - calls.push(target); + calls.push(realize(target)); if (options.fail) throw new Error(INJECTED); } return real(target); @@ -1063,20 +1081,22 @@ describe('writeSkills symlink attacks', () => { /** * The two symlink swap-race cases. * - * They are defensible only where the platform provides the `*at()` syscall - * family, so they are skipped off `SUPPORTS_DIR_FD` — the **same** capability - * probe the implementation gates on, never off a platform string, so a probe - * that silently reported "unsupported" could not also silently skip the tests - * that would have caught it. + * Defensible only where a destructive operation can be addressed relative to a + * pinned directory, which is true two ways: an `*at()` family + * (`SUPPORTS_DIR_FD`, false on every Node release to date) or descriptor + * addressing through procfs (`SUPPORTS_PROC_FD`, true on Linux). Gated on the + * **same** capability probes the implementation gates on, never on a platform + * string, so a probe that silently reported "unsupported" could not also silently + * skip the tests that would have caught it. * - * On Node that probe is false: `fs`/`fs.promises` expose no `renameat` or - * `unlinkat`, and `FileHandle` has no `rename`/`unlink`, so a destructive - * operation cannot be addressed relative to a pinned descriptor at all. These - * cases are therefore a documented residual exposure on this runtime — see - * `SUPPORTS_DIR_FD` in `safe-fs.ts`. They are written out - * in full so they become live the moment the probe flips. + * So these run on Linux, where `safe-fs.ts` addresses `/`'s children + * through `/proc/self/fd/` and the swap cannot redirect the write or the + * unlink. They are skipped on macOS and Windows, where the per-component `lstat` + * floor narrows the window without closing it and the exposure is documented + * instead — see `SUPPORTS_DIR_FD` in `safe-fs.ts`. They stay written out in full + * so they become live the moment either probe flips. */ -describe.skipIf(!SUPPORTS_DIR_FD)('writeSkills TOCTOU swap races', () => { +describe.skipIf(!SUPPORTS_DIR_FD && !SUPPORTS_PROC_FD)('writeSkills TOCTOU swap races', () => { /** * Fires the swap at the exact instant of an operation: renames * `/` aside and leaves a symlink to `outside` in its place, then @@ -1144,6 +1164,30 @@ describe.skipIf(!SUPPORTS_DIR_FD)('writeSkills TOCTOU swap races', () => { }); describe('filesystem capability probe', () => { + it('reports descriptor addressing only where procfs actually provides it', async () => { + // The Linux fast path is gated on this, and a hardcoded `true` would make + // the swap-race cases above pass vacuously on a platform that cannot + // support them. So assert the two halves of the real property: never true + // off Linux, and on Linux only when `/proc/self/fd/` resolves back to + // the inode the descriptor holds. + if (process.platform !== 'linux') { + expect(SUPPORTS_PROC_FD).toBe(false); + return; + } + const { open, lstat, stat } = await import('node:fs/promises'); + const handle = await open(tmpdir(), fsConstants.O_RDONLY | fsConstants.O_DIRECTORY); + try { + const address = `/proc/self/fd/${handle.fd}`; + const pinned = await handle.stat({ bigint: true }); + const resolved = await stat(address, { bigint: true }); + const usable = + (await lstat(address)).isSymbolicLink() && resolved.dev === pinned.dev && resolved.ino === pinned.ino; + expect(SUPPORTS_PROC_FD).toBe(usable); + } finally { + await handle.close(); + } + }); + it('is a real feature test rather than a hardcoded answer', async () => { // Node exposes no `*at()` family, so TypeScript // runs a per-component lstat check instead and the two diff --git a/packages/client/src/safe-fs.ts b/packages/client/src/safe-fs.ts index 6e73707b..64c0d6d2 100644 --- a/packages/client/src/safe-fs.ts +++ b/packages/client/src/safe-fs.ts @@ -2,27 +2,40 @@ * Symlink-refusing filesystem primitives: writing a file under a directory an * attacker may be racing you for. * - * A path check is only as good as the last path resolution after it, and on Node - * that gap cannot be fully closed — see {@link SUPPORTS_DIR_FD}. What is here is a - * per-component `lstat` check hardened as far as Node allows: every directory is - * opened with `O_NOFOLLOW`, every temp file is created exclusively in the target's - * own directory, and the pinned directory's identity is re-checked immediately - * before each destructive step. + * Split out because none of this knows what a skill is: it is the "write a file + * under a directory an attacker may be racing you for" problem, solved once. + * `skills-fs.ts` is the only caller today. + * + * The whole point is that a path check is only as good as the last path + * resolution after it. There are two implementations of that idea here, and which + * one runs is a platform property rather than a configuration choice: + * + * - **Linux — the swap window is closed.** Node exposes no `*at()` family (see + * {@link SUPPORTS_DIR_FD}), but it does not need one: a directory is pinned to a + * descriptor and its children are addressed as `/proc/self/fd//`, + * which the kernel resolves from the pinned *inode* rather than from the name. + * Renaming or symlinking the directory afterwards cannot redirect the + * operation. Gated on {@link SUPPORTS_PROC_FD}. + * - **Everywhere else — the window is narrowed, not closed.** A per-component + * `lstat` check, hardened as far as Node allows: every directory is opened with + * `O_NOFOLLOW`, every temp file is created exclusively in the target's own + * directory, and the pinned directory's identity is re-checked immediately + * before each destructive step. That last check is the floor, and a floor is not + * a fix — see {@link SUPPORTS_DIR_FD}. * - * **Platform bound — the floor is what runs everywhere, deliberately.** The - * Python SDK closes the swap window on POSIX with a descriptor walk; this module - * cannot, on any platform, because Node exposes no `*at()` family at all (see - * {@link SUPPORTS_DIR_FD}). So unlike Python, where the racy floor is the - * Windows-only fallback, here it is the *only* implementation — Linux included. * Windows is additionally not a supported or tested platform for this release: * reparse-point checks (`GetFileAttributesW`, or opening with * `FILE_FLAG_OPEN_REPARSE_POINT`) are **not implemented, by decision rather than * oversight**, since neither SDK repository has a Windows CI runner and Node - * gives this module no primitive that would make them meaningful. + * gives this module no primitive that would make them meaningful. That is also + * why a Linux-only fast path is an acceptable shape for the fix rather than a + * half-measure: the platforms it leaves on the floor are macOS, which is a + * development target, and Windows, which is out of scope. * - * The consequence is a single sentence, and it belongs in every deployment - * review: write permission on the managed root is *the* security boundary for - * skills materialization, so the privilege-separated deployment the README + * The consequence for the platforms on the floor is a single sentence, and it + * belongs in every deployment review: write permission on the managed root **or + * on any of its ancestors** is *the* security boundary for skills + * materialization, so the privilege-separated deployment the README * documents — reconcile identity separate from agent identity — is not advice but * the mitigation. Relatedly, this bound retroactively lowers the priority of the * Windows reserved-device-name work in `skills-fs.ts`: that code stays, because @@ -31,9 +44,10 @@ */ import { randomBytes } from 'node:crypto'; -import { constants as fsConstants } from 'node:fs'; +import { closeSync, constants as fsConstants, fstatSync, lstatSync, openSync, statSync } from 'node:fs'; import * as fsPromises from 'node:fs/promises'; import { type FileHandle, lstat, mkdir, open, rename, unlink } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; import path from 'node:path'; /** @@ -81,6 +95,75 @@ export const SUPPORTS_DIR_FD: boolean = (() => { return AT_FAMILY.every((name) => typeof exported[name] === 'function'); })(); +/** procfs's per-process descriptor table, where each open fd appears as a magic symlink. */ +const PROC_SELF_FD = '/proc/self/fd'; + +/** + * Whether children can be addressed relative to a held descriptor by *path*, + * through `/proc/self/fd//`. + * + * This is the way around {@link SUPPORTS_DIR_FD} on Linux. Node cannot pass a + * directory descriptor to `rename`, `unlink` or `open` — but it does not have to: + * the kernel resolves the `/proc/self/fd/` component to *the inode the + * descriptor holds*, not to the name it was opened under. So a path built on that + * prefix has the same property an `*at()` call would: renaming or symlinking the + * directory's name afterwards cannot redirect the operation, because the name is + * no longer part of the resolution. Every remaining component gets the usual + * `O_NOFOLLOW` treatment. + * + * `false` off Linux, where procfs does not exist — macOS has `/dev/fd/`, but + * it is not a directory-traversable prefix, so `/dev/fd//` does not + * resolve and there is nothing to gate on. Those platforms keep the per-component + * `lstat` floor described at {@link SUPPORTS_DIR_FD}, and for them write + * permission on the managed root *and its ancestors* remains the security + * boundary. Windows is not a supported platform for this release (see the module + * docblock), so a Linux-only fast path is the accepted design rather than a gap. + * + * A genuine probe rather than a platform string alone, for the same reason + * {@link SUPPORTS_DIR_FD} is: the swap-race tests are gated on this constant, so a + * wrong hardcoded answer would silently skip the tests that would have caught it. + * Linux without a mounted `/proc` — some minimal containers — answers `false` here + * and falls back correctly. The probe checks the property the fast path actually + * depends on: that the entry is procfs's magic symlink, and that resolving it + * reaches the very inode the descriptor is pinned to. + */ +export const SUPPORTS_PROC_FD: boolean = (() => { + if (process.platform !== 'linux') return false; + let fd: number | null = null; + try { + fd = openSync(tmpdir(), fsConstants.O_RDONLY | (fsConstants.O_DIRECTORY ?? 0)); + if (!lstatSync(`${PROC_SELF_FD}/${fd}`).isSymbolicLink()) return false; + const pinned = fstatSync(fd, { bigint: true }); + const resolved = statSync(`${PROC_SELF_FD}/${fd}`, { bigint: true }); + return resolved.dev === pinned.dev && resolved.ino === pinned.ino; + } catch { + return false; + } finally { + if (fd !== null) closeSync(fd); + } +})(); + +/** + * The prefix to build child paths on for a directory this process holds open. + * + * Returns the descriptor address on the fast path and `realPath` otherwise, so a + * caller writes `path.join(directoryAddress(handle, dir), name)` once and gets + * descriptor-relative addressing where the platform has it and the previous + * path-based behaviour where it does not. + * + * The returned string is for the *filesystem*, not for people: it is never what a + * report or an error message should show a caller. `skills-fs.ts` keeps the real + * path alongside it for that. + */ +export function directoryAddress(handle: FileHandle, realPath: string): string { + return SUPPORTS_PROC_FD ? `${PROC_SELF_FD}/${handle.fd}` : realPath; +} + +/** Whether `directory` is a descriptor address rather than an ordinary path. */ +function isDescriptorAddressed(directory: string): boolean { + return directory === PROC_SELF_FD || directory.startsWith(`${PROC_SELF_FD}/`); +} + /** * The destructive filesystem operations, as a replaceable record. * @@ -157,11 +240,22 @@ export async function openOrCreateDirectory(directory: string): Promise` reports procfs's magic + * symlink, not a directory. */ async function assertUnswapped(directory: string, handle: FileHandle): Promise { + if (isDescriptorAddressed(directory)) return; const pinned = await identityOf(handle); const onDisk = await lstat(directory, { bigint: true }); if (!onDisk.isDirectory() || Number(onDisk.dev) !== pinned.dev || onDisk.ino !== pinned.ino) { @@ -247,28 +341,19 @@ export async function atomicWrite( } /** - * `atomicWrite` against a directory this module does not already hold open. - * - * The handle is taken with `O_NOFOLLOW`, so a directory swapped for a symlink - * after the caller validated it fails the write instead of redirecting it. - */ -export async function atomicWriteIn(directory: string, name: string, data: Uint8Array): Promise { - const handle = await openDirectoryNoFollow(directory); - try { - await atomicWrite(directory, name, data, handle); - } finally { - await handle.close(); - } -} -/** * Removes `/` without following a trailing symlink. * * `unlink` never follows a *trailing* symlink, so a symlinked target would have * the link removed rather than its victim — but it does resolve the directory * above it, which is what makes a swapped `/` a delete primitive with - * an attacker-chosen target. The identity re-check narrows that window as far as - * Node permits; see {@link SUPPORTS_DIR_FD} for why it cannot be closed here. + * an attacker-chosen target. Pass a `directory` produced by + * {@link directoryAddress} and that resolution starts at the pinned inode, which + * closes the window; off {@link SUPPORTS_PROC_FD} the identity re-check narrows it + * as far as Node permits without closing it. + * + * `fsOps.unlink` is the one and only unlink call site for a managed file, so tests + * can intercept it. */ export async function unlinkNoFollow(directory: string, name: string, pinned: FileHandle): Promise { const target = path.join(directory, name); diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index 0af5ddaf..87a9ac86 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -13,15 +13,22 @@ * key, a corrupt manifest suppresses every destructive action, and an incomplete * retrieval suppresses pruning. Content is re-verified immediately before the * write, because a `Skill` can also be constructed directly by a caller. + * + * The managed root is pinned to a descriptor once, at the top of `writeSkills`, + * and held for the whole reconcile. Every path below it is built from + * {@link PinnedDirectory.address} rather than from the root's name, so on Linux + * the kernel resolves each operation from the pinned inode and a root swapped for + * a symlink mid-reconcile cannot redirect one. See `safe-fs.ts` for the mechanism + * and for what the other platforms get instead. */ import { createHash } from 'node:crypto'; import { constants as fsConstants } from 'node:fs'; -import { lstat, mkdir, open, readdir, realpath, rmdir, stat } from 'node:fs/promises'; +import { type FileHandle, lstat, mkdir, open, readdir, realpath, rmdir, stat } from 'node:fs/promises'; import path from 'node:path'; import { atomicWrite, - atomicWriteIn, + directoryAddress, openDirectoryNoFollow, openOrCreateDirectory, tempNamePattern, @@ -65,6 +72,35 @@ export const MANIFEST_VERSION = 1; /** The single file each skill materializes to, under `//`. */ export const SKILL_FILENAME = 'SKILL.md'; +/** + * A directory held open for the duration of an operation, with both of the ways + * it needs to be named. + * + * The two are the same string off the Linux fast path, and the split exists + * because they answer different questions: + * + * - `path` is the real, `realpath`'d location. It is what containment checks + * compare against, what the manifest records, and the only one of the two that + * may ever reach a caller — in a `ReconcileAction.path` or an error message. + * - `address` is what filesystem calls are issued against. On Linux it is + * `/proc/self/fd/`, which the kernel resolves from the pinned inode, so + * nothing done to the directory's *name* afterwards can redirect the call. + * + * Mixing them up is the bug this type exists to make hard, in both directions: + * an operation issued against `path` is racy, and an `address` shown to a caller + * is meaningless. + */ +type PinnedDirectory = { + readonly path: string; + readonly address: string; + readonly handle: FileHandle; +}; + +/** Pairs a freshly opened directory handle with the two names for it. */ +function pin(realPath: string, handle: FileHandle): PinnedDirectory { + return { path: realPath, address: directoryAddress(handle, realPath), handle }; +} + /** * Prefix on every error describing content that could not be retrieved. Tests * assert on it, so it lives in one place. @@ -186,7 +222,40 @@ export async function writeSkills( const deadline = performance.now() + timeout * 1000; const rootPath = await resolveRoot(root); - const { manifest, error: manifestError } = await loadManifest(rootPath); + + // Pin the root before anything reads or writes below it, and hold it until the + // reconcile is done. `resolveRoot` validated a *name*; from here on every path + // is built from the descriptor instead, so a root swapped for a symlink after + // that validation cannot redirect an operation into the attacker's directory. + // + // A swap that lands in the window before this open fails the open, and that is + // a run-level `error` action rather than a throw. The distinction is + // deliberate: `resolveRoot` throws because an unusable root is a caller + // mistake, whereas a root that was a real directory a moment ago and is a + // symlink now is an attack in flight, which belongs in the report next to + // every other refusal. + let rootHandle: FileHandle; + try { + rootHandle = await openDirectoryNoFollow(rootPath); + } catch (error) { + return createReconcileReport([runError(`the skills root could not be pinned: ${messageOf(error)}`)]); + } + + try { + return await reconcile(pin(rootPath, rootHandle), skills, { prune, timeout, onUnavailable, deadline }); + } finally { + await rootHandle.close().catch(() => undefined); + } +} + +/** Everything `writeSkills` does once the managed root is pinned. */ +async function reconcile( + root: PinnedDirectory, + skills: ReadonlyArray | '*', + options: { prune: boolean; timeout: number; onUnavailable: OnUnavailable; deadline: number }, +): Promise { + const { prune, timeout, onUnavailable, deadline } = options; + const { manifest, error: manifestError } = await loadManifest(root); const entries: Record = (manifest.entries as Record) ?? {}; const actions: ReconcileAction[] = []; @@ -195,7 +264,7 @@ export async function writeSkills( const { requests, incomplete: retrievalIncomplete } = await resolveRequests(skills, deadline, onUnavailable); - const { actions: written, timedOut } = await writeAll(rootPath, requests, entries, deadline, timeout); + const { actions: written, timedOut } = await writeAll(root, requests, entries, deadline, timeout); actions.push(...written); const incomplete = retrievalIncomplete || timedOut; @@ -208,7 +277,7 @@ export async function writeSkills( // other destructive action, and skipped once the deadline is gone: this is // hygiene, not part of converging on the requested set. if (manifestError === null && performance.now() < deadline) { - actions.push(...(await sweepOrphanTemps(rootPath, sweepableKeys(requests, entries)))); + actions.push(...(await sweepOrphanTemps(root, sweepableKeys(requests, entries)))); } // Pruning is destructive, so it needs a trustworthy picture of both sides: a @@ -216,10 +285,10 @@ export async function writeSkills( // retrieval that failed, or a deadline that expired mid-write — means we do not // know what is still current. Either way, deleting would be a guess. if (prune && manifestError === null && !incomplete) { - actions.push(...(await pruneEntries(rootPath, entries, new Set(requests.map((r) => r.key))))); + actions.push(...(await pruneEntries(root, entries, new Set(requests.map((r) => r.key))))); } - if (manifestError === null) actions.push(...(await rewriteManifest(rootPath, manifest, entries))); + if (manifestError === null) actions.push(...(await rewriteManifest(root, manifest, entries))); return createReconcileReport(actions); } @@ -234,8 +303,28 @@ function runError(message: string): ReconcileAction { return createReconcileAction({ key: '', action: 'error', error: message }); } +/** + * The descriptor addressing, as it appears quoted inside an `errno` message. + * + * Anchored on the literal procfs prefix and a run of digits, so it matches what + * {@link directoryAddress} builds and not a customer path that merely contains the + * words. + */ +const DESCRIPTOR_ADDRESS_IN_MESSAGE = /\/proc\/self\/fd\/\d+\//g; + +/** + * An error's message, with descriptor addressing stripped back out. + * + * On Linux a child is addressed as `/proc/self/fd//`, so a raw `errno` + * message quotes that instead of a path anyone recognizes. Removing the prefix + * leaves the name relative to whichever directory was pinned — `a/SKILL.md` for + * the root, `SKILL.md` for a skill directory — which is the form the rest of the + * messages in this file already use. How the write was made safe is not something + * an operator should have to decode out of an error string. + */ function messageOf(error: unknown): string { - return error instanceof Error ? error.message : String(error); + const raw = error instanceof Error ? error.message : String(error); + return raw.replace(DESCRIPTOR_ADDRESS_IN_MESSAGE, ''); } /** @@ -246,7 +335,7 @@ function messageOf(error: unknown): string { * manifest rewrite and orphan every file already written in this run. */ async function writeAll( - root: string, + root: PinnedDirectory, requests: readonly PendingWrite[], entries: Record, deadline: number, @@ -345,7 +434,7 @@ function serializeManifest(manifest: Record): Buffer { /** Writes the updated manifest. Returns an error action, or nothing. */ async function rewriteManifest( - root: string, + root: PinnedDirectory, manifest: Record, entries: Record, ): Promise { @@ -358,7 +447,7 @@ async function rewriteManifest( // Two-space indent, sorted keys, non-ASCII escaped, and no trailing newline — // byte-for-byte the shared on-disk form; see `serializeManifest`. const serialized = serializeManifest(updated); - await atomicWriteIn(root, MANIFEST_FILENAME, serialized); + await atomicWrite(root.address, MANIFEST_FILENAME, serialized, root.handle); } catch (error) { return [runError(`the skills manifest could not be written: ${messageOf(error)}`)]; } @@ -525,6 +614,12 @@ function resolveAll(deadline: number, onUnavailable: OnUnavailable): { requests: * An unusable root is a caller error rather than a per-skill outcome, so this * throws. Only the leaf directory is ever created — recursively creating missing * ancestors would let a typo scatter a directory tree. + * + * What this establishes is that a *name* was usable at one instant. It is not a + * defense on its own and never was: the caller pins the returned path to a + * descriptor immediately and addresses everything below it from there, because + * between this function's last `realpath` and any later operation the name can be + * swapped for a symlink by anyone with write access to the root's parent. */ async function resolveRoot(root: string): Promise { if (typeof root !== 'string' || root.length === 0) { @@ -583,18 +678,22 @@ async function resolveRoot(root: string): Promise { * * An absent manifest is not corrupt — that is simply a fresh root. */ -async function loadManifest(root: string): Promise<{ manifest: Record; error: string | null }> { +async function loadManifest( + root: PinnedDirectory, +): Promise<{ manifest: Record; error: string | null }> { const fresh = { manifestVersion: MANIFEST_VERSION, entries: {} }; let text: string; try { - // `readRegularFile` rather than a plain `readFile` for the same reason the - // skill files use it: the manifest lives in the skills root, so anyone able - // to swap a managed file for a FIFO can do it here too, and an open that - // blocks forever would hang the reconcile before the deadline is ever - // consulted. A non-regular file becomes the corrupt-manifest refusal below, - // which is the fail-closed outcome an unreadable manifest already had. - text = (await readRegularFile(path.join(root, MANIFEST_FILENAME))).toString('utf-8'); + // `readRegularFile` rather than a plain `readFile`, for the same reason the + // skill files use it: the manifest lives in the skills root, so anyone able to + // swap a managed file for a FIFO can do it here too, and an open that blocks + // forever would hang the reconcile before the deadline is ever consulted. + // Addressing through `root.address` stops the *swap*; it does nothing about a + // FIFO already sitting at the resolved path, so both defenses are needed. A + // non-regular file becomes the corrupt-manifest refusal below, which is the + // fail-closed outcome an unreadable manifest already had. + text = (await readRegularFile(path.join(root.address, MANIFEST_FILENAME))).toString('utf-8'); } catch (error) { if ((error as NodeJS.ErrnoException).code === 'ENOENT') return { manifest: fresh, error: null }; return { @@ -709,6 +808,14 @@ async function pathExists(target: string): Promise { * The containment check runs even when `skillDir` does not exist yet: `realpath` * on the deepest existing ancestor plus the remainder is what a fresh key under a * valid root has to satisfy. + * + * `root` is the real path; `skillDir` and `target` are addresses, which on the + * Linux fast path resolve through the pinned root's descriptor. `realpath` + * collapses those to the real location, so the comparison against `root` reads + * the same on both paths. Kept as **defense in depth** rather than as the + * boundary: on Linux the descriptor addressing is what makes containment + * structural, and these checks now catch a misbehaving key or a symlink planted + * inside the root rather than a swap of the root itself. */ async function unsafePathReason( root: string, @@ -736,7 +843,11 @@ async function unsafePathReason( } /** Reconciles one verified skill against the managed root. */ -async function writeOne(root: string, skill: Skill, entries: Record): Promise { +async function writeOne( + root: PinnedDirectory, + skill: Skill, + entries: Record, +): Promise { const key = skill.key; const failed = (message: string): ReconcileAction => createReconcileAction({ key, action: 'error', version: skill.version, error: message }); @@ -749,11 +860,14 @@ async function writeOne(root: string, skill: Skill, entries: Record { /** * Performs the write itself. Returns a failure reason, or `null` on success. * - * Decides nothing — `writeOne` above it has already decided *whether* to write. - * The directory is pinned to a handle and its identity is re-checked immediately - * before the rename, which is as much of the symlink-swap defense as Node permits - * (see `safe-fs.ts`). + * Split out of `writeOne` because everything above it decides *whether* to write + * and this decides nothing: the directory is pinned to a handle and the write is + * then addressed relative to that handle (see `safe-fs.ts`). + * + * `skillDir` already resolves through the pinned *root*, so it cannot have been + * redirected. The per-skill directory is pinned in turn and `SKILL.md` addressed + * relative to it, because the skill directory's own name is the next thing an + * attacker with write access can swap — and off the fast path `skillDir` is a + * plain path, where that swap is exactly the window the identity re-check inside + * `atomicWrite` is left to narrow. */ async function writeThroughPinnedDirectory( skillDir: string, @@ -891,7 +1011,7 @@ async function writeThroughPinnedDirectory( } try { - await atomicWrite(skillDir, SKILL_FILENAME, encoded, handle); + await atomicWrite(directoryAddress(handle, skillDir), SKILL_FILENAME, encoded, handle); } catch (error) { return `'${relative}' could not be written: ${messageOf(error)}`; } finally { @@ -959,7 +1079,7 @@ function pruneError(key: string, message: string, version: unknown = null): Reco * opt-out. */ async function pruneEntries( - root: string, + root: PinnedDirectory, entries: Record, requested: ReadonlySet, ): Promise { @@ -995,16 +1115,17 @@ async function pruneEntries( /** Removes one managed skill file, and its directory when that empties it. */ async function pruneOne( - root: string, + root: PinnedDirectory, relative: string, key: string, entries: Record, ): Promise { - const skillDir = path.join(root, key); + const skillDir = path.join(root.address, key); const target = path.join(skillDir, SKILL_FILENAME); + const reportedPath = path.join(root.path, key, SKILL_FILENAME); const version = (entries[relative] as Record).version; - const unsafe = await unsafePathReason(root, skillDir, target, key, false); + const unsafe = await unsafePathReason(root.path, skillDir, target, key, false); if (unsafe !== null) return pruneError(key, `'${relative}' was not removed: ${unsafe}`, version); let removedFromDisk = false; @@ -1016,16 +1137,18 @@ async function pruneOne( return pruneError(key, `'${relative}' was not removed: ${messageOf(error)}`, version); } try { - await unlinkNoFollow(skillDir, SKILL_FILENAME, handle); + await unlinkNoFollow(directoryAddress(handle, skillDir), SKILL_FILENAME, handle); } catch (error) { return pruneError(key, `'${relative}' could not be removed: ${messageOf(error)}`, version); } finally { await handle.close().catch(() => undefined); } removedFromDisk = true; - // Path-based, and safe that way: rmdir never follows a trailing symlink (it - // fails ENOTDIR) and only ever succeeds on an empty directory. The customer - // keeps their own files here too, so a failure is expected and ignored. + // Addressed through the pinned root rather than the skill directory's own + // handle, which is already closed — and safe path-based regardless: rmdir + // never follows a trailing symlink (it fails ENOTDIR) and only ever succeeds + // on an empty directory. The customer keeps their own files here too, so a + // failure is expected and ignored. await rmdir(skillDir).catch(() => undefined); } @@ -1037,7 +1160,7 @@ async function pruneOne( key, action: 'removed', version: isValidSkillVersion(version) ? version : null, - path: target, + path: reportedPath, }); } @@ -1067,7 +1190,7 @@ function sweepableKeys(requests: readonly PendingWrite[], entries: Record): Promise { +async function sweepOrphanTemps(root: PinnedDirectory, keys: ReadonlySet): Promise { const actions: ReconcileAction[] = []; for (const key of keys) actions.push(...(await sweepSkillDirectory(root, key))); return actions; @@ -1091,10 +1214,10 @@ async function sweepOrphanTemps(root: string, keys: ReadonlySet): Promis * That is a visible failure rather than a corrupt file, and concurrent reconciles * against a single managed root are outside the contract either way. */ -async function sweepSkillDirectory(root: string, key: string): Promise { +async function sweepSkillDirectory(root: PinnedDirectory, key: string): Promise { if (keyRejectionReason(key) !== null) return []; - const skillDir = path.join(root, key); + const skillDir = path.join(root.address, key); const pattern = tempNamePattern(SKILL_FILENAME); let handle: Awaited>; @@ -1104,11 +1227,16 @@ async function sweepSkillDirectory(root: string, key: string): Promise Date: Wed, 16 Sep 2026 14:41:04 -0400 Subject: [PATCH 08/55] docs(client): sweep review-closeout comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removes the internal-process references from this PR's comments and docs while keeping every technical constraint they carried: - The SEC-8985 ticket reference in the root-swap test header. The defect description around it is what a reader needs and stays as-is. - "the security review names", "reopening the security review", and "the security review asked for one; we declined" in agents.md and the prune test. The decisions and their reasoning stay; only the appeal to an internal process is gone. - The Windows bullet's reference to another repository's CI runner, and the Python contrast for where the racy floor sits. - Two re-introduced "Split out of ..." refactor narratives and the "retroactively lowers the priority" framing, which described project sequencing rather than the code. Note on the rebase onto the restacked #33: that PR's Bugbot fix routes the manifest read through `readRegularFile`, and this PR independently rewrote the same line to address the manifest via `root.address` with a plain `readFile`. The conflict resolution keeps both — `/proc/self/fd` addressing defeats a directory *swap*, but does nothing about a FIFO already sitting at the resolved path, so dropping `readRegularFile` would have restored the hang. The manifest FIFO test times out without it. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 2 +- packages/client/agents.md | 6 +++--- .../src/__tests__/skills-fs-root-swap.test.ts | 9 +++++---- .../client/src/__tests__/skills-fs.test.ts | 2 +- packages/client/src/safe-fs.ts | 20 ++++++++----------- packages/client/src/skills-fs.ts | 6 +++--- 6 files changed, 21 insertions(+), 24 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 2d04a8d5..63a4f3d7 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -339,7 +339,7 @@ example above, materializes only what the resolved variation actually asked for. **Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly, and it is a platform bound rather than a detail. Node exposes no `renameat`/`unlinkat`/`openat`, so no destructive step can be addressed relative to a pinned directory descriptor *directly* — and what the SDK can do about that differs by platform: - **On Linux, the swap window is closed.** The managed root is opened `O_RDONLY|O_DIRECTORY|O_NOFOLLOW` once and that descriptor is held for the whole reconcile, and every child — each `//`, each `SKILL.md`, each temp file, the manifest — is addressed as `/proc/self/fd//`. The kernel resolves that prefix to the inode the descriptor holds rather than to the name it was opened under, which gives it the same property an `*at()` call would: renaming the root, or replacing it with a symlink, after validation cannot redirect a write or a delete. -- **On macOS and Windows, the window is narrowed but not closed.** There is no traversable descriptor prefix (`/dev/fd//` does not resolve), so each check is a per-component `lstat` taken immediately before the path-based operation — a check-then-use race. An attacker who already has **write permission on the managed root, or on any of its ancestor directories**, can win that race and redirect a write or a delete outside the root. Windows additionally has no reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) in this release and is not a tested platform — neither SDK repository has a Windows CI runner. +- **On macOS and Windows, the window is narrowed but not closed.** There is no traversable descriptor prefix (`/dev/fd//` does not resolve), so each check is a per-component `lstat` taken immediately before the path-based operation — a check-then-use race. An attacker who already has **write permission on the managed root, or on any of its ancestor directories**, can win that race and redirect a write or a delete outside the root. Windows additionally has no reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) in this release and is not a tested platform — there is no Windows CI runner. So on those two platforms, write permission on the managed root **and on its ancestors** is *the* security boundary here. Keep them writable only by the identity running the reconcile — see [privilege separation](#privilege-separation-the-agent-must-not-be-able-to-rewrite-its-own-skills). The ancestor requirement is not a technicality: for a root of `.claude/skills`, the parent is `.claude`, which an agent identity often owns, and that alone is enough. diff --git a/packages/client/agents.md b/packages/client/agents.md index 0e874551..15c4ab14 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -267,10 +267,10 @@ Three specifics inside those two functions that a later contributor is most like ### 7. "Fixing" the platform bound, or adding a writability field to `ReconcileReport` -Two decisions here look like unfinished work and are not. Do not reverse either without reopening the security review. +Two decisions here look like unfinished work and are not. Neither should be reversed without re-examining the threat model each was decided against. -- **Windows reparse-point checks (`GetFileAttributesW`, `FILE_FLAG_OPEN_REPARSE_POINT`) are deliberately not implemented.** Windows is not a supported or tested platform for this release, neither repository has a Windows CI runner, and Node gives this module no `*at()` primitive that would make such checks meaningful anyway — the racy per-component `lstat` floor is universal here, Linux included, not a Windows-only fallback as it is in Python. The bound is documented in `safe-fs.ts` and the README instead. It also retroactively lowers the priority of the reserved-device-name work in `skills-fs.ts`: keep that code, since it keeps a root written on Linux usable when read from Windows, but do not read it as evidence that Windows is hardened. If Windows becomes supported, add the CI runner first and revisit both together. -- **`ReconcileReport` must not grow a "managed root is writable" field.** The security review asked for one; we declined, and the reasoning is load-bearing. The SDK knows only its *own* identity, which trivially has write access — it just wrote there — and cannot know which identity will later run the agent. Any check it could perform would answer a different question than the one asked and would manufacture false confidence exactly where caution is wanted. The real mitigation is deployment-side: run the reconcile as a different identity than the agent, so the `0644`/`0755` modes deny something and a prompt-injected agent cannot rewrite its own instructions or the manifest. The operator's verification steps live in the README. +- **Windows reparse-point checks (`GetFileAttributesW`, `FILE_FLAG_OPEN_REPARSE_POINT`) are deliberately not implemented.** Windows is not a supported or tested platform for this release, there is no Windows CI runner, and Node gives this module no `*at()` primitive that would make such checks meaningful anyway — off the Linux fast path the racy per-component `lstat` floor is what runs everywhere, rather than being a Windows-only fallback. The bound is documented in `safe-fs.ts` and the README instead. Keep the reserved-device-name handling in `skills-fs.ts`, since it keeps a root written on Linux usable when read from Windows, but do not read it as evidence that Windows is hardened. If Windows becomes supported, add the CI runner first and revisit both together. +- **`ReconcileReport` must not grow a "managed root is writable" field.** It has been asked for and rejected, and the reasoning is load-bearing. The SDK knows only its *own* identity, which trivially has write access — it just wrote there — and cannot know which identity will later run the agent. Any check it could perform would answer a different question than the one asked and would manufacture false confidence exactly where caution is wanted. The real mitigation is deployment-side: run the reconcile as a different identity than the agent, so the `0644`/`0755` modes deny something and a prompt-injected agent cannot rewrite its own instructions or the manifest. The operator's verification steps live in the README. --- diff --git a/packages/client/src/__tests__/skills-fs-root-swap.test.ts b/packages/client/src/__tests__/skills-fs-root-swap.test.ts index 8565481a..5c7c1eb1 100644 --- a/packages/client/src/__tests__/skills-fs-root-swap.test.ts +++ b/packages/client/src/__tests__/skills-fs-root-swap.test.ts @@ -2,7 +2,7 @@ * `writeSkills` — the swap one level up: the managed *root* replaced by a * symlink, not `/`. * - * The defect these were written against (SEC-8985 row 2): `resolveRoot` + * The defect these were written against: `resolveRoot` * validated the root once and returned a path, and nothing held it open. Each * write and each prune then opened `/` *by path* with * `O_NOFOLLOW | O_DIRECTORY` and pinned that. `O_NOFOLLOW` guards only the final @@ -25,9 +25,10 @@ * * Every test states the contract rather than the mechanism: nothing lands outside * the root, no outside file is overwritten, no outside file is removed. The - * trigger matches both the pre-fix and post-fix spellings of the child path (see - * the hook below), so these same bodies failed against the unfixed code and pass - * against the fixed one — which is the only thing that makes them evidence. + * trigger matches both the path-based and descriptor-based spellings of the child + * path (see the hook below), so these same bodies fail against an unpinned root + * and pass against a pinned one — which is the only thing that makes them + * evidence. * * In its own file because the swap has to land *before* the per-skill directory * is opened, which means intercepting `mkdir` and `open` from `node:fs/promises` diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index ef81a3be..abb576ff 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -1801,7 +1801,7 @@ describe('writeSkills corrupt manifest', () => { // ─── A well-formed manifest naming a path the SDK could not have written ───── /** - * The three literal cases the security review names for the prune path. + * The three literal cases that matter for the prune path. * * The distinction from the corrupt-manifest block above is the whole point: a * corrupt manifest suppresses every destructive action wholesale, so those tests diff --git a/packages/client/src/safe-fs.ts b/packages/client/src/safe-fs.ts index 64c0d6d2..e097f614 100644 --- a/packages/client/src/safe-fs.ts +++ b/packages/client/src/safe-fs.ts @@ -2,12 +2,8 @@ * Symlink-refusing filesystem primitives: writing a file under a directory an * attacker may be racing you for. * - * Split out because none of this knows what a skill is: it is the "write a file - * under a directory an attacker may be racing you for" problem, solved once. - * `skills-fs.ts` is the only caller today. - * - * The whole point is that a path check is only as good as the last path - * resolution after it. There are two implementations of that idea here, and which + * A path check is only as good as the last path resolution after it. There are + * two implementations of that idea here, and which * one runs is a platform property rather than a configuration choice: * * - **Linux — the swap window is closed.** Node exposes no `*at()` family (see @@ -26,8 +22,8 @@ * Windows is additionally not a supported or tested platform for this release: * reparse-point checks (`GetFileAttributesW`, or opening with * `FILE_FLAG_OPEN_REPARSE_POINT`) are **not implemented, by decision rather than - * oversight**, since neither SDK repository has a Windows CI runner and Node - * gives this module no primitive that would make them meaningful. That is also + * oversight**, since there is no Windows CI runner and Node gives this module no + * primitive that would make them meaningful. That is also * why a Linux-only fast path is an acceptable shape for the fix rather than a * half-measure: the platforms it leaves on the floor are macOS, which is a * development target, and Windows, which is out of scope. @@ -37,10 +33,10 @@ * on any of its ancestors** is *the* security boundary for skills * materialization, so the privilege-separated deployment the README * documents — reconcile identity separate from agent identity — is not advice but - * the mitigation. Relatedly, this bound retroactively lowers the priority of the - * Windows reserved-device-name work in `skills-fs.ts`: that code stays, because - * it keeps a managed root written on Linux usable when read from Windows, but it - * is not evidence that Windows is a hardened target. It is not. + * the mitigation. The Windows reserved-device-name handling in `skills-fs.ts` + * stays for a narrower reason than it may appear to serve: it keeps a managed + * root written on Linux usable when read from Windows. It is not evidence that + * Windows is a hardened target. */ import { randomBytes } from 'node:crypto'; diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index 87a9ac86..a67baf58 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -986,9 +986,9 @@ async function readRegularFile(target: string): Promise { /** * Performs the write itself. Returns a failure reason, or `null` on success. * - * Split out of `writeOne` because everything above it decides *whether* to write - * and this decides nothing: the directory is pinned to a handle and the write is - * then addressed relative to that handle (see `safe-fs.ts`). + * Decides nothing — `writeOne` above it has already decided *whether* to write. + * The directory is pinned to a handle and the write is then addressed relative to + * that handle (see `safe-fs.ts`). * * `skillDir` already resolves through the pinned *root*, so it cannot have been * redirected. The per-skill directory is pinned in turn and `SKILL.md` addressed From 6197fe3b7475a8166e0bfdb2cccecb27800f6457 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 4 Sep 2026 11:16:04 -0400 Subject: [PATCH 09/55] =?UTF-8?q?feat(client):=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20the=20FDv2=20delivery=20transport?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Node port of the Python SDK's FDv2 transport, behind the same SkillStore seam. Deliberately mechanical: same protocol boundary, same constants, same wire semantics, same diagnostics vocabulary. Nothing above the seam changed — the accessors, integrity verification, and writeSkills are untouched. The design's original Plan A — a bespoke poller against a gonfalon /private/flagdlv route — is dead. That route is authenticated by Cognito machine-token OAuth scopes with no per-tenant authorization, and the security review ruled out both relaxing its auth and shipping a machine credential to customer hosts. This uses GET /sdk/poll and GET /sdk/stream with the environment's server-side SDK key instead, which is also the channel payload signing will eventually cover. skills-fdv2.ts — FDv2SkillStore: authenticate, poll or stream, maintain selector/basis state, deserialize inline-resource/skill objects, hold them keyed by (key, objectVersion), and serve the seam. Bounded retries with capped jittered backoff, Retry-After honoured, If-None-Match/304. Platform globals only — fetch, AbortController, TextDecoder — so the content path adds no dependency. The trap, stated loudly and asserted in both directions: objectVersion is the skill's own version — the one {key, version} pins — while version is the payload version. Confusing them fails silently. seamObjectFromPut is the only place the translation happens. Flag and segment objects arrive on the same connection and are skipped cleanly rather than throwing; throwing is the unknown-kind reconnect loop this feature must not reproduce. Changes commit at payload-transferred, so an interrupted full transfer leaves last known good intact. One deliberate divergence from Python, documented on the class: the snapshot collapses to one object per key at its newest version, because this SDK has no newestByKey above the seam yet and //SKILL.md is a single path. End-to-end behaviour matches Python; getObject still resolves a pinned version out of the full set. contentHash is read from the envelope and verification semantics are unchanged. The field has not shipped on the write path yet, so its absence is made loudly diagnosable — an error per object naming missing_content_hash, a summary per wholly-hashless payload, and a diagnostics counter — rather than a silent empty store. There is deliberately no fallback that skips verification. skills-watch.ts — watchSkills/SkillWatcher pull the change-listener re-reconcile forward out of phase 4. A delete-object reaches a live stream in seconds, so a revoked skill's SKILL.md now leaves the disk within a debounce interval instead of at the next restart, which materially improves the review's AV-1. onUnavailable: 'keep' stays the default, as the review endorses. Server-side only: a mobile key or client-side environment ID throws from the constructor. Skill content is customer-confidential and payload assignment is shared across auth types, so this is the SDK-side half of that boundary. close() aborts the signal rather than only setting a flag, and every timer is unref'ed, so a background store is never why node stays up. Two stale doc comments on the SkillStore type corrected: the transport is no longer "future", and addListener now has a consumer. 103 tests against an in-process fake FDv2 endpoint that implements the contract: put/delete, objectVersion vs version, mixed and unknown-kind payloads, 304, basis round-tripping, reconnect/backoff, Retry-After, bounded retries, the snapshot collapse, and a missing-contentHash envelope producing withheld skills with the right reason code. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 40 +- packages/client/agents.md | 32 +- .../client/src/__tests__/skills-fdv2.test.ts | 1417 +++++++++++++++++ packages/client/src/index.ts | 4 + packages/client/src/skills-fdv2.ts | 1289 +++++++++++++++ packages/client/src/skills-watch.ts | 221 +++ packages/client/src/types.ts | 11 +- 7 files changed, 3006 insertions(+), 8 deletions(-) create mode 100644 packages/client/src/__tests__/skills-fdv2.test.ts create mode 100644 packages/client/src/skills-fdv2.ts create mode 100644 packages/client/src/skills-watch.ts diff --git a/packages/client/README.md b/packages/client/README.md index 63a4f3d7..63e99a81 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -229,7 +229,7 @@ if (!result.enabled) { Agent Skills are versioned `SKILL.md` documents managed in LaunchDarkly, attachable to AI Config variations by reference. This package surfaces which skills a config references, retrieves their content, and materializes them onto disk where agent runtimes (the Claude Agent SDK and friends) can discover them. -**Content delivery is not wired up in this release.** Everything below runs against the `SkillStore` seam; the shipped default is *absent*, so the accessors throw an actionable error until a store is configured. `InMemorySkillStore` is provided for local development, tests, and bring-your-own-content. The real transport ships in a follow-up. +**Everything below runs against the `SkillStore` seam**, and the shipped default is *absent*, so the accessors throw an actionable error until a store is configured. `InMemorySkillStore` is provided for local development, tests, and bring-your-own-content; `FDv2SkillStore` is the delivery transport that receives content from LaunchDarkly — see [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). ```ts import { createHash } from 'node:crypto'; @@ -326,6 +326,44 @@ example above, materializes only what the resolved variation actually asked for. | `allSkills()` | Every verified skill the store holds. | | `writeSkills(skills, root, options?)` | Materialize to `//SKILL.md`. Accepts `Skill` / `SkillReference` / key strings, or the literal `'*'`. Returns a `ReconcileReport`. Throws for a caller error — an unusable `root`, a bare string other than `'*'` — as distinct from the per-skill `error` actions in the report. | | `InMemorySkillStore` | A `SkillStore` backed by a plain object. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`. Holds one object per key, so it answers a version pin with what it has and lets the accessor refuse a mismatch. | +| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | +| `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | + +#### Receiving skills from LaunchDarkly + +`InMemorySkillStore` is for tests and bring-your-own-content. In production, skill content arrives through `FDv2SkillStore`, which speaks LaunchDarkly's SDK-facing FDv2 delivery channel — the same `GET /sdk/poll` and `GET /sdk/stream` endpoints the base SDK's FDv2 data source uses, authenticated with the environment's server-side SDK key. + +```ts +import { FDv2SkillStore, initClient, watchSkills } from '@launchdarkly/ai-server'; + +const store = new FDv2SkillStore(process.env.LD_SDK_KEY!).start(); +await store.waitForSkills(10_000); +await initClient({ skillStore: store }); + +// Materialize now, and re-materialize whenever delivery changes. +const { report, watcher } = await watchSkills('*', '.claude/skills'); +try { + // ... +} finally { + await watcher.close(); + await store.close(); +} +``` + +**Nothing above the store changes.** The accessors, integrity verification, and `writeSkills` are transport-agnostic: they see raw objects through the `SkillStore` seam and cannot tell which store produced them. Everything documented above about verification and reconcile semantics applies unchanged. + +**Server-side only.** Skills are for server-side agent runtimes and skill content is customer-confidential. A mobile key (`mob-…`) or a client-side environment ID throws from the constructor. + +**Streaming is the default, and it is what makes revocation fast.** A `delete-object` reaches a live stream in seconds; with `mode: 'poll'` it arrives within one `pollIntervalMs`. Paired with `watchSkills`, a revoked skill's `SKILL.md` leaves the disk without a restart. During an outage the store keeps serving the last content it received and `writeSkills`' default `onUnavailable: 'keep'` leaves managed files alone — an outage must not read as "everything was revoked". + +**The connection also carries your flags.** A client cannot request only the skill payload, so a skills-enabled environment delivers flag and segment objects on the same connection. They are skipped, not evaluated — this store does no evaluation of any kind — and `diagnostics.objectsIgnored` counts them. + +> **Beta caveats, worth knowing before you deploy.** Payload signing does not exist on this channel yet, so delivery is TLS-only and the content hash establishes self-consistency, not origin authenticity. FDv2 is opt-in per account: without it the endpoints return HTTP 403, which the store reports as a fatal error naming the setting. `ld-relay` does not speak the FDv2 endpoints, so relay-only deployments cannot receive skills. + +**If every skill comes back empty, check `diagnostics.hashlessObjects`.** Verification requires `contentHash` on the delivered object and withholds anything without one, so a nonzero count there means skills are being withheld rather than that the environment has none. The store logs an error per hashless object and one summary per wholly-hashless payload, both naming the reason. There is deliberately no fallback that skips verification: a hash the SDK computed from the content it was handed would certify the content against itself. + +| Export | Description | +|---|---| | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | | `createSkillOutcome(init)` | Build a frozen `SkillOutcome`. Exported for tests and for wrapping your own retrieval in the same shape. | | `SKILL_OBJECT_KIND` | `'skill'` — the delivery object kind. | diff --git a/packages/client/agents.md b/packages/client/agents.md index 15c4ab14..dba6568c 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -30,11 +30,39 @@ No other `@launchdarkly/ai-*` package may define or duplicate these. They import | `src/registry.ts` | `Registry`, `globalRegistry`, `compose` | | `src/skills-core.ts` | Agent Skills internals shared by the two layers above it: the store and telemetry seams, module state, integrity verification, store resolution | | `src/skills.ts` | `skillRefs`, the content accessors, `InMemorySkillStore`, and the documented test-injection hooks | +| `src/skills-fdv2.ts` | Agent Skills delivery transport — the FDv2 protocol, the `objectVersion`/`version` translation, the held object set, and `FDv2SkillStore`. Sits **below** the store seam; imports `skills-core` only, and nothing imports it | +| `src/skills-watch.ts` | Agent Skills eager re-reconcile — `watchSkills` / `SkillWatcher`, wiring the store's change listener to `writeSkills`. Sits **above** `skills-fs` and modifies none of it | | `src/skills-fs.ts` | `writeSkills` — the manifest format, on-disk filenames, and reconcile semantics | | `src/safe-fs.ts` | Symlink-refusing filesystem primitives. Knows nothing about skills; owns the single interceptable rename and unlink call sites | | `src/index.ts` | Public barrel — the only surface handler packages import from | -The Agent Skills modules are a deliberate split, and the dependencies run **one way only**: `types` ← `skills-core` ← `skills`, and `types` + `safe-fs` + `skills-core` ← `skills-fs`. `skills-core.ts` imports neither `skills.ts` nor `skills-fs.ts`. Do not add an edge that closes a cycle; the reason the store and the emitter live in `skills-core.ts` is so the accessor layer and the filesystem layer cannot disagree about whether one is configured. +The Agent Skills modules are a deliberate split, and the dependencies run **one way only**: `types` ← `skills-core` ← `skills`, and `types` + `safe-fs` + `skills-core` ← `skills-fs`. `skills-core.ts` imports neither `skills.ts` nor `skills-fs.ts`. The two later modules extend the same shape: `skills-fdv2.ts` sits *below* the seam and imports only `skills-core` + `types` — nothing imports it — and `skills-watch.ts` sits *above* `skills-fs` and imports it without being imported back. Do not add an edge that closes a cycle; the reason the store and the emitter live in `skills-core.ts` is so the accessor layer and the filesystem layer cannot disagree about whether one is configured. + +--- + +## Agent Skills — the delivery transport, and the one field that will bite you + +`FDv2SkillStore` speaks LaunchDarkly's SDK-facing FDv2 channel (`GET /sdk/poll`, `GET /sdk/stream`, server-side SDK key in `Authorization`, `basis` + `mv` params, `If-None-Match`/304). It lives below the seam and produces raw objects in the shape `SkillStore` documents; **nothing above the seam knows it exists**. The transport design was replaced wholesale late in this feature's life and cost zero changes above this line, which is the strongest evidence the seam is drawn correctly. If a transport change ever seems to require editing an accessor, verification, or `writeSkills`, the adapter boundary is wrong. + +**`objectVersion` is the skill's version. `version` is the payload's.** On the wire a skill `put-object` carries both, and they are not interchangeable: + +```json +{"key":"pdf-extraction","kind":"inline-resource","category":"skill", + "objectVersion":3,"version":42, + "object":{"contentType":"text/markdown","content":"…","contentHash":"…","name":"…"}} +``` + +`objectVersion` (3) is what a `{key, version}` reference pins and what becomes the seam's `version`. `version` (42) is the version of the *payload* the object arrived in — it moves when anything in the environment moves, including a flag with nothing to do with skills. Reading it as the skill's version fails **silently**: the object verifies, the hash matches, and the caller gets content under a version number that means nothing. Flags and segments carry only `version` and omit both `category` and `objectVersion`, which is exactly why the two fields look interchangeable. `seamObjectFromPut` is the only place the translation happens, and the `version translation` suite asserts it in both directions. + +**Skills are identified by `kind === 'inline-resource' && category === 'skill'`; everything else is ignored, not rejected.** An environment's payload assignment carries the flagging payload alongside the agent-skill payload, so flag and segment objects arrive as a matter of course. Throwing on an unrecognised kind is the unknown-kind reconnect loop this feature must not reproduce — a flag-delivery outage caused by a skills rollout. + +**Changes commit at `payload-transferred`, not as objects arrive.** A payload version is the unit of consistency: a half-applied full transfer would publish a state the server never described, and would briefly empty the store — which, with pruning on, is the difference between a reconcile and deleting a customer's skill files. An interrupted transfer therefore leaves last known good intact, and listeners fire once per commit. + +**A hashless object is held, not dropped.** Verification withholds it with `missing_content_hash`; the transport's job is to make that loud (an error per object, a summary per wholly-hashless payload, `diagnostics.hashlessObjects`) rather than to work around it. Dropping it at the transport would report `absent` — indistinguishable from "no such skill" — and would let a prune delete the last known-good copy on disk. Never synthesize a hash from the delivered content: that certifies the content against itself and verifies nothing. + +**`SkillObjectSet.snapshot` collapses to one object per key, and that is load-bearing here in a way it is not in Python.** `//SKILL.md` is a single path, so a whole-store consumer must see one object per key or a `'*'` reconcile writes the same path twice and `allSkills` returns two versions of one skill. The Python SDK collapses in `newest_by_key`, above the seam; that helper is not in this SDK yet, so the collapse happens in the adapter. End-to-end behaviour is identical, and `getObject` still resolves a pinned version out of the full set. When `newestByKey` lands in `skills-core.ts`, move it and delete the note on the class. + +**`close` aborts the signal, it does not just set a flag.** The delivery task spends its life awaiting a stream read, and a flag it never checks would leave a healthy stream running until the process exited. Every backoff timer and the `waitForSkills` timer are `unref`ed for the same reason: a background store must not be why `node` stays up. --- @@ -62,6 +90,8 @@ export { parseUsage, normalizeMode, parseAiConfig } from './tracking.js'; export { skillRefs, getSkill, getSkillResult, getSkills, allSkills, InMemorySkillStore } from './skills.js'; export { SKILL_OBJECT_KIND, MAX_SKILL_CONTENT_BYTES } from './skills-core.js'; export { writeSkills, SKILL_FILENAME, MANIFEST_FILENAME, MANIFEST_VERSION } from './skills-fs.js'; +export { FDv2SkillStore, DEFAULT_BASE_URI, SDK_DATA_MODEL_VERSION } from './skills-fdv2.js'; +export { watchSkills, SkillWatcher, DEFAULT_DEBOUNCE_MS } from './skills-watch.js'; export type { WriteSkillsOptions } from './skills-fs.js'; export { createSkill, createSkillOutcome, createSkillReference } from './types.js'; export type { diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts new file mode 100644 index 00000000..033e31f3 --- /dev/null +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -0,0 +1,1417 @@ +/** + * The FDv2 skill delivery transport. + * + * Two layers, deliberately, mirroring the Python suite: + * + * - **A real fake endpoint.** `FakeFDv2Endpoint` is an in-process + * `node:http` server implementing the wire contract — `basis` and `mv` query + * parameters, `Authorization`, `If-None-Match`/304, the `{"events": [...]}` + * polling envelope, and SSE for streaming. The store under test opens real + * sockets against it, so request construction and header handling are + * exercised rather than mocked. This is what stands in for a live server while + * the backend work is unmerged. + * - **The protocol reader driven directly.** Wire semantics — which objects are + * skills, `objectVersion` versus `version`, revocation, mixed payloads — are + * asserted against `ProtocolReader`, which has no I/O, so those cases read as + * the contract they are instead of as a server script. + */ + +import { createHash } from 'node:crypto'; +import { existsSync, readFileSync } from 'node:fs'; +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { _clearState, _setStore, allSkills, getSkill, getSkillResult, InMemorySkillStore } from '../skills.js'; +import { SKILL_OBJECT_KIND } from '../skills-core.js'; +import { + _warnedHashless, + backoffDelayMs, + classifyStatus, + decodePollBody, + FDV2_OBJECT_CATEGORY, + FDV2_OBJECT_KIND, + FDv2SkillStore, + isSkillEvent, + ProtocolReader, + RecoverableTransportError, + type Requester, + retryAfterMs, + SkillObjectSet, + seamObjectFromPut, + tombstoneFromDelete, +} from '../skills-fdv2.js'; +import { watchSkills } from '../skills-watch.js'; +import type { RawSkillObject } from '../types.js'; + +const SDK_KEY = 'sdk-00000000-0000-4000-8000-000000000000'; +const SKILL_BODY = '---\nname: PDF Extraction\n---\nExtract text from PDFs.\n'; + +const hash = (content: string): string => createHash('sha256').update(content, 'utf8').digest('hex'); + +// ─── Wire builders — one place that knows the shape ────────────────────────── + +type WireEvent = { event: string; data?: unknown }; + +function putSkill( + key = 'pdf-extraction', + { + objectVersion = 3 as unknown, + payloadVersion = 42, + content = SKILL_BODY, + contentHash = null as string | null, + omitHash = false, + omitObjectVersion = false, + } = {}, +): Record { + const envelope: Record = { + contentType: 'text/markdown', + content, + name: 'PDF Extraction', + description: 'Extracts text', + }; + if (!omitHash) envelope.contentHash = contentHash ?? hash(content); + const wire: Record = { + key, + kind: FDV2_OBJECT_KIND, + category: FDV2_OBJECT_CATEGORY, + version: payloadVersion, + object: envelope, + }; + if (!omitObjectVersion) wire.objectVersion = objectVersion; + return wire; +} + +function deleteSkill(key = 'pdf-extraction', { objectVersion = 3 as unknown, payloadVersion = 43 } = {}) { + return { + key, + kind: FDV2_OBJECT_KIND, + category: FDV2_OBJECT_CATEGORY, + objectVersion, + version: payloadVersion, + }; +} + +/** A flag `put-object`: no `category`, no `objectVersion`. */ +function putFlag(key = 'my-flag', version = 17) { + return { key, kind: 'flag', version, object: { key, version, on: true, variations: [true, false] } }; +} + +function putSegment(key = 'beta-users', version = 4) { + return { key, kind: 'segment', version, object: { key, version, included: [] } }; +} + +function serverIntent(code = 'xfer-full') { + return { payloads: [{ id: 'agent-skill', target: 1, intentCode: code, reason: 'test' }] }; +} + +const transferred = (state = 'basis-1', version = 42) => ({ state, version }); + +const events = (...pairs: Array<[string, unknown]>): WireEvent[] => pairs.map(([event, data]) => ({ event, data })); + +function fullPayload(objectEvents: Array<[string, unknown]>, state = 'basis-1'): WireEvent[] { + return events(['server-intent', serverIntent('xfer-full')], ...objectEvents, [ + 'payload-transferred', + transferred(state), + ]); +} + +// ─── The fake endpoint ─────────────────────────────────────────────────────── + +type RecordedRequest = { + path: string; + query: Record; + authorization?: string; + ifNoneMatch?: string; + accept?: string; +}; + +/** + * An in-process server implementing the SDK-facing FDv2 contract. + * + * Scripted per request: `queuePoll` appends a response for the next `/sdk/poll`, + * `queueStream` appends a sequence of SSE events for the next `/sdk/stream`. + * Every request's path, query and headers are recorded in `requests` so the tests + * can assert on what the store actually sent — which is the only way `basis` + * round-tripping and `If-None-Match` can be checked at all. + */ +class FakeFDv2Endpoint { + readonly requests: RecordedRequest[] = []; + holdStreamOpen = false; + private readonly polls: Array<{ + status: number; + events: WireEvent[]; + etag?: string; + retryAfter?: string; + }> = []; + private readonly streams: WireEvent[][] = []; + private readonly held = new Set(); + private server!: Server; + private port = 0; + + async listen(): Promise { + this.server = createServer((req, res) => this.route(req, res)); + await new Promise((resolve) => this.server.listen(0, '127.0.0.1', resolve)); + const address = this.server.address(); + this.port = typeof address === 'object' && address !== null ? address.port : 0; + } + + get baseUri(): string { + return `http://127.0.0.1:${this.port}`; + } + + queuePoll( + payloadEvents: WireEvent[] = [], + extra: { status?: number; etag?: string; retryAfter?: string } = {}, + ): void { + this.polls.push({ + status: extra.status ?? 200, + events: payloadEvents, + etag: extra.etag, + retryAfter: extra.retryAfter, + }); + } + + queueStream(payloadEvents: WireEvent[]): void { + this.streams.push(payloadEvents); + } + + private route(req: IncomingMessage, res: ServerResponse): void { + const url = new URL(req.url ?? '/', this.baseUri); + this.requests.push({ + path: url.pathname, + query: Object.fromEntries(url.searchParams.entries()), + authorization: req.headers.authorization, + ifNoneMatch: req.headers['if-none-match'] as string | undefined, + accept: req.headers.accept, + }); + if (url.pathname === '/sdk/poll') this.servePoll(res); + else if (url.pathname === '/sdk/stream') this.serveStream(res); + else { + res.writeHead(404); + res.end(); + } + } + + private servePoll(res: ServerResponse): void { + const queued = this.polls.shift() ?? { status: 304, events: [] }; + const headers: Record = {}; + if (queued.etag) headers.ETag = queued.etag; + if (queued.retryAfter) headers['Retry-After'] = queued.retryAfter; + if (queued.status === 200) { + const body = JSON.stringify({ events: queued.events }); + res.writeHead(200, { ...headers, 'Content-Type': 'application/json' }); + res.end(body); + return; + } + res.writeHead(queued.status, headers); + res.end(); + } + + private serveStream(res: ServerResponse): void { + const payloadEvents = this.streams.shift() ?? []; + res.writeHead(200, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache' }); + for (const event of payloadEvents) { + res.write(`event: ${event.event}\ndata: ${JSON.stringify(event.data ?? null)}\n\n`); + } + if (this.holdStreamOpen) { + // Held so a test can assert on the store's state without racing the + // reconnect path; released on `close`. + this.held.add(res); + return; + } + res.end(); + } + + async close(): Promise { + for (const res of this.held) res.end(); + this.held.clear(); + await new Promise((resolve) => this.server.close(() => resolve())); + } +} + +// ─── Fixtures ──────────────────────────────────────────────────────────────── + +let endpoint: FakeFDv2Endpoint; +let openStores: FDv2SkillStore[]; +let tempRoots: string[]; +let warnSpy: ReturnType; +let errorSpy: ReturnType; + +beforeEach(async () => { + endpoint = new FakeFDv2Endpoint(); + await endpoint.listen(); + openStores = []; + tempRoots = []; + _clearState(); + // The hashless-object error is deduped per process; per test here. + _warnedHashless.clear(); + warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); +}); + +afterEach(async () => { + for (const store of openStores) await store.close(); + await endpoint.close(); + for (const root of tempRoots) await rm(root, { recursive: true, force: true }); + _clearState(); + vi.restoreAllMocks(); +}); + +function pollStore(options: Record = {}): FDv2SkillStore { + const store = new FDv2SkillStore(SDK_KEY, { + baseUri: endpoint.baseUri, + mode: 'poll', + pollIntervalMs: 20, + initialBackoffMs: 5, + maxBackoffMs: 20, + ...options, + }); + openStores.push(store); + return store; +} + +function streamStore(options: Record = {}): FDv2SkillStore { + const store = new FDv2SkillStore(SDK_KEY, { + baseUri: endpoint.baseUri, + mode: 'stream', + initialBackoffMs: 5, + maxBackoffMs: 20, + ...options, + }); + openStores.push(store); + return store; +} + +async function scratchRoot(): Promise { + const root = await mkdtemp(path.join(tmpdir(), 'ld-skills-fdv2-')); + tempRoots.push(root); + return root; +} + +async function waitUntil(predicate: () => boolean, timeoutMs = 5000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (predicate()) return true; + await new Promise((resolve) => setTimeout(resolve, 5)); + } + return predicate(); +} + +const logged = (spy: ReturnType): string => spy.mock.calls.map((call) => String(call[0])).join('\n'); + +const consoleErrors = (): string => logged(errorSpy); + +// ─── Identifying skill objects, and ignoring everything else ───────────────── + +describe('object identification', () => { + it('identifies a skill by kind and category together', () => { + expect(isSkillEvent(putSkill())).toBe(true); + }); + + it('does not treat a flag as a skill', () => { + expect(isSkillEvent(putFlag())).toBe(false); + }); + + it('does not treat a segment as a skill', () => { + expect(isSkillEvent(putSegment())).toBe(false); + }); + + it('requires the category: inline-resource is a broad kind', () => { + expect(isSkillEvent({ ...putSkill(), category: 'prompt-template' })).toBe(false); + }); + + it('requires the kind: a skill category under another kind is not a skill', () => { + expect(isSkillEvent({ ...putSkill(), kind: 'some-future-kind' })).toBe(false); + }); + + it('documents that flags omit both category and objectVersion', () => { + expect('category' in putFlag()).toBe(false); + expect('objectVersion' in putFlag()).toBe(false); + }); + + it.each([null, undefined, 'skill', 3, []])('does not treat %s as a skill', (value) => { + expect(isSkillEvent(value)).toBe(false); + }); +}); + +// ─── objectVersion is not version. This is the whole ballgame. ─────────────── + +describe('version translation', () => { + it('turns objectVersion into the seam version', () => { + expect(seamObjectFromPut(putSkill('pdf-extraction', { objectVersion: 3, payloadVersion: 42 }))?.version).toBe(3); + }); + + it('never lets the payload version reach the seam', () => { + // The failure this guards against is silent: a store that read `version` + // would serve verifiable content under a version number that means nothing, + // and every pinned reference would resolve to the wrong thing with no error. + const raw = seamObjectFromPut(putSkill('pdf-extraction', { objectVersion: 3, payloadVersion: 42 })); + expect(raw?.version).not.toBe(42); + expect(Object.values(raw ?? {})).not.toContain(42); + }); + + it('distinguishes the two even when the payload version is lower', () => { + expect(seamObjectFromPut(putSkill('k', { objectVersion: 99, payloadVersion: 1 }))?.version).toBe(99); + }); + + it('does not default a missing objectVersion from the payload version', () => { + const raw = seamObjectFromPut(putSkill('k', { omitObjectVersion: true })); + expect(raw).not.toBeNull(); + expect('version' in (raw as RawSkillObject)).toBe(false); + }); + + it('carries an explicitly null objectVersion through rather than inventing one', () => { + // Carried, not invented: verification reports `invalid_version`. + expect(seamObjectFromPut(putSkill('k', { objectVersion: null }))?.version).toBeNull(); + }); + + it('translates objectVersion on a delete too', () => { + expect(tombstoneFromDelete(deleteSkill('k', { objectVersion: 3, payloadVersion: 43 }))?.objectVersion).toBe(3); + }); + + it('reads a delete with no usable objectVersion as revoking every version', () => { + expect(tombstoneFromDelete(deleteSkill('k', { objectVersion: null }))?.objectVersion).toBeNull(); + }); + + it('drops a keyless put, which has no identity to store it under', () => { + const { key: _dropped, ...keyless } = putSkill(); + expect(seamObjectFromPut(keyless)).toBeNull(); + }); + + it('copies the envelope verbatim', () => { + const raw = seamObjectFromPut(putSkill()); + expect(raw?.content).toBe(SKILL_BODY); + expect(raw?.contentHash).toBe(hash(SKILL_BODY)); + expect(raw?.name).toBe('PDF Extraction'); + expect(raw?.contentType).toBe('text/markdown'); + }); + + it('leaves an absent envelope field absent rather than defaulting it', () => { + const wire = putSkill(); + delete (wire.object as Record).name; + const raw = seamObjectFromPut(wire); + expect(raw).not.toBeNull(); + expect('name' in (raw as RawSkillObject)).toBe(false); + }); +}); + +// ─── The protocol reader ───────────────────────────────────────────────────── + +function drive(reader: ProtocolReader, payloadEvents: WireEvent[]) { + return payloadEvents.map((event) => reader.handle(event.event, event.data)); +} + +describe('protocol reader', () => { + it('commits a full transfer at payload-transferred', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + const outcomes = drive(reader, fullPayload([['put-object', putSkill()]])); + expect(held.size).toBe(1); + expect(outcomes.at(-1)?.committed).toBe(true); + expect(outcomes.at(-1)?.basis).toBe('basis-1'); + }); + + it('shows nothing before payload-transferred', () => { + // A payload version is the unit of consistency; half of one is not a state. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, events(['server-intent', serverIntent('xfer-full')], ['put-object', putSkill()])); + expect(held.size).toBe(0); + }); + + it('leaves last known good intact when a full transfer is interrupted', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill('pdf-extraction', { objectVersion: 1 })]])); + expect(held.get('pdf-extraction', null)).not.toBeNull(); + + drive( + reader, + events( + ['server-intent', serverIntent('xfer-full')], + ['put-object', putSkill('pdf-extraction', { objectVersion: 2 })], + ), + ); + expect(held.get('pdf-extraction', null)?.version).toBe(1); + }); + + it('replaces rather than merges on a full transfer', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill('first')]])); + drive(reader, fullPayload([['put-object', putSkill('second')]], 'basis-2')); + expect(held.get('first', null)).toBeNull(); + expect(held.get('second', null)).not.toBeNull(); + }); + + it('applies deltas over what is held on a change transfer', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill('first')]])); + drive( + reader, + events( + ['server-intent', serverIntent('xfer-changes')], + ['put-object', putSkill('second')], + ['payload-transferred', transferred('basis-2')], + ), + ); + expect(held.get('first', null)).not.toBeNull(); + expect(held.get('second', null)).not.toBeNull(); + }); + + it('revokes a skill on delete-object', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill()]])); + drive( + reader, + events( + ['server-intent', serverIntent('xfer-changes')], + ['delete-object', deleteSkill()], + ['payload-transferred', transferred('basis-2')], + ), + ); + expect(held.get('pdf-extraction', null)).toBeNull(); + expect(reader.diagnostics.objectsRevoked).toBe(1); + }); + + it('notifies a delete with a tombstone carrying no content', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill()]])); + const outcomes = drive( + reader, + events( + ['server-intent', serverIntent('xfer-changes')], + ['delete-object', deleteSkill()], + ['payload-transferred', transferred('basis-2')], + ), + ); + const changes = outcomes.at(-1)?.changes ?? []; + expect(changes).toEqual([{ key: 'pdf-extraction', version: 3 }]); + expect('content' in changes[0]).toBe(false); + }); + + it('leaves the other version held when one version is deleted', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive( + reader, + fullPayload([ + ['put-object', putSkill('pdf-extraction', { objectVersion: 2 })], + ['put-object', putSkill('pdf-extraction', { objectVersion: 3 })], + ]), + ); + drive( + reader, + events( + ['server-intent', serverIntent('xfer-changes')], + ['delete-object', deleteSkill('pdf-extraction', { objectVersion: 3 })], + ['payload-transferred', transferred('basis-2')], + ), + ); + expect(held.get('pdf-extraction', 2)).not.toBeNull(); + expect(held.get('pdf-extraction', null)?.version).toBe(2); + }); + + it('skips flag and segment objects cleanly', () => { + // The mixed payload is the normal case, not an edge one: an environment's + // assignment carries the flagging payload alongside the agent-skill payload. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + const outcomes = drive( + reader, + fullPayload([ + ['put-object', putFlag('flag-a')], + ['put-object', putSkill('pdf-extraction')], + ['put-object', putSegment('beta-users')], + ['put-object', putFlag('flag-b')], + ['delete-object', putFlag('flag-c')], + ]), + ); + expect(held.size).toBe(1); + expect(held.get('pdf-extraction', null)).not.toBeNull(); + expect(reader.diagnostics.objectsIgnored).toBe(4); + expect(reader.diagnostics.skillObjectsReceived).toBe(1); + expect(outcomes.every((o) => !o.fatal && !o.disconnect)).toBe(true); + }); + + it('ignores an unknown kind rather than treating it as fatal', () => { + // Erroring here is the unknown-kind reconnect loop this feature must not + // reproduce — a flag-delivery outage caused by a skills rollout. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + const outcomes = drive( + reader, + fullPayload([['put-object', { key: 'x', kind: 'quantum-widget', version: 1, object: { a: 1 } }]]), + ); + expect(held.size).toBe(0); + expect(outcomes.every((o) => !o.fatal && !o.disconnect)).toBe(true); + }); + + it('ignores an unknown event name', () => { + const outcome = new ProtocolReader(new SkillObjectSet()).handle('some-future-event', { anything: true }); + expect(outcome.fatal).toBeUndefined(); + expect(outcome.disconnect).toBeUndefined(); + }); + + it('does nothing on a heartbeat', () => { + expect(new ProtocolReader(new SkillObjectSet()).handle('heart-beat', null)).toEqual({}); + }); + + it('abandons the in-flight payload on an error event', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill('pdf-extraction', { objectVersion: 1 })]])); + const outcomes = drive( + reader, + events( + ['server-intent', serverIntent('xfer-full')], + ['put-object', putSkill('pdf-extraction', { objectVersion: 2 })], + ['error', { payloadId: 'agent-skill', reason: 'backend unavailable' }], + ), + ); + expect(outcomes.at(-1)?.disconnect).toBeTruthy(); + expect(held.get('pdf-extraction', null)?.version).toBe(1); + }); + + it('asks for a reconnect on goodbye', () => { + const outcome = new ProtocolReader(new SkillObjectSet()).handle('goodbye', { + reason: 'rebalancing', + silent: false, + }); + expect(outcome.disconnect).toBeTruthy(); + expect(outcome.fatal).toBeFalsy(); + }); + + it('treats a catastrophic goodbye as fatal', () => { + const outcome = new ProtocolReader(new SkillObjectSet()).handle('goodbye', { + reason: 'no', + silent: false, + catastrophe: true, + }); + expect(outcome.fatal).toBeTruthy(); + }); + + it('holds everything and commits on transfer-none', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill()]])); + drive(reader, events(['server-intent', serverIntent('none')], ['payload-transferred', transferred('basis-2')])); + expect(held.size).toBe(1); + }); + + it('treats an object arriving with no intent as a delta', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, events(['put-object', putSkill()], ['payload-transferred', transferred('basis-1')])); + expect(held.size).toBe(1); + }); +}); + +// ─── The held object set ───────────────────────────────────────────────────── + +describe('the held object set', () => { + const raws: RawSkillObject[] = [ + { key: 'a', version: 1, content: 'x', contentHash: hash('x') }, + { key: 'a', version: 4, content: 'y', contentHash: hash('y') }, + { key: 'b', version: 2, content: 'z', contentHash: hash('z') }, + { key: 'malformed', version: 'not-a-version', content: 'q' }, + ]; + + const filled = (): SkillObjectSet => { + const set = new SkillObjectSet(); + for (const raw of raws) set.put({ ...raw }); + return set; + }; + + it('resolves a pin to the pinned version, not the newest', () => { + expect(filled().get('a', 1)?.content).toBe('x'); + expect(filled().get('a', null)?.content).toBe('y'); + }); + + it('falls through to the version-less entry for a pin it cannot satisfy well', () => { + // A malformed object must reach verification and be withheld with a signal, + // rather than reading as simply absent. + expect(filled().get('malformed', 7)?.content).toBe('q'); + }); + + it('answers null for a key it does not hold', () => { + expect(filled().get('missing', null)).toBeNull(); + expect(filled().get('a', 99)).toBeNull(); + }); + + it('collapses the snapshot to one object per key at its newest version', () => { + // `//SKILL.md` is a single path, so a whole-store consumer must + // see one object per key. See the class docstring for why the collapse lives + // here in this SDK and in `newest_by_key` in Python. + const snapshot = filled().snapshot(); + const forA = Object.values(snapshot).filter((raw) => raw.key === 'a'); + expect(forA).toHaveLength(1); + expect(forA[0].version).toBe(4); + expect(Object.keys(snapshot)).toHaveLength(3); + }); + + it('keeps a malformed object in the snapshot so verification withholds it', () => { + expect(Object.values(filled().snapshot()).some((raw) => raw.key === 'malformed')).toBe(true); + }); +}); + +// ─── The store against the fake endpoint ───────────────────────────────────── + +describe('polling against the endpoint', () => { + it('makes a polled skill retrievable through the accessors', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')?.version).toBe(3); + }); + + it('sends the SDK key and the data model version', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(endpoint.requests[0].path).toBe('/sdk/poll'); + expect(endpoint.requests[0].authorization).toBe(SDK_KEY); + expect(endpoint.requests[0].query.mv).toBe('1'); + }); + + it('sends no basis on the first request', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(endpoint.requests[0].query.basis).toBeUndefined(); + }); + + it('echoes the basis from payload-transferred on the next request', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]], 'selector-abc')); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(await waitUntil(() => endpoint.requests.length >= 2)).toBe(true); + expect(endpoint.requests[1].query.basis).toBe('selector-abc'); + }); + + it('advances the basis across successive payloads', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]], 'basis-1')); + endpoint.queuePoll( + events( + ['server-intent', serverIntent('xfer-changes')], + ['put-object', putSkill('second')], + ['payload-transferred', transferred('basis-2')], + ), + ); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => endpoint.requests.length >= 3)).toBe(true); + expect(endpoint.requests.slice(0, 3).map((r) => r.query.basis)).toEqual([undefined, 'basis-1', 'basis-2']); + }); + + it('returns an ETag as If-None-Match', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]]), { etag: 'W/"v1"' }); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(await waitUntil(() => endpoint.requests.length >= 2)).toBe(true); + expect(endpoint.requests[1].ifNoneMatch).toBe('W/"v1"'); + }); + + it('keeps held content across a 304', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]]), { etag: 'W/"v1"' }); + endpoint.queuePoll([], { status: 304 }); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(await waitUntil(() => endpoint.requests.length >= 3)).toBe(true); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + expect(store.diagnostics.payloadsTransferred).toBe(1); + expect(store.failed).toBeNull(); + }); + + it('releases waitForSkills on a 304 before any payload', async () => { + // A reconnect with a cached basis has nothing to transfer; boot must not + // block on a payload the server has no reason to send. + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + }); + + it('yields only the skill from a mixed payload', async () => { + endpoint.queuePoll( + fullPayload([ + ['put-object', putFlag('flag-a')], + ['put-object', putSegment('beta')], + ['put-object', putSkill('pdf-extraction')], + ['put-object', putFlag('flag-b')], + ]), + ); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + const held = store.allObjects(SKILL_OBJECT_KIND); + expect(Object.keys(held)).toHaveLength(1); + expect(Object.values(held)[0].key).toBe('pdf-extraction'); + expect(store.diagnostics.objectsIgnored).toBe(3); + }); + + it('removes a revoked skill', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + endpoint.queuePoll( + events( + ['server-intent', serverIntent('xfer-changes')], + ['delete-object', deleteSkill()], + ['payload-transferred', transferred('basis-2')], + ), + ); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + expect( + await waitUntil( + () => store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction') === null && store.diagnostics.objectsRevoked === 1, + ), + ).toBe(true); + }); + + it('serves only the kind it holds', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(store.getObject('flag', 'pdf-extraction')).toBeNull(); + expect(store.allObjects('flag')).toEqual({}); + }); +}); + +describe('streaming against the endpoint', () => { + it('lands a streamed payload', async () => { + endpoint.holdStreamOpen = true; + endpoint.queueStream(fullPayload([['put-object', putSkill()]])); + const store = streamStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + }); + + it('advertises text/event-stream', async () => { + endpoint.holdStreamOpen = true; + endpoint.queueStream(fullPayload([['put-object', putSkill()]])); + const store = streamStore(); + store.start(); + await store.waitForSkills(5000); + expect(endpoint.requests[0].path).toBe('/sdk/stream'); + expect(endpoint.requests[0].accept).toBe('text/event-stream'); + }); + + it('applies a streamed revocation without a restart', async () => { + endpoint.holdStreamOpen = true; + endpoint.queueStream([ + ...fullPayload([['put-object', putSkill()]]), + ...events( + ['server-intent', serverIntent('xfer-changes')], + ['delete-object', deleteSkill()], + ['payload-transferred', transferred('basis-2')], + ), + ]); + const store = streamStore(); + store.start(); + expect( + await waitUntil( + () => store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction') === null && store.diagnostics.objectsRevoked === 1, + ), + ).toBe(true); + }); + + it('reconnects with the basis it reached', async () => { + endpoint.queueStream(fullPayload([['put-object', putSkill()]], 'basis-1')); + endpoint.queueStream(events(['heart-beat', null])); + const store = streamStore(); + store.start(); + expect(await waitUntil(() => endpoint.requests.length >= 2)).toBe(true); + expect(endpoint.requests[1].query.basis).toBe('basis-1'); + }); + + it('keeps content across a reconnect', async () => { + endpoint.queueStream(fullPayload([['put-object', putSkill()]])); + endpoint.queueStream(events(['heart-beat', null])); + const store = streamStore(); + store.start(); + expect(await waitUntil(() => endpoint.requests.length >= 2)).toBe(true); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + }); + + it('returns promptly from close while a stream is open', async () => { + // The delivery task spends its life awaiting a read; aborting the signal is + // what interrupts it. A flag it never checks would leave a healthy stream + // running until the process exited. + endpoint.holdStreamOpen = true; + endpoint.queueStream(fullPayload([['put-object', putSkill()]])); + const store = streamStore(); + store.start(); + await store.waitForSkills(5000); + const started = Date.now(); + await store.close(); + expect(Date.now() - started).toBeLessThan(1000); + }); + + it('does not report an interrupted stream as a failure', async () => { + endpoint.holdStreamOpen = true; + endpoint.queueStream(fullPayload([['put-object', putSkill()]])); + const store = streamStore(); + store.start(); + await store.waitForSkills(5000); + await store.close(); + expect(store.failed).toBeNull(); + }); +}); + +// ─── Failure handling ──────────────────────────────────────────────────────── + +/** Throws a scripted sequence, so backoff is asserted without real sockets. */ +class ScriptedRequester implements Requester { + readonly calls: Array<[string | null, string | null]> = []; + + constructor(private readonly outcomes: unknown[] = []) {} + + private next(): unknown { + return this.outcomes.length > 0 ? this.outcomes.shift() : new RecoverableTransportError('scripted failure'); + } + + async poll(basis: string | null, etag: string | null): Promise { + this.calls.push([basis, etag]); + throw this.next(); + } + + async stream(basis: string | null): Promise { + this.calls.push([basis, null]); + throw this.next(); + } +} + +describe('failure handling', () => { + it('stops on 403 and names the protocol control flag', async () => { + endpoint.queuePoll([], { status: 403 }); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toContain('403'); + expect(store.failed).toContain('fdv2-protocol-control'); + expect(consoleErrors()).toContain('fdv2-protocol-control'); + }); + + it('stops on 401', async () => { + endpoint.queuePoll([], { status: 401 }); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toContain('401'); + }); + + it('releases waitForSkills on a fatal failure rather than hanging', async () => { + endpoint.queuePoll([], { status: 401 }); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(store.failed).not.toBeNull(); + }); + + it('keeps last known good servable after a fatal failure', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + endpoint.queuePoll([], { status: 403 }); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + }); + + it('retries a 500', async () => { + endpoint.queuePoll([], { status: 500 }); + endpoint.queuePoll([], { status: 503 }); + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(store.failed).toBeNull(); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + }); + + it('resets the failure count on success', async () => { + endpoint.queuePoll([], { status: 500 }); + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(await waitUntil(() => store.diagnostics.connectionFailures === 0)).toBe(true); + }); + + it('bounds retries', async () => { + const store = new FDv2SkillStore(SDK_KEY, { + mode: 'poll', + pollIntervalMs: 5, + initialBackoffMs: 1, + maxBackoffMs: 2, + maxConsecutiveFailures: 3, + requester: new ScriptedRequester(), + }); + openStores.push(store); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toContain('gave up after 4 consecutive failures'); + }); + + it('honours a Retry-After header off the wire', async () => { + endpoint.queuePoll([], { status: 429, retryAfter: '0' }); + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + // If Retry-After were ignored the 5s backoff would blow the timeout. + const store = pollStore({ initialBackoffMs: 5000 }); + store.start(); + expect(await store.waitForSkills(3000)).toBe(true); + }); + + it('parses Retry-After into milliseconds', () => { + expect(retryAfterMs(new Headers({ 'Retry-After': '2' }))).toBe(2000); + expect(retryAfterMs(new Headers({ 'Retry-After': 'Wed, 21 Oct 2026 07:28:00 GMT' }))).toBeNull(); + expect(retryAfterMs(new Headers())).toBeNull(); + }); + + it('classifies statuses into recoverable and fatal', () => { + expect(classifyStatus(500).constructor.name).toBe('RecoverableTransportError'); + expect(classifyStatus(429).constructor.name).toBe('RecoverableTransportError'); + expect(classifyStatus(401).constructor.name).toBe('FatalTransportError'); + expect(classifyStatus(403).constructor.name).toBe('FatalTransportError'); + expect(classifyStatus(404).constructor.name).toBe('FatalTransportError'); + expect(classifyStatus(400).constructor.name).toBe('FatalTransportError'); + }); + + it('makes backoff exponential and capped', () => { + expect(backoffDelayMs(1, 1000, 30_000, 0)).toBe(1000); + expect(backoffDelayMs(2, 1000, 30_000, 0)).toBe(2000); + expect(backoffDelayMs(3, 1000, 30_000, 0)).toBe(4000); + expect(backoffDelayMs(20, 1000, 30_000, 0)).toBe(30_000); + }); + + it('never lets jitter exceed the cap', () => { + for (let attempt = 1; attempt < 12; attempt += 1) { + for (let i = 0; i < 50; i += 1) { + const delay = backoffDelayMs(attempt, 1000, 5000); + expect(delay).toBeGreaterThanOrEqual(0); + expect(delay).toBeLessThanOrEqual(5000); + } + } + }); + + it('treats a malformed polling envelope as recoverable', () => { + expect(() => decodePollBody('{"nope":1}')).toThrow(/no 'events' array/); + expect(() => decodePollBody('not json')).toThrow(/not valid JSON/); + }); + + it('does not let a throwing listener kill delivery', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill('first')]])); + endpoint.queuePoll( + events( + ['server-intent', serverIntent('xfer-changes')], + ['put-object', putSkill('second')], + ['payload-transferred', transferred('basis-2')], + ), + ); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.addListener(SKILL_OBJECT_KIND, () => { + throw new Error('boom'); + }); + store.start(); + expect(await waitUntil(() => store.getObject(SKILL_OBJECT_KIND, 'second') !== null)).toBe(true); + expect(store.failed).toBeNull(); + }); +}); + +// ─── The contentHash gap ───────────────────────────────────────────────────── + +describe('the missing contentHash', () => { + // The blocking backend gap, asserted as behaviour rather than assumed. An + // envelope with no `contentHash` must produce a *withheld* skill with the + // `missing_content_hash` reason — loudly, diagnosably, and without a crash. + // There is deliberately no fallback that skips verification: a hash the SDK + // computed from the content it was handed would certify the content against + // itself and verify nothing. + + it('withholds a hashless skill with the right reason', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill('pdf-extraction', { omitHash: true })]])); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const outcome = await getSkillResult('pdf-extraction'); + expect(outcome.skill).toBeNull(); + expect(outcome.reason).toBe('integrity_failure'); + expect(await getSkill('pdf-extraction')).toBeNull(); + expect(await allSkills()).toEqual([]); + }); + + it('still holds the object, so the outcome is not "absent"', async () => { + // Holding it is what makes the failure diagnosable. Dropping it at the + // transport would report `absent` — indistinguishable from "no such skill" — + // and would let a prune delete the last known-good copy already on disk. + endpoint.queuePoll(fullPayload([['put-object', putSkill('pdf-extraction', { omitHash: true })]])); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + const raw = store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction'); + expect(raw).not.toBeNull(); + expect('contentHash' in (raw as RawSkillObject)).toBe(false); + _setStore(store); + expect((await getSkillResult('pdf-extraction')).reason).not.toBe('absent'); + }); + + it('counts hashless objects', async () => { + endpoint.queuePoll( + fullPayload([ + ['put-object', putSkill('a', { omitHash: true })], + ['put-object', putSkill('b', { omitHash: true })], + ['put-object', putSkill('c')], + ]), + ); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(store.diagnostics.hashlessObjects).toBe(2); + expect(store.diagnostics.skillObjectsReceived).toBe(3); + }); + + it('logs an error naming the reason code', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill('pdf-extraction', { omitHash: true })]])); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + const errors = consoleErrors(); + expect(errors).toContain('missing_content_hash'); + expect(errors).toContain('pdf-extraction'); + expect(errors).toContain('contentHash'); + }); + + it('says so once when a whole payload is hashless', async () => { + endpoint.queuePoll( + fullPayload([ + ['put-object', putSkill('a', { omitHash: true })], + ['put-object', putSkill('b', { omitHash: true })], + ]), + ); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + const summaries = consoleErrors() + .split('\n') + .filter((line) => line.includes('No skill content will resolve')); + expect(summaries).toHaveLength(1); + expect(summaries[0]).toContain('All 2 skill object(s)'); + }); + + it('does not claim total failure for a partly hashed payload', async () => { + endpoint.queuePoll( + fullPayload([ + ['put-object', putSkill('a', { omitHash: true })], + ['put-object', putSkill('b')], + ]), + ); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(consoleErrors()).not.toContain('No skill content will resolve'); + }); + + it('distinguishes a mismatched hash from a missing one', async () => { + // `missing_content_hash` and `hash_mismatch` must not collapse: one is a + // backend gap and the other is possible tampering. + endpoint.queuePoll( + fullPayload([['put-object', putSkill('pdf-extraction', { contentHash: hash('something else') })]]), + ); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + expect(store.diagnostics.hashlessObjects).toBe(0); + _setStore(store); + expect((await getSkillResult('pdf-extraction')).reason).toBe('integrity_failure'); + }); + + it('resolves a hashed skill end to end', async () => { + // The positive control: everything above is a gap, not a broken adapter. + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const skill = await getSkill('pdf-extraction'); + expect(skill?.key).toBe('pdf-extraction'); + expect(skill?.version).toBe(3); + expect(new TextDecoder().decode(skill?.content)).toBe(SKILL_BODY); + expect(skill?.contentHash).toBe(hash(SKILL_BODY)); + expect(skill?.name).toBe('PDF Extraction'); + }); + + it('resolves a pinned reference to the pinned objectVersion', async () => { + endpoint.queuePoll( + fullPayload([ + ['put-object', putSkill('pdf-extraction', { objectVersion: 2, content: 'v2 body' })], + ['put-object', putSkill('pdf-extraction', { objectVersion: 5, content: 'v5 body' })], + ]), + ); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const pinned = await getSkill('pdf-extraction', { version: 2 }); + expect(new TextDecoder().decode(pinned?.content)).toBe('v2 body'); + expect((await getSkill('pdf-extraction'))?.version).toBe(5); + }); + + it('does not resolve the payload version as a skill version', async () => { + // The end-to-end form of the objectVersion/version assertion. Asking for the + // payload version resolves nothing — reported `absent`, because the store + // answers "I hold no such version" rather than answering with the wrong one. + endpoint.queuePoll( + fullPayload([['put-object', putSkill('pdf-extraction', { objectVersion: 3, payloadVersion: 42 })]]), + ); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const byPayloadVersion = await getSkillResult('pdf-extraction', { version: 42 }); + expect(byPayloadVersion.skill).toBeNull(); + expect(byPayloadVersion.reason).toBe('absent'); + expect(await getSkill('pdf-extraction', { version: 3 })).not.toBeNull(); + }); +}); + +// ─── Server-side only ──────────────────────────────────────────────────────── + +describe('server-side only', () => { + it('refuses a mobile key', () => { + expect(() => new FDv2SkillStore('mob-00000000-0000-4000-8000-000000000000')).toThrow(/mobile key/); + }); + + it('refuses a client-side environment ID', () => { + expect(() => new FDv2SkillStore('0123456789abcdef01234567')).toThrow(/client-side/); + }); + + it('refuses an empty credential', () => { + expect(() => new FDv2SkillStore(' ')).toThrow(/server-side SDK key/); + }); + + it('accepts a server-side key', () => { + expect(new FDv2SkillStore(SDK_KEY)).toBeTruthy(); + }); + + it('warns but allows an unrecognised credential shape', () => { + // Private instances and test doubles issue keys without the public prefix. + new FDv2SkillStore('my-private-instance-credential'); + expect(logged(warnSpy)).toContain('server-side SDK key'); + }); + + it('refuses an unknown mode', () => { + expect(() => new FDv2SkillStore(SDK_KEY, { mode: 'mobile' as never })).toThrow(/stream/); + }); +}); + +// ─── The eager re-reconcile ────────────────────────────────────────────────── + +describe('watchSkills', () => { + it('prunes a revoked skill without a restart', async () => { + // AV-1, closed at this layer. The store's change listener drives the + // reconcile, so the file goes away seconds after the delete-object rather + // than at the next process start. + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + endpoint.queuePoll( + events( + ['server-intent', serverIntent('xfer-changes')], + ['delete-object', deleteSkill()], + ['payload-transferred', transferred('basis-2')], + ), + ); + endpoint.queuePoll([], { status: 304 }); + + const store = pollStore({ pollIntervalMs: 100 }); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const root = path.join(await scratchRoot(), 'skills'); + const { report, watcher } = await watchSkills('*', root, { debounceMs: 20 }); + try { + const written = path.join(root, 'pdf-extraction', 'SKILL.md'); + expect(await readFile(written, 'utf8')).toBe(SKILL_BODY); + expect(report.actions.some((a) => a.action === 'written')).toBe(true); + expect(await waitUntil(() => !existsSync(written), 10_000)).toBe(true); + } finally { + await watcher.close(); + } + }); + + it('rewrites a new version without a restart', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill('pdf-extraction', { content: 'first' })]])); + endpoint.queuePoll( + events( + ['server-intent', serverIntent('xfer-full')], + ['put-object', putSkill('pdf-extraction', { objectVersion: 4, content: 'second' })], + ['payload-transferred', transferred('basis-2')], + ), + ); + endpoint.queuePoll([], { status: 304 }); + + const store = pollStore({ pollIntervalMs: 100 }); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const root = path.join(await scratchRoot(), 'skills'); + const { watcher } = await watchSkills('*', root, { debounceMs: 20 }); + try { + const written = path.join(root, 'pdf-extraction', 'SKILL.md'); + expect(await readFile(written, 'utf8')).toBe('first'); + expect(await waitUntil(() => existsSync(written) && readFileSync(written, 'utf8') === 'second', 10_000)).toBe( + true, + ); + } finally { + await watcher.close(); + } + }); + + it('coalesces a burst of changes into few reconciles', async () => { + endpoint.queuePoll( + fullPayload(Array.from({ length: 12 }, (_, i) => ['put-object', putSkill(`skill-${i}`)] as [string, unknown])), + ); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const root = path.join(await scratchRoot(), 'skills'); + const { watcher } = await watchSkills('*', root, { debounceMs: 50 }); + try { + await new Promise((resolve) => setTimeout(resolve, 400)); + // Twelve objects committed in one payload fire twelve listener calls; + // without coalescing that is twelve reconciles of one root. + expect(watcher.reconciles).toBeLessThanOrEqual(2); + } finally { + await watcher.close(); + } + }); + + it('keeps last known good during an outage', async () => { + // `onUnavailable: 'keep'` is the endorsed default: an outage must not read as + // "everything was revoked". + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + endpoint.queuePoll([], { status: 500 }); + const store = pollStore({ pollIntervalMs: 20 }); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const root = path.join(await scratchRoot(), 'skills'); + const { watcher } = await watchSkills('*', root, { debounceMs: 20 }); + try { + const written = path.join(root, 'pdf-extraction', 'SKILL.md'); + expect(await readFile(written, 'utf8')).toBe(SKILL_BODY); + // `lastError` rather than `connectionFailures`: the counter resets on the + // next successful poll, so asserting on it races the retry. + expect(await waitUntil(() => store.diagnostics.lastError !== null, 10_000)).toBe(true); + await new Promise((resolve) => setTimeout(resolve, 200)); + expect(await readFile(written, 'utf8')).toBe(SKILL_BODY); + } finally { + await watcher.close(); + } + }); + + it('refuses a store with no addListener, loudly', async () => { + _setStore({ + getObject: () => null, + allObjects: () => ({}), + }); + await expect(watchSkills('*', await scratchRoot())).rejects.toThrow(/addListener/); + }); + + it('throws when no store is configured', async () => { + await expect(watchSkills('*', await scratchRoot())).rejects.toThrow(/configured skill store/); + }); + + it('can also be driven by the in-memory store', async () => { + // The watcher is wired to the seam, not to the FDv2 store. + const store = new InMemorySkillStore(); + store.put({ key: 'a', version: 1, content: 'body', contentHash: hash('body') }); + _setStore(store); + + const root = path.join(await scratchRoot(), 'skills'); + const { watcher } = await watchSkills('*', root, { debounceMs: 20 }); + try { + const written = path.join(root, 'a', 'SKILL.md'); + expect(await readFile(written, 'utf8')).toBe('body'); + store.put({ key: 'a', version: 2, content: 'new body', contentHash: hash('new body') }); + expect(await waitUntil(() => existsSync(written) && readFileSync(written, 'utf8') === 'new body', 10_000)).toBe( + true, + ); + } finally { + await watcher.close(); + } + }); +}); + +// ─── Lifecycle ─────────────────────────────────────────────────────────────── + +describe('lifecycle', () => { + it('makes start idempotent', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + expect(store.start()).toBe(store); + expect(store.start()).toBe(store); + expect(await store.waitForSkills(5000)).toBe(true); + }); + + it('makes close idempotent', async () => { + const store = pollStore(); + store.start(); + await store.close(); + await store.close(); + }); + + it('keeps answering from what it received after close', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + store.start(); + await store.waitForSkills(5000); + await store.close(); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + }); + + it('times out waitForSkills rather than hanging', async () => { + const store = new FDv2SkillStore(SDK_KEY, { + mode: 'poll', + pollIntervalMs: 60_000, + requester: new ScriptedRequester([new Promise(() => {})]), + }); + openStores.push(store); + expect(await store.waitForSkills(50)).toBe(false); + }); + + it('satisfies the seam before it starts', () => { + const store = new FDv2SkillStore(SDK_KEY); + expect(store.getObject(SKILL_OBJECT_KIND, 'anything')).toBeNull(); + expect(store.allObjects(SKILL_OBJECT_KIND)).toEqual({}); + }); +}); diff --git a/packages/client/src/index.ts b/packages/client/src/index.ts index 7ab0f5ee..4695d0a0 100644 --- a/packages/client/src/index.ts +++ b/packages/client/src/index.ts @@ -21,8 +21,12 @@ export { getClient, initClient, inspectConfig, shutdown, shutdownTelemetry, wait export { compose, globalRegistry, Registry } from './registry.js'; export { allSkills, getSkill, getSkillResult, getSkills, InMemorySkillStore, skillRefs } from './skills.js'; export { MAX_SKILL_CONTENT_BYTES, SKILL_OBJECT_KIND } from './skills-core.js'; +export type { FDv2Mode, FDv2SkillStoreOptions, StoreDiagnostics } from './skills-fdv2.js'; +export { DEFAULT_BASE_URI, FDv2SkillStore, SDK_DATA_MODEL_VERSION } from './skills-fdv2.js'; export type { WriteSkillsOptions } from './skills-fs.js'; export { MANIFEST_FILENAME, MANIFEST_VERSION, SKILL_FILENAME, writeSkills } from './skills-fs.js'; +export type { WatchSkillsOptions } from './skills-watch.js'; +export { DEFAULT_DEBOUNCE_MS, SkillWatcher, watchSkills } from './skills-watch.js'; export type { ConfigArgs, GraphArgs, diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts new file mode 100644 index 00000000..78b2c2d6 --- /dev/null +++ b/packages/client/src/skills-fdv2.ts @@ -0,0 +1,1289 @@ +/** + * Agent Skills — the FDv2 delivery transport. + * + * The store implementation that actually talks to LaunchDarkly. It sits *below* + * the `SkillStore` seam, not above it: it produces raw wire objects in the shape + * `skills-core.ts` documents, and everything above — the accessors, integrity + * verification, the `Skill` type, materialization — is unchanged and unaware of + * it. That is the whole point of the seam, and the fact that replacing the + * transport design wholesale cost nothing above this line is the evidence it was + * drawn in the right place. + * + * Layering: + * + * ``` + * @launchdarkly/ai-server + * └─ SkillStore (types.ts) ── structurally typed accessor surface + * └─ FDv2SkillStore (this file) ── deserialize, hold, serve + * └─ the SDK-facing FDv2 channel on FDCore + * GET /sdk/poll, GET /sdk/stream, authenticated with the + * environment's server-side SDK key + * ``` + * + * Dependencies run one way. This module imports `skills-core.ts` for the seam's + * kind constant and nothing else from the feature; `skills.ts` and `skills-fs.ts` + * do not import it. It uses only platform globals — `fetch`, `AbortController`, + * `TextDecoder` — so the content path adds no dependency. + * + * **There is no bespoke private route here, deliberately.** An earlier design had + * this adapter poll `/private/flagdlv/payloads/{id}/latest/obj/skill/{key}`. + * Those are gonfalon private endpoints authenticated by Cognito machine-token + * OAuth scopes with no per-tenant authorization; the security review ruled out + * both relaxing that auth and shipping a machine credential to a customer host. + * This transport uses the genuinely SDK-facing channel instead, which is also the + * channel payload signing will eventually cover. Do not reintroduce the private + * route. + * + * This is a port of the Python SDK's `skills_fdv2.py` and is deliberately + * mechanical: same protocol boundary, same constants, same wire semantics, same + * diagnostics vocabulary. A change to one belongs in both. + * + * What this module does *not* do, on purpose: + * + * - **It does not verify content.** Verification lives at the accessor boundary + * in `skills-core.ts` so that it applies to every store equally, including + * `InMemorySkillStore` and a customer's own. + * - **It does not skip verification when the wire envelope has no + * `contentHash`.** A hashless object is stored verbatim and *withheld* by + * verification with `missing_content_hash`; this module's job is to make that + * outcome loud rather than to paper over it. + * - **It does not evaluate anything.** No flags, no segments, no targeting. + */ + +import { SKILL_OBJECT_KIND } from './skills-core.js'; +import type { RawSkillObject, SkillStore } from './types.js'; +import { isValidSkillVersion } from './types.js'; + +// --------------------------------------------------------------------------- +// The wire contract +// --------------------------------------------------------------------------- + +/** + * The FDv2 `kind` skills are delivered under. + * + * Distinct from `SKILL_OBJECT_KIND` (`'skill'`), which is the *seam* value the + * SDK asks a store for. Translating this pair — kind `inline-resource` plus + * category `skill` — onto that single value is exactly the adapter's job. + */ +export const FDV2_OBJECT_KIND = 'inline-resource'; + +/** The `category` that narrows `inline-resource` to an agent skill. */ +export const FDV2_OBJECT_CATEGORY = 'skill'; + +/** + * Where the SDK-facing FDv2 endpoints live. Overridable for Federal instances, + * private instances, and the fake endpoint the tests run against. + */ +export const DEFAULT_BASE_URI = 'https://sdk.launchdarkly.com'; + +export const POLL_PATH = '/sdk/poll'; +export const STREAM_PATH = '/sdk/stream'; + +/** + * The `mv` request parameter — the SDK data model version this adapter speaks. + * + * Overridable through `new FDv2SkillStore(key, { dataModelVersion })` because it + * is the one request parameter this side cannot verify: the LaunchDarkly base + * SDK's own FDv2 data source does not send `mv` at all today, and the streamer + * branch that carries skills is unmerged, so the value the server expects has not + * been observed. Confirm it with FDN before Beta rather than trusting this + * default. + */ +export const SDK_DATA_MODEL_VERSION = 1; + +const EVENT_SERVER_INTENT = 'server-intent'; +const EVENT_PUT_OBJECT = 'put-object'; +const EVENT_DELETE_OBJECT = 'delete-object'; +const EVENT_PAYLOAD_TRANSFERRED = 'payload-transferred'; +const EVENT_HEARTBEAT = 'heart-beat'; +const EVENT_GOODBYE = 'goodbye'; +const EVENT_ERROR = 'error'; + +const INTENT_TRANSFER_FULL = 'xfer-full'; +const INTENT_TRANSFER_CHANGES = 'xfer-changes'; +const INTENT_TRANSFER_NONE = 'none'; + +/** + * The skill object envelope's fields, copied through verbatim. + * + * `contentHash` is listed here and is the field the whole content path waits on. + * Nothing here is coerced, defaulted, or normalized — everything a store serves + * is untrusted input and is revalidated above the seam, so a transport that + * "helpfully" filled in a field would be forging the very thing verification + * exists to check. + */ +const ENVELOPE_FIELDS = ['contentType', 'content', 'contentHash', 'name', 'description'] as const; + +export type FDv2Mode = 'stream' | 'poll'; + +const MOBILE_KEY_PREFIX = 'mob-'; +const SERVER_KEY_PREFIX = 'sdk-'; + +/** + * A client-side environment ID: bare lowercase hex, no prefix. Server-side keys + * and mobile keys both carry a prefix, so "hex with no prefix" is an unambiguous + * client-side credential rather than a heuristic. + */ +const CLIENT_SIDE_ID = /^[0-9a-f]{20,}$/; + +function warn(message: string): void { + // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; delivery problems must be visible + console.warn(`[LaunchDarkly] ${message}`); +} + +function error(message: string): void { + // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; delivery problems must be visible + console.error(`[LaunchDarkly] ${message}`); +} + +// --------------------------------------------------------------------------- +// Server-side only +// --------------------------------------------------------------------------- + +/** + * Refuses a mobile key or a client-side environment ID. + * + * Skills are for server-side agent runtimes. The payload assignment that carries + * them is shared by every auth type, so the skill payload ID is appended for + * mobile and environment-ID auth too — which means a client-side credential may + * well *succeed* against these endpoints and deliver customer-confidential skill + * content to a client-side process. Throwing here is the SDK-side half of that + * boundary; excluding skills at assignment time is the platform-side half, and is + * an open ask on FDN. + * + * Throws rather than warning, because there is no degraded mode that is correct: + * a store built on the wrong credential should not exist. + */ +export function requireServerSideCredential(sdkKey: unknown): string { + if (typeof sdkKey !== 'string' || sdkKey.trim() === '') { + throw new Error('FDv2SkillStore requires a LaunchDarkly server-side SDK key (sdk-...); none was given.'); + } + const key = sdkKey.trim(); + if (key.startsWith(MOBILE_KEY_PREFIX)) { + throw new Error( + 'FDv2SkillStore was given a mobile key (mob-...). Agent Skills are a server-side feature: skill content is ' + + "customer-confidential and is never delivered to a mobile or client-side process. Use the environment's " + + 'server-side SDK key (sdk-...).', + ); + } + if (CLIENT_SIDE_ID.test(key)) { + throw new Error( + 'FDv2SkillStore was given what looks like a client-side environment ID. Agent Skills are a server-side ' + + 'feature: skill content is customer-confidential and is never delivered to a client-side process. Use the ' + + "environment's server-side SDK key (sdk-...).", + ); + } + if (!key.startsWith(SERVER_KEY_PREFIX)) { + // Not rejected: private instances and test doubles issue credentials that do + // not carry the public prefix, and refusing them would break a deployment + // that is perfectly correct. The two shapes above are refused because they + // are unambiguously *not* server-side. + warn( + 'The credential given to FDv2SkillStore does not look like a LaunchDarkly server-side SDK key (sdk-...). ' + + 'Skills are delivered only to server-side credentials; if this is a client-side or mobile credential the ' + + 'connection will be rejected or will deliver nothing.', + ); + } + return key; +} + +// --------------------------------------------------------------------------- +// Diagnostics — and the contentHash gap in particular +// --------------------------------------------------------------------------- + +/** + * What the transport has seen. Read-only from a caller's perspective. + * + * Not part of the `SkillStore` seam — nothing above the seam reads this — but the + * difference between "this environment has no skills" and "every skill was + * withheld" is the single most confusing failure this feature can produce, and a + * counter a caller can assert on beats reading logs. + */ +export type StoreDiagnostics = { + /** Completed `payload-transferred` commits since the store started. */ + readonly payloadsTransferred: number; + /** `put-object` events identified as skills, across all payloads. */ + readonly skillObjectsReceived: number; + /** + * Objects skipped because they were not skills — flags, segments, and any + * future kind. Skipping is the contract, not a failure; the count exists so a + * mixed payload is visibly mixed. + */ + readonly objectsIgnored: number; + /** `delete-object` events applied to skills. */ + readonly objectsRevoked: number; + /** + * Skill objects whose envelope carried no `contentHash`. + * + * **Nonzero means skills are being withheld.** Verification withholds a + * hashless object with `missing_content_hash`, so every one of these is a skill + * that will never resolve. The field exists so that outcome is a number a + * caller can read rather than an empty store they have to explain. + */ + readonly hashlessObjects: number; + /** Recoverable transport failures since the last successful transfer. */ + readonly connectionFailures: number; + /** The most recent transport error, if any. Human-readable; do not parse. */ + readonly lastError: string | null; +}; + +const HASHLESS_ADVICE = + "The delivered skill object carries no 'contentHash', so integrity verification withholds it with reason_code " + + "'missing_content_hash' and its content will never resolve. This is not a fault in this store and not something " + + 'the SDK can work around: verification hashes the verbatim bytes and compares, and there is nothing to compare ' + + 'against. The field is specified as an additive sha256-over-verbatim-UTF-8 value on the skill envelope ' + + '(LaunchDarkly AIC-2905) and has not shipped yet. Until it does, expect an empty result from every skill accessor.'; + +/** + * `(key, version)` pairs already reported hashless. + * + * Module-scoped so the error is one per object per process rather than one per + * re-delivered payload; exported for the tests, which need to clear it. + */ +export const _warnedHashless = new Set(); + +/** + * One error per `(key, version)` whose envelope had no `contentHash`. + * + * At error level rather than warn, and per object rather than once per process, + * because this is the difference between a broken deployment and an + * empty-by-design one — the exact confusion the blocking gap produces. + */ +function warnHashless(raw: RawSkillObject): void { + const identity = `${String(raw.key)}:${String(raw.version)}`; + if (_warnedHashless.has(identity)) return; + _warnedHashless.add(identity); + error( + `Skill '${String(raw.key)}' version ${String(raw.version)} arrived without a contentHash and will be ` + + `withheld. ${HASHLESS_ADVICE}`, + ); +} + +/** + * One error per committed payload in which *nothing* the store now holds can + * possibly verify. + * + * Fires at delivery time, so the condition is visible in a process that boots, + * materializes nothing, and exits — which is the shape a skills deployment fails + * in. The accessor boundary's own withholding summary only speaks once a caller + * asks. + */ +function warnIfNothingCanVerify(held: RawSkillObject[]): void { + if (held.length === 0) return; + const hashless = held.filter((raw) => typeof raw.contentHash !== 'string'); + if (hashless.length !== held.length) return; + error( + `All ${held.length} skill object(s) in the delivered payload arrived without a contentHash. No skill content ` + + `will resolve from this store. ${HASHLESS_ADVICE}`, + ); +} + +// --------------------------------------------------------------------------- +// Deserialization — where objectVersion is not version +// --------------------------------------------------------------------------- + +/** A `delete-object` narrowed to the identity it revokes. */ +export type Tombstone = { readonly key: string; readonly objectVersion: number | null }; + +/** + * Whether one `put-object` / `delete-object` payload is a skill. + * + * `kind === 'inline-resource' && category === 'skill'`, and nothing else. Both + * halves are required: `inline-resource` is a broad kind that may carry other + * categories, and flags and segments omit `category` entirely. + * + * Every other kind is **ignored, not rejected**. An environment's payload + * assignment carries the flagging payload alongside the agent-skill payload, so a + * connection delivers flag and segment objects as a matter of course. Throwing on + * them would turn a normal payload into a permanent failure — which is exactly + * the unknown-kind reconnect loop this feature must not reproduce. + */ +export function isSkillEvent(data: unknown): boolean { + if (typeof data !== 'object' || data === null || Array.isArray(data)) return false; + const candidate = data as { kind?: unknown; category?: unknown }; + return candidate.kind === FDV2_OBJECT_KIND && candidate.category === FDV2_OBJECT_CATEGORY; +} + +/** + * Translates one FDv2 skill `put-object` into a seam-shaped raw object. + * + * `null` when the event cannot be filed at all — only when `key` is not a string, + * since a keyless object has no identity to store it under and no key to + * attribute a failure to. Every other defect is carried through verbatim so that + * *verification* withholds it, with a reason code and an integrity signal, rather + * than the transport dropping it silently. A silent drop is indistinguishable + * from "no such skill" and would additionally let a prune delete the last + * known-good copy on disk. + * + * **The translation this whole module exists to get right:** + * + * ``` + * wire `objectVersion` → seam `version` (the skill's own version) + * wire `version` → dropped (the *payload* version) + * ``` + * + * `objectVersion` is what a `{key, version}` reference pins. `version` is the + * version of the payload the object arrived in — it changes when anything in the + * environment changes, including a flag that has nothing to do with skills. + * Reading it as the skill's version resolves the wrong content with no error + * anywhere: the object verifies, the hash matches, and the caller is handed a + * skill under a version number that means nothing. Flags and segments carry only + * `version`, which is why the two fields look interchangeable and are not. + */ +export function seamObjectFromPut(data: Record): RawSkillObject | null { + const { key } = data as { key?: unknown }; + if (typeof key !== 'string' || key === '') { + warn( + "An FDv2 skill put-object carried no string 'key' and could not be stored under any identity; it was dropped.", + ); + return null; + } + + const raw: RawSkillObject = { key }; + + // The single translation. Written as a property-presence test rather than a + // defaulted read so an explicitly-null objectVersion stays null and reaches + // verification as `invalid_version`, instead of being invented here. + if ('objectVersion' in data) raw.version = data.objectVersion; + + const envelope = data.object; + if (typeof envelope === 'object' && envelope !== null && !Array.isArray(envelope)) { + for (const field of ENVELOPE_FIELDS) { + if (field in envelope) raw[field] = (envelope as Record)[field]; + } + } + return raw; +} + +/** + * Narrows one FDv2 skill `delete-object` to the identity it revokes. + * + * A delete for an inline resource **is revocation** — the object leaves the + * payload, this store drops it, the accessors stop resolving it, and the next + * reconcile prunes its files. Same `objectVersion` translation as a put. + * + * An `objectVersion` of `null` means the delete named no usable version, and is + * read as "revoke every version of this key". That is the safe direction: the + * alternative is ignoring an unparseable revocation and continuing to serve + * content LaunchDarkly has withdrawn. + */ +export function tombstoneFromDelete(data: Record): Tombstone | null { + const { key } = data as { key?: unknown }; + if (typeof key !== 'string' || key === '') { + warn("An FDv2 skill delete-object carried no string 'key'; it was ignored."); + return null; + } + const objectVersion = data.objectVersion; + return { key, objectVersion: isValidSkillVersion(objectVersion) ? objectVersion : null }; +} + +// --------------------------------------------------------------------------- +// The held object set +// --------------------------------------------------------------------------- + +/** + * Raw skill objects held in memory, keyed by `(key, objectVersion)`. + * + * Several versions of one key coexist, because they coexist in a real payload: + * the newest version of every skill plus every version a variation currently + * pins. An object too malformed to carry a usable version is still held, under + * its key alone, so verification withholds it with a signal rather than the + * transport dropping it into indistinguishable absence. + * + * **`snapshot` collapses to one object per key at its newest version, and that is + * load-bearing here in a way it is not in Python.** `//SKILL.md` is a + * single path, so a whole-store consumer must see one object per key or a `'*'` + * reconcile writes the same path twice in one run and `allSkills` returns a list + * holding two versions of one skill. The Python SDK collapses in + * `newest_by_key`, above the seam; that helper has not been ported to this SDK + * yet, so the collapse happens here instead. The observable end-to-end behaviour + * is identical, and `getObject` still resolves a pinned version out of the full + * set — which is the case the collapse must not break. When `newestByKey` lands + * in `skills-core.ts`, move it and delete this note. + */ +export class SkillObjectSet { + private versions = new Map>(); + private loose = new Map(); + + put(raw: RawSkillObject): void { + const key = raw.key as string; + const { version } = raw; + if (isValidSkillVersion(version)) { + const held = this.versions.get(key) ?? new Map(); + held.set(version, raw); + this.versions.set(key, held); + } else { + this.loose.set(key, raw); + } + } + + /** + * Removes what `tombstone` revokes; returns the raw objects that went away. + * + * A tombstone with no usable version removes every version of the key — see + * {@link tombstoneFromDelete} for why that is the safe reading. + */ + delete(tombstone: Tombstone): RawSkillObject[] { + const removed: RawSkillObject[] = []; + if (tombstone.objectVersion === null) { + for (const raw of this.versions.get(tombstone.key)?.values() ?? []) removed.push(raw); + this.versions.delete(tombstone.key); + const loose = this.loose.get(tombstone.key); + if (loose) { + removed.push(loose); + this.loose.delete(tombstone.key); + } + return removed; + } + + const held = this.versions.get(tombstone.key); + const gone = held?.get(tombstone.objectVersion); + if (gone) { + removed.push(gone); + held?.delete(tombstone.objectVersion); + } + if (held && held.size === 0) this.versions.delete(tombstone.key); + return removed; + } + + /** + * The object for `key` at `version`, or the newest held when `version` is null. + * + * Falls through to the version-less entry when the pin matches nothing + * well-formed, so a malformed object reaches verification and is withheld with a + * signal rather than reading as simply absent. + */ + get(key: string, version: number | null): RawSkillObject | null { + const held = this.versions.get(key); + if (version !== null) return held?.get(version) ?? this.loose.get(key) ?? null; + if (held && held.size > 0) { + const newest = Math.max(...held.keys()); + return held.get(newest) ?? null; + } + return this.loose.get(key) ?? null; + } + + /** One entry per skill key, at its newest version. See the class docstring. */ + snapshot(): Record { + const out: Record = {}; + for (const [key, held] of this.versions) { + if (held.size === 0) continue; + const newest = Math.max(...held.keys()); + const raw = held.get(newest); + if (raw) out[`${key}:${newest}`] = raw; + } + for (const [key, raw] of this.loose) { + if (!this.versions.has(key)) out[key] = raw; + } + return out; + } + + /** Every object held, one per `(key, version)`. Diagnostics, not the seam. */ + allRaw(): RawSkillObject[] { + const out: RawSkillObject[] = []; + for (const held of this.versions.values()) out.push(...held.values()); + out.push(...this.loose.values()); + return out; + } + + /** Adopts `other`'s contents wholesale — how a full transfer commits. */ + replaceWith(other: SkillObjectSet): void { + this.versions = other.versions; + this.loose = other.loose; + } + + copy(): SkillObjectSet { + const clone = new SkillObjectSet(); + clone.versions = new Map([...this.versions].map(([key, held]) => [key, new Map(held)])); + clone.loose = new Map(this.loose); + return clone; + } + + get size(): number { + let total = this.loose.size; + for (const held of this.versions.values()) total += held.size; + return total; + } +} + +// --------------------------------------------------------------------------- +// The protocol state machine — pure, no I/O +// --------------------------------------------------------------------------- + +/** What one event did. Aggregated by the caller; nothing here does I/O. */ +export type TransferOutcome = { + committed?: boolean; + changes?: RawSkillObject[]; + basis?: string | null; + fatal?: string | null; + disconnect?: string | null; +}; + +type MutableDiagnostics = { -readonly [K in keyof StoreDiagnostics]: StoreDiagnostics[K] }; + +function freshDiagnostics(): MutableDiagnostics { + return { + payloadsTransferred: 0, + skillObjectsReceived: 0, + objectsIgnored: 0, + objectsRevoked: 0, + hashlessObjects: 0, + connectionFailures: 0, + lastError: null, + }; +} + +/** + * Applies FDv2 events to an object set. Pure — no sockets, no timers, no clock. + * + * Split out so the protocol is testable without a server: every wire case in + * `skills-fdv2.test.ts` drives this directly, and the HTTP layer above it only has + * to turn bytes into `[event name, data]` pairs. + * + * **Changes are buffered and committed at `payload-transferred`**, matching how + * the base SDK's FDv2 data source applies a change set. A payload version is the + * unit of consistency: applying half of one would publish a state the server + * never described, and on a full transfer it would briefly empty the store — + * which, with pruning on, is the difference between a reconcile and deleting a + * customer's skill files. Listeners therefore fire once per commit, not once per + * object, which is also exactly the granularity the re-reconcile wants. + */ +export class ProtocolReader { + readonly diagnostics = freshDiagnostics(); + private intent: string | null = null; + private pending: SkillObjectSet | null = null; + private changes: RawSkillObject[] = []; + + constructor(private readonly committed: SkillObjectSet) {} + + /** Routes one event. Unknown event names are ignored, by contract. */ + handle(name: string, data: unknown): TransferOutcome { + switch (name) { + case EVENT_SERVER_INTENT: + return this.serverIntent(data); + case EVENT_PUT_OBJECT: + return this.putObject(data); + case EVENT_DELETE_OBJECT: + return this.deleteObject(data); + case EVENT_PAYLOAD_TRANSFERRED: + return this.payloadTransferred(data); + case EVENT_ERROR: + return this.error(data); + case EVENT_GOODBYE: + return this.goodbye(data); + case EVENT_HEARTBEAT: + return {}; + default: + return {}; + } + } + + private serverIntent(data: unknown): TransferOutcome { + const payloads = (data as { payloads?: unknown } | null)?.payloads; + if (!Array.isArray(payloads) || payloads.length === 0) { + return { disconnect: 'server-intent carried no payload description' }; + } + const first = payloads[0] as { intentCode?: unknown } | null; + const intent = typeof first?.intentCode === 'string' ? first.intentCode : null; + this.intent = intent; + this.changes = []; + if (intent === INTENT_TRANSFER_FULL) { + // A fresh set: the payload about to arrive replaces everything held. Built + // alongside the live set rather than in place, so an interrupted transfer + // leaves last-known-good intact. + this.pending = new SkillObjectSet(); + } else if (intent === INTENT_TRANSFER_CHANGES) { + this.pending = this.committed.copy(); + } else if (intent === INTENT_TRANSFER_NONE) { + // The payload we hold is current. Nothing to apply, nothing to replace. + this.pending = null; + } else { + // Any future intent code. Ignored rather than guessed at, for the same + // reason an unknown kind is: guessing could empty the store. + this.pending = null; + } + return {}; + } + + private target(): SkillObjectSet | null { + if (this.pending === null && (this.intent === INTENT_TRANSFER_FULL || this.intent === INTENT_TRANSFER_CHANGES)) { + // An object arrived before any server-intent. Treat it as a delta against + // what we hold rather than dropping it. + this.pending = this.committed.copy(); + } + return this.pending; + } + + private putObject(data: unknown): TransferOutcome { + if (!isSkillEvent(data)) { + this.diagnostics.objectsIgnored += 1; + return {}; + } + if (this.pending === null && this.intent === null) this.intent = INTENT_TRANSFER_CHANGES; + const target = this.target(); + if (target === null) return {}; + + const raw = seamObjectFromPut(data as Record); + if (raw === null) return {}; + target.put(raw); + this.changes.push(raw); + this.diagnostics.skillObjectsReceived += 1; + if (typeof raw.contentHash !== 'string') { + this.diagnostics.hashlessObjects += 1; + warnHashless(raw); + } + return {}; + } + + private deleteObject(data: unknown): TransferOutcome { + if (!isSkillEvent(data)) { + this.diagnostics.objectsIgnored += 1; + return {}; + } + if (this.pending === null && this.intent === null) this.intent = INTENT_TRANSFER_CHANGES; + const target = this.target(); + if (target === null) return {}; + + const tombstone = tombstoneFromDelete(data as Record); + if (tombstone === null) return {}; + target.delete(tombstone); + this.diagnostics.objectsRevoked += 1; + // A tombstone, not a skill object: it carries identity and no content, so a + // listener that only needs "something changed" works unchanged while one that + // reads content sees no `content` field. Documented on `addListener`. + this.changes.push({ key: tombstone.key, version: tombstone.objectVersion }); + return {}; + } + + private payloadTransferred(data: unknown): TransferOutcome { + const state = (data as { state?: unknown } | null)?.state; + if (this.pending !== null) { + this.committed.replaceWith(this.pending); + warnIfNothingCanVerify(this.committed.allRaw()); + } + this.pending = null; + this.intent = null; + const { changes } = this; + this.changes = []; + this.diagnostics.payloadsTransferred += 1; + return { + committed: true, + changes, + basis: typeof state === 'string' && state !== '' ? state : null, + }; + } + + private error(data: unknown): TransferOutcome { + const reason = (data as { reason?: unknown } | null)?.reason; + // An error abandons the in-flight payload and keeps what is committed. + this.pending = null; + this.intent = null; + this.changes = []; + return { disconnect: `server sent error: ${String(reason)}` }; + } + + private goodbye(data: unknown): TransferOutcome { + const parsed = (data ?? {}) as { reason?: unknown; silent?: unknown; catastrophe?: unknown }; + this.pending = null; + this.intent = null; + this.changes = []; + if (parsed.silent !== true) { + warn(`FDv2 connection closing: ${String(parsed.reason)}`); + } + if (parsed.catastrophe === true) { + return { fatal: `server sent a catastrophic goodbye: ${String(parsed.reason)}` }; + } + return { disconnect: `server said goodbye: ${String(parsed.reason)}` }; + } +} + +// --------------------------------------------------------------------------- +// HTTP +// --------------------------------------------------------------------------- + +/** A failure retrying cannot fix: bad credential, forbidden, wrong URI. */ +export class FatalTransportError extends Error {} + +/** A failure worth retrying. Carries a server-requested delay when given one. */ +export class RecoverableTransportError extends Error { + constructor( + message: string, + readonly retryAfterMs: number | null = null, + ) { + super(message); + } +} + +const FORBIDDEN_ADVICE = + "FDv2 is opt-in per account: the 'fdv2-protocol-control' setting defaults to 'forbid', which is served as " + + 'HTTP 403. Skill delivery over this channel needs that flag flipped for the account, and needs the ' + + 'FDCore/streamer inline-resource support merged and deployed.'; + +/** `Retry-After` in milliseconds, when the server sent a usable one. */ +export function retryAfterMs(headers: Headers | null | undefined): number | null { + const raw = headers?.get('Retry-After'); + if (raw === null || raw === undefined) return null; + const seconds = Number(raw.trim()); + // The HTTP-date form is legal and rare; falling back to our own backoff is + // better than parsing a date to honour it approximately. + if (!Number.isFinite(seconds)) return null; + return Math.max(0, seconds * 1000); +} + +/** Turns an HTTP error status into the right error type. */ +export function classifyStatus(status: number, headers?: Headers | null): Error { + if (status === 401) { + return new FatalTransportError( + 'LaunchDarkly rejected the SDK key (HTTP 401). Skill delivery cannot start. Check that the key is the ' + + "environment's server-side SDK key.", + ); + } + if (status === 403) return new FatalTransportError(`LaunchDarkly returned HTTP 403. ${FORBIDDEN_ADVICE}`); + if (status === 404) { + return new FatalTransportError( + 'LaunchDarkly returned HTTP 404 for the FDv2 endpoint. Check the base URI, and that this instance serves ' + + '/sdk/poll and /sdk/stream.', + ); + } + if ([400, 405, 406, 414, 501].includes(status)) { + return new FatalTransportError( + `LaunchDarkly returned HTTP ${status}, which retrying will not fix. The request this adapter sent was not ` + + `understood; the 'mv' data model version (${SDK_DATA_MODEL_VERSION}) is the parameter most likely to be wrong.`, + ); + } + return new RecoverableTransportError(`LaunchDarkly returned HTTP ${status}`, retryAfterMs(headers)); +} + +export type PollResult = { + readonly notModified: boolean; + readonly events: Array<[string, unknown]>; + readonly etag: string | null; +}; + +/** + * Unwraps `{"events": [...]}`. + * + * Polling and streaming carry the *identical* event objects — polling just wraps + * them in an envelope — which is why the protocol state machine above is shared + * and neither mode has its own copy of the semantics. + */ +export function decodePollBody(body: string): Array<[string, unknown]> { + let parsed: unknown; + try { + parsed = JSON.parse(body); + } catch (cause) { + throw new RecoverableTransportError( + `polling response was not valid JSON: ${cause instanceof Error ? cause.message : String(cause)}`, + ); + } + const wrapped = parsed as { events?: unknown } | null; + if (typeof wrapped !== 'object' || wrapped === null || !Array.isArray(wrapped.events)) { + throw new RecoverableTransportError("polling response had no 'events' array"); + } + const events: Array<[string, unknown]> = []; + for (const entry of wrapped.events) { + if (typeof entry !== 'object' || entry === null) continue; + const { event, data } = entry as { event?: unknown; data?: unknown }; + if (typeof event === 'string') events.push([event, data]); + } + return events; +} + +/** + * Decodes an SSE byte stream into `[event name, data]` pairs. + * + * Minimal on purpose — this consumes one LaunchDarkly endpoint, not the whole + * spec: `event:`/`data:` fields, multi-line `data` joined with newlines, a blank + * line dispatching, and `:` comments skipped. + */ +export async function* iterSse(body: ReadableStream): AsyncGenerator<[string, unknown], void, undefined> { + const reader = body.getReader(); + const decoder = new TextDecoder(); + let buffer = ''; + let name: string | null = null; + let dataLines: string[] = []; + + const dispatch = (): [string, unknown] | null => { + if (name === null) return null; + const payload = dataLines.join('\n'); + const eventName = name; + name = null; + dataLines = []; + if (payload === '') return [eventName, null]; + try { + return [eventName, JSON.parse(payload)]; + } catch { + warn(`Discarding FDv2 '${eventName}' event whose data was not JSON`); + return null; + } + }; + + try { + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + buffer += decoder.decode(value, { stream: true }); + let newline = buffer.indexOf('\n'); + while (newline !== -1) { + const line = buffer.slice(0, newline).replace(/\r$/, ''); + buffer = buffer.slice(newline + 1); + if (line === '') { + const dispatched = dispatch(); + if (dispatched !== null) yield dispatched; + } else if (!line.startsWith(':')) { + const colon = line.indexOf(':'); + const field = colon === -1 ? line : line.slice(0, colon); + let value2 = colon === -1 ? '' : line.slice(colon + 1); + if (value2.startsWith(' ')) value2 = value2.slice(1); + if (field === 'event') name = value2; + else if (field === 'data') dataLines.push(value2); + } + newline = buffer.indexOf('\n'); + } + } + } finally { + try { + reader.releaseLock(); + } catch { + // Already released, or the stream errored. Nothing to recover. + } + } +} + +/** What the store needs from a transport. Swapped wholesale in tests. */ +export type Requester = { + poll(basis: string | null, etag: string | null, signal: AbortSignal): Promise; + stream(basis: string | null, signal: AbortSignal): Promise>; +}; + +/** + * The only place this module opens a connection. + * + * Platform globals only, on purpose: this package's runtime dependencies are + * `@opentelemetry/api` and `dotenv`, and its LaunchDarkly base-SDK dependency is + * an optional peer, so the content path must not smuggle in an HTTP client. + */ +export class FetchRequester implements Requester { + private readonly baseUri: string; + + constructor( + private readonly sdkKey: string, + baseUri: string, + private readonly dataModelVersion: number, + ) { + this.baseUri = baseUri.replace(/\/+$/, ''); + } + + private url(path: string, basis: string | null): string { + const params = new URLSearchParams({ mv: String(this.dataModelVersion) }); + if (basis) params.set('basis', basis); + return `${this.baseUri}${path}?${params.toString()}`; + } + + /** + * One `GET /sdk/poll`. Honours `If-None-Match` and returns 304 as a + * first-class outcome rather than as an error. + */ + async poll(basis: string | null, etag: string | null, signal: AbortSignal): Promise { + const headers: Record = { Authorization: this.sdkKey, Accept: 'application/json' }; + if (etag) headers['If-None-Match'] = etag; + + let response: Response; + try { + response = await fetch(this.url(POLL_PATH, basis), { headers, signal }); + } catch (cause) { + if (signal.aborted) throw cause; + throw new RecoverableTransportError( + `polling request failed: ${cause instanceof Error ? cause.message : String(cause)}`, + ); + } + + if (response.status === 304) return { notModified: true, events: [], etag }; + if (!response.ok) throw classifyStatus(response.status, response.headers); + return { + notModified: false, + events: decodePollBody(await response.text()), + etag: response.headers.get('ETag') ?? etag, + }; + } + + /** Opens `GET /sdk/stream` and yields `[event name, data]` pairs. */ + async stream(basis: string | null, signal: AbortSignal): Promise> { + const headers = { + Authorization: this.sdkKey, + Accept: 'text/event-stream', + 'Cache-Control': 'no-cache', + }; + + let response: Response; + try { + response = await fetch(this.url(STREAM_PATH, basis), { headers, signal }); + } catch (cause) { + if (signal.aborted) throw cause; + throw new RecoverableTransportError( + `streaming request failed: ${cause instanceof Error ? cause.message : String(cause)}`, + ); + } + + if (!response.ok) throw classifyStatus(response.status, response.headers); + if (response.body === null) throw new RecoverableTransportError('the FDv2 stream carried no body'); + return iterSse(response.body); + } +} + +// --------------------------------------------------------------------------- +// Backoff +// --------------------------------------------------------------------------- + +/** + * Exponential backoff with decorrelating jitter, capped at `maximumMs`. + * + * Jitter is subtractive over the *whole* range rather than added on top, so the + * cap is a real ceiling: a fleet of agent processes restarted together must not + * reconnect in lockstep, and must not exceed the interval the cap promises. + */ +export function backoffDelayMs(attempt: number, baseMs: number, maximumMs: number, jitter = 0.5): number { + const ceiling = Math.min(maximumMs, baseMs * 2 ** Math.max(0, attempt - 1)); + return ceiling * (1 - jitter * Math.random()); +} + +function sleep(ms: number, signal: AbortSignal): Promise { + return new Promise((resolve) => { + if (signal.aborted) { + resolve(); + return; + } + const timer = setTimeout(() => { + signal.removeEventListener('abort', onAbort); + resolve(); + }, ms); + // Unreffed so a pending backoff never holds the process open: the store is a + // background concern, not a reason for `node` to keep running. + (timer as unknown as { unref?: () => void }).unref?.(); + const onAbort = (): void => { + clearTimeout(timer); + resolve(); + }; + signal.addEventListener('abort', onAbort, { once: true }); + }); +} + +// --------------------------------------------------------------------------- +// The store +// --------------------------------------------------------------------------- + +export type FDv2SkillStoreOptions = { + /** + * `'stream'` by default. Prefer it: a `delete-object` reaches a live stream in + * seconds, which is what makes revocation seconds-latent instead of + * restart-latent, and is why the change-listener re-reconcile is worth wiring + * at all. `'poll'` exists for environments that cannot hold a long-lived + * connection, and revocation there is one `pollIntervalMs` late. + */ + readonly mode?: FDv2Mode; + readonly baseUri?: string; + readonly pollIntervalMs?: number; + readonly initialBackoffMs?: number; + readonly maxBackoffMs?: number; + /** + * Bounds the retry loop. On exceeding it the transport stops, logs an error, + * and the store keeps serving last known good rather than pretending to be + * live — `failed` reports it. + */ + readonly maxConsecutiveFailures?: number; + readonly dataModelVersion?: number; + /** Test seam: a transport double in place of `FetchRequester`. */ + readonly requester?: Requester; +}; + +/** + * A `SkillStore` fed by LaunchDarkly's SDK-facing FDv2 delivery channel. + * + * The transport half of Agent Skills. Constructed with the environment's + * server-side SDK key, started explicitly, and passed to `initClient`: + * + * ```ts + * const store = new FDv2SkillStore(process.env.LD_SDK_KEY!).start(); + * await store.waitForSkills(10_000); + * await initClient({ skillStore: store }); + * + * const skill = await getSkill('pdf-extraction'); + * // ... + * await store.close(); + * ``` + * + * **Server-side only.** Skills are for server-side agent runtimes and skill + * content is customer-confidential. A mobile key or a client-side environment ID + * throws from the constructor. + * + * **Delivery is in the background; retrieval is not.** `SkillStore` is a + * synchronous seam, so a background task owns the connection and fills memory, + * and `getObject` only ever reads what has already arrived. Nothing here blocks a + * retrieval on the network. The corollary is that a process which calls + * `getSkill` immediately after `start()` may see an empty store; `waitForSkills` + * is how you order boot against the first payload. + * + * **Last known good survives an outage.** A transport failure never empties the + * store and never makes `getObject` throw: it keeps serving what it last + * received, which is what makes `writeSkills({ onUnavailable: 'keep' })` correct. + * `diagnostics` and `failed` report the degradation. + * + * **What arrives is untrusted.** This store holds raw wire objects verbatim and + * verifies nothing — integrity verification lives at the accessor boundary so it + * applies to every store equally. In particular an object with no `contentHash` + * is held and then *withheld* by verification; see + * {@link StoreDiagnostics.hashlessObjects}. + */ +export class FDv2SkillStore implements SkillStore { + private readonly objects = new SkillObjectSet(); + private readonly reader = new ProtocolReader(this.objects); + private readonly listeners = new Map unknown>>(); + private readonly requester: Requester; + private readonly mode: FDv2Mode; + private readonly pollIntervalMs: number; + private readonly initialBackoffMs: number; + private readonly maxBackoffMs: number; + private readonly maxConsecutiveFailures: number; + + private basis: string | null = null; + private etag: string | null = null; + private controller: AbortController | null = null; + private loop: Promise | null = null; + private failedReason: string | null = null; + private firstPayload = false; + private readonly firstPayloadWaiters: Array<() => void> = []; + + constructor(sdkKey: string, options: FDv2SkillStoreOptions = {}) { + const key = requireServerSideCredential(sdkKey); + this.mode = options.mode ?? 'stream'; + if (this.mode !== 'stream' && this.mode !== 'poll') { + throw new Error(`mode must be 'stream' or 'poll', got ${JSON.stringify(options.mode)}`); + } + this.pollIntervalMs = options.pollIntervalMs ?? 30_000; + if (this.pollIntervalMs <= 0) { + throw new Error(`pollIntervalMs must be positive, got ${JSON.stringify(options.pollIntervalMs)}`); + } + this.initialBackoffMs = options.initialBackoffMs ?? 1_000; + this.maxBackoffMs = options.maxBackoffMs ?? 30_000; + this.maxConsecutiveFailures = options.maxConsecutiveFailures ?? 10; + this.requester = + options.requester ?? + new FetchRequester(key, options.baseUri ?? DEFAULT_BASE_URI, options.dataModelVersion ?? SDK_DATA_MODEL_VERSION); + } + + // -- lifecycle --------------------------------------------------------- + + /** + * Starts delivery. Idempotent; returns `this` so it chains. + * + * Does not await: use `waitForSkills` when boot ordering matters. + */ + start(): this { + if (this.loop !== null) return this; + this.controller = new AbortController(); + this.loop = this.run(this.controller.signal); + return this; + } + + /** + * Stops delivery. Idempotent, and safe to call twice. + * + * Held content is *not* dropped: a closed store still answers from what it + * received, so shutting the transport down does not turn into an integrity + * failure or an empty reconcile mid-flight. `shutdown()` is what detaches the + * store from the accessors. + * + * Aborting the signal is what interrupts an open stream: the delivery task + * spends its life awaiting a read, and a flag it never checks would leave a + * healthy stream running until the process exited. + */ + async close(): Promise { + this.controller?.abort(); + const loop = this.loop; + this.loop = null; + this.releaseWaiters(); + if (loop) await loop; + } + + /** + * Resolves once the first payload has been committed, or after `timeoutMs`. + * + * `true` means a payload arrived — not that any skill in it verified, and not + * that the environment has any skills. Boot ordering is all this answers; + * `diagnostics` answers the rest. + */ + waitForSkills(timeoutMs = 10_000): Promise { + if (this.firstPayload) return Promise.resolve(true); + return new Promise((resolve) => { + const timer = setTimeout(() => resolve(this.firstPayload), timeoutMs); + (timer as unknown as { unref?: () => void }).unref?.(); + this.firstPayloadWaiters.push(() => { + clearTimeout(timer); + resolve(this.firstPayload); + }); + }); + } + + private releaseWaiters(): void { + while (this.firstPayloadWaiters.length > 0) this.firstPayloadWaiters.pop()?.(); + } + + private markFirstPayload(): void { + this.firstPayload = true; + this.releaseWaiters(); + } + + /** Why delivery stopped for good, or `null` while it is running. */ + get failed(): string | null { + return this.failedReason; + } + + /** A snapshot of what the transport has seen. See {@link StoreDiagnostics}. */ + get diagnostics(): StoreDiagnostics { + return { ...this.reader.diagnostics }; + } + + // -- the SkillStore seam ---------------------------------------------- + + getObject(kind: string, key: string, version?: number | null): RawSkillObject | null { + if (kind !== SKILL_OBJECT_KIND) return null; + return this.objects.get(key, version ?? null); + } + + allObjects(kind: string): Record { + if (kind !== SKILL_OBJECT_KIND) return {}; + return this.objects.snapshot(); + } + + /** + * Registers `fn` to be called once per committed change. + * + * Fires **once per changed object at payload-transferred**, not as objects + * stream in: a payload version is the unit of consistency, and a listener that + * reacted to a half-applied full transfer would see the store briefly empty. + * `watchSkills` is the intended consumer. + * + * A put notifies with the raw skill object. A revocation notifies with a + * `{ key, version }` tombstone — it names what went away and carries no + * content, since there is none. A listener that only needs "something changed" + * works with both; one that reads content must check for `content` rather than + * assume it. + * + * `fn` runs inline on the delivery task. Keep it cheap and non-blocking: work + * done there delays the next event. An exception it throws is logged and + * swallowed, because a broken listener must not be able to kill delivery. + */ + addListener(kind: string, fn: (raw: RawSkillObject) => unknown): void { + const existing = this.listeners.get(kind); + if (existing) existing.push(fn); + else this.listeners.set(kind, [fn]); + } + + private notify(changes: RawSkillObject[]): void { + const listeners = this.listeners.get(SKILL_OBJECT_KIND) ?? []; + for (const raw of changes) { + for (const listener of listeners) { + try { + listener(raw); + } catch (cause) { + error( + `A skill store change listener threw; delivery continues: ${ + cause instanceof Error ? cause.message : String(cause) + }`, + ); + } + } + } + } + + // -- the delivery loop ------------------------------------------------- + + private async run(signal: AbortSignal): Promise { + let failures = 0; + while (!signal.aborted) { + try { + if (this.mode === 'stream') await this.streamOnce(signal); + else await this.pollOnce(signal); + failures = 0; + this.reader.diagnostics.connectionFailures = 0; + } catch (cause) { + if (signal.aborted) return; + if (cause instanceof FatalTransportError) { + this.giveUp(cause.message); + return; + } + if (!(cause instanceof RecoverableTransportError)) { + this.giveUp(`unexpected error in skill delivery: ${cause instanceof Error ? cause.message : String(cause)}`); + return; + } + failures += 1; + this.reader.diagnostics.connectionFailures = failures; + this.reader.diagnostics.lastError = cause.message; + if (failures > this.maxConsecutiveFailures) { + this.giveUp(`gave up after ${failures} consecutive failures; last error: ${cause.message}`); + return; + } + const delay = cause.retryAfterMs ?? backoffDelayMs(failures, this.initialBackoffMs, this.maxBackoffMs); + warn(`Skill delivery failed (${cause.message}); retrying in ${Math.round(delay)}ms`); + await sleep(delay, signal); + continue; + } + + if (this.mode === 'poll') await sleep(this.pollIntervalMs, signal); + } + } + + private giveUp(reason: string): void { + this.failedReason = reason; + this.reader.diagnostics.lastError = reason; + error( + `Skill delivery has stopped and will not retry: ${reason}. The store keeps serving the last content it ` + + 'received; skills will not update until the process restarts with a working connection.', + ); + // Release anyone waiting on a first payload that is never coming, rather than + // making them eat the full timeout. + this.markFirstPayload(); + } + + private apply(name: string, data: unknown): TransferOutcome { + const outcome = this.reader.handle(name, data); + if (outcome.committed) { + if (outcome.basis) this.basis = outcome.basis; + this.markFirstPayload(); + if (outcome.changes && outcome.changes.length > 0) this.notify(outcome.changes); + } + return outcome; + } + + private dispatch(outcome: TransferOutcome): void { + if (outcome.fatal) throw new FatalTransportError(outcome.fatal); + if (outcome.disconnect) throw new RecoverableTransportError(outcome.disconnect); + } + + private async pollOnce(signal: AbortSignal): Promise { + const result = await this.requester.poll(this.basis, this.etag, signal); + this.etag = result.etag; + if (result.notModified) { + // A 304 is a successful, current answer: the payload we hold is the payload + // the server has. It counts as a first payload so a boot that reconnects + // with a cached basis is not blocked on a transfer the server has no reason + // to send. + this.markFirstPayload(); + return; + } + for (const [name, data] of result.events) { + this.dispatch(this.apply(name, data)); + } + } + + private async streamOnce(signal: AbortSignal): Promise { + const events = await this.requester.stream(this.basis, signal); + for await (const [name, data] of events) { + if (signal.aborted) return; + this.dispatch(this.apply(name, data)); + } + // A stream that ends without a goodbye is a dropped connection, not a + // completed operation: reconnect through the backoff path. + throw new RecoverableTransportError('the FDv2 stream closed unexpectedly'); + } +} diff --git a/packages/client/src/skills-watch.ts b/packages/client/src/skills-watch.ts new file mode 100644 index 00000000..d2fa26fe --- /dev/null +++ b/packages/client/src/skills-watch.ts @@ -0,0 +1,221 @@ +/** + * Agent Skills — re-reconcile on delivery, so revocation does not wait for a restart. + * + * `writeSkills` is a one-shot reconcile: it materializes what the store holds + * now. That was the whole story while the only transport was a hand-populated + * store, and the design accordingly deferred an eager re-reconcile — revocation + * would take effect at the next process restart, which the security review filed + * as AV-1. + * + * A streaming FDv2 connection changes the premise. A `delete-object` reaches a + * live connection in **seconds**, and the store already publishes a change + * listener, so the gap between "LaunchDarkly revoked this skill" and "its + * `SKILL.md` is off the agent's disk" collapses from a process lifetime to a + * debounce interval. That is the single largest resilience improvement available + * at this layer, which is why it is here rather than in a later phase. + * + * `onUnavailable: 'keep'` stays the default, deliberately and per the review: an + * outage must not read as "everything was revoked". A watcher that pruned on a + * failed retrieval would convert every transport blip into deletion of a + * customer's skill files. + * + * Layering: this module sits *above* `skills-fs.ts` and calls `writeSkills` + * without modifying it. Nothing in the reconcile, the accessors, or verification + * knows this file exists. + */ + +import { getStore, SKILL_OBJECT_KIND } from './skills-core.js'; +import { type WriteSkillsOptions, writeSkills } from './skills-fs.js'; +import type { ReconcileReport, Skill, SkillReference } from './types.js'; + +/** + * How long a change waits for its neighbours before a reconcile runs. + * + * A full payload transfer commits many objects at once and the listener fires per + * object, so without coalescing a payload of forty skills would run forty + * reconciles against one root. Half a second is far below the seconds-scale + * latency this feature is trying to achieve and far above the microseconds a + * commit's listener calls take. + */ +export const DEFAULT_DEBOUNCE_MS = 500; + +export type WatchSkillsOptions = WriteSkillsOptions & { + /** Coalescing window, in **milliseconds**. Default {@link DEFAULT_DEBOUNCE_MS}. */ + debounceMs?: number; + /** Called with each re-reconcile's report. Exceptions are logged, not thrown. */ + onReconcile?: (report: ReconcileReport) => unknown; +}; + +function error(message: string): void { + // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; a failing reconcile must be visible + console.error(`[LaunchDarkly] ${message}`); +} + +/** + * A running re-reconcile. Returned by {@link watchSkills}; stop it with `close`. + * + * One watcher owns one root. **Do not point two watchers at the same root**, and + * do not run `writeSkills` against a watched root concurrently: the reconcile's + * own contract is one root, one reconcile at a time, because two interleaved runs + * lose the loser's manifest entries and leave the files it wrote unmanaged. This + * class enforces that for its *own* reconciles — they are chained, never + * overlapped — and cannot enforce it against a caller who reconciles the same + * root by hand. + */ +export class SkillWatcher { + private timer: ReturnType | null = null; + private running: Promise = Promise.resolve(); + private pending = false; + private closed = false; + private completed = 0; + + constructor( + private readonly request: ReadonlyArray | '*', + private readonly root: string, + private readonly options: WriteSkillsOptions, + private readonly debounceMs: number, + private readonly onReconcile?: (report: ReconcileReport) => unknown, + ) {} + + /** + * The store's change listener. Schedules a reconcile; runs nothing inline. + * + * Deliberately trivial. It is called from the delivery task, where a reconcile + * — which does filesystem I/O, an fsync per file, and a manifest rewrite — + * would stall event processing for the duration and, on a stream, let the + * connection's read buffer back up behind a disk write. The argument is + * ignored: a put's raw object and a revocation's tombstone both mean the same + * thing here, which is "the store is not what it was". + */ + readonly notify = (): void => { + if (this.closed) return; + // Restarting the timer rather than letting the first one win is what makes a + // burst collapse into one reconcile that sees the *settled* state. + if (this.timer !== null) clearTimeout(this.timer); + this.timer = setTimeout(() => { + this.timer = null; + this.schedule(); + }, this.debounceMs); + (this.timer as unknown as { unref?: () => void }).unref?.(); + }; + + private schedule(): void { + if (this.closed) return; + if (this.pending) return; + this.pending = true; + // Chained onto whatever is already running: two concurrent reconciles of one + // root interleave on the manifest and lose entries. + this.running = this.running.then(async () => { + this.pending = false; + await this.reconcileOnce(); + }); + } + + private async reconcileOnce(): Promise { + if (this.closed) return; + let report: ReconcileReport; + try { + report = await writeSkills(this.request, this.root, this.options); + } catch (cause) { + // A watcher that died on one bad reconcile would silently stop tracking + // revocations, which is worse than a noisy one. + error( + `A skill re-reconcile threw; the watcher continues: ${cause instanceof Error ? cause.message : String(cause)}`, + ); + return; + } + this.completed += 1; + if (this.onReconcile) { + try { + this.onReconcile(report); + } catch (cause) { + error(`A watchSkills callback threw: ${cause instanceof Error ? cause.message : String(cause)}`); + } + } + } + + /** + * How many re-reconciles have completed since the watcher started. + * + * Excludes the initial reconcile {@link watchSkills} awaits, which is the + * caller's own result. + */ + get reconciles(): number { + return this.completed; + } + + /** + * Stops watching. Idempotent. Does not undo anything already on disk. + * + * Awaits an in-flight reconcile rather than abandoning one, because a reconcile + * interrupted between its content writes and its manifest rewrite is the one + * case the manifest format has to recover from — worth avoiding when we control + * the timing. + */ + async close(): Promise { + this.closed = true; + if (this.timer !== null) { + clearTimeout(this.timer); + this.timer = null; + } + await this.running; + } +} + +/** + * Reconciles now, then re-reconciles whenever delivery changes. + * + * Every option `writeSkills` takes means the same thing here and is passed + * straight through; the reconcile's semantics are untouched. Resolves to the + * initial reconcile's report — so a caller can fail fast on a bad root or a + * corrupt manifest exactly as they would with `writeSkills` — paired with a + * {@link SkillWatcher} to close when the process is done: + * + * ```ts + * const { report, watcher } = await watchSkills('*', '.claude/skills'); + * try { + * // ... + * } finally { + * await watcher.close(); + * } + * ``` + * + * A revocation delivered over a streaming connection then prunes the skill's + * files within `debounceMs` of arriving, rather than at the next restart. + * + * Requires a store that implements the optional `addListener` half of the seam. + * Throws when no store is configured, and when the configured store has no + * `addListener` — the second case failing loudly rather than degrading to a + * one-shot reconcile, because a watcher that silently never fires looks exactly + * like a watcher whose skills never changed. + */ +export async function watchSkills( + skills: ReadonlyArray | '*', + root: string, + options: WatchSkillsOptions = {}, +): Promise<{ report: ReconcileReport; watcher: SkillWatcher }> { + const store = getStore(); + if (store === null) { + throw new Error( + 'watchSkills needs a configured skill store. Configure one with initClient({ skillStore: store }).', + ); + } + if (typeof store.addListener !== 'function') { + throw new Error( + 'watchSkills needs a skill store that implements addListener(kind, fn); the configured store does not, so ' + + 'delivery changes cannot be observed. Use writeSkills for a one-shot reconcile, or configure a store with a ' + + 'delivery transport (FDv2SkillStore).', + ); + } + + const { debounceMs = DEFAULT_DEBOUNCE_MS, onReconcile, ...writeOptions } = options; + if (debounceMs < 0) throw new Error(`debounceMs must not be negative, got ${JSON.stringify(debounceMs)}`); + + // The initial reconcile runs first, so its report is the caller's to inspect + // and a bad root throws out of `watchSkills` rather than into a log line. + const report = await writeSkills(skills, root, writeOptions); + + const watcher = new SkillWatcher(skills, root, writeOptions, debounceMs, onReconcile); + store.addListener(SKILL_OBJECT_KIND, watcher.notify); + return { report, watcher }; +} diff --git a/packages/client/src/types.ts b/packages/client/src/types.ts index ff7d66d1..381d8d50 100644 --- a/packages/client/src/types.ts +++ b/packages/client/src/types.ts @@ -258,14 +258,13 @@ export type RawSkillObject = { * Structural interface every source of skill content satisfies. * * Structurally typed on purpose, mirroring how {@link LDClientInterface} works in - * this package: pass any object carrying these methods. The future real transport - * — a poller against the FDv2 delivery route — drops in behind this interface - * without touching the public API. + * this package: pass any object carrying these methods. `FDv2SkillStore` — which + * streams or polls LaunchDarkly's SDK-facing FDv2 channel — drops in behind this + * interface without touching the public API, and did. * * `addListener` is part of the seam but **optional**: a store - * without it must still be accepted. Nothing in this SDK calls it today; it is - * declared so the delivery transport and both language implementations agree on - * the callback shape when it lands. + * without it must still be accepted. `watchSkills` is its one consumer, and + * refuses loudly rather than degrading when a configured store lacks it. * * Everything a store serves is untrusted input. The transport is not part of the * trust boundary — key, version, size, and content hash are revalidated at the From 7b560b82cf96daf2e85de0a7bbe9759d0bb0f59b Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 11 Sep 2026 16:12:04 -0400 Subject: [PATCH 10/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20read=20the=20skill's=20version=20off=20the=20wire=20key?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SDK-facing FDv2 channel delivers skills the way streamer #4681 and gonfalon #70638 spell them: object kinds are open strings, the agent-skill payload is classified `generic`, and every generic object carries only `key`, `kind`, `version` and `object`, exactly like a flag. A skill arrives under kind `skill` with its own version folded into the key as `:`. There is no `category` field and no `objectVersion` field; both came from an earlier streamer draft that never shipped. Identification is now the kind alone. The wire key is split in one place, `splitWireKey`, and both the put and the delete translation go through it. A key that will not split cleanly is held rather than dropped — version-less, or with the offending text as its version — so verification withholds it with `invalid_version` under a key the caller recognises; only a key with nothing before the delimiter is dropped, since there is no identity to hold it under. `SDK_DATA_MODEL_VERSION` and the `dataModelVersion` option go with it: the connection's `mv` parameter only accepts flag model versions, and generic payloads ignore it, so the request no longer sends one. The 403 and 400-class advice now says what a customer can act on rather than naming internal flags and unmerged branches. Ports python-ai-sdk efc4ca7. Co-Authored-By: Claude Fable 5.1 --- .../client/src/__tests__/skills-fdv2.test.ts | 175 ++++++++++++---- packages/client/src/index.ts | 2 +- packages/client/src/skills-fdv2.ts | 196 +++++++++++------- 3 files changed, 250 insertions(+), 123 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 033e31f3..4ffafb8c 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -4,16 +4,16 @@ * Two layers, deliberately, mirroring the Python suite: * * - **A real fake endpoint.** `FakeFDv2Endpoint` is an in-process - * `node:http` server implementing the wire contract — `basis` and `mv` query - * parameters, `Authorization`, `If-None-Match`/304, the `{"events": [...]}` + * `node:http` server implementing the wire contract — the `basis` query + * parameter, `Authorization`, `If-None-Match`/304, the `{"events": [...]}` * polling envelope, and SSE for streaming. The store under test opens real * sockets against it, so request construction and header handling are - * exercised rather than mocked. This is what stands in for a live server while - * the backend work is unmerged. + * exercised rather than mocked. * - **The protocol reader driven directly.** Wire semantics — which objects are - * skills, `objectVersion` versus `version`, revocation, mixed payloads — are - * asserted against `ProtocolReader`, which has no I/O, so those cases read as - * the contract they are instead of as a server script. + * skills, the skill's version in the wire `key` versus the payload's in + * `version`, revocation, mixed payloads — are asserted against + * `ProtocolReader`, which has no I/O, so those cases read as the contract they + * are instead of as a server script. */ import { createHash } from 'node:crypto'; @@ -31,7 +31,7 @@ import { backoffDelayMs, classifyStatus, decodePollBody, - FDV2_OBJECT_CATEGORY, + FDV2_KEY_DELIMITER, FDV2_OBJECT_KIND, FDv2SkillStore, isSkillEvent, @@ -41,6 +41,7 @@ import { retryAfterMs, SkillObjectSet, seamObjectFromPut, + splitWireKey, tombstoneFromDelete, } from '../skills-fdv2.js'; import { watchSkills } from '../skills-watch.js'; @@ -55,6 +56,18 @@ const hash = (content: string): string => createHash('sha256').update(content, ' type WireEvent = { event: string; data?: unknown }; +/** + * The wire `key` of one skill object: `:`. + * + * `null` builds a key with no version at all, which is how the tests spell a + * malformed object; anything else is spelled after the delimiter verbatim. + */ +function wireKey(key: string, objectVersion: unknown): string { + if (objectVersion === null) return key; + return `${key}${FDV2_KEY_DELIMITER}${String(objectVersion)}`; +} + +/** One skill `put-object` event's data, in the shape the wire delivers it. */ function putSkill( key = 'pdf-extraction', { @@ -63,7 +76,6 @@ function putSkill( content = SKILL_BODY, contentHash = null as string | null, omitHash = false, - omitObjectVersion = false, } = {}, ): Record { const envelope: Record = { @@ -73,28 +85,23 @@ function putSkill( description: 'Extracts text', }; if (!omitHash) envelope.contentHash = contentHash ?? hash(content); - const wire: Record = { - key, + return { + key: wireKey(key, objectVersion), kind: FDV2_OBJECT_KIND, - category: FDV2_OBJECT_CATEGORY, version: payloadVersion, object: envelope, }; - if (!omitObjectVersion) wire.objectVersion = objectVersion; - return wire; } function deleteSkill(key = 'pdf-extraction', { objectVersion = 3 as unknown, payloadVersion = 43 } = {}) { return { - key, + key: wireKey(key, objectVersion), kind: FDV2_OBJECT_KIND, - category: FDV2_OBJECT_CATEGORY, - objectVersion, version: payloadVersion, }; } -/** A flag `put-object`: no `category`, no `objectVersion`. */ +/** A flag `put-object`: the same envelope fields, a different `kind`. */ function putFlag(key = 'my-flag', version = 17) { return { key, kind: 'flag', version, object: { key, version, on: true, variations: [true, false] } }; } @@ -307,10 +314,17 @@ const consoleErrors = (): string => logged(errorSpy); // ─── Identifying skill objects, and ignoring everything else ───────────────── describe('object identification', () => { - it('identifies a skill by kind and category together', () => { + it('identifies a skill by the kind alone', () => { expect(isSkillEvent(putSkill())).toBe(true); }); + it('spells the kind as the bare category name', () => { + // Object kinds on the channel are open strings and the agent-skill payload + // is `generic`, so a skill arrives under the kind its producer registered — + // `skill` — not under a broader wrapper kind. + expect(FDV2_OBJECT_KIND).toBe('skill'); + }); + it('does not treat a flag as a skill', () => { expect(isSkillEvent(putFlag())).toBe(false); }); @@ -319,17 +333,18 @@ describe('object identification', () => { expect(isSkillEvent(putSegment())).toBe(false); }); - it('requires the category: inline-resource is a broad kind', () => { - expect(isSkillEvent({ ...putSkill(), category: 'prompt-template' })).toBe(false); + it('does not treat another generic kind as a skill', () => { + // A generic payload may carry other registered kinds one day. + expect(isSkillEvent({ ...putSkill(), kind: 'prompt-template' })).toBe(false); }); - it('requires the kind: a skill category under another kind is not a skill', () => { + it('does not treat a skill-shaped envelope under another kind as a skill', () => { expect(isSkillEvent({ ...putSkill(), kind: 'some-future-kind' })).toBe(false); }); - it('documents that flags omit both category and objectVersion', () => { - expect('category' in putFlag()).toBe(false); - expect('objectVersion' in putFlag()).toBe(false); + it('consults nothing but the kind', () => { + // No secondary field narrows the kind, and none may be required. + expect(Object.keys(putSkill()).sort()).toEqual(['key', 'kind', 'object', 'version']); }); it.each([null, undefined, 'skill', 3, []])('does not treat %s as a skill', (value) => { @@ -337,11 +352,22 @@ describe('object identification', () => { }); }); -// ─── objectVersion is not version. This is the whole ballgame. ─────────────── +// ─── The skill's version is in the wire key; `version` is the payload's. ───── describe('version translation', () => { - it('turns objectVersion into the seam version', () => { - expect(seamObjectFromPut(putSkill('pdf-extraction', { objectVersion: 3, payloadVersion: 42 }))?.version).toBe(3); + it('spells the wire key as key colon version', () => { + expect(putSkill('pdf-extraction', { objectVersion: 3 }).key).toBe('pdf-extraction:3'); + }); + + it('turns the version after the delimiter into the seam version', () => { + const raw = seamObjectFromPut(putSkill('pdf-extraction', { objectVersion: 3, payloadVersion: 42 })); + expect(raw?.version).toBe(3); + expect(typeof raw?.version).toBe('number'); + }); + + it('turns the key before the delimiter into the seam key', () => { + // A caller asks for `pdf-extraction`, never for `pdf-extraction:3`. + expect(seamObjectFromPut(putSkill('pdf-extraction', { objectVersion: 3 }))?.key).toBe('pdf-extraction'); }); it('never lets the payload version reach the seam', () => { @@ -357,23 +383,54 @@ describe('version translation', () => { expect(seamObjectFromPut(putSkill('k', { objectVersion: 99, payloadVersion: 1 }))?.version).toBe(99); }); - it('does not default a missing objectVersion from the payload version', () => { - const raw = seamObjectFromPut(putSkill('k', { omitObjectVersion: true })); + it('holds a key with no delimiter version-less', () => { + // Not defaulted from the payload version, and not dropped: verification + // reports `invalid_version` under a key the caller recognises. + const raw = seamObjectFromPut(putSkill('pdf-extraction', { objectVersion: null })); expect(raw).not.toBeNull(); + expect(raw?.key).toBe('pdf-extraction'); expect('version' in (raw as RawSkillObject)).toBe(false); }); - it('carries an explicitly null objectVersion through rather than inventing one', () => { - // Carried, not invented: verification reports `invalid_version`. - expect(seamObjectFromPut(putSkill('k', { objectVersion: null }))?.version).toBeNull(); + it.each([ + 'latest', + '', + '3.0', + '-1', + '1:2', + '３', + ])('carries a version that is not digits (%j) through as invalid', (spelling) => { + // Carried, not invented: verification reports `invalid_version` for the + // object rather than the transport reporting it absent. + const raw = seamObjectFromPut(putSkill('pdf-extraction', { objectVersion: spelling })); + expect(raw).not.toBeNull(); + expect(raw?.key).toBe('pdf-extraction'); + expect(raw?.version).toBe(spelling); + }); + + it('reads leading zeros as the same version', () => { + expect(seamObjectFromPut(putSkill('pdf-extraction', { objectVersion: '03' }))?.version).toBe(3); }); - it('translates objectVersion on a delete too', () => { - expect(tombstoneFromDelete(deleteSkill('k', { objectVersion: 3, payloadVersion: 43 }))?.objectVersion).toBe(3); + it('reads the wire key the same way on a delete', () => { + const tombstone = tombstoneFromDelete(deleteSkill('pdf-extraction', { objectVersion: 3, payloadVersion: 43 })); + expect(tombstone?.key).toBe('pdf-extraction'); + expect(tombstone?.objectVersion).toBe(3); }); - it('reads a delete with no usable objectVersion as revoking every version', () => { - expect(tombstoneFromDelete(deleteSkill('k', { objectVersion: null }))?.objectVersion).toBeNull(); + it.each([null, 'latest', '0'])('reads a delete with no usable version (%j) as revoking every version', (spelling) => { + const tombstone = tombstoneFromDelete(deleteSkill('pdf-extraction', { objectVersion: spelling })); + expect(tombstone?.key).toBe('pdf-extraction'); + expect(tombstone?.objectVersion).toBeNull(); + }); + + it.each([ + ':3', + '', + null, + 3, + ])('drops a put whose key (%j) carries no skill key, since it has no identity', (badKey) => { + expect(seamObjectFromPut({ ...putSkill(), key: badKey })).toBeNull(); }); it('drops a keyless put, which has no identity to store it under', () => { @@ -381,6 +438,32 @@ describe('version translation', () => { expect(seamObjectFromPut(keyless)).toBeNull(); }); + it('ignores a delete with no skill key', () => { + expect(tombstoneFromDelete({ ...deleteSkill(), key: ':3' })).toBeNull(); + }); + + it('splits both halves in one place', () => { + expect(splitWireKey('pdf-extraction:3')).toEqual({ key: 'pdf-extraction', hasVersion: true, version: 3 }); + expect(splitWireKey('pdf-extraction')).toEqual({ key: 'pdf-extraction', hasVersion: false }); + expect(splitWireKey('pdf-extraction:latest')).toEqual({ + key: 'pdf-extraction', + hasVersion: true, + version: 'latest', + }); + expect(splitWireKey(':3')).toBeNull(); + expect(splitWireKey('')).toBeNull(); + expect(splitWireKey(3)).toBeNull(); + }); + + it('round-trips the stored identity to the wire key', () => { + // `SkillObjectSet.snapshot` spells its opaque keys the way the wire does, so + // a held object can be matched back to the event that carried it. + const held = new SkillObjectSet(); + const wire = putSkill('pdf-extraction', { objectVersion: 3 }); + held.put(seamObjectFromPut(wire) as RawSkillObject); + expect(Object.keys(held.snapshot())).toEqual([wire.key]); + }); + it('copies the envelope verbatim', () => { const raw = seamObjectFromPut(putSkill()); expect(raw?.content).toBe(SKILL_BODY); @@ -672,14 +755,18 @@ describe('polling against the endpoint', () => { expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')?.version).toBe(3); }); - it('sends the SDK key and the data model version', async () => { + it('sends the SDK key and no data model version', async () => { + // No `mv`: that parameter selects the *flag* data model, the connection + // rejects any value but the flag default, and the generic agent-skill + // payload is served regardless of it. Sending `mv=1` — the skill payload's + // own model version — gets the whole connection refused. endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); const store = pollStore(); store.start(); await store.waitForSkills(5000); expect(endpoint.requests[0].path).toBe('/sdk/poll'); expect(endpoint.requests[0].authorization).toBe(SDK_KEY); - expect(endpoint.requests[0].query.mv).toBe('1'); + expect('mv' in endpoint.requests[0].query).toBe(false); }); it('sends no basis on the first request', async () => { @@ -901,14 +988,14 @@ class ScriptedRequester implements Requester { } describe('failure handling', () => { - it('stops on 403 and names the protocol control flag', async () => { + it('stops on 403 and explains why', async () => { endpoint.queuePoll([], { status: 403 }); const store = pollStore(); store.start(); expect(await waitUntil(() => store.failed !== null)).toBe(true); expect(store.failed).toContain('403'); - expect(store.failed).toContain('fdv2-protocol-control'); - expect(consoleErrors()).toContain('fdv2-protocol-control'); + expect(store.failed).toContain('opt-in'); + expect(consoleErrors()).toContain('opt-in'); }); it('stops on 401', async () => { @@ -1163,7 +1250,7 @@ describe('the missing contentHash', () => { expect(skill?.name).toBe('PDF Extraction'); }); - it('resolves a pinned reference to the pinned objectVersion', async () => { + it('resolves a pinned reference to the pinned version', async () => { endpoint.queuePoll( fullPayload([ ['put-object', putSkill('pdf-extraction', { objectVersion: 2, content: 'v2 body' })], @@ -1181,7 +1268,7 @@ describe('the missing contentHash', () => { }); it('does not resolve the payload version as a skill version', async () => { - // The end-to-end form of the objectVersion/version assertion. Asking for the + // The end-to-end form of the wire-key/version assertion. Asking for the // payload version resolves nothing — reported `absent`, because the store // answers "I hold no such version" rather than answering with the wrong one. endpoint.queuePoll( diff --git a/packages/client/src/index.ts b/packages/client/src/index.ts index 4695d0a0..da2851d9 100644 --- a/packages/client/src/index.ts +++ b/packages/client/src/index.ts @@ -22,7 +22,7 @@ export { compose, globalRegistry, Registry } from './registry.js'; export { allSkills, getSkill, getSkillResult, getSkills, InMemorySkillStore, skillRefs } from './skills.js'; export { MAX_SKILL_CONTENT_BYTES, SKILL_OBJECT_KIND } from './skills-core.js'; export type { FDv2Mode, FDv2SkillStoreOptions, StoreDiagnostics } from './skills-fdv2.js'; -export { DEFAULT_BASE_URI, FDv2SkillStore, SDK_DATA_MODEL_VERSION } from './skills-fdv2.js'; +export { DEFAULT_BASE_URI, FDv2SkillStore } from './skills-fdv2.js'; export type { WriteSkillsOptions } from './skills-fs.js'; export { MANIFEST_FILENAME, MANIFEST_VERSION, SKILL_FILENAME, writeSkills } from './skills-fs.js'; export type { WatchSkillsOptions } from './skills-watch.js'; diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 78b2c2d6..917282a5 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -61,14 +61,25 @@ import { isValidSkillVersion } from './types.js'; /** * The FDv2 `kind` skills are delivered under. * - * Distinct from `SKILL_OBJECT_KIND` (`'skill'`), which is the *seam* value the - * SDK asks a store for. Translating this pair — kind `inline-resource` plus - * category `skill` — onto that single value is exactly the adapter's job. + * Object kinds on the SDK-facing channel are open strings: the agent-skill + * payload is classified `generic` and every object in it carries the kind its + * producer registered, which for skills is the bare category name. Delivery + * lower-cases the kind, so an exact comparison is the whole test. The kind + * happens to equal `SKILL_OBJECT_KIND` today; they are still separate constants, + * because one is a wire value LaunchDarkly owns and the other is an SDK seam. */ -export const FDV2_OBJECT_KIND = 'inline-resource'; +export const FDV2_OBJECT_KIND = 'skill'; -/** The `category` that narrows `inline-resource` to an agent skill. */ -export const FDV2_OBJECT_CATEGORY = 'skill'; +/** + * What separates a skill's key from its version inside the object's wire `key`. + * + * A generic object is identified on the wire as `:` — the skill's + * own key, one delimiter, the skill's own version — because each version of a + * skill is a distinct object in the payload. Delivery forbids the delimiter + * inside a registered category and skill keys cannot contain it, so a + * well-formed wire key has exactly one. + */ +export const FDV2_KEY_DELIMITER = ':'; /** * Where the SDK-facing FDv2 endpoints live. Overridable for Federal instances, @@ -79,18 +90,6 @@ export const DEFAULT_BASE_URI = 'https://sdk.launchdarkly.com'; export const POLL_PATH = '/sdk/poll'; export const STREAM_PATH = '/sdk/stream'; -/** - * The `mv` request parameter — the SDK data model version this adapter speaks. - * - * Overridable through `new FDv2SkillStore(key, { dataModelVersion })` because it - * is the one request parameter this side cannot verify: the LaunchDarkly base - * SDK's own FDv2 data source does not send `mv` at all today, and the streamer - * branch that carries skills is unmerged, so the value the server expects has not - * been observed. Confirm it with FDN before Beta rather than trusting this - * default. - */ -export const SDK_DATA_MODEL_VERSION = 1; - const EVENT_SERVER_INTENT = 'server-intent'; const EVENT_PUT_OBJECT = 'put-object'; const EVENT_DELETE_OBJECT = 'delete-object'; @@ -279,7 +278,7 @@ function warnIfNothingCanVerify(held: RawSkillObject[]): void { } // --------------------------------------------------------------------------- -// Deserialization — where objectVersion is not version +// Deserialization — where the skill's version lives in the key, not in version // --------------------------------------------------------------------------- /** A `delete-object` narrowed to the identity it revokes. */ @@ -288,63 +287,96 @@ export type Tombstone = { readonly key: string; readonly objectVersion: number | /** * Whether one `put-object` / `delete-object` payload is a skill. * - * `kind === 'inline-resource' && category === 'skill'`, and nothing else. Both - * halves are required: `inline-resource` is a broad kind that may carry other - * categories, and flags and segments omit `category` entirely. - * - * Every other kind is **ignored, not rejected**. An environment's payload - * assignment carries the flagging payload alongside the agent-skill payload, so a - * connection delivers flag and segment objects as a matter of course. Throwing on - * them would turn a normal payload into a permanent failure — which is exactly - * the unknown-kind reconnect loop this feature must not reproduce. + * The kind alone decides it. Every other kind is **ignored, not rejected**, + * because flag and segment objects share the connection and erroring on them + * would turn a normal payload into a reconnect loop — exactly the unknown-kind + * failure this feature must not reproduce. */ export function isSkillEvent(data: unknown): boolean { if (typeof data !== 'object' || data === null || Array.isArray(data)) return false; - const candidate = data as { kind?: unknown; category?: unknown }; - return candidate.kind === FDV2_OBJECT_KIND && candidate.category === FDV2_OBJECT_CATEGORY; + return (data as { kind?: unknown }).kind === FDV2_OBJECT_KIND; } /** - * Translates one FDv2 skill `put-object` into a seam-shaped raw object. + * A skill object's wire `key`, split into the skill's key and version. + * + * `version` is a number when the wire carried one, the offending text when it + * did not, and absent (`undefined`, with `hasVersion === false`) when the wire + * key had no delimiter at all. + */ +export type WireIdentity = { readonly key: string; readonly hasVersion: boolean; readonly version?: unknown }; + +const DIGITS_ONLY = /^[0-9]+$/; + +/** + * Reads `:` off one object's wire `key`. * - * `null` when the event cannot be filed at all — only when `key` is not a string, - * since a keyless object has no identity to store it under and no key to - * attribute a failure to. Every other defect is carried through verbatim so that - * *verification* withholds it, with a reason code and an integrity signal, rather - * than the transport dropping it silently. A silent drop is indistinguishable - * from "no such skill" and would additionally let a prune delete the last - * known-good copy on disk. + * Lenient where leniency keeps the object diagnosable and strict only where + * there is nothing to diagnose: + * + * - No delimiter: the whole wire key is the skill key and there is no version, + * so the object is held version-less and verification reports + * `invalid_version` under a key the caller can recognise. + * - A version that is not a run of ASCII digits (`"pdf:latest"`, `"pdf:"`, + * `"a:1:2"`): the text is carried through *as the version*, for the same + * reason — the caller learns that `pdf` arrived broken, not that it is absent. + * - An empty key before the delimiter (`":3"`): there is no identity to hold it + * under, so `null`, and the caller drops it. + * + * Leading zeros are accepted (`"pdf:03"` is version 3) since the number is the + * identity a reference pins, not the spelling. + */ +export function splitWireKey(wireKey: unknown): WireIdentity | null { + if (typeof wireKey !== 'string' || wireKey === '') return null; + const delimiter = wireKey.indexOf(FDV2_KEY_DELIMITER); + if (delimiter === -1) return { key: wireKey, hasVersion: false }; + const key = wireKey.slice(0, delimiter); + if (key === '') return null; + const versionText = wireKey.slice(delimiter + 1); + if (DIGITS_ONLY.test(versionText)) return { key, hasVersion: true, version: Number(versionText) }; + return { key, hasVersion: true, version: versionText }; +} + +/** + * Translates one FDv2 skill `put-object` into a seam-shaped raw object. * * **The translation this whole module exists to get right:** * * ``` - * wire `objectVersion` → seam `version` (the skill's own version) - * wire `version` → dropped (the *payload* version) + * wire `key` → seam `key` and `version` (split on `:`) + * wire `version` → dropped (the *payload* version) * ``` * - * `objectVersion` is what a `{key, version}` reference pins. `version` is the - * version of the payload the object arrived in — it changes when anything in the - * environment changes, including a flag that has nothing to do with skills. - * Reading it as the skill's version resolves the wrong content with no error - * anywhere: the object verifies, the hash matches, and the caller is handed a - * skill under a version number that means nothing. Flags and segments carry only - * `version`, which is why the two fields look interchangeable and are not. + * Each version of a skill is its own object on the wire, identified as + * `:`; that version is what a `{key, version}` reference pins. The + * event's `version` field is the version of the payload the object arrived in and + * moves whenever anything in the environment moves, including a flag that has + * nothing to do with skills. Confusing them fails silently: the object verifies, + * the hash matches, and the caller is handed a skill under a version number that + * means nothing. + * + * `null` only when the wire `key` carries no skill key at all, since such an + * object has no identity to store it under. Every other defect is carried through + * verbatim so that *verification* withholds it, with a reason code and an + * integrity signal, rather than the transport dropping it silently. A silent drop + * is indistinguishable from "no such skill" and would additionally let a prune + * delete the last known-good copy on disk. */ export function seamObjectFromPut(data: Record): RawSkillObject | null { - const { key } = data as { key?: unknown }; - if (typeof key !== 'string' || key === '') { + const identity = splitWireKey(data.key); + if (identity === null) { warn( - "An FDv2 skill put-object carried no string 'key' and could not be stored under any identity; it was dropped.", + `An FDv2 skill put-object carried no usable 'key' (${JSON.stringify(data.key)}) and could not be stored ` + + 'under any identity; it was dropped.', ); return null; } - const raw: RawSkillObject = { key }; + const raw: RawSkillObject = { key: identity.key }; - // The single translation. Written as a property-presence test rather than a - // defaulted read so an explicitly-null objectVersion stays null and reaches - // verification as `invalid_version`, instead of being invented here. - if ('objectVersion' in data) raw.version = data.objectVersion; + // Absent stays absent and malformed stays malformed, so verification sees what + // arrived (as `invalid_version`) rather than something invented here. + if (identity.hasVersion) raw.version = identity.version; const envelope = data.object; if (typeof envelope === 'object' && envelope !== null && !Array.isArray(envelope)) { @@ -356,25 +388,30 @@ export function seamObjectFromPut(data: Record): RawSkillObject } /** - * Narrows one FDv2 skill `delete-object` to the identity it revokes. + * Narrows one FDv2 skill `delete-object` to the identity it revokes, reading the + * wire `key` the same way a put does. * - * A delete for an inline resource **is revocation** — the object leaves the - * payload, this store drops it, the accessors stop resolving it, and the next - * reconcile prunes its files. Same `objectVersion` translation as a put. + * A delete for a skill **is revocation** — the object leaves the payload, this + * store drops it, the accessors stop resolving it, and the next reconcile prunes + * its files. * * An `objectVersion` of `null` means the delete named no usable version, and is * read as "revoke every version of this key". That is the safe direction: the * alternative is ignoring an unparseable revocation and continuing to serve - * content LaunchDarkly has withdrawn. + * content LaunchDarkly has withdrawn. It also removes whatever a malformed put of + * the same wire key left held, since that was stored version-less under the same + * skill key. */ export function tombstoneFromDelete(data: Record): Tombstone | null { - const { key } = data as { key?: unknown }; - if (typeof key !== 'string' || key === '') { - warn("An FDv2 skill delete-object carried no string 'key'; it was ignored."); + const identity = splitWireKey(data.key); + if (identity === null) { + warn(`An FDv2 skill delete-object carried no usable 'key' (${JSON.stringify(data.key)}); it was ignored.`); return null; } - const objectVersion = data.objectVersion; - return { key, objectVersion: isValidSkillVersion(objectVersion) ? objectVersion : null }; + return { + key: identity.key, + objectVersion: identity.hasVersion && isValidSkillVersion(identity.version) ? identity.version : null, + }; } // --------------------------------------------------------------------------- @@ -382,7 +419,7 @@ export function tombstoneFromDelete(data: Record): Tombstone | // --------------------------------------------------------------------------- /** - * Raw skill objects held in memory, keyed by `(key, objectVersion)`. + * Raw skill objects held in memory, keyed by `(key, version)`. * * Several versions of one key coexist, because they coexist in a real payload: * the newest version of every skill plus every version a variation currently @@ -715,9 +752,8 @@ export class RecoverableTransportError extends Error { } const FORBIDDEN_ADVICE = - "FDv2 is opt-in per account: the 'fdv2-protocol-control' setting defaults to 'forbid', which is served as " + - 'HTTP 403. Skill delivery over this channel needs that flag flipped for the account, and needs the ' + - 'FDCore/streamer inline-resource support merged and deployed.'; + 'The FDv2 protocol is opt-in per LaunchDarkly account and is served as HTTP 403 while it is off. Skill ' + + 'delivery needs it enabled; contact LaunchDarkly support to enable it for your account.'; /** `Retry-After` in milliseconds, when the server sent a usable one. */ export function retryAfterMs(headers: Headers | null | undefined): number | null { @@ -748,7 +784,8 @@ export function classifyStatus(status: number, headers?: Headers | null): Error if ([400, 405, 406, 414, 501].includes(status)) { return new FatalTransportError( `LaunchDarkly returned HTTP ${status}, which retrying will not fix. The request this adapter sent was not ` + - `understood; the 'mv' data model version (${SDK_DATA_MODEL_VERSION}) is the parameter most likely to be wrong.`, + "understood. It carries only the SDK key and, after the first payload, a 'basis' selector, so check the " + + 'base URI and that the endpoint speaks FDv2.', ); } return new RecoverableTransportError(`LaunchDarkly returned HTTP ${status}`, retryAfterMs(headers)); @@ -869,15 +906,21 @@ export class FetchRequester implements Requester { constructor( private readonly sdkKey: string, baseUri: string, - private readonly dataModelVersion: number, ) { this.baseUri = baseUri.replace(/\/+$/, ''); } + /** + * The request URL: the path, plus `basis` once a payload has committed. + * + * Deliberately no `mv` (data model version). That parameter selects the *flag* + * data model and the connection rejects any value but the flag default; the + * agent-skill payload is generic, is served regardless of it, and has no model + * version of its own to ask for. + */ private url(path: string, basis: string | null): string { - const params = new URLSearchParams({ mv: String(this.dataModelVersion) }); - if (basis) params.set('basis', basis); - return `${this.baseUri}${path}?${params.toString()}`; + if (!basis) return `${this.baseUri}${path}`; + return `${this.baseUri}${path}?${new URLSearchParams({ basis }).toString()}`; } /** @@ -991,7 +1034,6 @@ export type FDv2SkillStoreOptions = { * live — `failed` reports it. */ readonly maxConsecutiveFailures?: number; - readonly dataModelVersion?: number; /** Test seam: a transport double in place of `FetchRequester`. */ readonly requester?: Requester; }; @@ -1066,9 +1108,7 @@ export class FDv2SkillStore implements SkillStore { this.initialBackoffMs = options.initialBackoffMs ?? 1_000; this.maxBackoffMs = options.maxBackoffMs ?? 30_000; this.maxConsecutiveFailures = options.maxConsecutiveFailures ?? 10; - this.requester = - options.requester ?? - new FetchRequester(key, options.baseUri ?? DEFAULT_BASE_URI, options.dataModelVersion ?? SDK_DATA_MODEL_VERSION); + this.requester = options.requester ?? new FetchRequester(key, options.baseUri ?? DEFAULT_BASE_URI); } // -- lifecycle --------------------------------------------------------- From 9407ddc1023a7719f40621f41bf2ada585fdc6d5 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 11 Sep 2026 16:13:58 -0400 Subject: [PATCH 11/55] =?UTF-8?q?feat(client):=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20name=20the=20payload=20a=20transfer=20completed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The protocol reader took payloads[0]'s intentCode and applied it to the skill object set, which is what the delivery protocol requires — one payload per credential, read the first intent, tolerate the rest — but it left the assumption behind that rule undocumented and unguarded. If the one-payload guarantee ever widens, an xfer-full for another payload would start an empty pending set and the next payload-transferred would publish it: every skill reported revoked, and with pruning on, a customer's files deleted. The first payload is still the payload that is read. What is new is that the reader now knows which payload skills actually arrive on — learnt from the intent's id, or from the (p::) selector, since no object or transfer event carries a payload id of its own — and declines to apply a transfer of any other, holding last known good, warning once, and counting it in diagnostics.payloadsIgnored. An intent describing more than one payload warns once on its own, because that is the one case the comparison cannot catch: another payload's transfer arriving before any skill has been seen has nothing to be compared against. Behaviour under one-payload delivery is unchanged, and a full transfer of the skill payload still empties it — every skill deleted is a real state the guard must not mask. Ports python-ai-sdk 1cf5235. Co-Authored-By: Claude Fable 5.1 --- .../client/src/__tests__/skills-fdv2.test.ts | 193 +++++++++++++++++- packages/client/src/skills-fdv2.ts | 132 +++++++++++- 2 files changed, 316 insertions(+), 9 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 4ffafb8c..a328f56d 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -110,8 +110,8 @@ function putSegment(key = 'beta-users', version = 4) { return { key, kind: 'segment', version, object: { key, version, included: [] } }; } -function serverIntent(code = 'xfer-full') { - return { payloads: [{ id: 'agent-skill', target: 1, intentCode: code, reason: 'test' }] }; +function serverIntent(code = 'xfer-full', payloadId = 'agent-skill') { + return { payloads: [{ id: payloadId, target: 1, intentCode: code, reason: 'test' }] }; } const transferred = (state = 'basis-1', version = 42) => ({ state, version }); @@ -696,6 +696,195 @@ describe('protocol reader', () => { }); }); +// ─── Which payload a transfer completed ────────────────────────────────────── + +const payloadWarnings = (fragment: string): string[] => + logged(warnSpy) + .split('\n') + .filter((line) => line.includes(fragment)); + +/** One payload's events, with the payload it belongs to named explicitly. */ +function skillPayload( + objectEvents: Array<[string, unknown]>, + { payloadId = 'agent-skill', code = 'xfer-full', state = 'basis-1' } = {}, +): WireEvent[] { + return events(['server-intent', serverIntent(code, payloadId)], ...objectEvents, [ + 'payload-transferred', + transferred(state), + ]); +} + +describe('payload identity', () => { + // Delivery provides one payload per credential and the protocol requires a + // client to read only the first payload intent, so today the payload read is + // the payload skills arrive on. These assert the behaviour that survives if + // the first of those stops holding: another payload's `xfer-full` must not + // publish an empty skill set, because with pruning on that deletes a + // customer's materialized files. + + it('reads only the first payload intent', () => { + // Reading only the first is what the protocol asks for, however many + // arrive — the point of the rest of this suite is to make that safe. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive( + reader, + events( + [ + 'server-intent', + { + payloads: [ + { id: 'agent-skill', target: 1, intentCode: 'xfer-full' }, + { id: 'env-flags', target: 2, intentCode: 'none' }, + ], + }, + ], + ['put-object', putSkill()], + ['payload-transferred', transferred()], + ), + ); + expect(held.size).toBe(1); + }); + + it('warns once about more than one payload intent', () => { + const reader = new ProtocolReader(new SkillObjectSet()); + const intent = { + payloads: [ + { id: 'env-flags', target: 1, intentCode: 'xfer-changes' }, + { id: 'agent-skill', target: 2, intentCode: 'xfer-changes' }, + ], + }; + reader.handle('server-intent', intent); + reader.handle('server-intent', intent); + expect(payloadWarnings('described 2 payloads')).toHaveLength(1); + }); + + it('warns about nothing for one payload intent', () => { + drive(new ProtocolReader(new SkillObjectSet()), skillPayload([['put-object', putSkill()]])); + expect(payloadWarnings('payload')).toEqual([]); + }); + + it("does not let another payload's full transfer empty the skills held", () => { + // The case this guard exists for. A flag payload's `xfer-full` starts an + // empty pending set; applying it at `payload-transferred` would publish + // every skill as revoked, which a reconcile with pruning on reads as + // "delete these files". + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, skillPayload([['put-object', putSkill()]])); + const outcomes = drive( + reader, + skillPayload([['put-object', putFlag()]], { payloadId: 'env-flags', state: 'basis-2' }), + ); + expect(held.get('pdf-extraction', null)).not.toBeNull(); + expect(reader.diagnostics.payloadsIgnored).toBe(1); + expect(payloadWarnings('was not applied')).toHaveLength(1); + // Nothing changed, so no listener is woken to reconcile against it. + expect(outcomes.at(-1)?.changes).toEqual([]); + }); + + it('warns once about a declined transfer however often it repeats', () => { + // A polling connection sees the other payload on every poll. + const reader = new ProtocolReader(new SkillObjectSet()); + drive(reader, skillPayload([['put-object', putSkill()]])); + const foreign = skillPayload([['put-object', putFlag()]], { payloadId: 'env-flags' }); + drive(reader, foreign); + drive(reader, foreign); + expect(payloadWarnings('was not applied')).toHaveLength(1); + expect(reader.diagnostics.payloadsIgnored).toBe(2); + }); + + it('still empties the skills on a full transfer of the skill payload', () => { + // Every skill deleted is a real state, and the guard must not mask it. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, skillPayload([['put-object', putSkill()]])); + drive(reader, skillPayload([], { state: 'basis-2' })); + expect(held.size).toBe(0); + expect(reader.diagnostics.payloadsIgnored).toBe(0); + }); + + it('identifies the payload as the skill payload from a revocation', () => { + // A payload that only revokes is still a payload skills arrive on. + const reader = new ProtocolReader(new SkillObjectSet()); + drive(reader, skillPayload([['delete-object', deleteSkill()]], { code: 'xfer-changes' })); + drive(reader, skillPayload([['put-object', putSkill()]], { payloadId: 'env-flags' })); + expect(reader.diagnostics.payloadsIgnored).toBe(1); + }); + + it('identifies the payload from the selector when no id is named', () => { + // `payload-transferred`'s selector is the only other place a completed + // transfer names its payload. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + const unnamed = { payloads: [{ target: 1, intentCode: 'xfer-full' }] }; + drive( + reader, + events( + ['server-intent', unnamed], + ['put-object', putSkill()], + ['payload-transferred', transferred('(p:agent-skill:53)')], + ), + ); + drive( + reader, + events( + ['server-intent', unnamed], + ['put-object', putFlag()], + ['payload-transferred', transferred('(p:env-flags:12)')], + ), + ); + expect(held.get('pdf-extraction', null)).not.toBeNull(); + expect(reader.diagnostics.payloadsIgnored).toBe(1); + }); + + it('applies an unidentifiable payload rather than withholding it', () => { + // A transfer naming no payload at all is the store's own, since delivery + // sends it one payload. Withholding it would break the common case to + // defend against a hypothetical one. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, skillPayload([['put-object', putSkill()]])); + drive( + reader, + events( + ['server-intent', { payloads: [{ intentCode: 'xfer-full' }] }], + ['put-object', putSkill('pdf-extraction', { objectVersion: 4 })], + ['payload-transferred', { version: 44 }], + ), + ); + expect(held.get('pdf-extraction', null)?.version).toBe(4); + expect(reader.diagnostics.payloadsIgnored).toBe(0); + }); + + it('leaves the first transfer of a connection as the residual', () => { + // Before a skill has arrived there is nothing to compare a payload + // against, so another payload's `xfer-full` arriving first cannot be told + // apart. The multiple-payload warning is the only signal there is, which + // is why it exists. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive( + reader, + events( + [ + 'server-intent', + { + payloads: [ + { id: 'env-flags', intentCode: 'xfer-full' }, + { id: 'agent-skill', intentCode: 'xfer-full' }, + ], + }, + ], + ['put-object', putFlag()], + ['payload-transferred', transferred()], + ), + ); + expect(held.size).toBe(0); + expect(payloadWarnings('described 2 payloads')).toHaveLength(1); + }); +}); + // ─── The held object set ───────────────────────────────────────────────────── describe('the held object set', () => { diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 917282a5..870ad7b2 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -113,6 +113,16 @@ const INTENT_TRANSFER_NONE = 'none'; */ const ENVELOPE_FIELDS = ['contentType', 'content', 'contentHash', 'name', 'description'] as const; +/** + * The payload identity inside a transfer's selector, `(p::)`. + * + * The selector is the only place a completed transfer names its own payload: + * `put-object`, `delete-object` and `payload-transferred` carry no payload id of + * their own. `ProtocolReader` reads it as a fallback for an intent that named no + * `id`. + */ +const PAYLOAD_SELECTOR = /\(p:([^:()]+):\d+\)/; + export type FDv2Mode = 'stream' | 'poll'; const MOBILE_KEY_PREFIX = 'mob-'; @@ -211,6 +221,11 @@ export type StoreDiagnostics = { readonly objectsIgnored: number; /** `delete-object` events applied to skills. */ readonly objectsRevoked: number; + /** + * Transfers not applied because they completed a payload other than the one + * skills arrive on. Zero while delivery sends one payload per connection. + */ + readonly payloadsIgnored: number; /** * Skill objects whose envelope carried no `contentHash`. * @@ -387,6 +402,20 @@ export function seamObjectFromPut(data: Record): RawSkillObject return raw; } +/** The payload id one payload intent names, when it names a usable one. */ +function payloadIdOf(intent: unknown): string | null { + if (typeof intent !== 'object' || intent === null) return null; + const { id } = intent as { id?: unknown }; + return typeof id === 'string' && id !== '' ? id : null; +} + +/** The payload id inside a transfer's selector, when it carries one. */ +function payloadIdFromSelector(state: unknown): string | null { + if (typeof state !== 'string') return null; + const match = PAYLOAD_SELECTOR.exec(state); + return match ? match[1] : null; +} + /** * Narrows one FDv2 skill `delete-object` to the identity it revokes, reading the * wire `key` the same way a put does. @@ -564,6 +593,7 @@ function freshDiagnostics(): MutableDiagnostics { skillObjectsReceived: 0, objectsIgnored: 0, objectsRevoked: 0, + payloadsIgnored: 0, hashlessObjects: 0, connectionFailures: 0, lastError: null, @@ -584,12 +614,32 @@ function freshDiagnostics(): MutableDiagnostics { * which, with pruning on, is the difference between a reconcile and deleting a * customer's skill files. Listeners therefore fire once per commit, not once per * object, which is also exactly the granularity the re-reconcile wants. + * + * **The first payload intent is read, and is assumed to be the skill payload.** + * Delivery provides one payload per credential and the protocol requires a client + * to ignore all but the first payload intent, so `payloads[0]` is both what + * arrives and what the protocol says to read. If that ever widens, an `xfer-full` + * for somebody else's payload would empty the skill set and the next + * `payload-transferred` would publish it empty — with pruning on, the difference + * between a reconcile and deleting a customer's files. This layer therefore learns + * which payload skills arrive on and declines to apply a transfer of any other, + * once at warning level and counted. The residual is the first transfer of a + * connection: before a skill has arrived there is nothing to compare a payload + * against. */ export class ProtocolReader { readonly diagnostics = freshDiagnostics(); private intent: string | null = null; private pending: SkillObjectSet | null = null; private changes: RawSkillObject[] = []; + // The payload the current intent describes, and the payload skills have + // actually arrived on. One payload per connection makes these the same + // payload; the class docstring says why they are kept apart regardless. + private intentPayloadId: string | null = null; + private skillPayloadId: string | null = null; + private skillsInPayload = 0; + private warnedMultiplePayloads = false; + private warnedForeignPayload = false; constructor(private readonly committed: SkillObjectSet) {} @@ -620,10 +670,14 @@ export class ProtocolReader { if (!Array.isArray(payloads) || payloads.length === 0) { return { disconnect: 'server-intent carried no payload description' }; } + if (payloads.length > 1) this.warnMultiplePayloads(payloads); + // The first payload only, as the protocol requires. const first = payloads[0] as { intentCode?: unknown } | null; const intent = typeof first?.intentCode === 'string' ? first.intentCode : null; this.intent = intent; + this.intentPayloadId = payloadIdOf(first); this.changes = []; + this.skillsInPayload = 0; if (intent === INTENT_TRANSFER_FULL) { // A fresh set: the payload about to arrive replaces everything held. Built // alongside the live set rather than in place, so an interrupted transfer @@ -665,6 +719,7 @@ export class ProtocolReader { target.put(raw); this.changes.push(raw); this.diagnostics.skillObjectsReceived += 1; + this.skillsInPayload += 1; if (typeof raw.contentHash !== 'string') { this.diagnostics.hashlessObjects += 1; warnHashless(raw); @@ -685,6 +740,8 @@ export class ProtocolReader { if (tombstone === null) return {}; target.delete(tombstone); this.diagnostics.objectsRevoked += 1; + // A revocation identifies the payload as ours just as a put does. + this.skillsInPayload += 1; // A tombstone, not a skill object: it carries identity and no content, so a // listener that only needs "something changed" works unchanged while one that // reads content sees no `content` field. Documented on `addListener`. @@ -694,12 +751,25 @@ export class ProtocolReader { private payloadTransferred(data: unknown): TransferOutcome { const state = (data as { state?: unknown } | null)?.state; - if (this.pending !== null) { + const payloadId = this.intentPayloadId ?? payloadIdFromSelector(state); + if (this.pending !== null && this.isForeignPayload(payloadId)) { + this.warnForeignPayload(payloadId); + this.diagnostics.payloadsIgnored += 1; + this.changes = []; + } else if (this.pending !== null) { this.committed.replaceWith(this.pending); warnIfNothingCanVerify(this.committed.allRaw()); + if (this.skillsInPayload > 0 && payloadId !== null) { + // Learnt, not configured: nothing below the seam is told which payload + // is which, so the payload that carried a skill put or revocation is + // the payload skills arrive on. + this.skillPayloadId = payloadId; + } } this.pending = null; this.intent = null; + this.intentPayloadId = null; + this.skillsInPayload = 0; const { changes } = this; this.changes = []; this.diagnostics.payloadsTransferred += 1; @@ -710,20 +780,24 @@ export class ProtocolReader { }; } - private error(data: unknown): TransferOutcome { - const reason = (data as { reason?: unknown } | null)?.reason; - // An error abandons the in-flight payload and keeps what is committed. + /** Drops the in-flight payload and keeps what is committed. */ + private abandonInFlight(): void { this.pending = null; this.intent = null; + this.intentPayloadId = null; + this.skillsInPayload = 0; this.changes = []; + } + + private error(data: unknown): TransferOutcome { + const reason = (data as { reason?: unknown } | null)?.reason; + this.abandonInFlight(); return { disconnect: `server sent error: ${String(reason)}` }; } private goodbye(data: unknown): TransferOutcome { const parsed = (data ?? {}) as { reason?: unknown; silent?: unknown; catastrophe?: unknown }; - this.pending = null; - this.intent = null; - this.changes = []; + this.abandonInFlight(); if (parsed.silent !== true) { warn(`FDv2 connection closing: ${String(parsed.reason)}`); } @@ -732,6 +806,50 @@ export class ProtocolReader { } return { disconnect: `server said goodbye: ${String(parsed.reason)}` }; } + + // -- payload identity ---------------------------------------------------- + + /** + * Whether a transfer completes a payload other than the one skills arrive on. + * + * `false` unless both payloads are known, so one-payload delivery and the + * first transfer of a connection behave exactly as they did before this check + * existed. + */ + private isForeignPayload(payloadId: string | null): boolean { + return this.skillPayloadId !== null && payloadId !== null && payloadId !== this.skillPayloadId; + } + + /** + * One warning per reader for an intent describing more than one payload. + * + * Not an error: reading only the first is what the protocol asks for. But it + * means the first payload is no longer *guaranteed* to be the skill payload, + * and an intent for another payload arriving before any skill has been seen is + * the one case `isForeignPayload` cannot catch. + */ + private warnMultiplePayloads(payloads: unknown[]): void { + if (this.warnedMultiplePayloads) return; + this.warnedMultiplePayloads = true; + warn( + `An FDv2 server-intent described ${payloads.length} payloads (${payloads + .map((p) => String(payloadIdOf(p))) + .join(', ')}). Only the first is read, as the protocol requires, and it is taken to be the payload skills ` + + 'arrive on. If skills stop resolving from this point, that is the assumption that broke; contact ' + + 'LaunchDarkly support.', + ); + } + + /** One warning per reader for a transfer this layer declined to apply. */ + private warnForeignPayload(payloadId: string | null): void { + if (this.warnedForeignPayload) return; + this.warnedForeignPayload = true; + warn( + `An FDv2 transfer of payload ${String(payloadId)} was not applied to the skills held, which arrive on ` + + `payload ${String(this.skillPayloadId)}. Applying it would have replaced them with whatever that payload ` + + 'carried — nothing, in the case of a flag payload. The skills held are unchanged.', + ); + } } // --------------------------------------------------------------------------- From 3aa45a87b541bd3e7c9c547f15dec8084d611694 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 11 Sep 2026 16:20:31 -0400 Subject: [PATCH 12/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20keep=20FDv2=20delivery=20alive=20through=20the=20failures=20?= =?UTF-8?q?a=20live=20stream=20actually=20has?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four fixes to the delivery loop, each of which stopped delivery — updates and revocations alike — for the process lifetime under an ordinary condition: - The consecutive-failure counter was reset only after `streamOnce` returned, and a stream never returns normally: it ends by being dropped. Every healthy, server-recycled connection therefore counted as a failure and a healthy server was given up on after maxConsecutiveFailures + 1 recycles. The count now resets at each committed payload inside `apply`, and still after a successful poll. - Only the connect was wrapped as recoverable, so a read timeout, reset or truncated chunk in the body reached the loop as whatever `fetch` threw and was read as a bug. `iterSse` now wraps the read itself — and only the read, so a protocol-reader or dispatch error still surfaces as the bug it is — and cancels the body on exit so an early stop does not leave a socket behind the reconnect. - `Retry-After` was honoured verbatim. The header may come from a proxy rather than LaunchDarkly, and a value in the hours would park revocation for that long; every retry delay is now clamped to `maxBackoffMs`. - There was no read timeout at all, so a stream that went quiet hung forever. `readTimeoutMs` is the one network timeout: in `'poll'` mode it bounds the whole request (default 10s), in `'stream'` mode the gap between reads (default 300s, well past LaunchDarkly's heartbeat). It is implemented as a `ReadDeadline` composed with the store's own abort signal, so closing the store during a connect or a read is still one abort, which the tests now assert against a host that accepts and never answers. Ports python-ai-sdk b6a25f9's loop fixes and c285ff5. Co-Authored-By: Claude Fable 5.1 --- .../client/src/__tests__/skills-fdv2.test.ts | 293 +++++++++++++++++- packages/client/src/skills-fdv2.ts | 223 +++++++++++-- 2 files changed, 476 insertions(+), 40 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index a328f56d..3d2a8746 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -20,6 +20,7 @@ import { createHash } from 'node:crypto'; import { existsSync, readFileSync } from 'node:fs'; import { mkdtemp, readFile, rm } from 'node:fs/promises'; import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'; +import { createServer as createTcpServer, type Socket, type Server as TcpServer } from 'node:net'; import { tmpdir } from 'node:os'; import path from 'node:path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; @@ -30,11 +31,15 @@ import { _warnedHashless, backoffDelayMs, classifyStatus, + DEFAULT_POLL_TIMEOUT_MS, + DEFAULT_STREAM_READ_TIMEOUT_MS, decodePollBody, FDV2_KEY_DELIMITER, FDV2_OBJECT_KIND, FDv2SkillStore, + FetchRequester, isSkillEvent, + type PollResult, ProtocolReader, RecoverableTransportError, type Requester, @@ -147,6 +152,7 @@ type RecordedRequest = { class FakeFDv2Endpoint { readonly requests: RecordedRequest[] = []; holdStreamOpen = false; + dropStreams = false; private readonly polls: Array<{ status: number; events: WireEvent[]; @@ -220,9 +226,17 @@ class FakeFDv2Endpoint { private serveStream(res: ServerResponse): void { const payloadEvents = this.streams.shift() ?? []; res.writeHead(200, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache' }); - for (const event of payloadEvents) { - res.write(`event: ${event.event}\ndata: ${JSON.stringify(event.data ?? null)}\n\n`); + const body = payloadEvents + .map((event) => `event: ${event.event}\ndata: ${JSON.stringify(event.data ?? null)}\n\n`) + .join(''); + if (this.dropStreams) { + // Kills the socket once the events have been flushed, rather than ending + // the chunked response cleanly: how a live stream actually dies, as a + // reset or a truncated chunk, not as an orderly end of body. + res.write(body, () => res.socket?.destroy()); + return; } + res.write(body); if (this.holdStreamOpen) { // Held so a test can assert on the store's state without racing the // reconnect path; released on `close`. @@ -1151,31 +1165,108 @@ describe('streaming against the endpoint', () => { await store.close(); expect(store.failed).toBeNull(); }); + + it('reconnects when the stream dies mid-body', async () => { + // A stream fails in its body far more often than at its connect. Treating + // such a failure as unexpected would stop delivery — including revocation + // — for the process lifetime the first time a socket died. + endpoint.dropStreams = true; + for (let i = 1; i <= 5; i += 1) { + endpoint.queueStream(fullPayload([['put-object', putSkill()]], `basis-${i}`)); + } + const store = streamStore({ maxConsecutiveFailures: 3 }); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(await waitUntil(() => endpoint.requests.length >= 5)).toBe(true); + expect(store.failed).toBeNull(); + expect(store.diagnostics.payloadsTransferred).toBeGreaterThanOrEqual(4); + }); + + it('reconnects when the stream goes quiet past readTimeoutMs', async () => { + // `readTimeoutMs` exists to bound a stream that has gone quiet so the loop + // can reconnect; tripping it must do that and not the opposite. + endpoint.holdStreamOpen = true; + endpoint.queueStream(fullPayload([['put-object', putSkill()]])); + endpoint.queueStream(fullPayload([['put-object', putSkill()]], 'basis-2')); + const store = streamStore({ readTimeoutMs: 100 }); + store.start(); + expect(await waitUntil(() => endpoint.requests.length >= 2)).toBe(true); + expect(store.failed).toBeNull(); + expect(await waitUntil(() => (store.diagnostics.lastError ?? '').includes('timed out'))).toBe(true); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + }); }); // ─── Failure handling ──────────────────────────────────────────────────────── -/** Throws a scripted sequence, so backoff is asserted without real sockets. */ +/** + * Plays a scripted sequence, so backoff is asserted without real sockets. + * + * Each outcome is an `Error` to throw, an array of `[event, data]` pairs to + * deliver (a stream that then ends, or one poll's events), or a promise to await + * as-is. Exhausted, it fails recoverably on every call. + */ class ScriptedRequester implements Requester { readonly calls: Array<[string | null, string | null]> = []; constructor(private readonly outcomes: unknown[] = []) {} - private next(): unknown { - return this.outcomes.length > 0 ? this.outcomes.shift() : new RecoverableTransportError('scripted failure'); + private async next(): Promise> { + const outcome = this.outcomes.length > 0 ? this.outcomes.shift() : new RecoverableTransportError('x'); + if (outcome instanceof Error) throw outcome; + if (outcome instanceof Promise) return outcome as Promise>; + return outcome as Array<[string, unknown]>; } - async poll(basis: string | null, etag: string | null): Promise { + async poll(basis: string | null, etag: string | null): Promise { this.calls.push([basis, etag]); - throw this.next(); + return { notModified: false, events: await this.next(), etag: null }; } - async stream(basis: string | null): Promise { + async stream(basis: string | null): Promise> { this.calls.push([basis, null]); - throw this.next(); + const scripted = await this.next(); + return (async function* () { + yield* scripted; + })(); + } +} + +const asPairs = (wire: WireEvent[]): Array<[string, unknown]> => wire.map((e) => [e.event, e.data]); + +/** + * A healthy server that recycles connections: every `stream` call succeeds, + * transfers a full payload, and then ends the connection, as LaunchDarkly and + * any proxy in between do to a long-lived stream. + */ +class RecyclingRequester implements Requester { + connections = 0; + + poll(): Promise { + throw new Error('not a polling double'); + } + + async stream(): Promise> { + this.connections += 1; + const scripted = asPairs(fullPayload([['put-object', putSkill()]], `basis-${this.connections}`)); + return (async function* () { + yield* scripted; + })(); } } +function scriptedStreamStore(requester: Requester, options: Record = {}): FDv2SkillStore { + const store = new FDv2SkillStore(SDK_KEY, { + mode: 'stream', + initialBackoffMs: 1, + maxBackoffMs: 2, + requester, + ...options, + }); + openStores.push(store); + return store; +} + describe('failure handling', () => { it('stops on 403 and explains why', async () => { endpoint.queuePoll([], { status: 403 }); @@ -1245,6 +1336,53 @@ describe('failure handling', () => { openStores.push(store); store.start(); expect(await waitUntil(() => store.failed !== null)).toBe(true); + // Four, not three: the bound is the number of failures *tolerated*, so the + // run that exceeds it is the one that gives up. + expect(store.failed).toContain('gave up after 4 consecutive failures'); + }); + + it('does not count recycled stream connections as failures', async () => { + // A streaming connection only ever ends by being dropped, so a loop that + // counted every drop as a failure would give up on a healthy server after + // maxConsecutiveFailures + 1 recycles, and delivery (including revocation) + // would silently stop for the process lifetime. + const requester = new RecyclingRequester(); + const store = scriptedStreamStore(requester, { maxConsecutiveFailures: 3 }); + store.start(); + expect(await waitUntil(() => requester.connections >= 8)).toBe(true); + expect(store.failed).toBeNull(); + expect(store.diagnostics.payloadsTransferred).toBeGreaterThanOrEqual(8); + // A drop is a failure until the next commit clears it, so the count may + // read 1 mid-reconnect. What it must never do is climb. + expect(store.diagnostics.connectionFailures).toBeLessThanOrEqual(1); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + }); + + it('resets the failure count on a stream commit', async () => { + const requester = new ScriptedRequester([ + new RecoverableTransportError('x'), + new RecoverableTransportError('x'), + new RecoverableTransportError('x'), + asPairs(fullPayload([['put-object', putSkill()]])), + ]); + const store = scriptedStreamStore(requester, { maxConsecutiveFailures: 3 }); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + // Three failures reach the bound, then a commit, then the exhausted + // requester fails on every reconnect. The count must start again at the + // commit: the stream's own drop is failure one, and three more connects are + // owed before giving up. Carrying the three over would give up on the drop + // itself, with no further connect at all. + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toContain('gave up after 4 consecutive failures'); + expect(store.failed).toContain('last error: x'); + expect(requester.calls).toHaveLength(7); + }); + + it('bounds stream retries', async () => { + const store = scriptedStreamStore(new ScriptedRequester(), { maxConsecutiveFailures: 3 }); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); expect(store.failed).toContain('gave up after 4 consecutive failures'); }); @@ -1257,6 +1395,28 @@ describe('failure handling', () => { expect(await store.waitForSkills(3000)).toBe(true); }); + it('neither dies on nor parks behind an unreasonable Retry-After', async () => { + // `Retry-After` is a request and `maxBackoffMs` is a promise: the header + // may come from a proxy rather than LaunchDarkly, and an hour would park + // revocation for that long. + const requester = new ScriptedRequester([ + new RecoverableTransportError('slow down', 3_600_000), + asPairs(fullPayload([['put-object', putSkill()]])), + ]); + const store = new FDv2SkillStore(SDK_KEY, { + mode: 'poll', + pollIntervalMs: 10_000, + initialBackoffMs: 5_000, + maxBackoffMs: 20, + requester, + }); + openStores.push(store); + store.start(); + expect(await store.waitForSkills(3000)).toBe(true); + expect(requester.calls.length).toBeGreaterThanOrEqual(2); + expect(store.failed).toBeNull(); + }); + it('parses Retry-After into milliseconds', () => { expect(retryAfterMs(new Headers({ 'Retry-After': '2' }))).toBe(2000); expect(retryAfterMs(new Headers({ 'Retry-After': 'Wed, 21 Oct 2026 07:28:00 GMT' }))).toBeNull(); @@ -1691,3 +1851,118 @@ describe('lifecycle', () => { expect(store.allObjects(SKILL_OBJECT_KIND)).toEqual({}); }); }); + +// ─── Timeouts ──────────────────────────────────────────────────────────────── + +/** + * A listening socket that accepts connections and never sends a byte. + * + * This is the host `readTimeoutMs` exists for: the TCP handshake completes, so + * nothing fails fast, and then no response ever comes. A request against it can + * only end by timing out, which makes the elapsed time a direct measurement of + * the timeout actually applied. + */ +class BlackHole { + private server!: TcpServer; + private readonly accepted: Socket[] = []; + baseUri = ''; + + async listen(): Promise { + this.server = createTcpServer((socket) => { + this.accepted.push(socket); + socket.on('error', () => {}); + }); + await new Promise((resolve) => this.server.listen(0, '127.0.0.1', resolve)); + const address = this.server.address(); + const port = typeof address === 'object' && address !== null ? address.port : 0; + this.baseUri = `http://127.0.0.1:${port}`; + } + + async close(): Promise { + for (const socket of this.accepted) socket.destroy(); + await new Promise((resolve) => this.server.close(() => resolve())); + } +} + +const requesterOf = (store: FDv2SkillStore): FetchRequester => + (store as unknown as { requester: FetchRequester }).requester; + +describe('timeouts', () => { + // `readTimeoutMs` is the only network timeout, and every request honours it. + // The bounds asserted here are loose on purpose: the point is that a request + // against an unresponsive host fails in roughly `readTimeoutMs` rather than + // in minutes, and that a regression back to a much longer default fails this + // suite quickly instead of hanging it. + let blackHole: BlackHole; + + beforeEach(async () => { + blackHole = new BlackHole(); + await blackHole.listen(); + }); + + afterEach(async () => { + await blackHole.close(); + }); + + it('fails a poll against an unresponsive host within readTimeoutMs', async () => { + const requester = new FetchRequester(SDK_KEY, blackHole.baseUri, 300); + const started = Date.now(); + await expect(requester.poll(null, null, new AbortController().signal)).rejects.toThrow(/timed out/); + const elapsed = Date.now() - started; + expect(elapsed).toBeGreaterThanOrEqual(200); + expect(elapsed).toBeLessThan(2000); + }); + + it('fails a stream connect against an unresponsive host within readTimeoutMs', async () => { + const requester = new FetchRequester(SDK_KEY, blackHole.baseUri, 300); + const started = Date.now(); + await expect(requester.stream(null, new AbortController().signal)).rejects.toThrow(RecoverableTransportError); + expect(Date.now() - started).toBeLessThan(2000); + }); + + it('reports the timeout and keeps going', async () => { + const store = new FDv2SkillStore(SDK_KEY, { + baseUri: blackHole.baseUri, + mode: 'poll', + pollIntervalMs: 50, + initialBackoffMs: 10, + maxBackoffMs: 50, + readTimeoutMs: 300, + }); + openStores.push(store); + store.start(); + expect(await waitUntil(() => store.diagnostics.connectionFailures >= 1)).toBe(true); + expect(store.failed).toBeNull(); + expect(store.diagnostics.lastError).toContain('timed out'); + }); + + it('returns promptly from close while a connect is pending', async () => { + // Before the connect returns there is no body read to interrupt; aborting + // the signal has to reach the pending `fetch` itself, or close waits on a + // host that will never speak. + const store = new FDv2SkillStore(SDK_KEY, { baseUri: blackHole.baseUri, mode: 'stream', readTimeoutMs: 60_000 }); + store.start(); + await new Promise((resolve) => setTimeout(resolve, 50)); + const started = Date.now(); + await store.close(); + expect(Date.now() - started).toBeLessThan(1000); + expect(store.failed).toBeNull(); + }); + + it('defaults the bound per mode', () => { + expect(DEFAULT_POLL_TIMEOUT_MS).toBe(10_000); + expect(DEFAULT_STREAM_READ_TIMEOUT_MS).toBe(300_000); + expect(requesterOf(new FDv2SkillStore(SDK_KEY, { mode: 'poll' })).readTimeoutMs).toBe(DEFAULT_POLL_TIMEOUT_MS); + expect(requesterOf(new FDv2SkillStore(SDK_KEY, { mode: 'stream' })).readTimeoutMs).toBe( + DEFAULT_STREAM_READ_TIMEOUT_MS, + ); + }); + + it.each(['poll', 'stream'] as const)('lets an explicit readTimeoutMs override the %s default', (mode) => { + expect(requesterOf(new FDv2SkillStore(SDK_KEY, { mode, readTimeoutMs: 42_000 })).readTimeoutMs).toBe(42_000); + }); + + it.each([0, -1, Number.POSITIVE_INFINITY, Number.NaN])('rejects a non-positive readTimeoutMs (%s)', (value) => { + expect(() => new FDv2SkillStore(SDK_KEY, { readTimeoutMs: value })).toThrow(/readTimeoutMs/); + }); +}); diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 870ad7b2..a5414ce3 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -90,6 +90,15 @@ export const DEFAULT_BASE_URI = 'https://sdk.launchdarkly.com'; export const POLL_PATH = '/sdk/poll'; export const STREAM_PATH = '/sdk/stream'; +/** Default `readTimeoutMs` in `'poll'` mode: the bound on one whole request. */ +export const DEFAULT_POLL_TIMEOUT_MS = 10_000; + +/** + * Default `readTimeoutMs` in `'stream'` mode: the longest gap tolerated between + * two reads. LaunchDarkly's heartbeats arrive well inside this. + */ +export const DEFAULT_STREAM_READ_TIMEOUT_MS = 300_000; + const EVENT_SERVER_INTENT = 'server-intent'; const EVENT_PUT_OBJECT = 'put-object'; const EVENT_DELETE_OBJECT = 'delete-object'; @@ -944,14 +953,100 @@ export function decodePollBody(body: string): Array<[string, unknown]> { return events; } +/** + * A signal that aborts when `parent` does, or when `ms` pass without `touch()`. + * + * This is the one network timeout. In `'poll'` mode nothing touches it, so it + * bounds the whole request; in `'stream'` mode every completed read touches it, + * so it bounds the gap between reads. `expired` tells the two abort causes + * apart: a store closing is not a failure, a stream gone quiet is. + */ +export type ReadDeadline = { + readonly signal: AbortSignal; + readonly parent: AbortSignal; + readonly ms: number; + readonly expired: boolean; + /** Restarts the clock. Call after each successful read. */ + touch(): void; + /** Stops the clock and detaches from `parent`. Idempotent. */ + clear(): void; +}; + +export function readDeadline(parent: AbortSignal, ms: number): ReadDeadline { + const controller = new AbortController(); + let timer: ReturnType | null = null; + let expired = false; + const stopTimer = (): void => { + if (timer !== null) clearTimeout(timer); + timer = null; + }; + const onParentAbort = (): void => { + stopTimer(); + controller.abort(parent.reason); + }; + const arm = (): void => { + stopTimer(); + if (controller.signal.aborted) return; + timer = setTimeout(() => { + expired = true; + controller.abort(new Error(`no response within ${ms}ms`)); + }, ms); + // Unreffed for the same reason the backoff timer is: a background store is + // not a reason for `node` to keep running. + (timer as unknown as { unref?: () => void }).unref?.(); + }; + if (parent.aborted) controller.abort(parent.reason); + else parent.addEventListener('abort', onParentAbort, { once: true }); + arm(); + return { + signal: controller.signal, + parent, + ms, + get expired() { + return expired; + }, + touch: arm, + clear: () => { + stopTimer(); + parent.removeEventListener('abort', onParentAbort); + }, + }; +} + +/** + * Presents a failed body read as retryable — unless the store is closing, in + * which case the abort is passed through untouched so the delivery loop reads it + * as the shutdown it is. + * + * A live stream dies mid-body far more often than it refuses to open: a read + * timeout on a stream that went quiet, a reset, a truncated chunk. Each of those + * arrives as whatever `fetch` threw, and the delivery loop retries only the + * transport errors this module defines — anything else it reads as a bug and + * stops for the process lifetime. Connecting is already wrapped in + * `FetchRequester.stream`; this is the same promise for the body. + */ +function readFailure(cause: unknown, what: string, deadline: ReadDeadline | undefined): unknown { + if (deadline?.parent.aborted) return cause; + if (deadline?.expired) return new RecoverableTransportError(`${what} timed out: no bytes in ${deadline.ms}ms`); + return new RecoverableTransportError(`${what} failed: ${cause instanceof Error ? cause.message : String(cause)}`); +} + /** * Decodes an SSE byte stream into `[event name, data]` pairs. * * Minimal on purpose — this consumes one LaunchDarkly endpoint, not the whole * spec: `event:`/`data:` fields, multi-line `data` joined with newlines, a blank * line dispatching, and `:` comments skipped. + * + * Only the read itself is wrapped as recoverable (see {@link readFailure}). + * Whatever the consumer's loop body throws while this generator is suspended at + * a `yield` — a protocol reader bug, a fatal goodbye — passes through `finally` + * untouched and still surfaces as what it is. */ -export async function* iterSse(body: ReadableStream): AsyncGenerator<[string, unknown], void, undefined> { +export async function* iterSse( + body: ReadableStream, + deadline?: ReadDeadline, +): AsyncGenerator<[string, unknown], void, undefined> { const reader = body.getReader(); const decoder = new TextDecoder(); let buffer = ''; @@ -975,7 +1070,14 @@ export async function* iterSse(body: ReadableStream): AsyncGenerator try { for (;;) { - const { done, value } = await reader.read(); + let chunk: ReadableStreamReadResult; + try { + chunk = await reader.read(); + } catch (cause) { + throw readFailure(cause, 'reading the FDv2 stream', deadline); + } + deadline?.touch(); + const { done, value } = chunk; if (done) break; buffer += decoder.decode(value, { stream: true }); let newline = buffer.indexOf('\n'); @@ -997,6 +1099,11 @@ export async function* iterSse(body: ReadableStream): AsyncGenerator } } } finally { + deadline?.clear(); + // Cancelling closes the connection underneath, so a consumer that stops + // early — a goodbye, an error event — does not leave a socket open behind + // the reconnect. Best effort: the stream may already be errored or closed. + reader.cancel().catch(() => {}); try { reader.releaseLock(); } catch { @@ -1017,6 +1124,10 @@ export type Requester = { * Platform globals only, on purpose: this package's runtime dependencies are * `@opentelemetry/api` and `dotenv`, and its LaunchDarkly base-SDK dependency is * an optional peer, so the content path must not smuggle in an HTTP client. + * + * `readTimeoutMs` is applied to every request through a {@link ReadDeadline}: + * connecting, waiting for headers and each body read are all bounded by the + * same value, and there is deliberately no separate connect timeout. */ export class FetchRequester implements Requester { private readonly baseUri: string; @@ -1024,6 +1135,7 @@ export class FetchRequester implements Requester { constructor( private readonly sdkKey: string, baseUri: string, + readonly readTimeoutMs: number, ) { this.baseUri = baseUri.replace(/\/+$/, ''); } @@ -1049,23 +1161,24 @@ export class FetchRequester implements Requester { const headers: Record = { Authorization: this.sdkKey, Accept: 'application/json' }; if (etag) headers['If-None-Match'] = etag; - let response: Response; + // Nothing touches the deadline, so it bounds the whole request: connect, + // headers and body together. + const deadline = readDeadline(signal, this.readTimeoutMs); try { - response = await fetch(this.url(POLL_PATH, basis), { headers, signal }); + const response = await fetch(this.url(POLL_PATH, basis), { headers, signal: deadline.signal }); + if (response.status === 304) return { notModified: true, events: [], etag }; + if (!response.ok) throw classifyStatus(response.status, response.headers); + return { + notModified: false, + events: decodePollBody(await response.text()), + etag: response.headers.get('ETag') ?? etag, + }; } catch (cause) { - if (signal.aborted) throw cause; - throw new RecoverableTransportError( - `polling request failed: ${cause instanceof Error ? cause.message : String(cause)}`, - ); + if (cause instanceof FatalTransportError || cause instanceof RecoverableTransportError) throw cause; + throw readFailure(cause, 'polling request', deadline); + } finally { + deadline.clear(); } - - if (response.status === 304) return { notModified: true, events: [], etag }; - if (!response.ok) throw classifyStatus(response.status, response.headers); - return { - notModified: false, - events: decodePollBody(await response.text()), - etag: response.headers.get('ETag') ?? etag, - }; } /** Opens `GET /sdk/stream` and yields `[event name, data]` pairs. */ @@ -1076,19 +1189,27 @@ export class FetchRequester implements Requester { 'Cache-Control': 'no-cache', }; + // The same deadline bounds the connect and then, touched by `iterSse` on + // every read, the gap between reads. + const deadline = readDeadline(signal, this.readTimeoutMs); let response: Response; try { - response = await fetch(this.url(STREAM_PATH, basis), { headers, signal }); + response = await fetch(this.url(STREAM_PATH, basis), { headers, signal: deadline.signal }); } catch (cause) { - if (signal.aborted) throw cause; - throw new RecoverableTransportError( - `streaming request failed: ${cause instanceof Error ? cause.message : String(cause)}`, - ); + deadline.clear(); + throw readFailure(cause, 'streaming request', deadline); } - if (!response.ok) throw classifyStatus(response.status, response.headers); - if (response.body === null) throw new RecoverableTransportError('the FDv2 stream carried no body'); - return iterSse(response.body); + if (!response.ok) { + deadline.clear(); + throw classifyStatus(response.status, response.headers); + } + if (response.body === null) { + deadline.clear(); + throw new RecoverableTransportError('the FDv2 stream carried no body'); + } + deadline.touch(); + return iterSse(response.body, deadline); } } @@ -1144,12 +1265,26 @@ export type FDv2SkillStoreOptions = { readonly mode?: FDv2Mode; readonly baseUri?: string; readonly pollIntervalMs?: number; + /** + * The only network timeout, in milliseconds. Its meaning and default follow + * the mode: in `'poll'` it bounds the whole request + * ({@link DEFAULT_POLL_TIMEOUT_MS}); in `'stream'` it bounds each wait for the + * next bytes ({@link DEFAULT_STREAM_READ_TIMEOUT_MS}), so a stream that goes + * quiet reconnects instead of hanging. Must be positive and finite when given. + */ + readonly readTimeoutMs?: number; readonly initialBackoffMs?: number; + /** + * Caps every delay between retries, including one the server asks for with + * `Retry-After`. The header may come from a proxy rather than LaunchDarkly, + * and a value in the hours would park revocation for that long. + */ readonly maxBackoffMs?: number; /** * Bounds the retry loop. On exceeding it the transport stops, logs an error, * and the store keeps serving last known good rather than pretending to be - * live — `failed` reports it. + * live — `failed` reports it. Only failures in a row count: a committed + * payload resets the count. */ readonly maxConsecutiveFailures?: number; /** Test seam: a transport double in place of `FetchRequester`. */ @@ -1212,6 +1347,11 @@ export class FDv2SkillStore implements SkillStore { private failedReason: string | null = null; private firstPayload = false; private readonly firstPayloadWaiters: Array<() => void> = []; + // Recoverable failures since the last committed payload. Reset at the commit + // rather than when a connection returns: a stream only ever ends by being + // dropped, so resetting on return would count every healthy, server-recycled + // connection as a failure. + private failures = 0; constructor(sdkKey: string, options: FDv2SkillStoreOptions = {}) { const key = requireServerSideCredential(sdkKey); @@ -1223,10 +1363,15 @@ export class FDv2SkillStore implements SkillStore { if (this.pollIntervalMs <= 0) { throw new Error(`pollIntervalMs must be positive, got ${JSON.stringify(options.pollIntervalMs)}`); } + const readTimeoutMs = + options.readTimeoutMs ?? (this.mode === 'stream' ? DEFAULT_STREAM_READ_TIMEOUT_MS : DEFAULT_POLL_TIMEOUT_MS); + if (typeof readTimeoutMs !== 'number' || !Number.isFinite(readTimeoutMs) || readTimeoutMs <= 0) { + throw new Error(`readTimeoutMs must be positive, got ${String(options.readTimeoutMs)}`); + } this.initialBackoffMs = options.initialBackoffMs ?? 1_000; this.maxBackoffMs = options.maxBackoffMs ?? 30_000; this.maxConsecutiveFailures = options.maxConsecutiveFailures ?? 10; - this.requester = options.requester ?? new FetchRequester(key, options.baseUri ?? DEFAULT_BASE_URI); + this.requester = options.requester ?? new FetchRequester(key, options.baseUri ?? DEFAULT_BASE_URI, readTimeoutMs); } // -- lifecycle --------------------------------------------------------- @@ -1357,13 +1502,14 @@ export class FDv2SkillStore implements SkillStore { // -- the delivery loop ------------------------------------------------- private async run(signal: AbortSignal): Promise { - let failures = 0; while (!signal.aborted) { try { if (this.mode === 'stream') await this.streamOnce(signal); else await this.pollOnce(signal); - failures = 0; - this.reader.diagnostics.connectionFailures = 0; + // A poll that returned is a current answer even when it committed + // nothing (HTTP 304). A stream never returns normally; its successes + // are counted at each commit in `apply`. + this.recordSuccess(); } catch (cause) { if (signal.aborted) return; if (cause instanceof FatalTransportError) { @@ -1374,14 +1520,22 @@ export class FDv2SkillStore implements SkillStore { this.giveUp(`unexpected error in skill delivery: ${cause instanceof Error ? cause.message : String(cause)}`); return; } - failures += 1; + this.failures += 1; + const failures = this.failures; this.reader.diagnostics.connectionFailures = failures; this.reader.diagnostics.lastError = cause.message; if (failures > this.maxConsecutiveFailures) { this.giveUp(`gave up after ${failures} consecutive failures; last error: ${cause.message}`); return; } - const delay = cause.retryAfterMs ?? backoffDelayMs(failures, this.initialBackoffMs, this.maxBackoffMs); + const requested = cause.retryAfterMs; + const delay = Math.min( + requested !== null && Number.isFinite(requested) + ? requested + : backoffDelayMs(failures, this.initialBackoffMs, this.maxBackoffMs), + // `Retry-After` is a request and `maxBackoffMs` is a promise. + this.maxBackoffMs, + ); warn(`Skill delivery failed (${cause.message}); retrying in ${Math.round(delay)}ms`); await sleep(delay, signal); continue; @@ -1391,6 +1545,11 @@ export class FDv2SkillStore implements SkillStore { } } + private recordSuccess(): void { + this.failures = 0; + this.reader.diagnostics.connectionFailures = 0; + } + private giveUp(reason: string): void { this.failedReason = reason; this.reader.diagnostics.lastError = reason; @@ -1407,6 +1566,8 @@ export class FDv2SkillStore implements SkillStore { const outcome = this.reader.handle(name, data); if (outcome.committed) { if (outcome.basis) this.basis = outcome.basis; + // A commit breaks the row of consecutive failures. + this.recordSuccess(); this.markFirstPayload(); if (outcome.changes && outcome.changes.length > 0) this.notify(outcome.changes); } From d4430b9ce6bc87ae50c522cfbd7f43830699cedd Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 11 Sep 2026 16:22:46 -0400 Subject: [PATCH 13/55] =?UTF-8?q?feat(client):=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20removeListener,=20and=20a=20no-store=20message=20th?= =?UTF-8?q?at=20names=20the=20delivery=20store?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `removeListener` joins `addListener` as the optional second half of change notification, on the `SkillStore` seam, on `InMemorySkillStore` and on `FDv2SkillStore`. `SkillWatcher.close` needs it to detach; without it a store held every watcher ever created for the rest of its life, and a closed watcher was still woken by every commit. The watcher probes for it, so a store without it keeps working at the cost of the listener staying registered. Both stores notify over a copy of the listener list, so a removal from inside a listener takes effect from the next commit rather than shifting its neighbours out from under the iteration. The no-store message now names `FDv2SkillStore` first, because it is the answer in production, and drops the sentence saying the delivery transport ships in a follow-up release, since it is in this one. Ports the interface half of python-ai-sdk dd2d55c and b6a25f9. Co-Authored-By: Claude Fable 5.1 --- .../client/src/__tests__/skills-fdv2.test.ts | 64 +++++++++++++++++++ packages/client/src/__tests__/skills.test.ts | 44 +++++++++++++ packages/client/src/skills-core.ts | 14 +++- packages/client/src/skills-fdv2.ts | 18 +++++- packages/client/src/skills-watch.ts | 22 ++++++- packages/client/src/skills.ts | 18 +++++- packages/client/src/types.ts | 14 +++- 7 files changed, 183 insertions(+), 11 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 3d2a8746..841fb68b 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -1806,6 +1806,70 @@ describe('watchSkills', () => { await watcher.close(); } }); + + it('detaches from the store on close, so a closed watcher is no longer notified', async () => { + const store = new InMemorySkillStore(); + store.put({ key: 'a', version: 1, content: 'first', contentHash: hash('first') }); + _setStore(store); + const listeners = (): unknown[] => + (store as unknown as { listeners: Map }).listeners.get(SKILL_OBJECT_KIND) ?? []; + + const root = path.join(await scratchRoot(), 'skills'); + const { watcher } = await watchSkills('*', root, { debounceMs: 20 }); + expect(listeners()).toContain(watcher.notify); + + await watcher.close(); + + expect(listeners()).not.toContain(watcher.notify); + store.put({ key: 'a', version: 4, content: 'second', contentHash: hash('second') }); + await new Promise((resolve) => setTimeout(resolve, 150)); + expect(await readFile(path.join(root, 'a', 'SKILL.md'), 'utf8')).toBe('first'); + expect(watcher.reconciles).toBe(0); + }); + + it('leaves no listeners behind across repeated watchers', async () => { + const store = new InMemorySkillStore(); + _setStore(store); + const listeners = (): unknown[] => + (store as unknown as { listeners: Map }).listeners.get(SKILL_OBJECT_KIND) ?? []; + const root = path.join(await scratchRoot(), 'skills'); + for (let i = 0; i < 5; i += 1) { + const { watcher } = await watchSkills('*', root, { debounceMs: 20 }); + expect(listeners()).toHaveLength(1); + await watcher.close(); + } + expect(listeners()).toEqual([]); + }); + + it('still closes against a store with no removeListener', async () => { + // `removeListener` is optional: an older store keeps working, at the cost + // of the listener staying registered. + const registered: unknown[] = []; + _setStore({ + getObject: () => null, + allObjects: () => ({}), + addListener: (_kind, fn) => { + registered.push(fn); + }, + }); + const { watcher } = await watchSkills('*', path.join(await scratchRoot(), 'skills'), { debounceMs: 20 }); + expect(registered).toEqual([watcher.notify]); + await watcher.close(); + await watcher.close(); + expect(registered).toEqual([watcher.notify]); + }); + + it('treats removing an unregistered listener from the FDv2 store as a no-op', () => { + const store = pollStore(); + const fn = (): void => {}; + store.removeListener(SKILL_OBJECT_KIND, fn); + store.addListener(SKILL_OBJECT_KIND, fn); + store.removeListener('flag', fn); + store.removeListener(SKILL_OBJECT_KIND, fn); + store.removeListener(SKILL_OBJECT_KIND, fn); + const listeners = (store as unknown as { listeners: Map }).listeners; + expect(listeners.get(SKILL_OBJECT_KIND)).toEqual([]); + }); }); // ─── Lifecycle ─────────────────────────────────────────────────────────────── diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index c27489c5..5cb916a6 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -420,6 +420,37 @@ describe('InMemorySkillStore', () => { expect(other).not.toHaveBeenCalled(); }); + it('stops notifying a listener once removed', () => { + const store = new InMemorySkillStore(); + const seen = vi.fn(); + store.addListener('skill', seen); + store.removeListener('skill', seen); + store.put(rawSkill({ key: 'a' })); + expect(seen).not.toHaveBeenCalled(); + }); + + it('removes one occurrence per removeListener call', () => { + const store = new InMemorySkillStore(); + const seen = vi.fn(); + store.addListener('skill', seen); + store.addListener('skill', seen); + store.removeListener('skill', seen); + store.put(rawSkill({ key: 'a' })); + expect(seen).toHaveBeenCalledTimes(1); + }); + + it('treats removing an unregistered listener as a no-op', () => { + const store = new InMemorySkillStore(); + const fn = vi.fn(); + store.removeListener('skill', fn); + store.addListener('skill', fn); + store.removeListener('flag', fn); + store.removeListener('skill', fn); + store.removeListener('skill', fn); + store.put(rawSkill({ key: 'a' })); + expect(fn).not.toHaveBeenCalled(); + }); + it('rejects a raw object with no string key', () => { expect(() => new InMemorySkillStore().put({ version: 1 } as RawSkillObject)).toThrow(); }); @@ -456,6 +487,19 @@ describe('store configuration', () => { await expect(getSkill('a')).rejects.toThrow(/InMemorySkillStore/); }); + it('the no-store message names the delivery store first', async () => { + // A deployment that hits this message must be pointed at the store that + // receives content from LaunchDarkly, not only at the development one. + const message = await getSkill('a').then( + () => '', + (cause: Error) => cause.message, + ); + expect(message).toContain('FDv2SkillStore'); + expect(message).toContain('InMemorySkillStore'); + expect(message.indexOf('FDv2SkillStore')).toBeLessThan(message.indexOf('InMemorySkillStore')); + expect(message).not.toContain('follow-up release'); + }); + it('shutdown clears the store', async () => { const store = new InMemorySkillStore(); store.put(rawSkill({ key: 'a' })); diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index ff76df75..6640a471 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -91,11 +91,19 @@ export type IntegrityReasonCode = | 'not_utf8' | 'over_size_cap'; +/** + * The first thing a user sees when no store is configured, so it names both + * stores. + * + * `FDv2SkillStore` comes first because it is the answer in production, and a + * message that offered only `InMemorySkillStore` would point a deployment at the + * development store. Callers match on "skill store"; keep that phrase if the + * wording changes. + */ export const NO_STORE_MESSAGE = 'No skill store is configured, so skill content cannot be retrieved. Configure one with ' + - 'initClient({ skillStore: store }) — InMemorySkillStore is available for local development ' + - 'and testing. Retrieving LaunchDarkly-delivered skill content additionally requires the ' + - 'delivery transport, which ships in a follow-up release.'; + 'initClient({ skillStore: store }) — FDv2SkillStore receives content from LaunchDarkly, and ' + + 'InMemorySkillStore is available for local development and testing.'; // --------------------------------------------------------------------------- // Telemetry seam diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index a5414ce3..7c7589ce 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1482,8 +1482,24 @@ export class FDv2SkillStore implements SkillStore { else this.listeners.set(kind, [fn]); } + /** + * Unregisters `fn` from `kind`. Safe to call from inside a listener: a removal + * during one commit takes effect from the next. + * + * Removes one occurrence; removing a callable that is not registered is a + * no-op, so `SkillWatcher.close` can detach unconditionally. + */ + removeListener(kind: string, fn: (raw: RawSkillObject) => unknown): void { + const listeners = this.listeners.get(kind); + if (!listeners) return; + const index = listeners.indexOf(fn); + if (index !== -1) listeners.splice(index, 1); + } + private notify(changes: RawSkillObject[]): void { - const listeners = this.listeners.get(SKILL_OBJECT_KIND) ?? []; + // A copy, so a listener removed mid-commit does not shift its neighbours + // out from under the iteration. + const listeners = [...(this.listeners.get(SKILL_OBJECT_KIND) ?? [])]; for (const raw of changes) { for (const listener of listeners) { try { diff --git a/packages/client/src/skills-watch.ts b/packages/client/src/skills-watch.ts index d2fa26fe..232cfa74 100644 --- a/packages/client/src/skills-watch.ts +++ b/packages/client/src/skills-watch.ts @@ -26,7 +26,7 @@ import { getStore, SKILL_OBJECT_KIND } from './skills-core.js'; import { type WriteSkillsOptions, writeSkills } from './skills-fs.js'; -import type { ReconcileReport, Skill, SkillReference } from './types.js'; +import type { ReconcileReport, Skill, SkillReference, SkillStore } from './types.js'; /** * How long a change waits for its neighbours before a reconcile runs. @@ -70,6 +70,7 @@ export class SkillWatcher { private completed = 0; constructor( + private readonly store: SkillStore, private readonly request: ReadonlyArray | '*', private readonly root: string, private readonly options: WriteSkillsOptions, @@ -151,8 +152,14 @@ export class SkillWatcher { * interrupted between its content writes and its manifest rewrite is the one * case the manifest format has to recover from — worth avoiding when we control * the timing. + * + * Detaches `notify` from the store first, so no further change reaches a + * watcher that is shutting down and the store no longer holds a reference to + * it. A store without the optional `removeListener` is left as it is rather + * than failing the close. */ async close(): Promise { + this.detach(); this.closed = true; if (this.timer !== null) { clearTimeout(this.timer); @@ -160,6 +167,13 @@ export class SkillWatcher { } await this.running; } + + private detach(): void { + if (this.closed) return; + if (typeof this.store.removeListener === 'function') { + this.store.removeListener(SKILL_OBJECT_KIND, this.notify); + } + } } /** @@ -187,7 +201,9 @@ export class SkillWatcher { * Throws when no store is configured, and when the configured store has no * `addListener` — the second case failing loudly rather than degrading to a * one-shot reconcile, because a watcher that silently never fires looks exactly - * like a watcher whose skills never changed. + * like a watcher whose skills never changed. The optional `removeListener` lets + * `SkillWatcher.close` detach; a store without it keeps working, at the cost of + * a listener that lives as long as the store does. */ export async function watchSkills( skills: ReadonlyArray | '*', @@ -215,7 +231,7 @@ export async function watchSkills( // and a bad root throws out of `watchSkills` rather than into a log line. const report = await writeSkills(skills, root, writeOptions); - const watcher = new SkillWatcher(skills, root, writeOptions, debounceMs, onReconcile); + const watcher = new SkillWatcher(store, skills, root, writeOptions, debounceMs, onReconcile); store.addListener(SKILL_OBJECT_KIND, watcher.notify); return { report, watcher }; } diff --git a/packages/client/src/skills.ts b/packages/client/src/skills.ts index 2d090918..8d189f43 100644 --- a/packages/client/src/skills.ts +++ b/packages/client/src/skills.ts @@ -92,7 +92,9 @@ export class InMemorySkillStore implements SkillStore { const { key } = raw; if (typeof key !== 'string') throw new Error("a raw skill object must carry a string 'key'"); this.objects[key] = raw; - for (const listener of this.listeners.get(SKILL_OBJECT_KIND) ?? []) listener(raw); + // A copy, so a listener that removes itself mid-notification does not + // shift its neighbours out from under the iteration. + for (const listener of [...(this.listeners.get(SKILL_OBJECT_KIND) ?? [])]) listener(raw); } /** @@ -130,6 +132,20 @@ export class InMemorySkillStore implements SkillStore { if (existing) existing.push(fn); else this.listeners.set(kind, [fn]); } + + /** + * Unregisters `fn` from `kind`, so a subsequent `put` no longer calls it. + * + * Removes one occurrence: a callable registered twice must be removed twice. + * Removing a callable that is not registered is a no-op, not an error, so a + * consumer that detaches on close can do so unconditionally. + */ + removeListener(kind: string, fn: (raw: RawSkillObject) => unknown): void { + const listeners = this.listeners.get(kind); + if (!listeners) return; + const index = listeners.indexOf(fn); + if (index !== -1) listeners.splice(index, 1); + } } // --------------------------------------------------------------------------- diff --git a/packages/client/src/types.ts b/packages/client/src/types.ts index 381d8d50..67c564b8 100644 --- a/packages/client/src/types.ts +++ b/packages/client/src/types.ts @@ -262,9 +262,16 @@ export type RawSkillObject = { * streams or polls LaunchDarkly's SDK-facing FDv2 channel — drops in behind this * interface without touching the public API, and did. * - * `addListener` is part of the seam but **optional**: a store - * without it must still be accepted. `watchSkills` is its one consumer, and - * refuses loudly rather than degrading when a configured store lacks it. + * `addListener` and `removeListener` are part of the seam but **optional**: a + * store without them must still be accepted. Nothing in the accessors calls + * either — they exist for the delivery transport to push updates through, and + * for a consumer such as `watchSkills` to stop receiving them. `watchSkills` is + * the one consumer: it refuses loudly rather than degrading when a configured + * store lacks `addListener`, and probes for `removeListener` on close, so a + * store without it keeps working at the cost of a listener that lives as long + * as the store does. A store that implements `addListener` should implement + * `removeListener` too; it removes one occurrence of `fn` under `kind` and is a + * no-op when `fn` is not registered. * * Everything a store serves is untrusted input. The transport is not part of the * trust boundary — key, version, size, and content hash are revalidated at the @@ -285,6 +292,7 @@ export type SkillStore = { getObject(kind: string, key: string, version?: number | null): RawSkillObject | null | undefined; allObjects(kind: string): Record; addListener?(kind: string, fn: (raw: RawSkillObject) => unknown): void; + removeListener?(kind: string, fn: (raw: RawSkillObject) => unknown): void; }; /** Builds a frozen {@link SkillReference}. */ From ede868a7963afa75845213507a2b0eb6ccd7f27f Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 11 Sep 2026 16:24:42 -0400 Subject: [PATCH 14/55] =?UTF-8?q?docs(client):=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20describe=20the=20FDv2=20wire=20contract=20as=20it?= =?UTF-8?q?=20ships?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit agents.md described the earlier streamer draft: `objectVersion` beside `version`, `inline-resource` plus `category`, `basis` plus `mv`. It now describes what the channel delivers — the skill's version folded into the wire key, identification by kind alone, no `mv` — and adds the three transport decisions that arrived with the loop fixes: the payload identity guard, the single read timeout, and which failures are recoverable. README's "Receiving skills from LaunchDarkly" gains `readTimeoutMs` with its per-mode default, `removeListener` on both stores and the seam, the `StoreDiagnostics` fields including `payloadsIgnored`, and the 403 wording that matches what the store now logs. Co-Authored-By: Claude Fable 5.1 --- packages/client/README.md | 13 ++++++++----- packages/client/agents.md | 25 ++++++++++++++++--------- 2 files changed, 24 insertions(+), 14 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 63e99a81..60758402 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -325,9 +325,10 @@ example above, materializes only what the resolved variation actually asked for. | `getSkills(refs)` | Batch form. Accepts `SkillReference` values and bare key strings (string = latest). Results follow input order; missing or unverifiable entries are omitted. | | `allSkills()` | Every verified skill the store holds. | | `writeSkills(skills, root, options?)` | Materialize to `//SKILL.md`. Accepts `Skill` / `SkillReference` / key strings, or the literal `'*'`. Returns a `ReconcileReport`. Throws for a caller error — an unusable `root`, a bare string other than `'*'` — as distinct from the per-skill `error` actions in the report. | -| `InMemorySkillStore` | A `SkillStore` backed by a plain object. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`. Holds one object per key, so it answers a version pin with what it has and lets the accessor refuse a mismatch. | -| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | -| `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | +| `InMemorySkillStore` | A `SkillStore` backed by a plain object. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Holds one object per key, so it answers a version pin with what it has and lets the accessor refuse a mismatch. | +| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | +| `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | +| `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked`, `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | #### Receiving skills from LaunchDarkly @@ -356,9 +357,11 @@ try { **Streaming is the default, and it is what makes revocation fast.** A `delete-object` reaches a live stream in seconds; with `mode: 'poll'` it arrives within one `pollIntervalMs`. Paired with `watchSkills`, a revoked skill's `SKILL.md` leaves the disk without a restart. During an outage the store keeps serving the last content it received and `writeSkills`' default `onUnavailable: 'keep'` leaves managed files alone — an outage must not read as "everything was revoked". +**One network timeout, and its default depends on the mode.** `readTimeoutMs` bounds every step of a request, connecting included. In `mode: 'poll'` it bounds the whole request and defaults to 10 seconds; in `mode: 'stream'` it bounds each wait for the next bytes and defaults to 300 seconds, well beyond LaunchDarkly's heartbeat interval. A stream that goes quiet past it reconnects rather than hanging, and a stream that dies mid-body — a reset, a truncated chunk — reconnects the same way. Every delay between retries, including one the server asks for with `Retry-After`, is capped at `maxBackoffMs`. + **The connection also carries your flags.** A client cannot request only the skill payload, so a skills-enabled environment delivers flag and segment objects on the same connection. They are skipped, not evaluated — this store does no evaluation of any kind — and `diagnostics.objectsIgnored` counts them. -> **Beta caveats, worth knowing before you deploy.** Payload signing does not exist on this channel yet, so delivery is TLS-only and the content hash establishes self-consistency, not origin authenticity. FDv2 is opt-in per account: without it the endpoints return HTTP 403, which the store reports as a fatal error naming the setting. `ld-relay` does not speak the FDv2 endpoints, so relay-only deployments cannot receive skills. +> **Beta caveats, worth knowing before you deploy.** Payload signing does not exist on this channel yet, so delivery is TLS-only and the content hash establishes self-consistency, not origin authenticity. The FDv2 protocol is opt-in per account: without it the endpoints return HTTP 403, which the store reports as a fatal error explaining what to do. `ld-relay` does not speak the FDv2 endpoints, so relay-only deployments cannot receive skills. **If every skill comes back empty, check `diagnostics.hashlessObjects`.** Verification requires `contentHash` on the delivered object and withholds anything without one, so a nonzero count there means skills are being withheld rather than that the environment has none. The store logs an error per hashless object and one summary per wholly-hashless payload, both naming the reason. There is deliberately no fallback that skips verification: a hash the SDK computed from the content it was handed would certify the content against itself. @@ -549,7 +552,7 @@ All types are re-exported from this package. Handler packages import them from h | `SkillReference` | A version-pinned pointer to a skill: `{ key, version }` | | `SkillOutcome` | What `getSkillResult()` resolves to: `{ skill, reason, detail }`. `skill` is non-null exactly when `reason` is `'ok'` | | `SkillOutcomeReason` | The closed set of retrieval outcomes: `'absent' \| 'integrity_failure' \| 'ok' \| 'store_unavailable' \| 'wrong_version'` | -| `SkillStore` | The structural seam skill content is retrieved through: `getObject(kind, key, version?)`, `allObjects`, optional `addListener` | +| `SkillStore` | The structural seam skill content is retrieved through: `getObject(kind, key, version?)`, `allObjects`, optional `addListener` / `removeListener` | | `RawSkillObject` | The wire shape a `SkillStore` serves, before verification. Every field is untrusted. | | `ReconcileReport` | The result of `writeSkills()`: `{ actions, ok, errors }` | | `ReconcileAction` | One outcome from a reconcile: `{ key, action, version, path, error }` | diff --git a/packages/client/agents.md b/packages/client/agents.md index dba6568c..5c112fac 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -30,7 +30,7 @@ No other `@launchdarkly/ai-*` package may define or duplicate these. They import | `src/registry.ts` | `Registry`, `globalRegistry`, `compose` | | `src/skills-core.ts` | Agent Skills internals shared by the two layers above it: the store and telemetry seams, module state, integrity verification, store resolution | | `src/skills.ts` | `skillRefs`, the content accessors, `InMemorySkillStore`, and the documented test-injection hooks | -| `src/skills-fdv2.ts` | Agent Skills delivery transport — the FDv2 protocol, the `objectVersion`/`version` translation, the held object set, and `FDv2SkillStore`. Sits **below** the store seam; imports `skills-core` only, and nothing imports it | +| `src/skills-fdv2.ts` | Agent Skills delivery transport — the FDv2 protocol, the wire-key/`version` translation, the held object set, and `FDv2SkillStore`. Sits **below** the store seam; imports `skills-core` only, and nothing imports it | | `src/skills-watch.ts` | Agent Skills eager re-reconcile — `watchSkills` / `SkillWatcher`, wiring the store's change listener to `writeSkills`. Sits **above** `skills-fs` and modifies none of it | | `src/skills-fs.ts` | `writeSkills` — the manifest format, on-disk filenames, and reconcile semantics | | `src/safe-fs.ts` | Symlink-refusing filesystem primitives. Knows nothing about skills; owns the single interceptable rename and unlink call sites | @@ -42,27 +42,34 @@ The Agent Skills modules are a deliberate split, and the dependencies run **one ## Agent Skills — the delivery transport, and the one field that will bite you -`FDv2SkillStore` speaks LaunchDarkly's SDK-facing FDv2 channel (`GET /sdk/poll`, `GET /sdk/stream`, server-side SDK key in `Authorization`, `basis` + `mv` params, `If-None-Match`/304). It lives below the seam and produces raw objects in the shape `SkillStore` documents; **nothing above the seam knows it exists**. The transport design was replaced wholesale late in this feature's life and cost zero changes above this line, which is the strongest evidence the seam is drawn correctly. If a transport change ever seems to require editing an accessor, verification, or `writeSkills`, the adapter boundary is wrong. +`FDv2SkillStore` speaks LaunchDarkly's SDK-facing FDv2 channel (`GET /sdk/poll`, `GET /sdk/stream`, server-side SDK key in `Authorization`, a `basis` param once a payload has committed, `If-None-Match`/304). It lives below the seam and produces raw objects in the shape `SkillStore` documents; **nothing above the seam knows it exists**. The transport design was replaced wholesale late in this feature's life and cost zero changes above this line, which is the strongest evidence the seam is drawn correctly. If a transport change ever seems to require editing an accessor, verification, or `writeSkills`, the adapter boundary is wrong. -**`objectVersion` is the skill's version. `version` is the payload's.** On the wire a skill `put-object` carries both, and they are not interchangeable: +**The skill's version is in the object's `key`. `version` is the payload's.** Each version of a skill is its own object on the wire, identified as `:`: ```json -{"key":"pdf-extraction","kind":"inline-resource","category":"skill", - "objectVersion":3,"version":42, +{"key":"pdf-extraction:3","kind":"skill","version":42, "object":{"contentType":"text/markdown","content":"…","contentHash":"…","name":"…"}} ``` -`objectVersion` (3) is what a `{key, version}` reference pins and what becomes the seam's `version`. `version` (42) is the version of the *payload* the object arrived in — it moves when anything in the environment moves, including a flag with nothing to do with skills. Reading it as the skill's version fails **silently**: the object verifies, the hash matches, and the caller gets content under a version number that means nothing. Flags and segments carry only `version` and omit both `category` and `objectVersion`, which is exactly why the two fields look interchangeable. `seamObjectFromPut` is the only place the translation happens, and the `version translation` suite asserts it in both directions. +The `3` after the delimiter is what a `{key, version}` reference pins and what becomes the seam's `version`, under the seam key `pdf-extraction`. `version` (42) is the version of the *payload* the object arrived in — it moves when anything in the environment moves, including a flag with nothing to do with skills. Reading it as the skill's version fails **silently**: the object verifies, the hash matches, and the caller gets content under a version number that means nothing. There is no separate field for the skill's version: the agent-skill payload is a *generic* payload, and generic objects carry only `key`, `kind`, `version` and `object`, exactly like a flag. `splitWireKey` is the only place the wire key is read, `seamObjectFromPut` and `tombstoneFromDelete` both go through it, and the `version translation` suite asserts the translation in both directions. A wire key that will not split cleanly is *held*, not dropped — version-less, or with the offending text as its version — so verification withholds it with `invalid_version` under a key the caller recognises; only a key with nothing before the delimiter is dropped, since there is no identity to hold it under. -**Skills are identified by `kind === 'inline-resource' && category === 'skill'`; everything else is ignored, not rejected.** An environment's payload assignment carries the flagging payload alongside the agent-skill payload, so flag and segment objects arrive as a matter of course. Throwing on an unrecognised kind is the unknown-kind reconnect loop this feature must not reproduce — a flag-delivery outage caused by a skills rollout. +**Skills are identified by `kind === 'skill'`; everything else is ignored, not rejected.** Object kinds on the SDK-facing channel are open strings, and the agent-skill payload is classified `generic`, so a skill arrives under the kind its producer registered — the bare category name — not under a broader wrapper kind with a narrowing field. An environment's payload assignment carries its flag payload alongside its agent-skill payload, so flag and segment objects arrive as a matter of course. Throwing on an unrecognised kind is the unknown-kind reconnect loop this feature must not reproduce — a flag-delivery outage caused by a skills rollout. + +**No `mv` parameter, deliberately.** It selects the *flag* data model, the connection rejects any value but the flag default, and the generic agent-skill payload is served regardless of it. Sending `mv=1` — the skill payload's own model version — gets the whole connection refused. **Changes commit at `payload-transferred`, not as objects arrive.** A payload version is the unit of consistency: a half-applied full transfer would publish a state the server never described, and would briefly empty the store — which, with pruning on, is the difference between a reconcile and deleting a customer's skill files. An interrupted transfer therefore leaves last known good intact, and listeners fire once per commit. +**The first payload intent is read, and is assumed to be the skill payload.** Delivery provides one payload per credential and the protocol requires a client to ignore all but the first payload intent, so `payloads[0]` is both what arrives and what the protocol says to read. The cost of that assumption is that an `xfer-full` for somebody *else's* payload would start an empty pending set, and the next `payload-transferred` would publish it — every skill reported revoked, and with pruning on, a customer's files deleted. `ProtocolReader` therefore learns which payload skills arrive on, from the intent's `id` or from the `(p::)` selector, and declines to apply a transfer of any other: once at warning level, counted in `diagnostics.payloadsIgnored`, holding last known good. A transfer that names no payload is applied, since one-payload delivery is the common case. The residual is the first transfer of a connection — before a skill has arrived there is nothing to compare against — which is what the separate warning on a multi-payload intent is for. + **A hashless object is held, not dropped.** Verification withholds it with `missing_content_hash`; the transport's job is to make that loud (an error per object, a summary per wholly-hashless payload, `diagnostics.hashlessObjects`) rather than to work around it. Dropping it at the transport would report `absent` — indistinguishable from "no such skill" — and would let a prune delete the last known-good copy on disk. Never synthesize a hash from the delivered content: that certifies the content against itself and verifies nothing. **`SkillObjectSet.snapshot` collapses to one object per key, and that is load-bearing here in a way it is not in Python.** `//SKILL.md` is a single path, so a whole-store consumer must see one object per key or a `'*'` reconcile writes the same path twice and `allSkills` returns two versions of one skill. The Python SDK collapses in `newest_by_key`, above the seam; that helper is not in this SDK yet, so the collapse happens in the adapter. End-to-end behaviour is identical, and `getObject` still resolves a pinned version out of the full set. When `newestByKey` lands in `skills-core.ts`, move it and delete the note on the class. -**`close` aborts the signal, it does not just set a flag.** The delivery task spends its life awaiting a stream read, and a flag it never checks would leave a healthy stream running until the process exited. Every backoff timer and the `waitForSkills` timer are `unref`ed for the same reason: a background store must not be why `node` stays up. +**There is one network timeout, not two.** `readTimeoutMs` is applied through a `ReadDeadline` composed with the store's own abort signal, so connect, headers and each body read share it. Its default is per mode: `DEFAULT_POLL_TIMEOUT_MS` bounds a whole poll request, `DEFAULT_STREAM_READ_TIMEOUT_MS` bounds the gap between reads on a stream, and tripping it on a stream that has gone quiet *reconnects* — the `timeouts` suite measures the bound against a socket that accepts and never answers. Do not add a separate connect timeout; the Python SDK cannot honour one and the two transports are kept option-for-option compatible. + +**A body read that fails is recoverable; a protocol-reader or dispatch error is not.** `iterSse` wraps the read itself — a reset, a truncated chunk, the read deadline — as `RecoverableTransportError`, because a live stream dies mid-body far more often than it refuses to open, and the delivery loop reads anything else as a bug and stops for the process lifetime. Whatever the consumer's loop body throws while the generator is suspended at a `yield` passes through untouched, so a bug still surfaces as one. The consecutive-failure counter resets at each *committed payload* in `apply`, not when a connection returns: a stream only ever ends by being dropped, so resetting on return would count every healthy, server-recycled connection as a failure and give up on a healthy server. Every retry delay, `Retry-After` included, is clamped to `maxBackoffMs`. + +**`close` aborts the signal, it does not just set a flag.** The delivery task spends its life awaiting a stream read, and a flag it never checks would leave a healthy stream running until the process exited. The store's signal is the parent of every `ReadDeadline`, so one abort reaches a pending connect and a pending read alike, and a store closing is passed through rather than reported as a failure. Every backoff timer, deadline timer and the `waitForSkills` timer are `unref`ed for the same reason: a background store must not be why `node` stays up. --- @@ -90,7 +97,7 @@ export { parseUsage, normalizeMode, parseAiConfig } from './tracking.js'; export { skillRefs, getSkill, getSkillResult, getSkills, allSkills, InMemorySkillStore } from './skills.js'; export { SKILL_OBJECT_KIND, MAX_SKILL_CONTENT_BYTES } from './skills-core.js'; export { writeSkills, SKILL_FILENAME, MANIFEST_FILENAME, MANIFEST_VERSION } from './skills-fs.js'; -export { FDv2SkillStore, DEFAULT_BASE_URI, SDK_DATA_MODEL_VERSION } from './skills-fdv2.js'; +export { FDv2SkillStore, DEFAULT_BASE_URI } from './skills-fdv2.js'; export { watchSkills, SkillWatcher, DEFAULT_DEBOUNCE_MS } from './skills-watch.js'; export type { WriteSkillsOptions } from './skills-fs.js'; export { createSkill, createSkillOutcome, createSkillReference } from './types.js'; From 4a5de2bc89fbf1d95e3b2915eaa903f040334e71 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 12:11:03 -0400 Subject: [PATCH 15/55] fix(client): four delivery defects from the Bugbot review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All four are real, and each gets a test that fails without its fix. **Snapshot keys break prune withhold.** `SkillObjectSet.snapshot` keyed its entries `key:version`, but `writeSkills('*')` derives its prune keep-set from the keys `allObjects` is keyed by. `pendingForRaw` could not parse `key:version` as a skill key, so an unverifiable object fell out of the keep-set and prune deleted the last known-good copy — the exact outcome withholding exists to prevent, and one the `contentHash` gap makes universal, since today every object arrives hashless. `snapshot` now keys by the bare skill key (matching `InMemorySkillStore`), and `pendingForRaw` reads the object's own `key` before falling back to the map key, so no store's choice of map key can reach the same end. `allRaw` remains the way to match a held object back to its wire event. **Empty transfer skips listener notify.** A full transfer revokes by omission — whatever it did not carry is gone, and no `delete-object` says so — so a full transfer that dropped every skill committed an empty store with an empty change list and woke nobody, leaving revoked files on disk until the process restarted. `payloadTransferred` now diffs the committed set against the pending one before the swap and reports the departures as tombstones, which also gives listeners that read versions both halves of a version move. **Foreign payload overwrites delivery basis.** A declined transfer returned the foreign selector, which `apply` then wrote into `basis`, asking the next poll or stream to resume from a payload this layer had just thrown away. Skill updates could stop arriving while every diagnostic still read healthy. A declined payload no longer moves the resume point. **Watcher misses updates during initial write.** `watchSkills` registered the change listener after the initial reconcile, so a payload committing during that reconcile's filesystem I/O reached nobody. The listener now goes on first, and the initial reconcile runs through the watcher's own chain via `runInitial`, so a notify arriving mid-run queues behind it rather than interleaving a second reconcile on one root. A failed initial reconcile closes the watcher instead of leaving a listener on a root that never reconciled. Note this touches `skills-fs.ts`, which the PR description describes as untouched: the `pendingForRaw` half of the first fix is a pre-existing weakness in the seam contract that the FDv2 store is simply the first store to expose. Full suite 1337 passing; `tsc --noEmit` and `biome check` clean. Co-Authored-By: Claude Opus 5 --- .../client/src/__tests__/skills-fdv2.test.ts | 195 +++++++++++++++++- packages/client/src/skills-fdv2.ts | 60 +++++- packages/client/src/skills-fs.ts | 16 +- packages/client/src/skills-watch.ts | 49 ++++- 4 files changed, 304 insertions(+), 16 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 841fb68b..ec9ae6be 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -49,6 +49,7 @@ import { splitWireKey, tombstoneFromDelete, } from '../skills-fdv2.js'; +import { writeSkills } from '../skills-fs.js'; import { watchSkills } from '../skills-watch.js'; import type { RawSkillObject } from '../types.js'; @@ -469,13 +470,17 @@ describe('version translation', () => { expect(splitWireKey(3)).toBeNull(); }); - it('round-trips the stored identity to the wire key', () => { - // `SkillObjectSet.snapshot` spells its opaque keys the way the wire does, so - // a held object can be matched back to the event that carried it. + it('keys the snapshot by the skill key, not the wire key', () => { + // `writeSkills('*')` builds its prune keep-set from these keys, so a + // `key:version` spelling here would make every unverifiable object fall out + // of the keep-set and take the copy already on disk with it. `allRaw` is + // what matches a held object back to the event that carried it. const held = new SkillObjectSet(); const wire = putSkill('pdf-extraction', { objectVersion: 3 }); held.put(seamObjectFromPut(wire) as RawSkillObject); - expect(Object.keys(held.snapshot())).toEqual([wire.key]); + expect(wire.key).toBe('pdf-extraction:3'); + expect(Object.keys(held.snapshot())).toEqual(['pdf-extraction']); + expect(held.allRaw()).toEqual([{ ...seamObjectFromPut(wire) }]); }); it('copies the envelope verbatim', () => { @@ -813,9 +818,61 @@ describe('payload identity', () => { const held = new SkillObjectSet(); const reader = new ProtocolReader(held); drive(reader, skillPayload([['put-object', putSkill()]])); - drive(reader, skillPayload([], { state: 'basis-2' })); + const outcomes = drive(reader, skillPayload([], { state: 'basis-2' })); expect(held.size).toBe(0); expect(reader.diagnostics.payloadsIgnored).toBe(0); + // And it reports the revocation, which is what wakes a listener. A full + // transfer revokes by omission — no `delete-object` says the skill is gone — + // so an empty change list here would commit an empty store silently and + // leave the revoked files on disk until the process restarted. + expect(outcomes.at(-1)?.changes).toEqual([{ key: 'pdf-extraction', version: 3 }]); + expect(reader.diagnostics.objectsRevoked).toBe(1); + }); + + it('reports the departures when a full transfer replaces the set', () => { + // The general form of the same thing: what the payload did not carry is + // gone, and a listener that reads versions needs both halves of a version + // move — the put for the arrival, the tombstone for the departure. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive( + reader, + skillPayload([ + ['put-object', putSkill('a')], + ['put-object', putSkill('b')], + ]), + ); + const outcomes = drive( + reader, + skillPayload([['put-object', putSkill('a', { objectVersion: 4 })]], { state: 'basis-2' }), + ); + expect(held.size).toBe(1); + // The arrival carries content, as a put's change always has; the departures + // are tombstones, spelled exactly as `delete-object`'s are. + const changes = outcomes.at(-1)?.changes ?? []; + expect(changes.map((raw) => ({ key: raw.key, version: raw.version }))).toEqual([ + { key: 'a', version: 4 }, + { key: 'a', version: 3 }, + { key: 'b', version: 3 }, + ]); + expect(changes[0].content).toBe(SKILL_BODY); + expect(changes.slice(1).every((raw) => !('content' in raw))).toBe(true); + }); + + it('does not adopt the selector of a payload it declined', () => { + // Ignoring a foreign payload's contents while adopting its resume point + // would ask the next poll or stream to resume from someone else's payload: + // skill updates could stop arriving while every diagnostic read healthy. + const reader = new ProtocolReader(new SkillObjectSet()); + const ours = drive(reader, skillPayload([['put-object', putSkill()]], { state: 'skills-basis' })); + expect(ours.at(-1)?.basis).toBe('skills-basis'); + + const outcomes = drive( + reader, + skillPayload([['put-object', putFlag()]], { payloadId: 'env-flags', state: 'flag-basis' }), + ); + expect(outcomes.at(-1)?.basis).toBeNull(); + expect(reader.diagnostics.payloadsIgnored).toBe(1); }); it('identifies the payload as the skill payload from a revocation', () => { @@ -1006,6 +1063,28 @@ describe('polling against the endpoint', () => { expect(endpoint.requests.slice(0, 3).map((r) => r.query.basis)).toEqual([undefined, 'basis-1', 'basis-2']); }); + it('keeps asking from the skill basis when another payload transfers', async () => { + // The wire half of the declined-payload case: the store must resume from the + // payload skills arrive on, not from the one it just threw away. + endpoint.queuePoll(fullPayload([['put-object', putSkill()]], 'skills-basis')); + endpoint.queuePoll( + events( + ['server-intent', serverIntent('xfer-full', 'env-flags')], + ['put-object', putFlag()], + ['payload-transferred', transferred('flag-basis')], + ), + ); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => endpoint.requests.length >= 3)).toBe(true); + expect(endpoint.requests.slice(0, 3).map((r) => r.query.basis)).toEqual([ + undefined, + 'skills-basis', + 'skills-basis', + ]); + }); + it('returns an ETag as If-None-Match', async () => { endpoint.queuePoll(fullPayload([['put-object', putSkill()]]), { etag: 'W/"v1"' }); endpoint.queuePoll([], { status: 304 }); @@ -1513,6 +1592,39 @@ describe('the missing contentHash', () => { expect((await getSkillResult('pdf-extraction')).reason).not.toBe('absent'); }); + it('leaves the copy on disk alone when a hashless payload replaces a good one', async () => { + // The whole point of withholding, asserted through the reconcile rather than + // through the accessors — and the case the hash gap makes universal, since + // today *every* object arrives hashless. `writeSkills('*')` derives its prune + // keep-set from the keys `allObjects` is keyed by, so a key it cannot parse + // as a skill key drops out of the keep-set and prune deletes the last + // known-good copy. Which is the outcome this transport was written to avoid. + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + endpoint.queuePoll( + fullPayload([['put-object', putSkill('pdf-extraction', { objectVersion: 4, omitHash: true })]], 'basis-2'), + ); + endpoint.queuePoll([], { status: 304 }); + + const store = pollStore({ pollIntervalMs: 20 }); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const root = path.join(await scratchRoot(), 'skills'); + const written = path.join(root, 'pdf-extraction', 'SKILL.md'); + await writeSkills('*', root); + expect(readFileSync(written, 'utf8')).toBe(SKILL_BODY); + + expect(await waitUntil(() => store.diagnostics.hashlessObjects > 0, 5000)).toBe(true); + const report = await writeSkills('*', root); + + expect(existsSync(written)).toBe(true); + expect(readFileSync(written, 'utf8')).toBe(SKILL_BODY); + expect(report.actions.some((action) => action.action === 'removed')).toBe(false); + // Reported against the skill's own key, so the failure names the skill. + expect(report.actions.some((action) => action.action === 'error' && action.key === 'pdf-extraction')).toBe(true); + }); + it('counts hashless objects', async () => { endpoint.queuePoll( fullPayload([ @@ -1775,6 +1887,79 @@ describe('watchSkills', () => { } }); + it('prunes when a full transfer drops every skill', async () => { + // A full transfer revokes by omission, so this payload carries no + // `delete-object` at all. The commit still has to wake the watcher, or the + // revoked file sits on disk until the process restarts. + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + endpoint.queuePoll( + events(['server-intent', serverIntent('xfer-full')], ['payload-transferred', transferred('basis-2')]), + ); + endpoint.queuePoll([], { status: 304 }); + + const store = pollStore({ pollIntervalMs: 100 }); + store.start(); + await store.waitForSkills(5000); + _setStore(store); + + const root = path.join(await scratchRoot(), 'skills'); + const { watcher } = await watchSkills('*', root, { debounceMs: 20 }); + try { + const written = path.join(root, 'pdf-extraction', 'SKILL.md'); + expect(readFileSync(written, 'utf8')).toBe(SKILL_BODY); + expect(await waitUntil(() => !existsSync(written), 10_000)).toBe(true); + } finally { + await watcher.close(); + } + }); + + it('reconciles again for a change that commits during the initial write', async () => { + // The listener is registered before the initial reconcile, so a payload that + // commits while that reconcile is doing its filesystem I/O is not lost. The + // fake store fires its listener from inside `allObjects` — that is, from + // inside the initial reconcile — and answers with the set as it was, so the + // second version only reaches disk if the watcher heard about it. + const verified = (content: string): RawSkillObject => ({ + key: 'a', + version: 1, + content, + contentHash: hash(content), + }); + + let fire: (() => void) | null = null; + let objects: Record = { a: verified('first') }; + let fired = false; + _setStore({ + getObject: (_kind: string, key: string) => objects[key] ?? null, + allObjects: () => { + const current = objects; + if (!fired && fire) { + fired = true; + objects = { a: verified('second') }; + fire(); + } + return current; + }, + addListener: (_kind: string, fn: () => void) => { + fire = fn; + }, + removeListener: () => { + fire = null; + }, + }); + + const root = path.join(await scratchRoot(), 'skills'); + const { report, watcher } = await watchSkills('*', root, { debounceMs: 10 }); + try { + expect(fired).toBe(true); + expect(report.actions.some((action) => action.action === 'written')).toBe(true); + const written = path.join(root, 'a', 'SKILL.md'); + expect(await waitUntil(() => readFileSync(written, 'utf8') === 'second', 5000)).toBe(true); + } finally { + await watcher.close(); + } + }); + it('refuses a store with no addListener, loudly', async () => { _setStore({ getObject: () => null, diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 7c7589ce..6b5f25f9 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -538,14 +538,25 @@ export class SkillObjectSet { return this.loose.get(key) ?? null; } - /** One entry per skill key, at its newest version. See the class docstring. */ + /** + * One entry per skill key, at its newest version, keyed by the bare skill key. + * + * The key is the *skill* key, never the wire key: `writeSkills('*')` derives its + * prune keep-set from these keys, and a key it cannot parse as a skill key + * drops out of the keep-set and takes the copy already on disk with it. A + * `key:version` spelling here therefore turns every unverifiable object into a + * deletion — precisely the outcome withholding exists to prevent. Matching a + * held object back to the event that carried it is what `allRaw` is for. + * + * See the class docstring for why the collapse to one object per key lives here. + */ snapshot(): Record { const out: Record = {}; for (const [key, held] of this.versions) { if (held.size === 0) continue; const newest = Math.max(...held.keys()); const raw = held.get(newest); - if (raw) out[`${key}:${newest}`] = raw; + if (raw) out[key] = raw; } for (const [key, raw] of this.loose) { if (!this.versions.has(key)) out[key] = raw; @@ -594,6 +605,31 @@ export type TransferOutcome = { disconnect?: string | null; }; +/** + * `(key, version)` as one comparable string. Objects with no usable version + * compare alike, which is what holding them under their key alone already means. + */ +function identityOf(raw: RawSkillObject): string { + return `${String(raw.key)}\u0000${isValidSkillVersion(raw.version) ? raw.version : ''}`; +} + +/** + * Tombstones for every object `next` no longer holds. + * + * A full transfer states the whole payload, so its revocations arrive as an + * absence rather than as an event; this recovers them. At `(key, version)` + * granularity to match `delete-object`, so a key whose version moved yields both + * a put for the arrival and a tombstone for the departure — what a listener that + * reads versions needs, and harmless to one that only needs "something changed". + */ +function revocationsBetween(current: SkillObjectSet, next: SkillObjectSet): RawSkillObject[] { + const surviving = new Set(next.allRaw().map(identityOf)); + return current + .allRaw() + .filter((raw) => !surviving.has(identityOf(raw))) + .map((raw) => ({ key: raw.key, version: isValidSkillVersion(raw.version) ? raw.version : null })); +} + type MutableDiagnostics = { -readonly [K in keyof StoreDiagnostics]: StoreDiagnostics[K] }; function freshDiagnostics(): MutableDiagnostics { @@ -761,11 +797,23 @@ export class ProtocolReader { private payloadTransferred(data: unknown): TransferOutcome { const state = (data as { state?: unknown } | null)?.state; const payloadId = this.intentPayloadId ?? payloadIdFromSelector(state); - if (this.pending !== null && this.isForeignPayload(payloadId)) { + const foreign = this.pending !== null && this.isForeignPayload(payloadId); + if (foreign) { this.warnForeignPayload(payloadId); this.diagnostics.payloadsIgnored += 1; this.changes = []; } else if (this.pending !== null) { + // A full transfer revokes by omission: whatever it did not carry is gone, + // and no `delete-object` ever says so. Diffed before the swap, so those + // departures reach listeners as tombstones like any other revocation. + // Without this, a full transfer that drops every skill commits an empty + // store with an empty change list and notifies nobody — and `watchSkills` + // would leave the revoked files on disk until the process restarted. + if (this.intent === INTENT_TRANSFER_FULL) { + const revoked = revocationsBetween(this.committed, this.pending); + this.changes.push(...revoked); + this.diagnostics.objectsRevoked += revoked.length; + } this.committed.replaceWith(this.pending); warnIfNothingCanVerify(this.committed.allRaw()); if (this.skillsInPayload > 0 && payloadId !== null) { @@ -785,7 +833,11 @@ export class ProtocolReader { return { committed: true, changes, - basis: typeof state === 'string' && state !== '' ? state : null, + // A declined payload must not move the resume point. Adopting the selector + // of a transfer whose contents this layer just threw away would ask the + // next poll or stream to resume from someone else's payload, and skill + // updates could stop arriving while every diagnostic still read healthy. + basis: foreign || typeof state !== 'string' || state === '' ? null : state, }; } diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index a67baf58..c3a363f4 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -49,6 +49,7 @@ import { } from './skills-core.js'; import type { OnUnavailable, + RawSkillObject, ReconcileAction, ReconcileActionKind, ReconcileReport, @@ -571,12 +572,21 @@ function unavailableRun( function pendingForRaw(objectKey: string, raw: unknown): PendingWrite { const skill = verifyRawSkill(raw); if (skill) return { key: skill.key, skill }; - if (!isValidSkillKey(objectKey)) { + + // The map key a store chooses is its own business — the seam never promised it + // spells a skill key, and a store that holds several versions of one key has + // reason to spell it `key:version`. So the object's own `key` is the first + // answer and the map key only the fallback: getting this order wrong drops an + // unverifiable object out of the requested set, and prune reads that as a + // revocation and deletes the last known-good copy. + const candidate = typeof raw === 'object' && raw !== null ? (raw as RawSkillObject).key : undefined; + const key = isValidSkillKey(candidate) ? candidate : objectKey; + if (!isValidSkillKey(key)) { return { key: '', error: 'the skill store served an object under an invalid key; it was withheld' }; } return { - key: objectKey, - error: `skill '${objectKey}' failed integrity verification and was withheld; the copy already on disk was left alone`, + key, + error: `skill '${key}' failed integrity verification and was withheld; the copy already on disk was left alone`, }; } diff --git a/packages/client/src/skills-watch.ts b/packages/client/src/skills-watch.ts index 232cfa74..3064cab5 100644 --- a/packages/client/src/skills-watch.ts +++ b/packages/client/src/skills-watch.ts @@ -112,6 +112,31 @@ export class SkillWatcher { }); } + /** + * Runs the first reconcile inside the watcher's own chain, and returns its report. + * + * Exists so {@link watchSkills} can register the change listener *before* the + * initial reconcile: a payload that commits while the reconcile is doing its + * filesystem I/O would otherwise reach nobody, and the on-disk set would stay + * stale until some later change happened along. A notify that lands during + * this run therefore queues behind it rather than interleaving a second + * reconcile on one root, which is the case that loses manifest entries. + * + * Unlike {@link reconcileOnce} the failure propagates — a bad root or a corrupt + * manifest is `watchSkills`'s to throw — and does not count toward + * {@link reconciles}, which reports re-reconciles only. + */ + async runInitial(): Promise { + const run = this.running.then(() => writeSkills(this.request, this.root, this.options)); + // The chain itself must survive a rejection: a rejected `running` would + // reject every reconcile chained after it. The caller still gets `run`. + this.running = run.then( + () => undefined, + () => undefined, + ); + return run; + } + private async reconcileOnce(): Promise { if (this.closed) return; let report: ReconcileReport; @@ -227,11 +252,27 @@ export async function watchSkills( const { debounceMs = DEFAULT_DEBOUNCE_MS, onReconcile, ...writeOptions } = options; if (debounceMs < 0) throw new Error(`debounceMs must not be negative, got ${JSON.stringify(debounceMs)}`); - // The initial reconcile runs first, so its report is the caller's to inspect - // and a bad root throws out of `watchSkills` rather than into a log line. - const report = await writeSkills(skills, root, writeOptions); - const watcher = new SkillWatcher(store, skills, root, writeOptions, debounceMs, onReconcile); + + // The listener goes on before the first reconcile, not after. A payload that + // commits during that reconcile — filesystem I/O and an fsync per file, so not + // a narrow window — would otherwise be delivered to nobody, and a revocation + // or version bump that landed at boot would sit on disk until the next change + // happened along. `runInitial` shares the watcher's reconcile chain, so a + // notify arriving mid-run queues behind it instead of reconciling the same + // root twice at once. store.addListener(SKILL_OBJECT_KIND, watcher.notify); + + // The initial reconcile is still awaited here, so its report is the caller's to + // inspect and a bad root throws out of `watchSkills` rather than into a log + // line. Nothing should be left watching a root that never reconciled, hence + // the close on the way out. + let report: ReconcileReport; + try { + report = await watcher.runInitial(); + } catch (cause) { + await watcher.close(); + throw cause; + } return { report, watcher }; } From 44ef4e205ae5133a045910d7cb6f504a7627ca89 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 13:09:26 -0400 Subject: [PATCH 16/55] =?UTF-8?q?docs(client):=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20sweep=20delivery-transport=20comments=20for=20a=20p?= =?UTF-8?q?ublic=20SDK?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removes references to internal process, tickets, design history and the Python SDK from the FDv2 transport, the watcher, the store seam docblock, and the tests added with them. Trims docblocks that restated the same rationale at class, method and inline level. The runtime hashless-object advice no longer names an internal ticket or a shipping timeline. No behavioural change. Co-Authored-By: Claude Fable 5.1 --- packages/client/agents.md | 6 +- .../client/src/__tests__/skills-fdv2.test.ts | 10 +- packages/client/src/skills-fdv2.ts | 143 ++++++------------ packages/client/src/skills-watch.ts | 21 +-- packages/client/src/types.ts | 21 ++- 5 files changed, 68 insertions(+), 133 deletions(-) diff --git a/packages/client/agents.md b/packages/client/agents.md index 5c112fac..8e017c64 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -42,7 +42,7 @@ The Agent Skills modules are a deliberate split, and the dependencies run **one ## Agent Skills — the delivery transport, and the one field that will bite you -`FDv2SkillStore` speaks LaunchDarkly's SDK-facing FDv2 channel (`GET /sdk/poll`, `GET /sdk/stream`, server-side SDK key in `Authorization`, a `basis` param once a payload has committed, `If-None-Match`/304). It lives below the seam and produces raw objects in the shape `SkillStore` documents; **nothing above the seam knows it exists**. The transport design was replaced wholesale late in this feature's life and cost zero changes above this line, which is the strongest evidence the seam is drawn correctly. If a transport change ever seems to require editing an accessor, verification, or `writeSkills`, the adapter boundary is wrong. +`FDv2SkillStore` speaks LaunchDarkly's SDK-facing FDv2 channel (`GET /sdk/poll`, `GET /sdk/stream`, server-side SDK key in `Authorization`, a `basis` param once a payload has committed, `If-None-Match`/304). It lives below the seam and produces raw objects in the shape `SkillStore` documents; **nothing above the seam knows it exists**. If a transport change ever seems to require editing an accessor, verification, or `writeSkills`, the adapter boundary is wrong. **The skill's version is in the object's `key`. `version` is the payload's.** Each version of a skill is its own object on the wire, identified as `:`: @@ -63,9 +63,9 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b **A hashless object is held, not dropped.** Verification withholds it with `missing_content_hash`; the transport's job is to make that loud (an error per object, a summary per wholly-hashless payload, `diagnostics.hashlessObjects`) rather than to work around it. Dropping it at the transport would report `absent` — indistinguishable from "no such skill" — and would let a prune delete the last known-good copy on disk. Never synthesize a hash from the delivered content: that certifies the content against itself and verifies nothing. -**`SkillObjectSet.snapshot` collapses to one object per key, and that is load-bearing here in a way it is not in Python.** `//SKILL.md` is a single path, so a whole-store consumer must see one object per key or a `'*'` reconcile writes the same path twice and `allSkills` returns two versions of one skill. The Python SDK collapses in `newest_by_key`, above the seam; that helper is not in this SDK yet, so the collapse happens in the adapter. End-to-end behaviour is identical, and `getObject` still resolves a pinned version out of the full set. When `newestByKey` lands in `skills-core.ts`, move it and delete the note on the class. +**`SkillObjectSet.snapshot` collapses to one object per key, keyed by the bare skill key.** `//SKILL.md` is a single path, so a whole-store consumer must see one object per key or a `'*'` reconcile writes the same path twice and `allSkills` returns two versions of one skill. The keys must be skill keys, not wire `key:version` keys, because `writeSkills('*')` derives its prune keep-set from them. `getObject` still resolves a pinned version out of the full set. -**There is one network timeout, not two.** `readTimeoutMs` is applied through a `ReadDeadline` composed with the store's own abort signal, so connect, headers and each body read share it. Its default is per mode: `DEFAULT_POLL_TIMEOUT_MS` bounds a whole poll request, `DEFAULT_STREAM_READ_TIMEOUT_MS` bounds the gap between reads on a stream, and tripping it on a stream that has gone quiet *reconnects* — the `timeouts` suite measures the bound against a socket that accepts and never answers. Do not add a separate connect timeout; the Python SDK cannot honour one and the two transports are kept option-for-option compatible. +**There is one network timeout, not two.** `readTimeoutMs` is applied through a `ReadDeadline` composed with the store's own abort signal, so connect, headers and each body read share it. Its default is per mode: `DEFAULT_POLL_TIMEOUT_MS` bounds a whole poll request, `DEFAULT_STREAM_READ_TIMEOUT_MS` bounds the gap between reads on a stream, and tripping it on a stream that has gone quiet *reconnects* — the `timeouts` suite measures the bound against a socket that accepts and never answers. Do not add a separate connect timeout. **A body read that fails is recoverable; a protocol-reader or dispatch error is not.** `iterSse` wraps the read itself — a reset, a truncated chunk, the read deadline — as `RecoverableTransportError`, because a live stream dies mid-body far more often than it refuses to open, and the delivery loop reads anything else as a bug and stops for the process lifetime. Whatever the consumer's loop body throws while the generator is suspended at a `yield` passes through untouched, so a bug still surfaces as one. The consecutive-failure counter resets at each *committed payload* in `apply`, not when a connection returns: a stream only ever ends by being dropped, so resetting on return would count every healthy, server-recycled connection as a failure and give up on a healthy server. Every retry delay, `Retry-After` included, is clamped to `maxBackoffMs`. diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index ec9ae6be..85a9a103 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -1,7 +1,7 @@ /** * The FDv2 skill delivery transport. * - * Two layers, deliberately, mirroring the Python suite: + * Two layers, deliberately: * * - **A real fake endpoint.** `FakeFDv2Endpoint` is an in-process * `node:http` server implementing the wire contract — the `basis` query @@ -990,8 +990,7 @@ describe('the held object set', () => { it('collapses the snapshot to one object per key at its newest version', () => { // `//SKILL.md` is a single path, so a whole-store consumer must - // see one object per key. See the class docstring for why the collapse lives - // here in this SDK and in `newest_by_key` in Python. + // see one object per key. const snapshot = filled().snapshot(); const forA = Object.values(snapshot).filter((raw) => raw.key === 'a'); expect(forA).toHaveLength(1); @@ -1781,9 +1780,8 @@ describe('server-side only', () => { describe('watchSkills', () => { it('prunes a revoked skill without a restart', async () => { - // AV-1, closed at this layer. The store's change listener drives the - // reconcile, so the file goes away seconds after the delete-object rather - // than at the next process start. + // The store's change listener drives the reconcile, so the file goes away + // seconds after the delete-object rather than at the next process start. endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); endpoint.queuePoll( events( diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 6b5f25f9..1beaf692 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1,13 +1,10 @@ /** * Agent Skills — the FDv2 delivery transport. * - * The store implementation that actually talks to LaunchDarkly. It sits *below* - * the `SkillStore` seam, not above it: it produces raw wire objects in the shape - * `skills-core.ts` documents, and everything above — the accessors, integrity - * verification, the `Skill` type, materialization — is unchanged and unaware of - * it. That is the whole point of the seam, and the fact that replacing the - * transport design wholesale cost nothing above this line is the evidence it was - * drawn in the right place. + * The store implementation that talks to LaunchDarkly. It sits *below* the + * `SkillStore` seam: it produces raw wire objects in the shape `skills-core.ts` + * documents, and everything above — the accessors, integrity verification, the + * `Skill` type, materialization — is unaware of it. * * Layering: * @@ -15,7 +12,7 @@ * @launchdarkly/ai-server * └─ SkillStore (types.ts) ── structurally typed accessor surface * └─ FDv2SkillStore (this file) ── deserialize, hold, serve - * └─ the SDK-facing FDv2 channel on FDCore + * └─ the SDK-facing FDv2 channel * GET /sdk/poll, GET /sdk/stream, authenticated with the * environment's server-side SDK key * ``` @@ -25,19 +22,6 @@ * do not import it. It uses only platform globals — `fetch`, `AbortController`, * `TextDecoder` — so the content path adds no dependency. * - * **There is no bespoke private route here, deliberately.** An earlier design had - * this adapter poll `/private/flagdlv/payloads/{id}/latest/obj/skill/{key}`. - * Those are gonfalon private endpoints authenticated by Cognito machine-token - * OAuth scopes with no per-tenant authorization; the security review ruled out - * both relaxing that auth and shipping a machine credential to a customer host. - * This transport uses the genuinely SDK-facing channel instead, which is also the - * channel payload signing will eventually cover. Do not reintroduce the private - * route. - * - * This is a port of the Python SDK's `skills_fdv2.py` and is deliberately - * mechanical: same protocol boundary, same constants, same wire semantics, same - * diagnostics vocabulary. A change to one belongs in both. - * * What this module does *not* do, on purpose: * * - **It does not verify content.** Verification lives at the accessor boundary @@ -45,8 +29,8 @@ * `InMemorySkillStore` and a customer's own. * - **It does not skip verification when the wire envelope has no * `contentHash`.** A hashless object is stored verbatim and *withheld* by - * verification with `missing_content_hash`; this module's job is to make that - * outcome loud rather than to paper over it. + * verification with `missing_content_hash`; this module makes that outcome + * loud rather than papering over it. * - **It does not evaluate anything.** No flags, no segments, no targeting. */ @@ -161,13 +145,10 @@ function error(message: string): void { /** * Refuses a mobile key or a client-side environment ID. * - * Skills are for server-side agent runtimes. The payload assignment that carries - * them is shared by every auth type, so the skill payload ID is appended for - * mobile and environment-ID auth too — which means a client-side credential may - * well *succeed* against these endpoints and deliver customer-confidential skill - * content to a client-side process. Throwing here is the SDK-side half of that - * boundary; excluding skills at assignment time is the platform-side half, and is - * an open ask on FDN. + * Skills are for server-side agent runtimes and skill content is + * customer-confidential. A client-side credential may succeed against these + * endpoints, so the SDK refuses one up front rather than deliver skill content + * to a client-side process. * * Throws rather than warning, because there is no degraded mode that is correct: * a store built on the wrong credential should not exist. @@ -252,10 +233,9 @@ export type StoreDiagnostics = { const HASHLESS_ADVICE = "The delivered skill object carries no 'contentHash', so integrity verification withholds it with reason_code " + - "'missing_content_hash' and its content will never resolve. This is not a fault in this store and not something " + - 'the SDK can work around: verification hashes the verbatim bytes and compares, and there is nothing to compare ' + - 'against. The field is specified as an additive sha256-over-verbatim-UTF-8 value on the skill envelope ' + - '(LaunchDarkly AIC-2905) and has not shipped yet. Until it does, expect an empty result from every skill accessor.'; + "'missing_content_hash' and its content will not resolve. The SDK cannot work around this: verification hashes " + + 'the verbatim bytes and compares, and there is nothing to compare against. Skill accessors return an empty ' + + 'result for this skill until LaunchDarkly delivers it with a contentHash.'; /** * `(key, version)` pairs already reported hashless. @@ -270,7 +250,7 @@ export const _warnedHashless = new Set(); * * At error level rather than warn, and per object rather than once per process, * because this is the difference between a broken deployment and an - * empty-by-design one — the exact confusion the blocking gap produces. + * empty-by-design one. */ function warnHashless(raw: RawSkillObject): void { const identity = `${String(raw.key)}:${String(raw.version)}`; @@ -313,8 +293,7 @@ export type Tombstone = { readonly key: string; readonly objectVersion: number | * * The kind alone decides it. Every other kind is **ignored, not rejected**, * because flag and segment objects share the connection and erroring on them - * would turn a normal payload into a reconnect loop — exactly the unknown-kind - * failure this feature must not reproduce. + * would turn a normal payload into a reconnect loop. */ export function isSkillEvent(data: unknown): boolean { if (typeof data !== 'object' || data === null || Array.isArray(data)) return false; @@ -465,16 +444,9 @@ export function tombstoneFromDelete(data: Record): Tombstone | * its key alone, so verification withholds it with a signal rather than the * transport dropping it into indistinguishable absence. * - * **`snapshot` collapses to one object per key at its newest version, and that is - * load-bearing here in a way it is not in Python.** `//SKILL.md` is a - * single path, so a whole-store consumer must see one object per key or a `'*'` - * reconcile writes the same path twice in one run and `allSkills` returns a list - * holding two versions of one skill. The Python SDK collapses in - * `newest_by_key`, above the seam; that helper has not been ported to this SDK - * yet, so the collapse happens here instead. The observable end-to-end behaviour - * is identical, and `getObject` still resolves a pinned version out of the full - * set — which is the case the collapse must not break. When `newestByKey` lands - * in `skills-core.ts`, move it and delete this note. + * `snapshot` collapses to one object per key at its newest version, because + * `//SKILL.md` is a single path and `allSkills` should return one + * entry per skill. `get` still resolves a pinned version out of the full set. */ export class SkillObjectSet { private versions = new Map>(); @@ -541,14 +513,10 @@ export class SkillObjectSet { /** * One entry per skill key, at its newest version, keyed by the bare skill key. * - * The key is the *skill* key, never the wire key: `writeSkills('*')` derives its - * prune keep-set from these keys, and a key it cannot parse as a skill key - * drops out of the keep-set and takes the copy already on disk with it. A - * `key:version` spelling here therefore turns every unverifiable object into a - * deletion — precisely the outcome withholding exists to prevent. Matching a - * held object back to the event that carried it is what `allRaw` is for. - * - * See the class docstring for why the collapse to one object per key lives here. + * The key must be the skill key, never the wire `key:version`: `writeSkills('*')` + * derives its prune keep-set from these keys, and a key it cannot parse as a + * skill key drops out of the keep-set and takes the copy already on disk with + * it. Use `allRaw` to see every held `(key, version)`. */ snapshot(): Record { const out: Record = {}; @@ -652,25 +620,18 @@ function freshDiagnostics(): MutableDiagnostics { * `skills-fdv2.test.ts` drives this directly, and the HTTP layer above it only has * to turn bytes into `[event name, data]` pairs. * - * **Changes are buffered and committed at `payload-transferred`**, matching how - * the base SDK's FDv2 data source applies a change set. A payload version is the - * unit of consistency: applying half of one would publish a state the server - * never described, and on a full transfer it would briefly empty the store — - * which, with pruning on, is the difference between a reconcile and deleting a - * customer's skill files. Listeners therefore fire once per commit, not once per - * object, which is also exactly the granularity the re-reconcile wants. + * **Changes are buffered and committed at `payload-transferred`.** A payload + * version is the unit of consistency: applying half of one would publish a state + * the server never described, and on a full transfer it would briefly empty the + * store — which, with pruning on, is the difference between a reconcile and + * deleting a customer's skill files. Listeners therefore fire once per commit. * - * **The first payload intent is read, and is assumed to be the skill payload.** - * Delivery provides one payload per credential and the protocol requires a client - * to ignore all but the first payload intent, so `payloads[0]` is both what - * arrives and what the protocol says to read. If that ever widens, an `xfer-full` - * for somebody else's payload would empty the skill set and the next - * `payload-transferred` would publish it empty — with pruning on, the difference - * between a reconcile and deleting a customer's files. This layer therefore learns - * which payload skills arrive on and declines to apply a transfer of any other, - * once at warning level and counted. The residual is the first transfer of a - * connection: before a skill has arrived there is nothing to compare a payload - * against. + * **The first payload intent is read, and is assumed to be the skill payload**, + * as the protocol requires. Because an `xfer-full` for a different payload would + * otherwise empty the skill set, this layer learns which payload skills arrive + * on and declines to apply a transfer of any other, once at warning level and + * counted. The first transfer of a connection is always applied: before a skill + * has arrived there is nothing to compare a payload against. */ export class ProtocolReader { readonly diagnostics = freshDiagnostics(); @@ -678,8 +639,8 @@ export class ProtocolReader { private pending: SkillObjectSet | null = null; private changes: RawSkillObject[] = []; // The payload the current intent describes, and the payload skills have - // actually arrived on. One payload per connection makes these the same - // payload; the class docstring says why they are kept apart regardless. + // actually arrived on. Kept apart so a transfer of some other payload can be + // recognised and declined. private intentPayloadId: string | null = null; private skillPayloadId: string | null = null; private skillsInPayload = 0; @@ -806,9 +767,6 @@ export class ProtocolReader { // A full transfer revokes by omission: whatever it did not carry is gone, // and no `delete-object` ever says so. Diffed before the swap, so those // departures reach listeners as tombstones like any other revocation. - // Without this, a full transfer that drops every skill commits an empty - // store with an empty change list and notifies nobody — and `watchSkills` - // would leave the revoked files on disk until the process restarted. if (this.intent === INTENT_TRANSFER_FULL) { const revoked = revocationsBetween(this.committed, this.pending); this.changes.push(...revoked); @@ -874,8 +832,7 @@ export class ProtocolReader { * Whether a transfer completes a payload other than the one skills arrive on. * * `false` unless both payloads are known, so one-payload delivery and the - * first transfer of a connection behave exactly as they did before this check - * existed. + * first transfer of a connection are always applied. */ private isForeignPayload(payloadId: string | null): boolean { return this.skillPayloadId !== null && payloadId !== null && payloadId !== this.skillPayloadId; @@ -1171,11 +1128,8 @@ export type Requester = { }; /** - * The only place this module opens a connection. - * - * Platform globals only, on purpose: this package's runtime dependencies are - * `@opentelemetry/api` and `dotenv`, and its LaunchDarkly base-SDK dependency is - * an optional peer, so the content path must not smuggle in an HTTP client. + * The only place this module opens a connection. Uses platform globals only, so + * the content path adds no HTTP client dependency. * * `readTimeoutMs` is applied to every request through a {@link ReadDeadline}: * connecting, waiting for headers and each body read are all bounded by the @@ -1309,10 +1263,8 @@ function sleep(ms: number, signal: AbortSignal): Promise { export type FDv2SkillStoreOptions = { /** * `'stream'` by default. Prefer it: a `delete-object` reaches a live stream in - * seconds, which is what makes revocation seconds-latent instead of - * restart-latent, and is why the change-listener re-reconcile is worth wiring - * at all. `'poll'` exists for environments that cannot hold a long-lived - * connection, and revocation there is one `pollIntervalMs` late. + * seconds. `'poll'` exists for environments that cannot hold a long-lived + * connection, and a revocation there arrives within one `pollIntervalMs`. */ readonly mode?: FDv2Mode; readonly baseUri?: string; @@ -1363,12 +1315,10 @@ export type FDv2SkillStoreOptions = { * content is customer-confidential. A mobile key or a client-side environment ID * throws from the constructor. * - * **Delivery is in the background; retrieval is not.** `SkillStore` is a - * synchronous seam, so a background task owns the connection and fills memory, - * and `getObject` only ever reads what has already arrived. Nothing here blocks a - * retrieval on the network. The corollary is that a process which calls - * `getSkill` immediately after `start()` may see an empty store; `waitForSkills` - * is how you order boot against the first payload. + * **Delivery is in the background; retrieval is not.** A background task owns + * the connection and fills memory, and `getObject` only ever reads what has + * already arrived. A process that calls `getSkill` immediately after `start()` + * may see an empty store; `waitForSkills` orders boot against the first payload. * * **Last known good survives an outage.** A transport failure never empties the * store and never makes `getObject` throw: it keeps serving what it last @@ -1514,8 +1464,7 @@ export class FDv2SkillStore implements SkillStore { * Registers `fn` to be called once per committed change. * * Fires **once per changed object at payload-transferred**, not as objects - * stream in: a payload version is the unit of consistency, and a listener that - * reacted to a half-applied full transfer would see the store briefly empty. + * stream in, so a listener never observes a half-applied transfer. * `watchSkills` is the intended consumer. * * A put notifies with the raw skill object. A revocation notifies with a diff --git a/packages/client/src/skills-watch.ts b/packages/client/src/skills-watch.ts index 3064cab5..82ea827b 100644 --- a/packages/client/src/skills-watch.ts +++ b/packages/client/src/skills-watch.ts @@ -2,22 +2,13 @@ * Agent Skills — re-reconcile on delivery, so revocation does not wait for a restart. * * `writeSkills` is a one-shot reconcile: it materializes what the store holds - * now. That was the whole story while the only transport was a hand-populated - * store, and the design accordingly deferred an eager re-reconcile — revocation - * would take effect at the next process restart, which the security review filed - * as AV-1. + * now. This module re-runs it whenever the store reports a change, so a + * `delete-object` that reaches a live connection takes a skill's `SKILL.md` off + * disk within a debounce interval rather than at the next process restart. * - * A streaming FDv2 connection changes the premise. A `delete-object` reaches a - * live connection in **seconds**, and the store already publishes a change - * listener, so the gap between "LaunchDarkly revoked this skill" and "its - * `SKILL.md` is off the agent's disk" collapses from a process lifetime to a - * debounce interval. That is the single largest resilience improvement available - * at this layer, which is why it is here rather than in a later phase. - * - * `onUnavailable: 'keep'` stays the default, deliberately and per the review: an - * outage must not read as "everything was revoked". A watcher that pruned on a - * failed retrieval would convert every transport blip into deletion of a - * customer's skill files. + * `onUnavailable: 'keep'` stays the default: an outage must not read as + * "everything was revoked". A watcher that pruned on a failed retrieval would + * convert every transport blip into deletion of a customer's skill files. * * Layering: this module sits *above* `skills-fs.ts` and calls `writeSkills` * without modifying it. Nothing in the reconcile, the accessors, or verification diff --git a/packages/client/src/types.ts b/packages/client/src/types.ts index 67c564b8..05f47280 100644 --- a/packages/client/src/types.ts +++ b/packages/client/src/types.ts @@ -258,20 +258,17 @@ export type RawSkillObject = { * Structural interface every source of skill content satisfies. * * Structurally typed on purpose, mirroring how {@link LDClientInterface} works in - * this package: pass any object carrying these methods. `FDv2SkillStore` — which - * streams or polls LaunchDarkly's SDK-facing FDv2 channel — drops in behind this - * interface without touching the public API, and did. + * this package: pass any object carrying these methods. `FDv2SkillStore`, which + * streams or polls LaunchDarkly's FDv2 channel, is one implementation; + * `InMemorySkillStore` is another. * * `addListener` and `removeListener` are part of the seam but **optional**: a - * store without them must still be accepted. Nothing in the accessors calls - * either — they exist for the delivery transport to push updates through, and - * for a consumer such as `watchSkills` to stop receiving them. `watchSkills` is - * the one consumer: it refuses loudly rather than degrading when a configured - * store lacks `addListener`, and probes for `removeListener` on close, so a - * store without it keeps working at the cost of a listener that lives as long - * as the store does. A store that implements `addListener` should implement - * `removeListener` too; it removes one occurrence of `fn` under `kind` and is a - * no-op when `fn` is not registered. + * store without them must still be accepted. The accessors call neither; they + * exist so a consumer such as `watchSkills` can observe delivery changes. + * `watchSkills` throws when the configured store lacks `addListener`, and uses + * `removeListener` on close when present. A store that implements `addListener` + * should implement `removeListener` too; it removes one occurrence of `fn` under + * `kind` and is a no-op when `fn` is not registered. * * Everything a store serves is untrusted input. The transport is not part of the * trust boundary — key, version, size, and content hash are revalidated at the From b73192e03b94d40f139b1c92bbf8fa90dcce5eee Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 13:17:54 -0400 Subject: [PATCH 17/55] =?UTF-8?q?docs(client):=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20finish=20the=20public-SDK=20comment=20sweep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-on to the sweep in a333b1d, covering what it left behind. The hashless-object advice is written to console.error on every withheld skill, so it is customer-facing text. It still claimed the SDK "cannot work around this" and that accessors stay empty "until LaunchDarkly delivers it with a contentHash" — a shipping prediction in a log line. Cut to the reason code, what it means, and where to go. The hashless tests assert on 'missing_content_hash', 'contentHash' and the separate whole-payload summary, all of which survive. Also removes three references customers have no use for: the `_warnedHashless` note about clearing it from the tests, `ProtocolReader`'s pointer at `skills-fdv2.test.ts`, and "test seam: a transport double" on `requester`, which is a public option whose docblock shows up as hover text. Trims the `NO_STORE_MESSAGE` maintainer note and three blocks in `skills-fs.ts` / `skills-watch.ts` that explained a rule by narrating the defect it fixed; the rules are kept. Reunites the skills export table in the README. The new "Receiving skills from LaunchDarkly" section had been inserted mid-table, splitting seven rows into a second table after ~35 lines of prose. The section now follows the whole table as a sibling of "Observability"; its heading is unchanged, so both in-page links still resolve. No behavioural change. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 42 ++++++++++++----------------- packages/client/src/skills-core.ts | 10 +------ packages/client/src/skills-fdv2.ts | 13 ++++----- packages/client/src/skills-fs.ts | 10 +++---- packages/client/src/skills-watch.ts | 26 +++++++----------- 5 files changed, 36 insertions(+), 65 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 60758402..01b68363 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -329,6 +329,23 @@ example above, materializes only what the resolved variation actually asked for. | `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | | `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked`, `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | +| `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | +| `createSkillOutcome(init)` | Build a frozen `SkillOutcome`. Exported for tests and for wrapping your own retrieval in the same shape. | +| `SKILL_OBJECT_KIND` | `'skill'` — the delivery object kind. | +| `SKILL_FILENAME` | `'SKILL.md'`. | +| `MANIFEST_FILENAME` | `'.launchdarkly-skills.json'` — add this to your `.gitignore` if you do not commit materialized skills. | +| `MANIFEST_VERSION` | `1`. | +| `MAX_SKILL_CONTENT_BYTES` | `10485760` (10 MiB) — the hard cap. Content above it is withheld. A backstop against absurd input, set well above the platform's own limit, which is enforced before delivery; do not read it as that limit. | + +`ReconcileReport` exposes `actions`, `ok` (true iff no action is an `error`), and `errors` (the error actions, in order), so callers never re-derive the filter. Each `ReconcileAction` carries `key`, `action` (`written` | `updated` | `skipped_current` | `removed` | `error`), and nullable `version` / `path` / `error`. A failure belonging to the whole run rather than one skill — a corrupt manifest, for instance — carries the **empty string** in `key`. + +**Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly, and it is a platform bound rather than a detail: Node exposes no `renameat`/`unlinkat`/`openat`, so no destructive step can be performed relative to a pinned directory descriptor. Every check here is therefore a per-component `lstat` taken immediately before the path-based operation — a check-then-use race, not a closed window — and unlike the Python SDK, which closes that window on POSIX by holding a descriptor opened `O_RDONLY|O_DIRECTORY|O_NOFOLLOW` for the whole reconcile, **this floor is what runs on every platform Node supports, Linux included.** An attacker who already has **write permission on the managed root** can therefore win a race to redirect a write or a delete outside it. Windows adds nothing worse and nothing better: reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) are deliberately not implemented in this release, and Windows is not a tested platform for it — neither SDK repository has a Windows CI runner. So write permission on the managed root is *the* security boundary here. Keep it writable only by the identity running the reconcile — see [privilege separation](#privilege-separation-the-agent-must-not-be-able-to-rewrite-its-own-skills). + +**Two constraints on skill keys, imposed here rather than by the data model.** A key becomes a single directory name, so `writeSkills` rejects — as a reported `error` action, on every platform — a key over 255 bytes and the 22 Windows reserved device names (`con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9`). Both are checked in the filesystem layer only: a key like `aux` remains valid everywhere else, so an AI Config referencing it still parses and its other skills still materialize. One residual the SDK cannot check for you: the 255-byte bound is per *path component*, not on the total path, so `//SKILL.md` can still exceed Windows' 260-character `MAX_PATH` if the root is deep and the key is long. The root is yours, so budget for it there. + +**Skill content is opaque to the SDK.** A verified `Skill` carries `content` as a `Uint8Array` — the exact bytes that were hashed — and the SDK never parses, decodes, or interprets it anywhere. There is deliberately no frontmatter accessor and no YAML dependency: a consumer that wants the frontmatter decodes the bytes and parses them itself, with whatever parser and bounds it trusts. + +**No LaunchDarkly telemetry is emitted for skills.** Signals go through an internal no-op emitter; `client.track()` is never called and no LD context is involved. #### Receiving skills from LaunchDarkly @@ -365,31 +382,6 @@ try { **If every skill comes back empty, check `diagnostics.hashlessObjects`.** Verification requires `contentHash` on the delivered object and withholds anything without one, so a nonzero count there means skills are being withheld rather than that the environment has none. The store logs an error per hashless object and one summary per wholly-hashless payload, both naming the reason. There is deliberately no fallback that skips verification: a hash the SDK computed from the content it was handed would certify the content against itself. -| Export | Description | -|---|---| -| `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | -| `createSkillOutcome(init)` | Build a frozen `SkillOutcome`. Exported for tests and for wrapping your own retrieval in the same shape. | -| `SKILL_OBJECT_KIND` | `'skill'` — the delivery object kind. | -| `SKILL_FILENAME` | `'SKILL.md'`. | -| `MANIFEST_FILENAME` | `'.launchdarkly-skills.json'` — add this to your `.gitignore` if you do not commit materialized skills. | -| `MANIFEST_VERSION` | `1`. | -| `MAX_SKILL_CONTENT_BYTES` | `10485760` (10 MiB) — the hard cap. Content above it is withheld. A backstop against absurd input, set well above the platform's own limit, which is enforced before delivery; do not read it as that limit. | - -`ReconcileReport` exposes `actions`, `ok` (true iff no action is an `error`), and `errors` (the error actions, in order), so callers never re-derive the filter. Each `ReconcileAction` carries `key`, `action` (`written` | `updated` | `skipped_current` | `removed` | `error`), and nullable `version` / `path` / `error`. A failure belonging to the whole run rather than one skill — a corrupt manifest, for instance — carries the **empty string** in `key`. - -**Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly, and it is a platform bound rather than a detail. Node exposes no `renameat`/`unlinkat`/`openat`, so no destructive step can be addressed relative to a pinned directory descriptor *directly* — and what the SDK can do about that differs by platform: - -- **On Linux, the swap window is closed.** The managed root is opened `O_RDONLY|O_DIRECTORY|O_NOFOLLOW` once and that descriptor is held for the whole reconcile, and every child — each `//`, each `SKILL.md`, each temp file, the manifest — is addressed as `/proc/self/fd//`. The kernel resolves that prefix to the inode the descriptor holds rather than to the name it was opened under, which gives it the same property an `*at()` call would: renaming the root, or replacing it with a symlink, after validation cannot redirect a write or a delete. -- **On macOS and Windows, the window is narrowed but not closed.** There is no traversable descriptor prefix (`/dev/fd//` does not resolve), so each check is a per-component `lstat` taken immediately before the path-based operation — a check-then-use race. An attacker who already has **write permission on the managed root, or on any of its ancestor directories**, can win that race and redirect a write or a delete outside the root. Windows additionally has no reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) in this release and is not a tested platform — there is no Windows CI runner. - -So on those two platforms, write permission on the managed root **and on its ancestors** is *the* security boundary here. Keep them writable only by the identity running the reconcile — see [privilege separation](#privilege-separation-the-agent-must-not-be-able-to-rewrite-its-own-skills). The ancestor requirement is not a technicality: for a root of `.claude/skills`, the parent is `.claude`, which an agent identity often owns, and that alone is enough. - -**Two constraints on skill keys, imposed here rather than by the data model.** A key becomes a single directory name, so `writeSkills` rejects — as a reported `error` action, on every platform — a key over 255 bytes and the 22 Windows reserved device names (`con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9`). Both are checked in the filesystem layer only: a key like `aux` remains valid everywhere else, so an AI Config referencing it still parses and its other skills still materialize. One residual the SDK cannot check for you: the 255-byte bound is per *path component*, not on the total path, so `//SKILL.md` can still exceed Windows' 260-character `MAX_PATH` if the root is deep and the key is long. The root is yours, so budget for it there. - -**Skill content is opaque to the SDK.** A verified `Skill` carries `content` as a `Uint8Array` — the exact bytes that were hashed — and the SDK never parses, decodes, or interprets it anywhere. There is deliberately no frontmatter accessor and no YAML dependency: a consumer that wants the frontmatter decodes the bytes and parses them itself, with whatever parser and bounds it trusts. - -**No LaunchDarkly telemetry is emitted for skills.** Signals go through an internal no-op emitter; `client.track()` is never called and no LD context is involved. - #### Observability: integrity failures are logged for your SIEM A skill that fails integrity verification is **withheld** — the accessor returns `null`, `writeSkills` reports an `error` action, and no unverified byte reaches your agent. Every failure additionally writes one line to `console.error` in a fixed, machine-parseable shape, so that withholding is *detectable* and not merely correct: diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 6640a471..2cd79ec5 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -91,15 +91,7 @@ export type IntegrityReasonCode = | 'not_utf8' | 'over_size_cap'; -/** - * The first thing a user sees when no store is configured, so it names both - * stores. - * - * `FDv2SkillStore` comes first because it is the answer in production, and a - * message that offered only `InMemorySkillStore` would point a deployment at the - * development store. Callers match on "skill store"; keep that phrase if the - * wording changes. - */ +/** What the accessors report when no store is configured. */ export const NO_STORE_MESSAGE = 'No skill store is configured, so skill content cannot be retrieved. Configure one with ' + 'initClient({ skillStore: store }) — FDv2SkillStore receives content from LaunchDarkly, and ' + diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 1beaf692..947eb341 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -233,15 +233,13 @@ export type StoreDiagnostics = { const HASHLESS_ADVICE = "The delivered skill object carries no 'contentHash', so integrity verification withholds it with reason_code " + - "'missing_content_hash' and its content will not resolve. The SDK cannot work around this: verification hashes " + - 'the verbatim bytes and compares, and there is nothing to compare against. Skill accessors return an empty ' + - 'result for this skill until LaunchDarkly delivers it with a contentHash.'; + "'missing_content_hash' and its content will not resolve. Contact LaunchDarkly support."; /** * `(key, version)` pairs already reported hashless. * * Module-scoped so the error is one per object per process rather than one per - * re-delivered payload; exported for the tests, which need to clear it. + * re-delivered payload. */ export const _warnedHashless = new Set(); @@ -616,9 +614,8 @@ function freshDiagnostics(): MutableDiagnostics { /** * Applies FDv2 events to an object set. Pure — no sockets, no timers, no clock. * - * Split out so the protocol is testable without a server: every wire case in - * `skills-fdv2.test.ts` drives this directly, and the HTTP layer above it only has - * to turn bytes into `[event name, data]` pairs. + * Split out so the protocol can be driven without a server: the HTTP layer above + * it only has to turn bytes into `[event name, data]` pairs. * * **Changes are buffered and committed at `payload-transferred`.** A payload * version is the unit of consistency: applying half of one would publish a state @@ -1291,7 +1288,7 @@ export type FDv2SkillStoreOptions = { * payload resets the count. */ readonly maxConsecutiveFailures?: number; - /** Test seam: a transport double in place of `FetchRequester`. */ + /** Replaces the built-in `fetch` transport. Intended for testing. */ readonly requester?: Requester; }; diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index c3a363f4..17398f71 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -573,12 +573,10 @@ function pendingForRaw(objectKey: string, raw: unknown): PendingWrite { const skill = verifyRawSkill(raw); if (skill) return { key: skill.key, skill }; - // The map key a store chooses is its own business — the seam never promised it - // spells a skill key, and a store that holds several versions of one key has - // reason to spell it `key:version`. So the object's own `key` is the first - // answer and the map key only the fallback: getting this order wrong drops an - // unverifiable object out of the requested set, and prune reads that as a - // revocation and deletes the last known-good copy. + // The seam never promised a store's map key spells a skill key — a store + // holding several versions of one key has reason to spell it `key:version`. + // So the object's own `key` is the first answer and the map key the fallback, + // which keeps an unverifiable object inside the requested set. const candidate = typeof raw === 'object' && raw !== null ? (raw as RawSkillObject).key : undefined; const key = isValidSkillKey(candidate) ? candidate : objectKey; if (!isValidSkillKey(key)) { diff --git a/packages/client/src/skills-watch.ts b/packages/client/src/skills-watch.ts index 82ea827b..1a1a0041 100644 --- a/packages/client/src/skills-watch.ts +++ b/packages/client/src/skills-watch.ts @@ -106,12 +106,10 @@ export class SkillWatcher { /** * Runs the first reconcile inside the watcher's own chain, and returns its report. * - * Exists so {@link watchSkills} can register the change listener *before* the - * initial reconcile: a payload that commits while the reconcile is doing its - * filesystem I/O would otherwise reach nobody, and the on-disk set would stay - * stale until some later change happened along. A notify that lands during - * this run therefore queues behind it rather than interleaving a second - * reconcile on one root, which is the case that loses manifest entries. + * Sharing the chain lets {@link watchSkills} register the change listener + * *before* this runs: a payload that commits during the reconcile's filesystem + * I/O still reaches the watcher, and queues behind this run rather than + * reconciling the same root concurrently. * * Unlike {@link reconcileOnce} the failure propagates — a bad root or a corrupt * manifest is `watchSkills`'s to throw — and does not count toward @@ -245,19 +243,13 @@ export async function watchSkills( const watcher = new SkillWatcher(store, skills, root, writeOptions, debounceMs, onReconcile); - // The listener goes on before the first reconcile, not after. A payload that - // commits during that reconcile — filesystem I/O and an fsync per file, so not - // a narrow window — would otherwise be delivered to nobody, and a revocation - // or version bump that landed at boot would sit on disk until the next change - // happened along. `runInitial` shares the watcher's reconcile chain, so a - // notify arriving mid-run queues behind it instead of reconciling the same - // root twice at once. + // The listener goes on before the first reconcile, so a payload committing + // during that reconcile's filesystem I/O still reaches the watcher. store.addListener(SKILL_OBJECT_KIND, watcher.notify); - // The initial reconcile is still awaited here, so its report is the caller's to - // inspect and a bad root throws out of `watchSkills` rather than into a log - // line. Nothing should be left watching a root that never reconciled, hence - // the close on the way out. + // The initial reconcile is awaited, so its report is the caller's to inspect + // and a bad root throws out of `watchSkills`. Nothing is left watching a root + // that never reconciled, hence the close on the way out. let report: ReconcileReport; try { report = await watcher.runInitial(); From 0a0e847da2cd70e246e3eb9b31a5aa31b28949e8 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 13:25:22 -0400 Subject: [PATCH 18/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20do=20not=20expire=20an=20unchanging=20environment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A reconnect whose basis is already current is answered with the `none` intent and no payload, then the connection is recycled with a goodbye. Nothing committed on such a connection, and only a commit cleared the consecutive-failure count, so every healthy recycle counted as a failure and delivery gave up for good after `maxConsecutiveFailures` of them — on an environment whose skills simply never changed. A `server-intent` that parses now clears the count, whether or not a payload follows, and a non-catastrophic goodbye reconnects without counting as a failure or appearing in the diagnostics. Dropped streams, read timeouts, HTTP errors, and `error` events count as before. Co-Authored-By: Claude Fable 5.1 --- .../client/src/__tests__/skills-fdv2.test.ts | 65 ++++++++++++++ packages/client/src/skills-fdv2.ts | 85 ++++++++++++++----- 2 files changed, 128 insertions(+), 22 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 85a9a103..a9e13635 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -688,6 +688,8 @@ describe('protocol reader', () => { }); expect(outcome.disconnect).toBeTruthy(); expect(outcome.fatal).toBeFalsy(); + // Expected, so the reconnect it asks for is not counted as a failure. + expect(outcome.expected).toBe(true); }); it('treats a catastrophic goodbye as fatal', () => { @@ -1333,6 +1335,38 @@ class RecyclingRequester implements Requester { } } +/** + * A healthy server with nothing to send: every connection is answered with the + * `none` intent — what we hold is already current — and then closed with a + * goodbye, which is how a long-lived stream is recycled. Nothing commits, + * because there is nothing to commit; given a first transfer, it delivers that + * on the first connection and nothing on any connection after. + */ +class UnchangingRequester implements Requester { + connections = 0; + + constructor(private readonly firstTransfer: WireEvent[] = []) {} + + poll(): Promise { + throw new Error('not a polling double'); + } + + async stream(): Promise> { + this.connections += 1; + const transfer = + this.connections === 1 && this.firstTransfer.length > 0 + ? this.firstTransfer + : events(['server-intent', serverIntent('none')]); + const scripted = asPairs([ + ...transfer, + ...events(['goodbye', { reason: 'server recycle', silent: true, catastrophe: false }]), + ]); + return (async function* () { + yield* scripted; + })(); + } +} + function scriptedStreamStore(requester: Requester, options: Record = {}): FDv2SkillStore { const store = new FDv2SkillStore(SDK_KEY, { mode: 'stream', @@ -1436,6 +1470,37 @@ describe('failure handling', () => { expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); }); + it('does not count an up-to-date connection against the retry bound', async () => { + // An environment whose skills never change is answered with the `none` + // intent and then recycled, so nothing ever commits. Clearing the failure + // row only at a commit would give up on this healthy server after + // maxConsecutiveFailures + 1 recycles, and no later revocation would ever + // be delivered. + const requester = new UnchangingRequester(); + const store = scriptedStreamStore(requester, { maxConsecutiveFailures: 3 }); + store.start(); + // A bound well inside the test timeout, so a loop that gave up fails here + // rather than by running out of time: each recycle costs a 1ms backoff. + expect(await waitUntil(() => requester.connections >= 8, 2000)).toBe(true); + expect(store.failed).toBeNull(); + expect(store.diagnostics.connectionFailures).toBe(0); + expect(store.diagnostics.lastError).toBeNull(); + }); + + it('keeps delivering after more recycles than the retry bound tolerates', async () => { + // The same server, but with a payload transferred first: the content it + // delivered has to survive the recycles, and delivery has to still be live + // afterwards rather than quietly given up on. + const requester = new UnchangingRequester(fullPayload([['put-object', putSkill()]], 'basis-1')); + const store = scriptedStreamStore(requester, { maxConsecutiveFailures: 3 }); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(await waitUntil(() => requester.connections >= 6, 2000)).toBe(true); + expect(store.failed).toBeNull(); + expect(store.diagnostics.connectionFailures).toBe(0); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + }); + it('resets the failure count on a stream commit', async () => { const requester = new ScriptedRequester([ new RecoverableTransportError('x'), diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 947eb341..2d1bc214 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -225,7 +225,11 @@ export type StoreDiagnostics = { * caller can read rather than an empty store they have to explain. */ readonly hashlessObjects: number; - /** Recoverable transport failures since the last successful transfer. */ + /** + * Recoverable transport failures in a row. Back to zero on any sign of a + * working server — a parsed `server-intent` or a committed payload — and + * never raised by a connection the server closes normally. + */ readonly connectionFailures: number; /** The most recent transport error, if any. Human-readable; do not parse. */ readonly lastError: string | null; @@ -569,6 +573,18 @@ export type TransferOutcome = { basis?: string | null; fatal?: string | null; disconnect?: string | null; + /** + * Set by an event that proves the connection reached a working server: a + * `server-intent` that parsed. True even for the `none` intent, which commits + * nothing. + */ + healthy?: boolean; + /** + * Set on a `disconnect` the server asked for while serving normally — a + * non-catastrophic `goodbye`. The connection still ends and is retried; it is + * not a failure. + */ + expected?: boolean; }; /** @@ -696,7 +712,9 @@ export class ProtocolReader { // reason an unknown kind is: guessing could empty the store. this.pending = null; } - return {}; + // An intent that parsed means the connection reached a working server, even + // when the intent is `none` and no payload will follow. + return { healthy: true }; } private target(): SkillObjectSet | null { @@ -820,7 +838,9 @@ export class ProtocolReader { if (parsed.catastrophe === true) { return { fatal: `server sent a catastrophic goodbye: ${String(parsed.reason)}` }; } - return { disconnect: `server said goodbye: ${String(parsed.reason)}` }; + // Expected: the server is closing a connection it was serving, which is how + // a long-lived stream gets recycled. + return { disconnect: `server said goodbye: ${String(parsed.reason)}`, expected: true }; } // -- payload identity ---------------------------------------------------- @@ -874,11 +894,19 @@ export class ProtocolReader { /** A failure retrying cannot fix: bad credential, forbidden, wrong URI. */ export class FatalTransportError extends Error {} -/** A failure worth retrying. Carries a server-requested delay when given one. */ +/** + * A connection that ended and is worth retrying. Carries a server-requested + * delay when given one. + */ export class RecoverableTransportError extends Error { constructor( message: string, readonly retryAfterMs: number | null = null, + /** + * Whether the server closed a connection it was serving normally. Retried + * like any other, but not counted against `maxConsecutiveFailures`. + */ + readonly expected = false, ) { super(message); } @@ -1284,8 +1312,9 @@ export type FDv2SkillStoreOptions = { /** * Bounds the retry loop. On exceeding it the transport stops, logs an error, * and the store keeps serving last known good rather than pretending to be - * live — `failed` reports it. Only failures in a row count: a committed - * payload resets the count. + * live — `failed` reports it. Only failures in a row count: reaching a working + * server resets the count, and a connection the server closes normally never + * counts at all. */ readonly maxConsecutiveFailures?: number; /** Replaces the built-in `fetch` transport. Intended for testing. */ @@ -1346,10 +1375,10 @@ export class FDv2SkillStore implements SkillStore { private failedReason: string | null = null; private firstPayload = false; private readonly firstPayloadWaiters: Array<() => void> = []; - // Recoverable failures since the last committed payload. Reset at the commit - // rather than when a connection returns: a stream only ever ends by being - // dropped, so resetting on return would count every healthy, server-recycled - // connection as a failure. + // Recoverable failures in a row, cleared by any sign of a working server: a + // `server-intent` that parsed, or a payload that committed. Not cleared when a + // connection returns, because a stream never returns normally — it only ends + // by being dropped, which is a failure, or by a goodbye, which is not. private failures = 0; constructor(sdkKey: string, options: FDv2SkillStoreOptions = {}) { @@ -1522,7 +1551,7 @@ export class FDv2SkillStore implements SkillStore { else await this.pollOnce(signal); // A poll that returned is a current answer even when it committed // nothing (HTTP 304). A stream never returns normally; its successes - // are counted at each commit in `apply`. + // are counted in `apply`, at each intent and each commit. this.recordSuccess(); } catch (cause) { if (signal.aborted) return; @@ -1534,23 +1563,31 @@ export class FDv2SkillStore implements SkillStore { this.giveUp(`unexpected error in skill delivery: ${cause instanceof Error ? cause.message : String(cause)}`); return; } - this.failures += 1; - const failures = this.failures; - this.reader.diagnostics.connectionFailures = failures; - this.reader.diagnostics.lastError = cause.message; - if (failures > this.maxConsecutiveFailures) { - this.giveUp(`gave up after ${failures} consecutive failures; last error: ${cause.message}`); - return; + // A goodbye from a server that was serving normally ends the connection + // without being a failure. It reconnects like one, but it neither counts + // against `maxConsecutiveFailures` nor shows up in the diagnostics: + // every recycle of an up-to-date stream arrives this way, so counting + // them would expire an environment whose skills never change. + if (!cause.expected) { + this.failures += 1; + this.reader.diagnostics.connectionFailures = this.failures; + this.reader.diagnostics.lastError = cause.message; + if (this.failures > this.maxConsecutiveFailures) { + this.giveUp(`gave up after ${this.failures} consecutive failures; last error: ${cause.message}`); + return; + } } const requested = cause.retryAfterMs; const delay = Math.min( requested !== null && Number.isFinite(requested) ? requested - : backoffDelayMs(failures, this.initialBackoffMs, this.maxBackoffMs), + : backoffDelayMs(this.failures, this.initialBackoffMs, this.maxBackoffMs), // `Retry-After` is a request and `maxBackoffMs` is a promise. this.maxBackoffMs, ); - warn(`Skill delivery failed (${cause.message}); retrying in ${Math.round(delay)}ms`); + if (!cause.expected) { + warn(`Skill delivery failed (${cause.message}); retrying in ${Math.round(delay)}ms`); + } await sleep(delay, signal); continue; } @@ -1578,9 +1615,13 @@ export class FDv2SkillStore implements SkillStore { private apply(name: string, data: unknown): TransferOutcome { const outcome = this.reader.handle(name, data); + // Any sign of a working server breaks the row of consecutive failures. A + // reconnect whose basis is already current is answered with the `none` + // intent and commits nothing, so waiting for a commit would leave an + // unchanging environment counting healthy connections against its bound. + if (outcome.healthy) this.recordSuccess(); if (outcome.committed) { if (outcome.basis) this.basis = outcome.basis; - // A commit breaks the row of consecutive failures. this.recordSuccess(); this.markFirstPayload(); if (outcome.changes && outcome.changes.length > 0) this.notify(outcome.changes); @@ -1590,7 +1631,7 @@ export class FDv2SkillStore implements SkillStore { private dispatch(outcome: TransferOutcome): void { if (outcome.fatal) throw new FatalTransportError(outcome.fatal); - if (outcome.disconnect) throw new RecoverableTransportError(outcome.disconnect); + if (outcome.disconnect) throw new RecoverableTransportError(outcome.disconnect, null, outcome.expected === true); } private async pollOnce(signal: AbortSignal): Promise { From a92ee61fb5773d98080fdd03f2da2399c4bf4757 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 13:38:52 -0400 Subject: [PATCH 19/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20resolve=20waitForSkills=20false=20when=20delivery=20gives=20?= =?UTF-8?q?up?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Giving up on delivery released first-payload waiters by marking a first payload, so `waitForSkills` resolved `true` after a fatal failure such as an unauthorized key. A caller gating boot on the return value went on to boot against a store that would never receive content. Release the waiters without marking a payload, so they resolve `false` promptly, and answer a later `waitForSkills` from `failed` rather than making it sit out the full timeout. A 304 before any payload still counts as a first payload, and `close()` still releases waiters with `false`. Co-Authored-By: Claude Fable 5.1 --- packages/client/README.md | 2 ++ .../client/src/__tests__/skills-fdv2.test.ts | 16 +++++++++++++++- packages/client/src/skills-fdv2.ts | 19 +++++++++++++------ 3 files changed, 30 insertions(+), 7 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 01b68363..7f8c3659 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -368,6 +368,8 @@ try { } ``` +**`waitForSkills` orders boot against the first payload.** It resolves `true` once a payload has been committed, or once a `304` confirms the payload already held is the current one. It resolves `false` if the wait times out, the store is closed, or delivery stops for good — a fatal error such as an unauthorized key resolves it right away rather than at the timeout, so a boot gated on the return value does not proceed on a dead store. Read `failed` to tell that case from a timeout. + **Nothing above the store changes.** The accessors, integrity verification, and `writeSkills` are transport-agnostic: they see raw objects through the `SkillStore` seam and cannot tell which store produced them. Everything documented above about verification and reconcile semantics applies unchanged. **Server-side only.** Skills are for server-side agent runtimes and skill content is customer-confidential. A mobile key (`mob-…`) or a client-side environment ID throws from the constructor. diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index a9e13635..c7a0e45c 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -1402,10 +1402,24 @@ describe('failure handling', () => { endpoint.queuePoll([], { status: 401 }); const store = pollStore(); store.start(); - expect(await store.waitForSkills(5000)).toBe(true); + const started = Date.now(); + // `false`, and promptly: a caller gating boot on the return value must not + // be told a payload arrived, nor be left to sit out the whole timeout. + expect(await store.waitForSkills(5000)).toBe(false); + expect(Date.now() - started).toBeLessThan(2000); expect(store.failed).not.toBeNull(); }); + it('resolves waitForSkills false immediately once delivery has given up', async () => { + endpoint.queuePoll([], { status: 401 }); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + const started = Date.now(); + expect(await store.waitForSkills(5000)).toBe(false); + expect(Date.now() - started).toBeLessThan(2000); + }); + it('keeps last known good servable after a fatal failure', async () => { endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); endpoint.queuePoll([], { status: 403 }); diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 2d1bc214..9a655511 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1437,14 +1437,21 @@ export class FDv2SkillStore implements SkillStore { } /** - * Resolves once the first payload has been committed, or after `timeoutMs`. + * Resolves once the first payload has been committed, or sooner if delivery + * stops for good, or after `timeoutMs`. * - * `true` means a payload arrived — not that any skill in it verified, and not - * that the environment has any skills. Boot ordering is all this answers; - * `diagnostics` answers the rest. + * `true` means a payload committed, or a 304 confirmed the payload already + * held is the current one — not that any skill in it verified, and not that + * the environment has any skills. `false` means the wait timed out, the store + * was closed, or delivery stopped for good and no payload will arrive; see + * `failed` to tell the last case from the others. Boot ordering is all this + * answers; `diagnostics` answers the rest. */ waitForSkills(timeoutMs = 10_000): Promise { if (this.firstPayload) return Promise.resolve(true); + // Delivery that has already stopped for good has no payload left to wait + // for, so answer now rather than after the timeout. + if (this.failedReason !== null) return Promise.resolve(false); return new Promise((resolve) => { const timer = setTimeout(() => resolve(this.firstPayload), timeoutMs); (timer as unknown as { unref?: () => void }).unref?.(); @@ -1609,8 +1616,8 @@ export class FDv2SkillStore implements SkillStore { 'received; skills will not update until the process restarts with a working connection.', ); // Release anyone waiting on a first payload that is never coming, rather than - // making them eat the full timeout. - this.markFirstPayload(); + // making them eat the full timeout. They resolve `false`: nothing arrived. + this.releaseWaiters(); } private apply(name: string, data: unknown): TransferOutcome { From fba8cb83ff075e2a3a8ac72d5d53e32f8de2030c Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 13:44:36 -0400 Subject: [PATCH 20/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20clear=20the=20SSE=20data=20buffer=20on=20every=20block?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A block with no `event:` field left its `data:` lines in the buffer, so the next event parsed as the two payloads concatenated, failed `JSON.parse` and was discarded. Dispatch now resets the buffered fields whether or not the block becomes an event. Co-Authored-By: Claude Fable 5.1 --- .../client/src/__tests__/skills-fdv2.test.ts | 43 +++++++++++++++++++ packages/client/src/skills-fdv2.ts | 8 +++- 2 files changed, 49 insertions(+), 2 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index c7a0e45c..61cfeeab 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -39,6 +39,7 @@ import { FDv2SkillStore, FetchRequester, isSkillEvent, + iterSse, type PollResult, ProtocolReader, RecoverableTransportError, @@ -1165,6 +1166,48 @@ describe('polling against the endpoint', () => { }); }); +describe('SSE framing', () => { + const sseBody = (text: string): ReadableStream => + new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode(text)); + controller.close(); + }, + }); + + const framed = async (text: string): Promise> => { + const seen: Array<[string, unknown]> = []; + for await (const event of iterSse(sseBody(text))) seen.push(event); + return seen; + }; + + it('drops a block with no event name without eating the next event', async () => { + expect(await framed('data: {"orphan":true}\n\nevent: heart-beat\ndata: {}\n\n')).toEqual([['heart-beat', {}]]); + }); + + it('keeps both named events around a nameless block', async () => { + expect( + await framed('event: heart-beat\ndata: {"n":1}\n\ndata: {"orphan":true}\n\nevent: heart-beat\ndata: {"n":2}\n\n'), + ).toEqual([ + ['heart-beat', { n: 1 }], + ['heart-beat', { n: 2 }], + ]); + }); + + it('ignores a comment between events', async () => { + expect( + await framed('event: heart-beat\ndata: {"n":1}\n\n: keep-alive\n\nevent: heart-beat\ndata: {"n":2}\n\n'), + ).toEqual([ + ['heart-beat', { n: 1 }], + ['heart-beat', { n: 2 }], + ]); + }); + + it('dispatches a named block with no data as a null payload', async () => { + expect(await framed('event: heart-beat\n\n')).toEqual([['heart-beat', null]]); + }); +}); + describe('streaming against the endpoint', () => { it('lands a streamed payload', async () => { endpoint.holdStreamOpen = true; diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 9a655511..bae15142 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1087,12 +1087,16 @@ export async function* iterSse( let name: string | null = null; let dataLines: string[] = []; + // Every block that ends clears the buffered fields, whether or not it turns + // into an event: a block with no `event:` field is the default `message` + // event, which this endpoint never sends, so dropping it is right — but its + // `data:` lines must not be left behind to corrupt the block that follows. const dispatch = (): [string, unknown] | null => { - if (name === null) return null; - const payload = dataLines.join('\n'); const eventName = name; + const payload = dataLines.join('\n'); name = null; dataLines = []; + if (eventName === null) return null; if (payload === '') return [eventName, null]; try { return [eventName, JSON.parse(payload)]; From c4e9d0a1b36783d3053634db54ac1819eb585412 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 14:41:34 -0400 Subject: [PATCH 21/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20bound=20the=20retry=20loop=20a=20goodbye=20cannot=20escape?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three defects in the delivery retry path, all in the same retry/HTTP-classification region. A `goodbye` is exempt from `maxConsecutiveFailures` because it is how a long-lived stream is recycled. That exemption was unconditional, so a server closing every connection with a non-catastrophic `goodbye` without ever sending a `server-intent` reconnected without limit, with nothing reaching `failed` or the diagnostics. Only a connection that reached a working server was being served normally, so only that one is exempt now; a `goodbye` on a connection that never got there delivered nothing and is counted, logged and bounded like any other failure. A genuine recycle sends `server-intent` first, which resets the count, so an unchanging environment is unaffected. `Number('')` is 0 and finite, so an empty `Retry-After` — a blank header from a proxy is enough — was honoured as no delay at all, burning every retry the bound allows in milliseconds and going permanently fatal. A blank value is now absent rather than zero, and an honoured `Retry-After` is floored at `initialBackoffMs` so a legal `Retry-After: 0` still waits. The `maxBackoffMs` clamp on the other side is unchanged. HTTP 400 was permanently fatal, but the request carries a `basis` selector and an etag, which can go stale and leave delivery stopped for the process lifetime. A 400 for a request carrying either now clears both and asks for a full transfer once; a 400 for a request carrying neither is the request itself being refused, and is still fatal with the same base-URI and FDv2 advice. Co-Authored-By: Claude Opus 5 --- .../client/src/__tests__/skills-fdv2.test.ts | 145 +++++++++++++++++- packages/client/src/skills-fdv2.ts | 100 +++++++++--- 2 files changed, 219 insertions(+), 26 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 61cfeeab..4d049ee4 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -46,6 +46,7 @@ import { type Requester, retryAfterMs, SkillObjectSet, + StaleRequestStateError, seamObjectFromPut, splitWireKey, tombstoneFromDelete, @@ -214,7 +215,9 @@ class FakeFDv2Endpoint { const queued = this.polls.shift() ?? { status: 304, events: [] }; const headers: Record = {}; if (queued.etag) headers.ETag = queued.etag; - if (queued.retryAfter) headers['Retry-After'] = queued.retryAfter; + // Sent even when blank: a proxy that emits an empty `Retry-After` is a case + // the store has to survive, so the fake has to be able to produce one. + if (queued.retryAfter !== undefined) headers['Retry-After'] = queued.retryAfter; if (queued.status === 200) { const body = JSON.stringify({ events: queued.events }); res.writeHead(200, { ...headers, 'Content-Type': 'application/json' }); @@ -1410,6 +1413,28 @@ class UnchangingRequester implements Requester { } } +/** + * A server that says goodbye and nothing else: every connection is closed with + * a silent, non-catastrophic goodbye, without a `server-intent` ever arriving. + * Indistinguishable from a recycle at the event level, but nothing was ever + * served, so reconnecting cannot make progress. + */ +class GoodbyeOnlyRequester implements Requester { + connections = 0; + + poll(): Promise { + throw new Error('not a polling double'); + } + + async stream(): Promise> { + this.connections += 1; + const scripted = asPairs(events(['goodbye', { reason: 'recycling', silent: true }])); + return (async function* () { + yield* scripted; + })(); + } +} + function scriptedStreamStore(requester: Requester, options: Record = {}): FDv2SkillStore { const store = new FDv2SkillStore(SDK_KEY, { mode: 'stream', @@ -1558,6 +1583,23 @@ describe('failure handling', () => { expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); }); + it('bounds a server that only ever says goodbye', async () => { + // A goodbye is exempt from the retry bound because it is how a healthy + // stream is recycled — but a connection that says goodbye without ever + // sending a `server-intent` served nothing. Exempting that too would + // reconnect without limit, and without `failed` or the diagnostics ever + // saying so. + const requester = new GoodbyeOnlyRequester(); + const store = scriptedStreamStore(requester, { maxConsecutiveFailures: 3 }); + store.start(); + expect(await waitUntil(() => store.failed !== null, 2000)).toBe(true); + expect(store.failed).toContain('gave up after 4 consecutive failures'); + expect(store.failed).toContain('server said goodbye: recycling'); + expect(requester.connections).toBe(4); + expect(store.diagnostics.connectionFailures).toBe(4); + expect(store.diagnostics.lastError).not.toBeNull(); + }); + it('resets the failure count on a stream commit', async () => { const requester = new ScriptedRequester([ new RecoverableTransportError('x'), @@ -1587,12 +1629,96 @@ describe('failure handling', () => { }); it('honours a Retry-After header off the wire', async () => { - endpoint.queuePoll([], { status: 429, retryAfter: '0' }); + endpoint.queuePoll([], { status: 429, retryAfter: '1' }); endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); - // If Retry-After were ignored the 5s backoff would blow the timeout. - const store = pollStore({ initialBackoffMs: 5000 }); + // A one-second request sitting between a 20ms backoff and a 5s cap, so the + // wait that follows can only have come from the header. + const store = pollStore({ initialBackoffMs: 20, maxBackoffMs: 5000 }); + const started = Date.now(); + store.start(); + expect(await store.waitForSkills(4000)).toBe(true); + expect(Date.now() - started).toBeGreaterThanOrEqual(900); + }); + + it('does not treat a blank Retry-After as no delay at all', async () => { + // `Number('')` is 0 and finite, so a header a proxy sent empty would win + // over the backoff and reconnect with no wait — burning every retry the + // bound allows in a few milliseconds and going permanently fatal. + for (let i = 0; i < 5; i += 1) endpoint.queuePoll([], { status: 503, retryAfter: '' }); + const store = pollStore({ initialBackoffMs: 100, maxBackoffMs: 100, maxConsecutiveFailures: 3 }); + const started = Date.now(); + store.start(); + expect(await waitUntil(() => store.failed !== null, 4000)).toBe(true); + // Three backoffs before the fourth failure gives up. Jitter can halve each, + // so 120ms is the floor; collapsed, the whole run lands in single digits. + expect(Date.now() - started).toBeGreaterThanOrEqual(120); + }); + + it('floors an honoured Retry-After at initialBackoffMs', async () => { + // `Retry-After: 0` is legal and means "try again now". Taken literally it + // is a busy loop against the retry bound, so it is honoured as the shortest + // delay the store was configured to wait. + const requester = new ScriptedRequester([ + new RecoverableTransportError('slow down', 0), + asPairs(fullPayload([['put-object', putSkill()]])), + ]); + const store = new FDv2SkillStore(SDK_KEY, { + mode: 'poll', + pollIntervalMs: 10_000, + initialBackoffMs: 200, + maxBackoffMs: 5_000, + requester, + }); + openStores.push(store); + const started = Date.now(); store.start(); expect(await store.waitForSkills(3000)).toBe(true); + expect(Date.now() - started).toBeGreaterThanOrEqual(180); + }); + + it('recovers from a 400 by dropping the basis and asking again', async () => { + // The request carries the SDK key, a `basis` selector and an etag. A + // selector the server has stopped accepting is rejected as a 400, and + // treating that as permanently fatal would stop delivery for the process + // lifetime over state the store could simply drop. + endpoint.queuePoll(fullPayload([['put-object', putSkill('first')]], 'basis-1'), { etag: 'etag-1' }); + endpoint.queuePoll([], { status: 400 }); + endpoint.queuePoll(fullPayload([['put-object', putSkill('second')]], 'basis-2')); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => store.getObject(SKILL_OBJECT_KIND, 'second') !== null)).toBe(true); + expect(store.failed).toBeNull(); + // The retry asks from scratch: no selector, and no etag that would let the + // server answer 304 for a basis it just rejected. + const retry = endpoint.requests[2]; + expect(retry.query.basis).toBeUndefined(); + expect(retry.ifNoneMatch).toBeUndefined(); + }); + + it('stops on a 400 for a request that carried no basis', async () => { + // Nothing left to drop: the request was already the from-scratch one, so + // the endpoint is refusing the request itself. + endpoint.queuePoll([], { status: 400 }); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toContain('400'); + expect(store.failed).toContain('base URI'); + expect(store.failed).toContain('FDv2'); + expect(endpoint.requests).toHaveLength(1); + }); + + it('stops on a second 400 after the basis has been dropped', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]], 'basis-1')); + endpoint.queuePoll([], { status: 400 }); + endpoint.queuePoll([], { status: 400 }); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toContain('400'); + expect(endpoint.requests).toHaveLength(3); + // The content the first transfer delivered is still servable. + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); }); it('neither dies on nor parks behind an unreasonable Retry-After', async () => { @@ -1621,6 +1747,10 @@ describe('failure handling', () => { expect(retryAfterMs(new Headers({ 'Retry-After': '2' }))).toBe(2000); expect(retryAfterMs(new Headers({ 'Retry-After': 'Wed, 21 Oct 2026 07:28:00 GMT' }))).toBeNull(); expect(retryAfterMs(new Headers())).toBeNull(); + // A blank value is absent, not zero. + expect(retryAfterMs(new Headers({ 'Retry-After': '' }))).toBeNull(); + expect(retryAfterMs(new Headers({ 'Retry-After': ' ' }))).toBeNull(); + expect(retryAfterMs(new Headers({ 'Retry-After': '0' }))).toBe(0); }); it('classifies statuses into recoverable and fatal', () => { @@ -1629,7 +1759,12 @@ describe('failure handling', () => { expect(classifyStatus(401).constructor.name).toBe('FatalTransportError'); expect(classifyStatus(403).constructor.name).toBe('FatalTransportError'); expect(classifyStatus(404).constructor.name).toBe('FatalTransportError'); - expect(classifyStatus(400).constructor.name).toBe('FatalTransportError'); + expect(classifyStatus(405).constructor.name).toBe('FatalTransportError'); + // A 400 may be the selector the request carried rather than the request + // itself, so it is retried once from scratch before it is fatal. + expect(classifyStatus(400)).toBeInstanceOf(StaleRequestStateError); + expect(classifyStatus(400)).toBeInstanceOf(RecoverableTransportError); + expect(classifyStatus(400).message).toContain('base URI'); }); it('makes backoff exponential and capped', () => { diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index bae15142..87f79f50 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -227,8 +227,10 @@ export type StoreDiagnostics = { readonly hashlessObjects: number; /** * Recoverable transport failures in a row. Back to zero on any sign of a - * working server — a parsed `server-intent` or a committed payload — and - * never raised by a connection the server closes normally. + * working server — a parsed `server-intent` or a committed payload — and not + * raised by a connection the server closes normally after reaching that + * point, which is how a long-lived stream is recycled. A connection closed + * without ever getting there delivered nothing, and does raise it. */ readonly connectionFailures: number; /** The most recent transport error, if any. Human-readable; do not parse. */ @@ -581,8 +583,9 @@ export type TransferOutcome = { healthy?: boolean; /** * Set on a `disconnect` the server asked for while serving normally — a - * non-catastrophic `goodbye`. The connection still ends and is retried; it is - * not a failure. + * non-catastrophic `goodbye`. The connection still ends and is retried. The + * caller decides whether it counts as a failure, since only a connection that + * had reached a working server was being served normally at all. */ expected?: boolean; }; @@ -903,8 +906,11 @@ export class RecoverableTransportError extends Error { message: string, readonly retryAfterMs: number | null = null, /** - * Whether the server closed a connection it was serving normally. Retried - * like any other, but not counted against `maxConsecutiveFailures`. + * Whether the server closed a connection it had been serving normally — + * a `goodbye` on a connection that reached a working server. Retried like + * any other, but neither logged as a failure nor counted against + * `maxConsecutiveFailures`. A `goodbye` on a connection that never got that + * far is not expected: it delivered nothing, so it counts. */ readonly expected = false, ) { @@ -912,6 +918,18 @@ export class RecoverableTransportError extends Error { } } +/** + * An HTTP 400 for a request carrying client state — the `basis` selector, or an + * `If-None-Match` etag. That state is the one part of the request that can go + * stale, so it is dropped and a full transfer requested once before the status + * is treated as fatal. + */ +export class StaleRequestStateError extends RecoverableTransportError {} + +const REQUEST_ADVICE = + 'The request this adapter sent was not understood. It carries only the SDK key and, after the first payload, ' + + "a 'basis' selector, so check the base URI and that the endpoint speaks FDv2."; + const FORBIDDEN_ADVICE = 'The FDv2 protocol is opt-in per LaunchDarkly account and is served as HTTP 403 while it is off. Skill ' + 'delivery needs it enabled; contact LaunchDarkly support to enable it for your account.'; @@ -920,7 +938,11 @@ const FORBIDDEN_ADVICE = export function retryAfterMs(headers: Headers | null | undefined): number | null { const raw = headers?.get('Retry-After'); if (raw === null || raw === undefined) return null; - const seconds = Number(raw.trim()); + const value = raw.trim(); + // A blank header is not a delay of zero. `Number('')` is 0 and finite, and a + // proxy that sends the header empty would otherwise collapse every backoff. + if (value === '') return null; + const seconds = Number(value); // The HTTP-date form is legal and rare; falling back to our own backoff is // better than parsing a date to honour it approximately. if (!Number.isFinite(seconds)) return null; @@ -942,11 +964,13 @@ export function classifyStatus(status: number, headers?: Headers | null): Error '/sdk/poll and /sdk/stream.', ); } - if ([400, 405, 406, 414, 501].includes(status)) { + // A 400 is the one rejection the adapter can act on: the selector it sent may + // be one the server no longer accepts. Recoverable so the selector can be + // dropped and a full transfer requested; fatal once that has been tried. + if (status === 400) return new StaleRequestStateError(`LaunchDarkly returned HTTP 400. ${REQUEST_ADVICE}`); + if ([405, 406, 414, 501].includes(status)) { return new FatalTransportError( - `LaunchDarkly returned HTTP ${status}, which retrying will not fix. The request this adapter sent was not ` + - "understood. It carries only the SDK key and, after the first payload, a 'basis' selector, so check the " + - 'base URI and that the endpoint speaks FDv2.', + `LaunchDarkly returned HTTP ${status}, which retrying will not fix. ${REQUEST_ADVICE}`, ); } return new RecoverableTransportError(`LaunchDarkly returned HTTP ${status}`, retryAfterMs(headers)); @@ -1317,8 +1341,10 @@ export type FDv2SkillStoreOptions = { * Bounds the retry loop. On exceeding it the transport stops, logs an error, * and the store keeps serving last known good rather than pretending to be * live — `failed` reports it. Only failures in a row count: reaching a working - * server resets the count, and a connection the server closes normally never - * counts at all. + * server resets the count, and a connection the server closes normally after + * that is exempt, so a stream being recycled never approaches the bound. A + * connection closed before any of that is a failure like any other, which is + * what bounds a server that does nothing but close connections. */ readonly maxConsecutiveFailures?: number; /** Replaces the built-in `fetch` transport. Intended for testing. */ @@ -1382,8 +1408,13 @@ export class FDv2SkillStore implements SkillStore { // Recoverable failures in a row, cleared by any sign of a working server: a // `server-intent` that parsed, or a payload that committed. Not cleared when a // connection returns, because a stream never returns normally — it only ends - // by being dropped, which is a failure, or by a goodbye, which is not. + // by being dropped, which is a failure, or by a goodbye, which is a failure + // only when the connection saying it never reached a working server. private failures = 0; + // Whether the connection now open has shown a sign of a working server. Reset + // per attempt: it is what tells a stream being recycled from one that says + // goodbye having delivered nothing, and only the former escapes the bound. + private reachedServer = false; constructor(sdkKey: string, options: FDv2SkillStoreOptions = {}) { const key = requireServerSideCredential(sdkKey); @@ -1558,6 +1589,7 @@ export class FDv2SkillStore implements SkillStore { private async run(signal: AbortSignal): Promise { while (!signal.aborted) { try { + this.reachedServer = false; if (this.mode === 'stream') await this.streamOnce(signal); else await this.pollOnce(signal); // A poll that returned is a current answer even when it committed @@ -1574,11 +1606,25 @@ export class FDv2SkillStore implements SkillStore { this.giveUp(`unexpected error in skill delivery: ${cause instanceof Error ? cause.message : String(cause)}`); return; } - // A goodbye from a server that was serving normally ends the connection - // without being a failure. It reconnects like one, but it neither counts - // against `maxConsecutiveFailures` nor shows up in the diagnostics: - // every recycle of an up-to-date stream arrives this way, so counting - // them would expire an environment whose skills never change. + if (cause instanceof StaleRequestStateError) { + // The selector and etag are the only client state in the request, so + // a rejection of a request carrying neither is the request itself + // being refused, and retrying cannot fix it. Carrying one, the state + // may be stale: drop it, ask for a full transfer, and let the next + // 400 be the fatal one. + if (this.basis === null && this.etag === null) { + this.giveUp(cause.message); + return; + } + this.basis = null; + this.etag = null; + } + // A connection the server closed while serving it normally ended + // without being a failure — see `dispatch` for which ones qualify. It + // reconnects like one, but it neither counts against + // `maxConsecutiveFailures` nor shows up in the diagnostics: every + // recycle of an up-to-date stream arrives this way, so counting them + // would expire an environment whose skills never change. if (!cause.expected) { this.failures += 1; this.reader.diagnostics.connectionFailures = this.failures; @@ -1591,7 +1637,10 @@ export class FDv2SkillStore implements SkillStore { const requested = cause.retryAfterMs; const delay = Math.min( requested !== null && Number.isFinite(requested) - ? requested + ? // A server asking for no delay still gets one: honouring + // `Retry-After: 0` literally would reconnect in a loop and burn + // the whole retry bound in milliseconds. + Math.max(requested, this.initialBackoffMs) : backoffDelayMs(this.failures, this.initialBackoffMs, this.maxBackoffMs), // `Retry-After` is a request and `maxBackoffMs` is a promise. this.maxBackoffMs, @@ -1630,6 +1679,7 @@ export class FDv2SkillStore implements SkillStore { // reconnect whose basis is already current is answered with the `none` // intent and commits nothing, so waiting for a commit would leave an // unchanging environment counting healthy connections against its bound. + if (outcome.healthy || outcome.committed) this.reachedServer = true; if (outcome.healthy) this.recordSuccess(); if (outcome.committed) { if (outcome.basis) this.basis = outcome.basis; @@ -1642,7 +1692,15 @@ export class FDv2SkillStore implements SkillStore { private dispatch(outcome: TransferOutcome): void { if (outcome.fatal) throw new FatalTransportError(outcome.fatal); - if (outcome.disconnect) throw new RecoverableTransportError(outcome.disconnect, null, outcome.expected === true); + if (outcome.disconnect) { + // A goodbye is only a normal end of service for a connection that got + // there: one that says goodbye without ever sending a `server-intent` + // delivered nothing, and is retried as the failure it is, under the bound. + // Exempting it would let a server that only ever says goodbye reconnect + // without limit and without any of it reaching `diagnostics` or `failed`. + const expected = outcome.expected === true && this.reachedServer; + throw new RecoverableTransportError(outcome.disconnect, null, expected); + } } private async pollOnce(signal: AbortSignal): Promise { From 65dbcaed36cc8b79396958ddfd3ab1a95a5e8209 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 14:52:56 -0400 Subject: [PATCH 22/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20report=20revocations=20and=20hashless=20payloads=20accuratel?= =?UTF-8?q?y?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three reporting defects in the FDv2 skill transport. What the store holds, what it serves, and what reaches listeners are all unchanged; only the counters and the log volume move. A full transfer that moves a key to a new version no longer counts as a revocation. `revocationsBetween` still diffs at `(key, version)`, so both halves of a version move reach listeners as a put and a tombstone, but `objectsRevoked` now counts only the keys that left the payload entirely — the counter operators alert on, so a routine version bump must not raise it. The wholly-hashless summary is deduped. It speaks when the condition becomes true and whenever the withheld objects change, and stays quiet for a store that has not moved: with `contentHash` absent from the wire, a polling connection answered with `xfer-changes` was logging the identical paragraph once per interval indefinitely. A store that recovers and relapses is reported again. The set backing both dedupes is now bounded, and is cleared outright once everything held verifies, so a long-running agent whose skills are versioned frequently no longer accumulates an entry per version for the life of the process. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 4 +- .../client/src/__tests__/skills-fdv2.test.ts | 133 +++++++++++++++++- packages/client/src/skills-fdv2.ts | 98 +++++++++++-- 3 files changed, 220 insertions(+), 15 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 7f8c3659..11481616 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -328,7 +328,7 @@ example above, materializes only what the resolved variation actually asked for. | `InMemorySkillStore` | A `SkillStore` backed by a plain object. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Holds one object per key, so it answers a version pin with what it has and lets the accessor refuse a mismatch. | | `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | -| `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked`, `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | +| `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object`, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | | `createSkillOutcome(init)` | Build a frozen `SkillOutcome`. Exported for tests and for wrapping your own retrieval in the same shape. | | `SKILL_OBJECT_KIND` | `'skill'` — the delivery object kind. | @@ -382,7 +382,7 @@ try { > **Beta caveats, worth knowing before you deploy.** Payload signing does not exist on this channel yet, so delivery is TLS-only and the content hash establishes self-consistency, not origin authenticity. The FDv2 protocol is opt-in per account: without it the endpoints return HTTP 403, which the store reports as a fatal error explaining what to do. `ld-relay` does not speak the FDv2 endpoints, so relay-only deployments cannot receive skills. -**If every skill comes back empty, check `diagnostics.hashlessObjects`.** Verification requires `contentHash` on the delivered object and withholds anything without one, so a nonzero count there means skills are being withheld rather than that the environment has none. The store logs an error per hashless object and one summary per wholly-hashless payload, both naming the reason. There is deliberately no fallback that skips verification: a hash the SDK computed from the content it was handed would certify the content against itself. +**If every skill comes back empty, check `diagnostics.hashlessObjects`.** Verification requires `contentHash` on the delivered object and withholds anything without one, so a nonzero count there means skills are being withheld rather than that the environment has none. The store logs an error per hashless object, and a summary whenever the store it holds becomes wholly hashless or the withheld objects change — both naming the reason, and neither repeated for a payload re-delivered unchanged. There is deliberately no fallback that skips verification: a hash the SDK computed from the content it was handed would certify the content against itself. #### Observability: integrity failures are logged for your SIEM diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 4d049ee4..6a35c04e 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -865,6 +865,58 @@ describe('payload identity', () => { expect(changes.slice(1).every((raw) => !('content' in raw))).toBe(true); }); + it('does not count a version bump as a revocation', () => { + // `objectsRevoked` is the kind of counter an operator alerts on, so a + // routine version move must not raise it. The departed version is still + // reported, because a listener that reads versions needs both halves. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, skillPayload([['put-object', putSkill('pdf-extraction', { objectVersion: 1 })]])); + const outcomes = drive( + reader, + skillPayload([['put-object', putSkill('pdf-extraction', { objectVersion: 2 })]], { state: 'basis-2' }), + ); + expect(reader.diagnostics.objectsRevoked).toBe(0); + expect((outcomes.at(-1)?.changes ?? []).map((raw) => ({ key: raw.key, version: raw.version }))).toEqual([ + { key: 'pdf-extraction', version: 2 }, + { key: 'pdf-extraction', version: 1 }, + ]); + }); + + it('counts only the keys a full transfer dropped altogether', () => { + // One key moves and one leaves: exactly one revocation, both tombstones. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive( + reader, + skillPayload([ + ['put-object', putSkill('a', { objectVersion: 1 })], + ['put-object', putSkill('b', { objectVersion: 1 })], + ]), + ); + const outcomes = drive( + reader, + skillPayload([['put-object', putSkill('a', { objectVersion: 2 })]], { state: 'basis-2' }), + ); + expect(reader.diagnostics.objectsRevoked).toBe(1); + expect((outcomes.at(-1)?.changes ?? []).map((raw) => raw.key)).toEqual(['a', 'a', 'b']); + }); + + it('counts a key that leaves once however many versions it held', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive( + reader, + skillPayload([ + ['put-object', putSkill('a', { objectVersion: 1 })], + ['put-object', putSkill('a', { objectVersion: 2 })], + ]), + ); + drive(reader, skillPayload([['put-object', putSkill('b', { objectVersion: 1 })]], { state: 'basis-2' })); + expect(reader.diagnostics.objectsRevoked).toBe(1); + expect(held.get('a', null)).toBeNull(); + }); + it('does not adopt the selector of a payload it declined', () => { // Ignoring a foreign payload's contents while adopting its resume point // would ask the next poll or stream to resume from someone else's payload: @@ -1819,6 +1871,11 @@ describe('the missing contentHash', () => { // computed from the content it was handed would certify the content against // itself and verify nothing. + const hashlessSummaries = (): string[] => + consoleErrors() + .split('\n') + .filter((line) => line.includes('No skill content will resolve')); + it('withholds a hashless skill with the right reason', async () => { endpoint.queuePoll(fullPayload([['put-object', putSkill('pdf-extraction', { omitHash: true })]])); const store = pollStore(); @@ -1917,13 +1974,83 @@ describe('the missing contentHash', () => { const store = pollStore(); store.start(); await store.waitForSkills(5000); - const summaries = consoleErrors() - .split('\n') - .filter((line) => line.includes('No skill content will resolve')); + const summaries = hashlessSummaries(); expect(summaries).toHaveLength(1); expect(summaries[0]).toContain('All 2 skill object(s)'); }); + it('does not repeat the summary for a store that has not moved', () => { + // `contentHash` is not on the wire yet, so a server answering `xfer-changes` + // rather than `xfer-none` re-delivers the same hashless payload on every + // poll. One problem is one paragraph, not one per interval. + const reader = new ProtocolReader(new SkillObjectSet()); + const payload = fullPayload([ + ['put-object', putSkill('a', { omitHash: true })], + ['put-object', putSkill('b', { omitHash: true })], + ]); + drive(reader, payload); + drive(reader, payload); + drive(reader, payload); + expect(hashlessSummaries()).toHaveLength(1); + }); + + it('speaks again when the hashless objects change', () => { + // A different set of withheld skills is a different problem. + const reader = new ProtocolReader(new SkillObjectSet()); + drive(reader, fullPayload([['put-object', putSkill('a', { omitHash: true })]])); + drive(reader, fullPayload([['put-object', putSkill('b', { omitHash: true })]], 'basis-2')); + expect(hashlessSummaries()).toHaveLength(2); + }); + + it('speaks again about a relapse after a recovery', () => { + const reader = new ProtocolReader(new SkillObjectSet()); + const broken = fullPayload([['put-object', putSkill('a', { omitHash: true })]]); + drive(reader, broken); + drive(reader, fullPayload([['put-object', putSkill('a')]], 'basis-2')); + drive(reader, broken); + expect(hashlessSummaries()).toHaveLength(2); + }); + + it('bounds what it remembers about hashless objects', () => { + // One entry per `(key, version)` for the life of the process would leak + // slowly in an agent whose skills are versioned often. + const reader = new ProtocolReader(new SkillObjectSet()); + for (let version = 1; version <= 600; version += 1) { + drive( + reader, + fullPayload( + [['put-object', putSkill('pdf-extraction', { objectVersion: version, omitHash: true })]], + 'basis-1', + ), + ); + } + expect(_warnedHashless.size).toBeLessThan(600); + }); + + it('forgets what it remembers once everything held verifies', () => { + const reader = new ProtocolReader(new SkillObjectSet()); + drive(reader, fullPayload([['put-object', putSkill('a', { omitHash: true })]])); + expect(_warnedHashless.size).toBeGreaterThan(0); + drive(reader, fullPayload([['put-object', putSkill('a')]], 'basis-2')); + expect(_warnedHashless.size).toBe(0); + }); + + it('still reports each hashless object in a payload separately', () => { + // The per-object dedupe the bound and the summary must not disturb. + const reader = new ProtocolReader(new SkillObjectSet()); + drive( + reader, + fullPayload([ + ['put-object', putSkill('a', { omitHash: true })], + ['put-object', putSkill('b', { omitHash: true })], + ]), + ); + const perObject = consoleErrors() + .split('\n') + .filter((line) => line.includes('arrived without a contentHash and will be withheld')); + expect(perObject).toHaveLength(2); + }); + it('does not claim total failure for a partly hashed payload', async () => { endpoint.queuePoll( fullPayload([ diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 87f79f50..cccee819 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -209,7 +209,16 @@ export type StoreDiagnostics = { * mixed payload is visibly mixed. */ readonly objectsIgnored: number; - /** `delete-object` events applied to skills. */ + /** + * Skill revocations applied: every `delete-object` event, plus every skill key + * a full transfer dropped altogether. + * + * A key whose version merely moved does not count. A full transfer states the + * whole payload, so a version bump arrives as a put for the new version and an + * absence where the old one was; both halves still reach listeners, because a + * listener that reads versions needs them, but a key that survives under a new + * version was never revoked. + */ readonly objectsRevoked: number; /** * Transfers not applied because they completed a payload other than the one @@ -242,13 +251,36 @@ const HASHLESS_ADVICE = "'missing_content_hash' and its content will not resolve. Contact LaunchDarkly support."; /** - * `(key, version)` pairs already reported hashless. + * What has already been reported hashless: one entry per `(key, version)`, plus + * one describing the store-wide summary last spoken. * - * Module-scoped so the error is one per object per process rather than one per - * re-delivered payload. + * Module-scoped so each error is one per object per process rather than one per + * re-delivered payload, which matters most in polling mode — the same payload + * arrives on every interval. */ export const _warnedHashless = new Set(); +/** + * Ceiling on remembered reports, so a process whose skills are versioned often + * cannot accumulate an entry per version indefinitely. Oldest out first; an + * evicted object can be reported a second time, which is the cheaper of the two + * failure modes. + */ +const HASHLESS_MEMORY_LIMIT = 512; + +/** Distinguishes the store-wide summary's entry from a per-object one. */ +const SUMMARY_MARKER = '\u0000summary\u0000'; + +function rememberHashless(entry: string): void { + // Insertion-ordered iteration makes the first entry the oldest. + while (_warnedHashless.size >= HASHLESS_MEMORY_LIMIT) { + const oldest = _warnedHashless.values().next().value; + if (oldest === undefined) break; + _warnedHashless.delete(oldest); + } + _warnedHashless.add(entry); +} + /** * One error per `(key, version)` whose envelope had no `contentHash`. * @@ -259,26 +291,55 @@ export const _warnedHashless = new Set(); function warnHashless(raw: RawSkillObject): void { const identity = `${String(raw.key)}:${String(raw.version)}`; if (_warnedHashless.has(identity)) return; - _warnedHashless.add(identity); + rememberHashless(identity); error( `Skill '${String(raw.key)}' version ${String(raw.version)} arrived without a contentHash and will be ` + `withheld. ${HASHLESS_ADVICE}`, ); } +/** Forgets the summary, so a relapse after a recovery is reported afresh. */ +function forgetHashlessSummary(): void { + for (const entry of _warnedHashless) { + if (entry.startsWith(SUMMARY_MARKER)) _warnedHashless.delete(entry); + } +} + /** - * One error per committed payload in which *nothing* the store now holds can - * possibly verify. + * One error per *distinct* committed store in which nothing held can possibly + * verify. * * Fires at delivery time, so the condition is visible in a process that boots, * materializes nothing, and exits — which is the shape a skills deployment fails * in. The accessor boundary's own withholding summary only speaks once a caller * asks. + * + * Spoken when the condition becomes true and whenever the hashless objects + * change, and not again for a store that has not moved: an unchanging payload + * re-delivered on every poll describes one problem, not one per interval. A + * store that recovers and relapses is reported again. */ function warnIfNothingCanVerify(held: RawSkillObject[]): void { - if (held.length === 0) return; const hashless = held.filter((raw) => typeof raw.contentHash !== 'string'); - if (hashless.length !== held.length) return; + if (hashless.length === 0) { + // Everything held verifies, so there is nothing outstanding to remember. + _warnedHashless.clear(); + return; + } + if (hashless.length !== held.length) { + forgetHashlessSummary(); + return; + } + + const summary = + SUMMARY_MARKER + + hashless + .map((raw) => `${String(raw.key)}:${String(raw.version)}`) + .sort() + .join('\u0000'); + if (_warnedHashless.has(summary)) return; + forgetHashlessSummary(); + rememberHashless(summary); error( `All ${held.length} skill object(s) in the delivered payload arrived without a contentHash. No skill content ` + `will resolve from this store. ${HASHLESS_ADVICE}`, @@ -615,6 +676,22 @@ function revocationsBetween(current: SkillObjectSet, next: SkillObjectSet): RawS .map((raw) => ({ key: raw.key, version: isValidSkillVersion(raw.version) ? raw.version : null })); } +/** + * How many of `revoked` are true revocations rather than version moves. + * + * Counted per key, not per tombstone: a key `next` still holds under some other + * version has moved, and only a key that left the payload entirely is gone. This + * is what `objectsRevoked` counts; `changes` carries every tombstone regardless. + */ +function keysFullyRevoked(revoked: RawSkillObject[], next: SkillObjectSet): number { + const departed = new Set(); + for (const raw of revoked) { + const key = String(raw.key); + if (next.get(key, null) === null) departed.add(key); + } + return departed.size; +} + type MutableDiagnostics = { -readonly [K in keyof StoreDiagnostics]: StoreDiagnostics[K] }; function freshDiagnostics(): MutableDiagnostics { @@ -788,7 +865,8 @@ export class ProtocolReader { if (this.intent === INTENT_TRANSFER_FULL) { const revoked = revocationsBetween(this.committed, this.pending); this.changes.push(...revoked); - this.diagnostics.objectsRevoked += revoked.length; + // Every departure is reported; only a key that left counts as revoked. + this.diagnostics.objectsRevoked += keysFullyRevoked(revoked, this.pending); } this.committed.replaceWith(this.pending); warnIfNothingCanVerify(this.committed.allRaw()); From 3ebc7aca257d50698c92196127de5ff60ca39b77 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 15:02:15 -0400 Subject: [PATCH 23/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20tighten=20the=20store's=20listener=20and=20wait=20lifecycle?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three lifecycle edges on the skill stores' listener and wait surfaces, all of which produced a store that looked live and was not. `addListener` accepted a listener under any kind and only ever notified 'skill', so `addListener('flag', fn)` was retained and never called — the same silent-watcher failure `watchSkills` already refuses loudly for a store with no `addListener` at all. Both shipped stores now throw for a kind they cannot notify. `removeListener` still tolerates any kind, so `SkillWatcher.close` can keep detaching unconditionally. `waitForSkills` pushed a resolver closure into `firstPayloadWaiters` and left it there when the timeout fired, retaining one entry per timed-out call for the store's lifetime. The timeout path now removes its own waiter. A wait started after `close()` saw no first payload and no failure, so it built a fresh timer and ate the full duration — while the README promised `false` for a closed store. `close` now records that it ran and `waitForSkills` short-circuits on it, as it already did for delivery that gave up. `failed` stays null: closing is not a failure. `start` refuses a closed store rather than opening a second delivery loop on one the caller shut down. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 10 +- .../client/src/__tests__/skills-fdv2.test.ts | 112 ++++++++++++++++++ packages/client/src/__tests__/skills.test.ts | 9 +- packages/client/src/skills-fdv2.ts | 69 +++++++++-- packages/client/src/skills.ts | 16 ++- packages/client/src/types.ts | 6 + 6 files changed, 205 insertions(+), 17 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 11481616..e5cf62d4 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -325,8 +325,8 @@ example above, materializes only what the resolved variation actually asked for. | `getSkills(refs)` | Batch form. Accepts `SkillReference` values and bare key strings (string = latest). Results follow input order; missing or unverifiable entries are omitted. | | `allSkills()` | Every verified skill the store holds. | | `writeSkills(skills, root, options?)` | Materialize to `//SKILL.md`. Accepts `Skill` / `SkillReference` / key strings, or the literal `'*'`. Returns a `ReconcileReport`. Throws for a caller error — an unusable `root`, a bare string other than `'*'` — as distinct from the per-skill `error` actions in the report. | -| `InMemorySkillStore` | A `SkillStore` backed by a plain object. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Holds one object per key, so it answers a version pin with what it has and lets the accessor refuse a mismatch. | -| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | +| `InMemorySkillStore` | A `SkillStore` backed by a plain object. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Holds one object per key, so it answers a version pin with what it has and lets the accessor refuse a mismatch. `addListener` throws for any kind but `'skill'`. | +| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | | `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object`, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | @@ -368,7 +368,11 @@ try { } ``` -**`waitForSkills` orders boot against the first payload.** It resolves `true` once a payload has been committed, or once a `304` confirms the payload already held is the current one. It resolves `false` if the wait times out, the store is closed, or delivery stops for good — a fatal error such as an unauthorized key resolves it right away rather than at the timeout, so a boot gated on the return value does not proceed on a dead store. Read `failed` to tell that case from a timeout. +**`waitForSkills` orders boot against the first payload.** It resolves `true` once a payload has been committed, or once a `304` confirms the payload already held is the current one. It resolves `false` if the wait times out, the store is closed, or delivery stops for good — a fatal error such as an unauthorized key resolves it right away rather than at the timeout, so a boot gated on the return value does not proceed on a dead store. Neither a closed store nor one that has given up waits at all, whether the wait was already pending when it happened or started afterwards. Read `failed` to tell a store that gave up from one that timed out; a store you closed yourself leaves `failed` as `null`, because closing is not a failure. + +**`close()` is final.** It stops delivery and does not undo it: `start()` throws on a closed store rather than opening a second connection, since a store that looks live and is not is worse than one that plainly refuses. Retrieval needs no restart — a closed store keeps answering from the content it received — so construct a new store if you need delivery again. + +**`addListener` observes skill changes only.** The store notifies changes to `'skill'` objects; flag and segment objects on the same connection are skipped, never dispatched. A listener registered under any other kind therefore throws rather than being accepted and silently never firing. `removeListener` tolerates any kind, so a consumer that detaches on close can do so unconditionally. `watchSkills` is the intended consumer of both. **Nothing above the store changes.** The accessors, integrity verification, and `writeSkills` are transport-agnostic: they see raw objects through the `SkillStore` seam and cannot tell which store produced them. Everything documented above about verification and reconcile semantics applies unchanged. diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 6a35c04e..72c42db2 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -2437,6 +2437,31 @@ describe('watchSkills', () => { const listeners = (store as unknown as { listeners: Map }).listeners; expect(listeners.get(SKILL_OBJECT_KIND)).toEqual([]); }); + + it('refuses a listener for a kind the FDv2 store cannot notify', () => { + const store = pollStore(); + expect(() => store.addListener('flag', vi.fn())).toThrow(/never fire/); + expect(() => store.addListener('segment', vi.fn())).toThrow(/never fire/); + }); + + it('retains no listener it refused, so nothing is left to leak', () => { + const store = pollStore(); + expect(() => store.addListener('flag', vi.fn())).toThrow(); + const listeners = (store as unknown as { listeners: Map }).listeners; + expect(listeners.has('flag')).toBe(false); + }); + + it('still lets a refused listener be detached unconditionally', async () => { + // `SkillWatcher.close` removes without knowing whether the add succeeded. + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + const fn = vi.fn(); + expect(() => store.addListener('flag', fn)).toThrow(); + expect(() => store.removeListener('flag', fn)).not.toThrow(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(fn).not.toHaveBeenCalled(); + }); }); // ─── Lifecycle ─────────────────────────────────────────────────────────────── @@ -2476,6 +2501,93 @@ describe('lifecycle', () => { expect(await store.waitForSkills(50)).toBe(false); }); + it('retains no waiter for a wait that timed out', async () => { + const store = new FDv2SkillStore(SDK_KEY, { + mode: 'poll', + pollIntervalMs: 60_000, + requester: new ScriptedRequester([new Promise(() => {})]), + }); + openStores.push(store); + const waiters = (store as unknown as { firstPayloadWaiters: unknown[] }).firstPayloadWaiters; + for (let i = 0; i < 3; i += 1) expect(await store.waitForSkills(10)).toBe(false); + // Every timed-out wait left behind is retained for the store's lifetime. + expect(waiters).toHaveLength(0); + }); + + it('resolves waitForSkills false at once for a wait started after close', async () => { + // A connection that is open and has delivered nothing: the store is neither + // holding a payload nor has it given up, which is the state in which a wait + // used to run its timeout out in full. + endpoint.holdStreamOpen = true; + endpoint.queueStream([]); + const store = streamStore(); + store.start(); + expect(await waitUntil(() => endpoint.requests.length > 0)).toBe(true); + await store.close(); + const started = Date.now(); + expect(await store.waitForSkills(3_000)).toBe(false); + expect(Date.now() - started).toBeLessThan(1_000); + }); + + it('resolves waitForSkills false at once for a store closed before it started', async () => { + const store = pollStore(); + await store.close(); + const started = Date.now(); + expect(await store.waitForSkills(3_000)).toBe(false); + expect(Date.now() - started).toBeLessThan(1_000); + }); + + it('does not report a clean close as a delivery failure', async () => { + endpoint.holdStreamOpen = true; + endpoint.queueStream([]); + const store = streamStore(); + store.start(); + expect(await waitUntil(() => endpoint.requests.length > 0)).toBe(true); + await store.close(); + expect(await store.waitForSkills(50)).toBe(false); + expect(store.failed).toBeNull(); + }); + + it('still answers waitForSkills true after close when a payload arrived', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + await store.close(); + expect(await store.waitForSkills(3_000)).toBe(true); + }); + + it('refuses to restart a closed store', async () => { + const store = pollStore(); + store.start(); + await store.close(); + expect(() => store.start()).toThrow(/cannot be restarted/); + }); + + it('opens no second delivery loop once closed', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore({ pollIntervalMs: 60_000 }); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + await store.close(); + const requests = endpoint.requests.length; + // The refusal itself is asserted above; what matters here is that the store + // opened nothing, rather than throwing after starting a second loop. + try { + store.start(); + } catch { + /* expected */ + } + await new Promise((resolve) => setTimeout(resolve, 50)); + expect(endpoint.requests).toHaveLength(requests); + }); + + it('refuses to restart a store closed before it ever started', async () => { + const store = pollStore(); + await store.close(); + expect(() => store.start()).toThrow(/cannot be restarted/); + }); + it('satisfies the seam before it starts', () => { const store = new FDv2SkillStore(SDK_KEY); expect(store.getObject(SKILL_OBJECT_KIND, 'anything')).toBeNull(); diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 5cb916a6..d9504bb2 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -412,10 +412,15 @@ describe('InMemorySkillStore', () => { expect(seen[0]).toBe(raw); }); - it('records but never fires a listener registered for another kind', () => { + it('refuses a listener for a kind it cannot notify', () => { + const store = new InMemorySkillStore(); + expect(() => store.addListener('flag', vi.fn())).toThrow(/never fire/); + }); + + it('keeps no listener it refused', () => { const store = new InMemorySkillStore(); const other = vi.fn(); - store.addListener('flag', other); + expect(() => store.addListener('flag', other)).toThrow(); store.put(rawSkill({ key: 'a' })); expect(other).not.toHaveBeenCalled(); }); diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index cccee819..ba6b1ff8 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1481,6 +1481,11 @@ export class FDv2SkillStore implements SkillStore { private controller: AbortController | null = null; private loop: Promise | null = null; private failedReason: string | null = null; + // Set by `close`, and deliberately not folded into `failedReason`: closing is + // the caller's own decision, not a delivery failure, so `failed` stays `null`. + // It is what lets `waitForSkills` answer a closed store immediately and + // `start` refuse to reopen one. + private closed = false; private firstPayload = false; private readonly firstPayloadWaiters: Array<() => void> = []; // Recoverable failures in a row, cleared by any sign of a working server: a @@ -1521,8 +1526,19 @@ export class FDv2SkillStore implements SkillStore { * Starts delivery. Idempotent; returns `this` so it chains. * * Does not await: use `waitForSkills` when boot ordering matters. + * + * Throws once the store has been closed. `close` is final — a store is not a + * connection to be reopened — and restarting delivery on one would produce a + * store that looks live and is not, which is the failure this whole surface is + * built to refuse. Construct a new store instead. */ start(): this { + if (this.closed) { + throw new Error( + 'this FDv2SkillStore has been closed and cannot be restarted; construct a new FDv2SkillStore instead. ' + + 'A closed store still answers from the content it received, so retrieval needs no restart.', + ); + } if (this.loop !== null) return this; this.controller = new AbortController(); this.loop = this.run(this.controller.signal); @@ -1537,11 +1553,17 @@ export class FDv2SkillStore implements SkillStore { * failure or an empty reconcile mid-flight. `shutdown()` is what detaches the * store from the accessors. * + * Final, and one-way: `start` throws afterwards rather than opening a second + * delivery loop. `failed` stays `null` — closing is not a failure — and + * `waitForSkills` answers `false` at once rather than waiting out its timeout + * for a payload that cannot arrive. + * * Aborting the signal is what interrupts an open stream: the delivery task * spends its life awaiting a read, and a flag it never checks would leave a * healthy stream running until the process exited. */ async close(): Promise { + this.closed = true; this.controller?.abort(); const loop = this.loop; this.loop = null; @@ -1559,22 +1581,38 @@ export class FDv2SkillStore implements SkillStore { * was closed, or delivery stopped for good and no payload will arrive; see * `failed` to tell the last case from the others. Boot ordering is all this * answers; `diagnostics` answers the rest. + * + * Neither a closed store nor one whose delivery has stopped for good waits: + * both answer immediately, whether the wait was already pending when it + * happened or started afterwards. A store that did receive a payload still + * answers `true` after close, matching what it will still serve. */ waitForSkills(timeoutMs = 10_000): Promise { if (this.firstPayload) return Promise.resolve(true); - // Delivery that has already stopped for good has no payload left to wait - // for, so answer now rather than after the timeout. - if (this.failedReason !== null) return Promise.resolve(false); + // A closed store, and delivery that has already stopped for good, both have + // no payload left to wait for: answer now rather than after the timeout. + if (this.closed || this.failedReason !== null) return Promise.resolve(false); return new Promise((resolve) => { - const timer = setTimeout(() => resolve(this.firstPayload), timeoutMs); - (timer as unknown as { unref?: () => void }).unref?.(); - this.firstPayloadWaiters.push(() => { + const waiter = (): void => { clearTimeout(timer); resolve(this.firstPayload); - }); + }; + const timer = setTimeout(() => { + // The timed-out waiter takes itself out of the list. Left in, every + // expired wait would be retained for the lifetime of the store. + this.dropWaiter(waiter); + resolve(this.firstPayload); + }, timeoutMs); + (timer as unknown as { unref?: () => void }).unref?.(); + this.firstPayloadWaiters.push(waiter); }); } + private dropWaiter(waiter: () => void): void { + const index = this.firstPayloadWaiters.indexOf(waiter); + if (index !== -1) this.firstPayloadWaiters.splice(index, 1); + } + private releaseWaiters(): void { while (this.firstPayloadWaiters.length > 0) this.firstPayloadWaiters.pop()?.(); } @@ -1622,8 +1660,20 @@ export class FDv2SkillStore implements SkillStore { * `fn` runs inline on the delivery task. Keep it cheap and non-blocking: work * done there delays the next event. An exception it throws is logged and * swallowed, because a broken listener must not be able to kill delivery. + * + * Throws for any `kind` but `'skill'`. This store notifies skill changes and + * nothing else — flag and segment objects on the same connection are skipped, + * never dispatched — so accepting a listener on another kind would hand back a + * watcher that silently never fires, which is indistinguishable from one whose + * objects never changed. */ addListener(kind: string, fn: (raw: RawSkillObject) => unknown): void { + if (kind !== SKILL_OBJECT_KIND) { + throw new Error( + `FDv2SkillStore notifies only '${SKILL_OBJECT_KIND}' changes, so a listener on ${JSON.stringify(kind)} ` + + `would never fire. Register it on '${SKILL_OBJECT_KIND}'.`, + ); + } const existing = this.listeners.get(kind); if (existing) existing.push(fn); else this.listeners.set(kind, [fn]); @@ -1634,7 +1684,10 @@ export class FDv2SkillStore implements SkillStore { * during one commit takes effect from the next. * * Removes one occurrence; removing a callable that is not registered is a - * no-op, so `SkillWatcher.close` can detach unconditionally. + * no-op, so `SkillWatcher.close` can detach unconditionally. Unlike + * `addListener` this tolerates any `kind` — a kind that holds no listeners is + * simply nothing to remove — so detaching never has to know which kind it + * attached under. */ removeListener(kind: string, fn: (raw: RawSkillObject) => unknown): void { const listeners = this.listeners.get(kind); diff --git a/packages/client/src/skills.ts b/packages/client/src/skills.ts index 8d189f43..9a7a1fa3 100644 --- a/packages/client/src/skills.ts +++ b/packages/client/src/skills.ts @@ -123,11 +123,18 @@ export class InMemorySkillStore implements SkillStore { /** * Registers `fn` to be called with each raw object `put` under `kind`. * - * Only `kind === SKILL_OBJECT_KIND` is ever notified, because `put` only - * accepts skill objects; a listener registered under any other kind is recorded - * and never fires. + * Throws for any `kind` but `'skill'`. `put` accepts skill objects and nothing + * else, so this store has no other kind to notify, and a listener it accepted + * on one would silently never fire — indistinguishable from a store whose + * objects never changed. `FDv2SkillStore.addListener` refuses the same way. */ addListener(kind: string, fn: (raw: RawSkillObject) => unknown): void { + if (kind !== SKILL_OBJECT_KIND) { + throw new Error( + `InMemorySkillStore notifies only '${SKILL_OBJECT_KIND}' changes, so a listener on ${JSON.stringify(kind)} ` + + `would never fire. Register it on '${SKILL_OBJECT_KIND}'.`, + ); + } const existing = this.listeners.get(kind); if (existing) existing.push(fn); else this.listeners.set(kind, [fn]); @@ -138,7 +145,8 @@ export class InMemorySkillStore implements SkillStore { * * Removes one occurrence: a callable registered twice must be removed twice. * Removing a callable that is not registered is a no-op, not an error, so a - * consumer that detaches on close can do so unconditionally. + * consumer that detaches on close can do so unconditionally. Unlike + * `addListener` this tolerates any `kind`, for the same reason. */ removeListener(kind: string, fn: (raw: RawSkillObject) => unknown): void { const listeners = this.listeners.get(kind); diff --git a/packages/client/src/types.ts b/packages/client/src/types.ts index 05f47280..76055ffd 100644 --- a/packages/client/src/types.ts +++ b/packages/client/src/types.ts @@ -270,6 +270,12 @@ export type RawSkillObject = { * should implement `removeListener` too; it removes one occurrence of `fn` under * `kind` and is a no-op when `fn` is not registered. * + * `addListener` should throw for a `kind` the store cannot notify rather than + * accept the listener and never call it — both shipped stores notify `'skill'` + * only, and do throw. A watcher that silently never fires is indistinguishable + * from one whose objects never changed. `removeListener` has no such constraint: + * a kind holding no listeners is simply nothing to remove. + * * Everything a store serves is untrusted input. The transport is not part of the * trust boundary — key, version, size, and content hash are revalidated at the * accessor boundary on every pass. From 7106c4d97081e37ceedcfa43f422fb168bfd2188 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 16:09:14 -0400 Subject: [PATCH 24/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20stream=20from=20stream.launchdarkly.com=20by=20default?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `FetchRequester` sent `GET /sdk/stream` to the polling host. LaunchDarkly serves streaming from `stream.launchdarkly.com` and polling from `sdk.launchdarkly.com`, matching the base server-side SDK's defaults, so the default streaming store would have connected to the wrong host on first contact with a real environment. Adds `DEFAULT_STREAM_URI` and a `streamUri` option. A `baseUri` given without `streamUri` is used for both endpoints, so a relay or private instance serving both from one host, and the test fake, keep working unchanged. The hook's biome checks were run by hand from the main checkout; this worktree has no node_modules for it to find. Co-Authored-By: Claude Fable 5.1 --- packages/client/README.md | 4 +- packages/client/agents.md | 2 +- .../client/src/__tests__/skills-fdv2.test.ts | 68 +++++++++++++++++++ packages/client/src/index.ts | 2 +- packages/client/src/skills-fdv2.ts | 44 +++++++++--- 5 files changed, 107 insertions(+), 13 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index e5cf62d4..3c498aac 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -326,7 +326,7 @@ example above, materializes only what the resolved variation actually asked for. | `allSkills()` | Every verified skill the store holds. | | `writeSkills(skills, root, options?)` | Materialize to `//SKILL.md`. Accepts `Skill` / `SkillReference` / key strings, or the literal `'*'`. Returns a `ReconcileReport`. Throws for a caller error — an unusable `root`, a bare string other than `'*'` — as distinct from the per-skill `error` actions in the report. | | `InMemorySkillStore` | A `SkillStore` backed by a plain object. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Holds one object per key, so it answers a version pin with what it has and lets the accessor refuse a mismatch. `addListener` throws for any kind but `'skill'`. | -| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | +| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | | `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object`, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | @@ -349,7 +349,7 @@ example above, materializes only what the resolved variation actually asked for. #### Receiving skills from LaunchDarkly -`InMemorySkillStore` is for tests and bring-your-own-content. In production, skill content arrives through `FDv2SkillStore`, which speaks LaunchDarkly's SDK-facing FDv2 delivery channel — the same `GET /sdk/poll` and `GET /sdk/stream` endpoints the base SDK's FDv2 data source uses, authenticated with the environment's server-side SDK key. +`InMemorySkillStore` is for tests and bring-your-own-content. In production, skill content arrives through `FDv2SkillStore`, which speaks LaunchDarkly's SDK-facing FDv2 delivery channel — the same `GET /sdk/poll` and `GET /sdk/stream` endpoints the base SDK's FDv2 data source uses, authenticated with the environment's server-side SDK key. Polling goes to `sdk.launchdarkly.com` and streaming to `stream.launchdarkly.com` by default; set `baseUri` alone to point both at one host, such as a relay or private instance, or `streamUri` as well when the two differ. ```ts import { FDv2SkillStore, initClient, watchSkills } from '@launchdarkly/ai-server'; diff --git a/packages/client/agents.md b/packages/client/agents.md index 8e017c64..382f10bb 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -97,7 +97,7 @@ export { parseUsage, normalizeMode, parseAiConfig } from './tracking.js'; export { skillRefs, getSkill, getSkillResult, getSkills, allSkills, InMemorySkillStore } from './skills.js'; export { SKILL_OBJECT_KIND, MAX_SKILL_CONTENT_BYTES } from './skills-core.js'; export { writeSkills, SKILL_FILENAME, MANIFEST_FILENAME, MANIFEST_VERSION } from './skills-fs.js'; -export { FDv2SkillStore, DEFAULT_BASE_URI } from './skills-fdv2.js'; +export { FDv2SkillStore, DEFAULT_BASE_URI, DEFAULT_STREAM_URI } from './skills-fdv2.js'; export { watchSkills, SkillWatcher, DEFAULT_DEBOUNCE_MS } from './skills-watch.js'; export type { WriteSkillsOptions } from './skills-fs.js'; export { createSkill, createSkillOutcome, createSkillReference } from './types.js'; diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 72c42db2..47cf7f05 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -31,8 +31,10 @@ import { _warnedHashless, backoffDelayMs, classifyStatus, + DEFAULT_BASE_URI, DEFAULT_POLL_TIMEOUT_MS, DEFAULT_STREAM_READ_TIMEOUT_MS, + DEFAULT_STREAM_URI, decodePollBody, FDV2_KEY_DELIMITER, FDV2_OBJECT_KIND, @@ -2709,3 +2711,69 @@ describe('timeouts', () => { expect(() => new FDv2SkillStore(SDK_KEY, { readTimeoutMs: value })).toThrow(/readTimeoutMs/); }); }); + +// ─── Endpoints ─────────────────────────────────────────────────────────────── + +describe('endpoints', () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + /** Runs one poll and one stream connect through a stubbed `fetch`; returns the URLs requested. */ + async function requestedUrls(requester: FetchRequester): Promise<{ poll: string; stream: string }> { + const urls: string[] = []; + vi.spyOn(globalThis, 'fetch').mockImplementation(async (input) => { + urls.push(String(input)); + return new Response(null, { status: 304 }); + }); + await requester.poll(null, null, new AbortController().signal); + await expect(requester.stream(null, new AbortController().signal)).rejects.toThrow(); + return { poll: urls[0], stream: urls[1] }; + } + + it('polls sdk.launchdarkly.com and streams from stream.launchdarkly.com by default', async () => { + expect(DEFAULT_BASE_URI).toBe('https://sdk.launchdarkly.com'); + expect(DEFAULT_STREAM_URI).toBe('https://stream.launchdarkly.com'); + const requester = requesterOf(new FDv2SkillStore(SDK_KEY)); + expect(requester.baseUri).toBe(DEFAULT_BASE_URI); + expect(requester.streamUri).toBe(DEFAULT_STREAM_URI); + const { poll, stream } = await requestedUrls(requester); + expect(poll).toBe('https://sdk.launchdarkly.com/sdk/poll'); + expect(stream).toBe('https://stream.launchdarkly.com/sdk/stream'); + }); + + it('sends both endpoints to a custom baseUri when no streamUri is given', async () => { + const requester = requesterOf(new FDv2SkillStore(SDK_KEY, { baseUri: 'https://relay.example.com/' })); + const { poll, stream } = await requestedUrls(requester); + expect(poll).toBe('https://relay.example.com/sdk/poll'); + expect(stream).toBe('https://relay.example.com/sdk/stream'); + }); + + it('lets streamUri differ from baseUri', async () => { + const requester = requesterOf( + new FDv2SkillStore(SDK_KEY, { baseUri: 'https://sdk.example.com', streamUri: 'https://stream.example.com/' }), + ); + const { poll, stream } = await requestedUrls(requester); + expect(poll).toBe('https://sdk.example.com/sdk/poll'); + expect(stream).toBe('https://stream.example.com/sdk/stream'); + }); + + it('keeps the default poll host when only streamUri is given', () => { + const requester = requesterOf(new FDv2SkillStore(SDK_KEY, { streamUri: 'https://stream.example.com' })); + expect(requester.baseUri).toBe(DEFAULT_BASE_URI); + expect(requester.streamUri).toBe('https://stream.example.com'); + }); + + it('carries the basis to whichever host the request goes to', async () => { + const requester = new FetchRequester(SDK_KEY, 'https://sdk.example.com', 1000, 'https://stream.example.com'); + const urls: string[] = []; + vi.spyOn(globalThis, 'fetch').mockImplementation(async (input) => { + urls.push(String(input)); + return new Response(null, { status: 304 }); + }); + await requester.poll('(p:a:1)', null, new AbortController().signal); + await expect(requester.stream('(p:a:1)', new AbortController().signal)).rejects.toThrow(); + expect(urls[0]).toBe('https://sdk.example.com/sdk/poll?basis=%28p%3Aa%3A1%29'); + expect(urls[1]).toBe('https://stream.example.com/sdk/stream?basis=%28p%3Aa%3A1%29'); + }); +}); diff --git a/packages/client/src/index.ts b/packages/client/src/index.ts index da2851d9..fe66821d 100644 --- a/packages/client/src/index.ts +++ b/packages/client/src/index.ts @@ -22,7 +22,7 @@ export { compose, globalRegistry, Registry } from './registry.js'; export { allSkills, getSkill, getSkillResult, getSkills, InMemorySkillStore, skillRefs } from './skills.js'; export { MAX_SKILL_CONTENT_BYTES, SKILL_OBJECT_KIND } from './skills-core.js'; export type { FDv2Mode, FDv2SkillStoreOptions, StoreDiagnostics } from './skills-fdv2.js'; -export { DEFAULT_BASE_URI, FDv2SkillStore } from './skills-fdv2.js'; +export { DEFAULT_BASE_URI, DEFAULT_STREAM_URI, FDv2SkillStore } from './skills-fdv2.js'; export type { WriteSkillsOptions } from './skills-fs.js'; export { MANIFEST_FILENAME, MANIFEST_VERSION, SKILL_FILENAME, writeSkills } from './skills-fs.js'; export type { WatchSkillsOptions } from './skills-watch.js'; diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index ba6b1ff8..89b2c979 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -66,11 +66,17 @@ export const FDV2_OBJECT_KIND = 'skill'; export const FDV2_KEY_DELIMITER = ':'; /** - * Where the SDK-facing FDv2 endpoints live. Overridable for Federal instances, - * private instances, and the fake endpoint the tests run against. + * Where `GET /sdk/poll` is served. Overridable for Federal instances, private + * instances, and relay deployments. */ export const DEFAULT_BASE_URI = 'https://sdk.launchdarkly.com'; +/** + * Where `GET /sdk/stream` is served. LaunchDarkly serves streaming from a + * different host than polling, matching the base server-side SDK's defaults. + */ +export const DEFAULT_STREAM_URI = 'https://stream.launchdarkly.com'; + export const POLL_PATH = '/sdk/poll'; export const STREAM_PATH = '/sdk/stream'; @@ -1267,14 +1273,19 @@ export type Requester = { * same value, and there is deliberately no separate connect timeout. */ export class FetchRequester implements Requester { - private readonly baseUri: string; + /** Origin `GET /sdk/poll` is sent to. */ + readonly baseUri: string; + /** Origin `GET /sdk/stream` is sent to. Defaults to `baseUri`. */ + readonly streamUri: string; constructor( private readonly sdkKey: string, baseUri: string, readonly readTimeoutMs: number, + streamUri: string = baseUri, ) { this.baseUri = baseUri.replace(/\/+$/, ''); + this.streamUri = streamUri.replace(/\/+$/, ''); } /** @@ -1285,9 +1296,9 @@ export class FetchRequester implements Requester { * agent-skill payload is generic, is served regardless of it, and has no model * version of its own to ask for. */ - private url(path: string, basis: string | null): string { - if (!basis) return `${this.baseUri}${path}`; - return `${this.baseUri}${path}?${new URLSearchParams({ basis }).toString()}`; + private url(origin: string, path: string, basis: string | null): string { + if (!basis) return `${origin}${path}`; + return `${origin}${path}?${new URLSearchParams({ basis }).toString()}`; } /** @@ -1302,7 +1313,7 @@ export class FetchRequester implements Requester { // headers and body together. const deadline = readDeadline(signal, this.readTimeoutMs); try { - const response = await fetch(this.url(POLL_PATH, basis), { headers, signal: deadline.signal }); + const response = await fetch(this.url(this.baseUri, POLL_PATH, basis), { headers, signal: deadline.signal }); if (response.status === 304) return { notModified: true, events: [], etag }; if (!response.ok) throw classifyStatus(response.status, response.headers); return { @@ -1331,7 +1342,7 @@ export class FetchRequester implements Requester { const deadline = readDeadline(signal, this.readTimeoutMs); let response: Response; try { - response = await fetch(this.url(STREAM_PATH, basis), { headers, signal: deadline.signal }); + response = await fetch(this.url(this.streamUri, STREAM_PATH, basis), { headers, signal: deadline.signal }); } catch (cause) { deadline.clear(); throw readFailure(cause, 'streaming request', deadline); @@ -1398,7 +1409,18 @@ export type FDv2SkillStoreOptions = { * connection, and a revocation there arrives within one `pollIntervalMs`. */ readonly mode?: FDv2Mode; + /** + * Origin for `GET /sdk/poll`. Default {@link DEFAULT_BASE_URI}. When given + * without `streamUri`, it is used for streaming too, which is what a relay or + * private instance serving both endpoints from one host needs. + */ readonly baseUri?: string; + /** + * Origin for `GET /sdk/stream`. Default {@link DEFAULT_STREAM_URI}, or + * `baseUri` when that is given, since LaunchDarkly serves streaming from a + * separate host but a relay or private instance usually does not. + */ + readonly streamUri?: string; readonly pollIntervalMs?: number; /** * The only network timeout, in milliseconds. Its meaning and default follow @@ -1517,7 +1539,11 @@ export class FDv2SkillStore implements SkillStore { this.initialBackoffMs = options.initialBackoffMs ?? 1_000; this.maxBackoffMs = options.maxBackoffMs ?? 30_000; this.maxConsecutiveFailures = options.maxConsecutiveFailures ?? 10; - this.requester = options.requester ?? new FetchRequester(key, options.baseUri ?? DEFAULT_BASE_URI, readTimeoutMs); + const baseUri = options.baseUri ?? DEFAULT_BASE_URI; + // A custom `baseUri` alone means one host serves both endpoints; only the + // LaunchDarkly defaults split them. + const streamUri = options.streamUri ?? (options.baseUri === undefined ? DEFAULT_STREAM_URI : baseUri); + this.requester = options.requester ?? new FetchRequester(key, baseUri, readTimeoutMs, streamUri); } // -- lifecycle --------------------------------------------------------- From f13fc193681b2e5b44f5726f9c6f1d3524cc54ee Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 15 Sep 2026 16:11:37 -0400 Subject: [PATCH 25/55] =?UTF-8?q?docs(client):=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20bring=20the=20transport=20contract=20notes=20up=20t?= =?UTF-8?q?o=20date?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The delivery fixes changed rules agents.md still stated the old way, on exactly the points it exists to pin down. The consecutive-failure rule was the wrong one: the counter resets on any sign of a working server, not only at a committed payload, because a reconnect whose basis is current is answered with the `none` intent and commits nothing. Records the `reachedServer` qualifier alongside it, since dropping either half brings back one of the two failures — an unchanging environment expiring, or a goodbye-only server reconnecting unbounded and unreported. Retry delays are now floored at `initialBackoffMs` as well as clamped to `maxBackoffMs`, and a blank `Retry-After` reads as "none given" rather than as zero. The hashless summary is per distinct withheld set, not per payload, and `_warnedHashless` is capped. Adds the contracts that were established without being written down: the split poll and stream origins, HTTP 400 being recoverable exactly once against a stale selector, `addListener` refusing a foreign kind in both stores, and `close` being final with `start` throwing afterwards. Separates `objectsRevoked`'s per-key counting from the per-tombstone `changes` it is computed from. The split-origin note is here because its absence is why the stream request went to the polling host: the fake endpoint serves both from one origin, so no test can catch that. Docs only; agents.md is not published. Co-Authored-By: Claude Opus 5 --- packages/client/agents.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/packages/client/agents.md b/packages/client/agents.md index 382f10bb..870c91bf 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -57,20 +57,30 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b **No `mv` parameter, deliberately.** It selects the *flag* data model, the connection rejects any value but the flag default, and the generic agent-skill payload is served regardless of it. Sending `mv=1` — the skill payload's own model version — gets the whole connection refused. +**Polling and streaming have different default hosts.** `GET /sdk/poll` is served from `DEFAULT_BASE_URI` (`sdk.launchdarkly.com`) and `GET /sdk/stream` from `DEFAULT_STREAM_URI` (`stream.launchdarkly.com`), matching the base server-side SDK. Sending the stream request to the polling host fails against a real environment, and the fake endpoint cannot catch it because it serves both from one origin. `baseUri` given alone therefore applies to both, since a relay or private instance usually serves both from one host; only the LaunchDarkly defaults split them, and `streamUri` overrides the stream origin on its own. + **Changes commit at `payload-transferred`, not as objects arrive.** A payload version is the unit of consistency: a half-applied full transfer would publish a state the server never described, and would briefly empty the store — which, with pruning on, is the difference between a reconcile and deleting a customer's skill files. An interrupted transfer therefore leaves last known good intact, and listeners fire once per commit. +**`objectsRevoked` counts revocations, not tombstones.** A full transfer revokes by omission, so `payloadTransferred` diffs the committed set against the pending one and pushes the departures into `changes` as tombstones at `(key, version)` granularity — a listener that reads versions needs both halves of a version move. The *counter* is coarser on purpose: only a key the payload dropped altogether counts, because a key surviving under a new version was never revoked, and this is a field operators alert on. `keysFullyRevoked` is where the two granularities part. + **The first payload intent is read, and is assumed to be the skill payload.** Delivery provides one payload per credential and the protocol requires a client to ignore all but the first payload intent, so `payloads[0]` is both what arrives and what the protocol says to read. The cost of that assumption is that an `xfer-full` for somebody *else's* payload would start an empty pending set, and the next `payload-transferred` would publish it — every skill reported revoked, and with pruning on, a customer's files deleted. `ProtocolReader` therefore learns which payload skills arrive on, from the intent's `id` or from the `(p::)` selector, and declines to apply a transfer of any other: once at warning level, counted in `diagnostics.payloadsIgnored`, holding last known good. A transfer that names no payload is applied, since one-payload delivery is the common case. The residual is the first transfer of a connection — before a skill has arrived there is nothing to compare against — which is what the separate warning on a multi-payload intent is for. -**A hashless object is held, not dropped.** Verification withholds it with `missing_content_hash`; the transport's job is to make that loud (an error per object, a summary per wholly-hashless payload, `diagnostics.hashlessObjects`) rather than to work around it. Dropping it at the transport would report `absent` — indistinguishable from "no such skill" — and would let a prune delete the last known-good copy on disk. Never synthesize a hash from the delivered content: that certifies the content against itself and verifies nothing. +**A hashless object is held, not dropped.** Verification withholds it with `missing_content_hash`; the transport's job is to make that loud (an error per object, a summary each time the held store becomes wholly hashless or its withheld set changes, `diagnostics.hashlessObjects`) rather than to work around it. Neither error repeats for a payload re-delivered unchanged, which matters most in polling mode, where the same payload arrives on every interval; `_warnedHashless` remembers what has been said and is capped, so a frequently versioned environment cannot accumulate an entry per version. Dropping it at the transport would report `absent` — indistinguishable from "no such skill" — and would let a prune delete the last known-good copy on disk. Never synthesize a hash from the delivered content: that certifies the content against itself and verifies nothing. **`SkillObjectSet.snapshot` collapses to one object per key, keyed by the bare skill key.** `//SKILL.md` is a single path, so a whole-store consumer must see one object per key or a `'*'` reconcile writes the same path twice and `allSkills` returns two versions of one skill. The keys must be skill keys, not wire `key:version` keys, because `writeSkills('*')` derives its prune keep-set from them. `getObject` still resolves a pinned version out of the full set. **There is one network timeout, not two.** `readTimeoutMs` is applied through a `ReadDeadline` composed with the store's own abort signal, so connect, headers and each body read share it. Its default is per mode: `DEFAULT_POLL_TIMEOUT_MS` bounds a whole poll request, `DEFAULT_STREAM_READ_TIMEOUT_MS` bounds the gap between reads on a stream, and tripping it on a stream that has gone quiet *reconnects* — the `timeouts` suite measures the bound against a socket that accepts and never answers. Do not add a separate connect timeout. -**A body read that fails is recoverable; a protocol-reader or dispatch error is not.** `iterSse` wraps the read itself — a reset, a truncated chunk, the read deadline — as `RecoverableTransportError`, because a live stream dies mid-body far more often than it refuses to open, and the delivery loop reads anything else as a bug and stops for the process lifetime. Whatever the consumer's loop body throws while the generator is suspended at a `yield` passes through untouched, so a bug still surfaces as one. The consecutive-failure counter resets at each *committed payload* in `apply`, not when a connection returns: a stream only ever ends by being dropped, so resetting on return would count every healthy, server-recycled connection as a failure and give up on a healthy server. Every retry delay, `Retry-After` included, is clamped to `maxBackoffMs`. +**A body read that fails is recoverable; a protocol-reader or dispatch error is not.** `iterSse` wraps the read itself — a reset, a truncated chunk, the read deadline — as `RecoverableTransportError`, because a live stream dies mid-body far more often than it refuses to open, and the delivery loop reads anything else as a bug and stops for the process lifetime. Whatever the consumer's loop body throws while the generator is suspended at a `yield` passes through untouched, so a bug still surfaces as one. Every retry delay is clamped to `maxBackoffMs` and floored at `initialBackoffMs` — a server asking for no delay with `Retry-After: 0` would otherwise reconnect in a loop and burn the whole retry bound in milliseconds, and a blank `Retry-After` must read as "no delay given" rather than as zero, because `Number("")` is `0` and finite. + +**What resets the failure counter, and what escapes it.** The counter resets on any sign of a working server — a `server-intent` that parsed, or a payload that committed — not when a connection returns: a stream only ever ends by being dropped, so resetting on return would count every healthy, server-recycled connection as a failure. Waiting for a *commit* is not enough either, because a reconnect whose basis is already current is answered with the `none` intent and commits nothing, which would expire an environment whose skills never change. A non-catastrophic `goodbye` is exempt from the counter, but only on a connection that had reached a working server, which is what `reachedServer` tracks per attempt: a server that says goodbye having sent no intent delivered nothing, and exempting it would let it reconnect without limit and without any of it reaching `diagnostics` or `failed`. Keep both halves — the reset and the `reachedServer` qualifier — or one of those two failures comes back. + +**HTTP 400 is recoverable exactly once.** The `basis` selector and the `If-None-Match` etag are the only client state in the request, and a selector the server no longer accepts is the one rejection the adapter can act on — so a 400 arrives as `StaleRequestStateError`, the state is dropped, and a full transfer is requested. A 400 for a request carrying neither is the request itself being refused, and is fatal. Do not make 400 unconditionally fatal again: that strands delivery for the process lifetime on a stale selector. 405, 406, 414 and 501 stay fatal, since none of them is about client state. **`close` aborts the signal, it does not just set a flag.** The delivery task spends its life awaiting a stream read, and a flag it never checks would leave a healthy stream running until the process exited. The store's signal is the parent of every `ReadDeadline`, so one abort reaches a pending connect and a pending read alike, and a store closing is passed through rather than reported as a failure. Every backoff timer, deadline timer and the `waitForSkills` timer are `unref`ed for the same reason: a background store must not be why `node` stays up. +**The store refuses two things loudly rather than degrading.** `addListener` throws for any kind but `'skill'`, in `FDv2SkillStore` and `InMemorySkillStore` alike: neither notifies anything else, and a listener accepted on another kind would silently never fire — indistinguishable from one whose objects never changed, which is the failure `watchSkills` already refuses. `removeListener` has no such constraint, so a consumer can detach unconditionally. And `close` is final: `start` throws afterwards rather than opening a second delivery loop, because a store that looks live and is not is worse than one that plainly refuses. Closing leaves `failed` as `null` — it is the caller's decision, not a delivery failure — and both a closed store and one that gave up answer `waitForSkills` immediately rather than waiting out a timeout for a payload that cannot arrive. + --- ## Public Exports (`src/index.ts`) From 47cdc18d20c3a70ee77521a9abc42c91b41002d3 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 16 Sep 2026 14:48:00 -0400 Subject: [PATCH 26/55] docs(client): sweep transport comments and correct hashlessObjects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Most of this PR's comments were already swept in a333b1d and e8a8816; this covers the remainder plus a doc/behavior mismatch. `StoreDiagnostics.hashlessObjects` documented itself as a gauge — "nonzero means skills are being withheld", "every one of these is a skill that will never resolve" — but it is incremented as each hashless put-object is read and never decreases. So it counts objects from payloads that never commit, and in polling mode it climbs on every re-delivery of an unchanged payload. Every other field on the type is cumulative ("since the store started", "across all payloads"), the README describes the type as what the transport has seen, and the tests assert exact running totals, so the counter is right and the comment was the outlier. Documented as cumulative, with both caveats named, and the README's troubleshooting note adjusted to match. Reported by Cursor Bugbot on #50. Making the field an actual gauge would also resolve it, but that changes the semantics of a public diagnostics field, so it is left as a decision rather than assumed. Also rephrases the one remaining "Split out so ..." docblock in skills-fdv2.ts, and restores the README security-posture section: the move in e8a8816 and the Linux /proc/self/fd rewrite in the base PR collided, leaving the pre-fix text claiming the racy floor runs on every platform and that the managed root alone is the boundary. Replaced with the base PR's current version, which describes both platform paths and the ancestor requirement. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 9 +++++++-- packages/client/src/skills-fdv2.ts | 21 ++++++++++++++------- 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 3c498aac..03de9227 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -339,7 +339,12 @@ example above, materializes only what the resolved variation actually asked for. `ReconcileReport` exposes `actions`, `ok` (true iff no action is an `error`), and `errors` (the error actions, in order), so callers never re-derive the filter. Each `ReconcileAction` carries `key`, `action` (`written` | `updated` | `skipped_current` | `removed` | `error`), and nullable `version` / `path` / `error`. A failure belonging to the whole run rather than one skill — a corrupt manifest, for instance — carries the **empty string** in `key`. -**Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly, and it is a platform bound rather than a detail: Node exposes no `renameat`/`unlinkat`/`openat`, so no destructive step can be performed relative to a pinned directory descriptor. Every check here is therefore a per-component `lstat` taken immediately before the path-based operation — a check-then-use race, not a closed window — and unlike the Python SDK, which closes that window on POSIX by holding a descriptor opened `O_RDONLY|O_DIRECTORY|O_NOFOLLOW` for the whole reconcile, **this floor is what runs on every platform Node supports, Linux included.** An attacker who already has **write permission on the managed root** can therefore win a race to redirect a write or a delete outside it. Windows adds nothing worse and nothing better: reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) are deliberately not implemented in this release, and Windows is not a tested platform for it — neither SDK repository has a Windows CI runner. So write permission on the managed root is *the* security boundary here. Keep it writable only by the identity running the reconcile — see [privilege separation](#privilege-separation-the-agent-must-not-be-able-to-rewrite-its-own-skills). +**Security posture.** `writeSkills` is writing LaunchDarkly-delivered content to your disk, so it fails closed: skill keys are re-validated locally, content is hash-verified again immediately before writing, writes go through a temp file in the target's own directory and an atomic rename at mode `0644`, symlinked roots/directories/targets are refused, a target that is not a regular file (a FIFO, a device node) is refused rather than read, and a corrupt manifest suppresses every destructive action. One limitation is worth stating plainly, and it is a platform bound rather than a detail. Node exposes no `renameat`/`unlinkat`/`openat`, so no destructive step can be addressed relative to a pinned directory descriptor *directly* — and what the SDK can do about that differs by platform: + +- **On Linux, the swap window is closed.** The managed root is opened `O_RDONLY|O_DIRECTORY|O_NOFOLLOW` once and that descriptor is held for the whole reconcile, and every child — each `//`, each `SKILL.md`, each temp file, the manifest — is addressed as `/proc/self/fd//`. The kernel resolves that prefix to the inode the descriptor holds rather than to the name it was opened under, which gives it the same property an `*at()` call would: renaming the root, or replacing it with a symlink, after validation cannot redirect a write or a delete. +- **On macOS and Windows, the window is narrowed but not closed.** There is no traversable descriptor prefix (`/dev/fd//` does not resolve), so each check is a per-component `lstat` taken immediately before the path-based operation — a check-then-use race. An attacker who already has **write permission on the managed root, or on any of its ancestor directories**, can win that race and redirect a write or a delete outside the root. Windows additionally has no reparse-point checks (`GetFileAttributesW` / `FILE_FLAG_OPEN_REPARSE_POINT`) in this release and is not a tested platform — there is no Windows CI runner. + +So on those two platforms, write permission on the managed root **and on its ancestors** is *the* security boundary here. Keep them writable only by the identity running the reconcile — see [privilege separation](#privilege-separation-the-agent-must-not-be-able-to-rewrite-its-own-skills). The ancestor requirement is not a technicality: for a root of `.claude/skills`, the parent is `.claude`, which an agent identity often owns, and that alone is enough. **Two constraints on skill keys, imposed here rather than by the data model.** A key becomes a single directory name, so `writeSkills` rejects — as a reported `error` action, on every platform — a key over 255 bytes and the 22 Windows reserved device names (`con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9`). Both are checked in the filesystem layer only: a key like `aux` remains valid everywhere else, so an AI Config referencing it still parses and its other skills still materialize. One residual the SDK cannot check for you: the 255-byte bound is per *path component*, not on the total path, so `//SKILL.md` can still exceed Windows' 260-character `MAX_PATH` if the root is deep and the key is long. The root is yours, so budget for it there. @@ -386,7 +391,7 @@ try { > **Beta caveats, worth knowing before you deploy.** Payload signing does not exist on this channel yet, so delivery is TLS-only and the content hash establishes self-consistency, not origin authenticity. The FDv2 protocol is opt-in per account: without it the endpoints return HTTP 403, which the store reports as a fatal error explaining what to do. `ld-relay` does not speak the FDv2 endpoints, so relay-only deployments cannot receive skills. -**If every skill comes back empty, check `diagnostics.hashlessObjects`.** Verification requires `contentHash` on the delivered object and withholds anything without one, so a nonzero count there means skills are being withheld rather than that the environment has none. The store logs an error per hashless object, and a summary whenever the store it holds becomes wholly hashless or the withheld objects change — both naming the reason, and neither repeated for a payload re-delivered unchanged. There is deliberately no fallback that skips verification: a hash the SDK computed from the content it was handed would certify the content against itself. +**If every skill comes back empty, check `diagnostics.hashlessObjects`.** Verification requires `contentHash` on the delivered object and withholds anything without one, so a nonzero count there means hashless objects have arrived — skills are being withheld rather than the environment having none. It is a cumulative count of what the transport has seen, not the current size of the withheld set. The store logs an error per hashless object, and a summary whenever the store it holds becomes wholly hashless or the withheld objects change — both naming the reason, and neither repeated for a payload re-delivered unchanged. There is deliberately no fallback that skips verification: a hash the SDK computed from the content it was handed would certify the content against itself. #### Observability: integrity failures are logged for your SIEM diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 89b2c979..ebad0f26 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -232,12 +232,18 @@ export type StoreDiagnostics = { */ readonly payloadsIgnored: number; /** - * Skill objects whose envelope carried no `contentHash`. + * Skill objects whose envelope carried no `contentHash`, across all payloads. * - * **Nonzero means skills are being withheld.** Verification withholds a - * hashless object with `missing_content_hash`, so every one of these is a skill - * that will never resolve. The field exists so that outcome is a number a - * caller can read rather than an empty store they have to explain. + * Verification withholds a hashless object with `missing_content_hash`, so a + * nonzero count means such objects have arrived and the skills they carry will + * not resolve. The field exists so that outcome is a number a caller can read + * rather than an empty store they have to explain. + * + * Like the rest of this type it is **cumulative and never decreases.** Objects + * are counted as their events are read, so the count includes objects from a + * payload that never committed, and in polling mode it rises again every time + * an unchanged payload is re-delivered. Read it as "this has happened", not as + * the size of the currently withheld set. */ readonly hashlessObjects: number; /** @@ -716,8 +722,9 @@ function freshDiagnostics(): MutableDiagnostics { /** * Applies FDv2 events to an object set. Pure — no sockets, no timers, no clock. * - * Split out so the protocol can be driven without a server: the HTTP layer above - * it only has to turn bytes into `[event name, data]` pairs. + * Kept free of transport concerns so the protocol can be driven without a + * server: the HTTP layer above it only has to turn bytes into + * `[event name, data]` pairs. * * **Changes are buffered and committed at `payload-transferred`.** A payload * version is the unit of consistency: applying half of one would publish a state From 937131dc2e6964a387f3768926ae3fb448c5358f Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 16 Sep 2026 15:23:47 -0400 Subject: [PATCH 27/55] =?UTF-8?q?fix(client)!:=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20three=20contract=20fixes=20and=20the=20tests=20that?= =?UTF-8?q?=20pin=20them?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each of the three was pinned the wrong way by an existing test, which is why all three survived review. The test rewrites are the substance of this change as much as the source fixes are; each one was verified by mutation. **1. Unpublish `SKILL_OBJECT_KIND` and `MAX_SKILL_CONTENT_BYTES`.** Both were re-exported from the package root. Neither belongs there, and their absence is itself the contract: - `SKILL_OBJECT_KIND` is an SDK-side seam string, not the wire format. It is what the accessors hand `SkillStore.getObject`, and an adapter is free to map it onto whatever its transport actually uses. Exporting it advertises a claim this side cannot make and could not walk back once a caller depended on it. - `MAX_SKILL_CONTENT_BYTES` is a local enforcement bound on content the platform produces, set well above the platform's own limit precisely so that limit can move without this constant following. Publishing it semver-locks a number this side does not own, and a caller pre-flighting "will my skill fit?" against it reads the backstop rather than the real bound. The test that should have caught this asserted the opposite — titled "the five fixed values", with a comment claiming callers need the cap to pre-check content, which is the exact reading the contract rejects. It now asserts the three public constants' values and the two names' *absence*, mirroring `test_skills.py`. Tests that need the cap keep reading it from `skills-core.js`, which is where it lives. **2. A store answering under a different key is `integrity_failure`.** It returned `wrong_version`, which names a version mismatch specifically — there is deliberately no `wrong_key` to parallel it. Content was delivered and its identity did not verify, which is the one outcome a caller is expected to fail closed on; filed under `wrong_version`, skill substitution was invisible to a fail-closed caller. The JSDoc already described the intent correctly; only the token was wrong. The no-signal/no-log half is deliberate and unchanged: the check runs *after* verification has passed, so the eight-token `reason_code` vocabulary does not cover it and neither telemetry surface fires. The test now asserts that asymmetry in both directions — the outcome is `integrity_failure`, the recording emitter saw nothing, and no `ld.skills.integrity_failure` record was logged — and branches on the token rather than matching `detail` text, which is for a human. **3. `AgentControl Skill Revoked Received` omits `version` rather than emitting null.** `recordRevoked` built its properties unconditionally, so a manifest entry with a malformed version published `version: null`. It now adds the property only for a valid version, and the caller passes the raw manifest value through instead of pre-collapsing it to null. `skill_key` is now redacted through `isValidSkillKey` there too, as Python's `record_revoked` does. The key comes from the manifest — a file on disk anything with write access to the root can author — so it is exactly as attacker-controlled as a wire object. Reachability note: `pruneEntries` already refuses an entry whose key fails validation before `pruneOne` runs, so this is the second line of defense behind that guard; the end-to-end test asserts the body stays out of telemetry either way, and a direct call pins the redaction itself, which end-to-end cannot reach. Also in scope, same theme: - `getSkill` asserts `content` is a `Uint8Array`. `toEqual` alone passes for any structurally-equal value, including a plain array of the same numbers. - The `SkillOutcomeReason` exhaustiveness check named the union through `types.js`. It is the *root* export that is fixed — a union reachable only from the implementation module is not reachable by a supported import — so it now goes through the package index. The file's two root-export suites are merged while we are in here. - README and `agents.md` documented both constants as public API, and the resolution-reason table had no row for a key mismatch. The README's `integrity_failure` row also promised an `ld.skills.integrity_failure` record for every case reaching that token, which the key-mismatch path does not write. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 24 ++- packages/client/agents.md | 15 +- .../client/src/__tests__/skills-fs.test.ts | 93 ++++++++++- packages/client/src/__tests__/skills.test.ts | 147 ++++++++++++------ packages/client/src/index.ts | 1 - packages/client/src/skills-core.ts | 32 +++- packages/client/src/skills-fs.ts | 5 +- 7 files changed, 254 insertions(+), 63 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 03de9227..5c64d7e8 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -331,11 +331,20 @@ example above, materializes only what the resolved variation actually asked for. | `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object`, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | | `createSkillOutcome(init)` | Build a frozen `SkillOutcome`. Exported for tests and for wrapping your own retrieval in the same shape. | -| `SKILL_OBJECT_KIND` | `'skill'` — the delivery object kind. | | `SKILL_FILENAME` | `'SKILL.md'`. | | `MANIFEST_FILENAME` | `'.launchdarkly-skills.json'` — add this to your `.gitignore` if you do not commit materialized skills. | | `MANIFEST_VERSION` | `1`. | -| `MAX_SKILL_CONTENT_BYTES` | `10485760` (10 MiB) — the hard cap. Content above it is withheld. A backstop against absurd input, set well above the platform's own limit, which is enforced before delivery; do not read it as that limit. | + +Two further constants exist and are deliberately **not** exported. `MAX_SKILL_CONTENT_BYTES` +(`10485760`, 10 MiB) is the hard cap above which content is withheld — a local backstop against +absurd input, set well above the platform's own limit, which is enforced before delivery. +Exporting it would semver-lock a number this side does not own, and pre-flighting "will my skill +fit?" against it would read the backstop rather than the real limit; the `over_size_cap` reason +string reports the bound when it is what withheld content, which is the supported way to learn it. +`SKILL_OBJECT_KIND` (`'skill'`) is the string this SDK hands a store, and a store adapter maps +whatever its transport calls a skill onto it — publishing it would advertise an SDK-side seam as +the wire contract. Both live in this package's `skills-core` module and stay internal to it — the +package publishes a single entry point, so neither is reachable by a supported import. `ReconcileReport` exposes `actions`, `ok` (true iff no action is an `error`), and `errors` (the error actions, in order), so callers never re-derive the filter. Each `ReconcileAction` carries `key`, `action` (`written` | `updated` | `skipped_current` | `removed` | `error`), and nullable `version` / `path` / `error`. A failure belonging to the whole run rather than one skill — a corrupt manifest, for instance — carries the **empty string** in `key`. @@ -429,7 +438,7 @@ Optional fields are **omitted, never null** — the absence of `observed_hash` m | `missing_content` | `content` is absent or not a string. | | `missing_content_hash` | `contentHash` is absent or not a string. | | `not_utf8` | The content has no UTF-8 encoding (a lone surrogate), so there are no bytes LaunchDarkly could have hashed. | -| `over_size_cap` | The content exceeds `MAX_SKILL_CONTENT_BYTES`, so it is inauthentic whatever it hashes to. | +| `over_size_cap` | The content exceeds the internal `MAX_SKILL_CONTENT_BYTES` cap, so it is inauthentic whatever it hashes to. The reason string names the bound. | | `hash_mismatch` | The content does not hash to the `contentHash` delivered alongside it. | These eight tokens are the whole vocabulary, and the Python SDK emits the same eight for the same conditions — including identical JSON key order — so one parser and one alert rule cover a polyglot fleet. @@ -447,8 +456,9 @@ switch (outcome.reason) { case 'ok': return outcome.skill; // non-null exactly here case 'integrity_failure': - // Content and its declared digest disagreed. Do not proceed on a fallback: - // the safe interpretation is that skill delivery is being tampered with. + // Content and its declared digest disagreed, or the store answered under a + // different key than the one requested. Do not proceed on a fallback: the + // safe interpretation is that skill delivery is being tampered with. console.error(`refusing to start: ${outcome.detail}`); process.exit(1); case 'absent': @@ -468,8 +478,8 @@ switch (outcome.reason) { |---|---|---| | `ok` | the skill | Retrieved and verified. | | `absent` | `null` | The store holds nothing under that key. Not configured, not yet delivered, or revoked. | -| `integrity_failure` | `null` | Content failed verification and was withheld. The `ld.skills.integrity_failure` record above was written for the same failure. | -| `wrong_version` | `null` | A version was pinned and the store answered with a different one. | +| `integrity_failure` | `null` | Content was delivered and its identity did not verify, so it was withheld. Two cases reach this token: content that failed hash/size/shape verification, for which the `ld.skills.integrity_failure` record above was also written; and a store that answered under a **different key** than the one requested, which is reported here but records no signal and writes no log record (the check runs after verification has already passed). This is the outcome to fail closed on. | +| `wrong_version` | `null` | A version was pinned and the store answered with a different one. Only a version mismatch — a key mismatch is `integrity_failure`, and there is deliberately no `wrong_key`. | | `store_unavailable` | `null` | The store threw. Nothing was retrieved, so nothing is known either way. | `detail` carries the human-readable reason for every non-`ok` outcome and is `null` for `ok`. It is safe to log or surface: it names the key, the requested and held versions, and the failure category, and never skill content or a filesystem path. diff --git a/packages/client/agents.md b/packages/client/agents.md index 870c91bf..6ccf91ad 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -105,7 +105,6 @@ export { parseUsage, normalizeMode, parseAiConfig } from './tracking.js'; // Agent Skills export { skillRefs, getSkill, getSkillResult, getSkills, allSkills, InMemorySkillStore } from './skills.js'; -export { SKILL_OBJECT_KIND, MAX_SKILL_CONTENT_BYTES } from './skills-core.js'; export { writeSkills, SKILL_FILENAME, MANIFEST_FILENAME, MANIFEST_VERSION } from './skills-fs.js'; export { FDv2SkillStore, DEFAULT_BASE_URI, DEFAULT_STREAM_URI } from './skills-fdv2.js'; export { watchSkills, SkillWatcher, DEFAULT_DEBOUNCE_MS } from './skills-watch.js'; @@ -119,6 +118,15 @@ export type { When adding a new export, add it here. Handler packages must never import from sub-paths (e.g. `@launchdarkly/ai-server/dist/client`). +`MAX_SKILL_CONTENT_BYTES` and `SKILL_OBJECT_KIND` are deliberately **not** among them, and that +absence is a contract with an absence assertion behind it — do not re-export either from +`index.ts`. The cap is a local enforcement bound on content the platform produces, not a value this +SDK defines, so exporting it would semver-lock a number this side does not own; the `over_size_cap` +reason string already reports the bound when it is what withheld content. The kind is the string +this SDK hands a `SkillStore`, and an adapter maps whatever its transport calls a skill onto it, so +publishing it would advertise an SDK-side seam as the wire format — a claim this side cannot make +and could not walk back once a caller depended on it. Both stay internal to `skills-core.ts`. + --- ## Key Types @@ -347,11 +355,14 @@ Two decisions here look like unfinished work and are not. Neither should be reve | `resolveFromStore` — the store threw (also sets `unavailable`) | `store_unavailable` | | `resolveFromStore` — `raw` is not an object (null, an array, a scalar) | `absent` | | `resolveFromStore` — `verifyRawSkill` returned `null` | `integrity_failure` | + | `resolveFromStore` — `skill.key !== key` (the store answered under another key) | `integrity_failure` | | `resolveFromStore` — `skill.version !== wantedVersion` | `wrong_version` | | `resolveFromStore` — success | `ok` | | `skills-fs.ts` `resolveReference` — deadline exhausted, or no store configured (both set `unavailable`) | `store_unavailable` | - Adding a seventh row means answering "which of the five does a caller see?" before writing the code. `unavailable` stays a separate field rather than folding into `reason`: it is narrower, it is what suppresses pruning, and the two bottom rows above never reach an accessor at all. + The two `integrity_failure` rows are **not** interchangeable, and the difference is the easy thing to get wrong. The `verifyRawSkill` row fires *inside* verification, so it records the `AgentControl Skill Integrity Failure` signal and writes the `ld.skills.integrity_failure` log record with one of the eight `IntegrityReasonCode` tokens. The key-mismatch row fires *after* verification has already passed, so it does neither — it is an outcome reason and nothing else, with no signal, no log record, and no `reason_code`. Do not "fix" that by adding an emission at the key check: the eight-token vocabulary does not cover it, so it would need a ninth token (`key_mismatch`) and a matching change in the Python SDK. A test asserts the silence in both directions. + + Adding an eighth row means answering "which of the five does a caller see?" before writing the code. `unavailable` stays a separate field rather than folding into `reason`: it is narrower, it is what suppresses pruning, and the two bottom rows above never reach an accessor at all. - `wantedVersion` is passed **into** `SkillStore.getObject(kind, key, version)`, and the post-hoc `skill.version !== wantedVersion` check is kept anyway. The parameter is there because a store may hold several versions of one key and only the store can pick between them; the equality check is a **defense**, not the selection mechanism, because the store is untrusted. Removing either one is wrong: without the parameter a satisfiable pin gets reported as `wrong_version`, and without the check a lying store gets its answer through. `InMemorySkillStore` holds one object per key and so ignores the parameter by design — it answers with what it has rather than with `null`, so a pin it cannot satisfy reports `wrong_version` rather than `absent`. Giving it real multi-version semantics is a separate change that pulls in `allObjects` and `allSkills`. - `getSkill`'s contract — "resolves to `null`, never rejects; throws only when no store is configured" — is **frozen**. It is documented in its JSDoc and in the README, and every existing caller treats that `null` as "no skill", so a reason must be added *alongside* it (as `getSkillResult` was) and never by changing what `getSkill` returns. `getSkillResult` is a projection of the same `Resolution` and shares the single throw; it records no telemetry and writes no log record of its own, because the integrity record already fired inside verification before the resolution returned and reporting it again would double-count one failure. `getSkills` and `allSkills` deliberately have no reporting equivalents yet. - This package has no logger abstraction — every `console.*` call site carries a `biome-ignore` saying so. Introducing a `logger` option is a package-wide API decision affecting unrelated call sites, not a skills change; until one exists, the integrity record has to be self-describing in the string it logs, which is why the event name appears both in the prefix and in the JSON. diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index abb576ff..eb4383c4 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -28,7 +28,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { fsOps, SUPPORTS_DIR_FD, SUPPORTS_PROC_FD } from '../safe-fs.js'; import { _clearState, _setEmitterForTesting, _setStore, InMemorySkillStore, skillRefs } from '../skills.js'; -import { MAX_SKILL_CONTENT_BYTES } from '../skills-core.js'; +import { MAX_SKILL_CONTENT_BYTES, recordRevoked } from '../skills-core.js'; import { writeSkills } from '../skills-fs.js'; import type { RawSkillObject, ReconcileAction, ReconcileReport, Skill, SkillStore } from '../types.js'; import { createSkill, isValidSkillKey, parseAiConfig } from '../types.js'; @@ -1996,6 +1996,97 @@ describe('writeSkills telemetry', () => { expect(props.language).toBe('typescript'); }); + it('omits version from the Revoked signal when the manifest entry is malformed', async () => { + // The absent half of the bullet above, and the one that needs asserting: + // `version` is *omitted*, not recorded as null. The manifest is a file on + // disk, so the value in hand here is whatever was written there — a caller + // reads "no usable version was recorded" off the key's absence rather than + // off a null, and the signal never carries a null for a field that was not + // known. `toHaveProperty` is the assertion that distinguishes the two; + // reading `props.version` and comparing to undefined would pass either way. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + await mkdir(path.join(root, 'gone'), { recursive: true }); + await writeFile(path.join(root, 'gone', SKILL_MD), SKILL_BODY, 'utf-8'); + await writeManifest(root, { + manifestVersion: 1, + entries: { + [`gone/${SKILL_MD}`]: { + key: 'gone', + version: 'seven', + sha256: hash(SKILL_BODY), + writtenAt: '2026-08-14T19:00:00Z', + }, + }, + }); + + await writeSkills([], root); + + const [props] = emitter.signals(REVOKED_SIGNAL); + expect(props).toBeDefined(); + expect(props.skill_key).toBe('gone'); + expect(props.removed_from_disk).toBe(true); + expect(props).not.toHaveProperty('version'); + expect(Object.keys(props).sort()).toEqual(['language', 'removed_from_disk', 'skill_key']); + }); + + it('publishes no skill body when a manifest entry carries the body as its key', async () => { + // The manifest is as attacker-controlled as a wire object — anything with + // write access to the root can author it — so the key it carries gets the + // same shape check the wire key does before it reaches a signal. + // + // Note where this is actually stopped today: `pruneEntries` rejects an entry + // whose key fails validation before `pruneOne` runs, so the run reports a + // prune `error` and no Revoked signal is emitted at all. Either way the body + // stays out of telemetry, which is what this asserts. The redaction inside + // `recordRevoked` is the second line of defense behind that guard, and it is + // pinned directly below — an end-to-end test cannot reach it. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + const hostileKey = SKILL_BODY; + await writeManifest(root, { + manifestVersion: 1, + entries: { + [`${hostileKey}/${SKILL_MD}`]: { + key: hostileKey, + version: 1, + sha256: hash(SKILL_BODY), + writtenAt: '2026-08-14T19:00:00Z', + }, + }, + }); + + const report = await writeSkills([], root); + + for (const [, props] of emitter.records) { + for (const value of Object.values(props)) { + expect(String(value)).not.toContain('Do the thing.'); + } + } + // The entry is reported rather than silently ignored: a manifest path this + // SDK could not have written is left in place and surfaced as an error. + expect(report.ok).toBe(false); + expect(emitter.signals(REVOKED_SIGNAL)).toEqual([]); + }); + + it('redacts a manifest-supplied key that is not a valid skill key', async () => { + // Called directly because the prune path's own key validation refuses the + // entry first (see the test above), so this branch is unreachable end to + // end. It is still the parity requirement and the defense that holds if that + // guard is ever relaxed — and it asserts the placeholder's *effect*, the + // body being absent, rather than its exact spelling, which no spec fixes. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + + recordRevoked(SKILL_BODY, 3); + + const [props] = emitter.signals(REVOKED_SIGNAL); + expect(props.skill_key).not.toContain('Do the thing.'); + expect(isValidSkillKey(props.skill_key)).toBe(false); + expect(props.version).toBe(3); + expect(props.removed_from_disk).toBe(true); + }); + it('records no Revoked signal when pruning is disabled', async () => { const emitter = new RecordingEmitter(); _setEmitterForTesting(emitter); diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index d9504bb2..6e3e19bc 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -46,8 +46,8 @@ import { InMemorySkillStore, skillRefs, } from '../skills.js'; -import { allRawObjects, MAX_SKILL_CONTENT_BYTES, requireStore } from '../skills-core.js'; -import type { RawSkillObject, Skill, SkillOutcomeReason, SkillStore } from '../types.js'; +import { allRawObjects, MAX_SKILL_CONTENT_BYTES, requireStore, SKILL_OBJECT_KIND } from '../skills-core.js'; +import type { RawSkillObject, Skill, SkillStore } from '../types.js'; import { createReconcileAction, createReconcileReport, @@ -263,14 +263,34 @@ describe('ReconcileReport ok and errors', () => { // ─── Package exports ─────────────────────────────────────────────────── describe('package exports', () => { - it('exports the five fixed values from the package root with exact values', () => { - // These are API, not implementation detail: a caller needs MANIFEST_FILENAME - // to gitignore the manifest and MAX_SKILL_CONTENT_BYTES to pre-check content. - expect(packageIndex.SKILL_OBJECT_KIND).toBe('skill'); + it('exports the three fixed values from the package root with exact values', () => { + // These three are API, not implementation detail: a caller needs + // MANIFEST_FILENAME to gitignore the manifest, and all three describe an + // on-disk layout this SDK defines and a caller may have to agree with. expect(packageIndex.SKILL_FILENAME).toBe('SKILL.md'); expect(packageIndex.MANIFEST_FILENAME).toBe('.launchdarkly-skills.json'); expect(packageIndex.MANIFEST_VERSION).toBe(1); - expect(packageIndex.MAX_SKILL_CONTENT_BYTES).toBe(10 * 1024 * 1024); + }); + + it('does not export the object kind or the content cap from the package root', () => { + // The absence is itself the contract, so it gets an assertion — an + // accidental re-export from index.ts is caught here rather than shipping. + // + // SKILL_OBJECT_KIND is an SDK-side seam string rather than the wire format: + // it is what the accessors hand SkillStore.getObject, and an adapter is free + // to map it onto whatever its transport actually uses. MAX_SKILL_CONTENT_BYTES + // is a local enforcement bound on content the platform produces, set well + // above the platform's own limit precisely so that limit can move without + // this constant following — a caller pre-flighting "will my skill fit?" + // against it would be reading the backstop, not the real bound. + expect('SKILL_OBJECT_KIND' in packageIndex).toBe(false); + expect('MAX_SKILL_CONTENT_BYTES' in packageIndex).toBe(false); + + // Still reachable through the implementation module, for the store + // implementer who has to agree with them. Asserted here rather than left + // implicit so the absence above reads as "not published" rather than "gone". + expect(SKILL_OBJECT_KIND).toBe('skill'); + expect(MAX_SKILL_CONTENT_BYTES).toBe(10 * 1024 * 1024); }); it('exports the skills functions and the in-memory store from the package root', () => { @@ -302,6 +322,35 @@ describe('package exports', () => { const exhaustive: Record = { keep: true, raise: true }; expect(Object.keys(exhaustive).sort()).toEqual(['keep', 'raise']); }); + + it('exports getSkillResult and the outcome factory from the package root', () => { + expect(typeof packageIndex.getSkillResult).toBe('function'); + expect(typeof packageIndex.createSkillOutcome).toBe('function'); + }); + + it('the SkillOutcomeReason union admits exactly the five reason tokens', () => { + // The five tokens are API — customers branch on them, and every SDK + // publishes the same five for the same conditions. Adding a sixth here + // should force a matching change in the other SDKs, not just a green test. + // + // Named through the package index rather than through types.js: it is the + // *root* export that is fixed, and a union reachable only from the + // implementation module is not reachable by a supported import. + const exhaustive: Record = { + absent: true, + integrity_failure: true, + ok: true, + store_unavailable: true, + wrong_version: true, + }; + expect(Object.keys(exhaustive).sort()).toEqual([ + 'absent', + 'integrity_failure', + 'ok', + 'store_unavailable', + 'wrong_version', + ]); + }); }); // ─── skillRefs ───────────────────────────────────────────────────────── @@ -627,6 +676,10 @@ describe('getSkill', () => { expect(found).not.toBeNull(); expect(found?.key).toBe('pdf-extraction'); expect(found?.version).toBe(2); + // The declared type, asserted before the value: `content` is an opaque byte + // buffer and never a string, and `toEqual` alone passes for any + // structurally-equal value — including a plain array of the same numbers. + expect(found?.content).toBeInstanceOf(Uint8Array); expect(found?.content).toEqual(SKILL_BODY_BYTES); expect(found?.contentHash).toBe(hash(SKILL_BODY)); expect(found?.name).toBe('Test Skill'); @@ -653,11 +706,25 @@ describe('getSkill', () => { expect(await getSkill('nope')).toBeNull(); }); - it('withholds a store answering under a different key', async () => { + it('withholds a store answering under a different key as integrity_failure, silently', async () => { // The key needs the same post-fetch defense the version already has. // Identity is read off the object itself, and the store is untrusted. An // answer served under a different key would otherwise be handed back under // the key the caller asked for while carrying its own. + // + // The token is `integrity_failure`, not `wrong_version`: content was + // delivered and its identity did not verify, which is the one outcome a + // caller is expected to fail closed on. `wrong_version` names a version + // mismatch specifically and there is deliberately no `wrong_key`, so a + // substituting store filed under it would be invisible to a caller + // branching on the token. Pinned here because the choice is not recoverable + // from the message. + // + // The silence is the other half, and it is asserted in both directions: the + // check runs *after* verifyRawSkill has already passed, so neither §3.24 + // detection surface fires. A recorded signal or a logged record here would + // mean the check had migrated into verification — a real change, not a + // cosmetic one, since it would need a ninth reason_code to go with it. const aliasing: SkillStore = { getObject() { return rawSkill({ key: 'other-key' }); @@ -668,12 +735,34 @@ describe('getSkill', () => { }; _setStore(aliasing); - expect(await getSkill('asked-for')).toBeNull(); + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let errorCalls: unknown[][] = []; + let collapsed: Skill | null; + let outcome: Awaited>; + try { + collapsed = await getSkill('asked-for'); + outcome = await getSkillResult('asked-for'); + } finally { + // Read the calls out before restoring: `mockRestore` also resets the + // recorded history, so a read afterwards sees nothing. + errorCalls = [...spy.mock.calls]; + spy.mockRestore(); + } - const outcome = await getSkillResult('asked-for'); + expect(collapsed).toBeNull(); expect(outcome.skill).toBeNull(); - expect(outcome.reason).toBe('wrong_version'); - expect(outcome.detail).toBe("skill 'asked-for' is not available: the store answered under key 'other-key'"); + expect(outcome.reason).toBe('integrity_failure'); + // Branch on the token, not on the message: `detail` is for a human. Assert + // only that it is present and carries no skill body. + expect(outcome.detail).toBeTruthy(); + expect(outcome.detail).not.toContain('Do the thing.'); + + expect(emitter.records).toEqual([]); + const lines = errorCalls.map(([first]) => String(first)); + expect(lines.filter((line) => line.includes('ld.skills.integrity_failure'))).toEqual([]); }); }); @@ -1386,7 +1475,7 @@ describe('getSkillResult', () => { * different construction site in `resolveFromStore`, which is what makes the * mapping — not just the union — the thing under test. */ - const cases: Array<[SkillOutcomeReason, () => SkillStore, () => Promise]> = [ + const cases: Array<[packageIndex.SkillOutcomeReason, () => SkillStore, () => Promise]> = [ ['ok', () => storeHolding(rawSkill({ key: 'a' })), () => getSkillResult('a')], ['absent', () => new InMemorySkillStore(), () => getSkillResult('a')], [ @@ -1682,35 +1771,3 @@ describe('createSkillOutcome', () => { expect(outcome.reason).toBe('ok'); }); }); - -// ─── Package exports ─────────────────────────────────────────────────── - -describe('getSkillResult package exports', () => { - it('exports the accessor and the outcome factory from the package root', async () => { - // Imported here rather than at the top of the file so this stays a - // self-contained check of the barrel. - const pkg = await import('../index.js'); - expect(typeof pkg.getSkillResult).toBe('function'); - expect(typeof pkg.createSkillOutcome).toBe('function'); - }); - - it('the SkillOutcomeReason union admits exactly the five reason tokens', () => { - // The five tokens are API — customers branch on them, and every SDK - // publishes the same five for the same conditions. Adding a sixth here - // should force a matching change in the other SDKs, not just a green test. - const exhaustive: Record = { - absent: true, - integrity_failure: true, - ok: true, - store_unavailable: true, - wrong_version: true, - }; - expect(Object.keys(exhaustive).sort()).toEqual([ - 'absent', - 'integrity_failure', - 'ok', - 'store_unavailable', - 'wrong_version', - ]); - }); -}); diff --git a/packages/client/src/index.ts b/packages/client/src/index.ts index fe66821d..af11ecf8 100644 --- a/packages/client/src/index.ts +++ b/packages/client/src/index.ts @@ -20,7 +20,6 @@ export type { InspectConfigResult } from './lifecycle.js'; export { getClient, initClient, inspectConfig, shutdown, shutdownTelemetry, waitForTelemetry } from './lifecycle.js'; export { compose, globalRegistry, Registry } from './registry.js'; export { allSkills, getSkill, getSkillResult, getSkills, InMemorySkillStore, skillRefs } from './skills.js'; -export { MAX_SKILL_CONTENT_BYTES, SKILL_OBJECT_KIND } from './skills-core.js'; export type { FDv2Mode, FDv2SkillStoreOptions, StoreDiagnostics } from './skills-fdv2.js'; export { DEFAULT_BASE_URI, DEFAULT_STREAM_URI, FDv2SkillStore } from './skills-fdv2.js'; export type { WriteSkillsOptions } from './skills-fs.js'; diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 2cd79ec5..4c61725f 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -282,13 +282,23 @@ export function recordMaterialized( * allowlist is maintained in one place: every signal this SDK can emit is visible * in this section of this module, and nothing outside it touches `emit`. */ -export function recordRevoked(skillKey: string, version: number | null): void { - emit(SIGNAL_REVOKED, { - skill_key: skillKey, - version, +export function recordRevoked(skillKey: unknown, version: unknown): void { + // Both fields come off the manifest, which is a file on disk anything with + // write access to the root can author — exactly as attacker-controlled as a + // wire object. Same rule as `recordIntegrityFailure`: shape-check, then + // redact, so a hand-edited manifest cannot plant an arbitrary string in a + // signal that is otherwise body-free. + const safeKey = isValidSkillKey(skillKey) ? skillKey : ''; + const properties: Record = { + skill_key: safeKey, removed_from_disk: true, language: LANGUAGE, - }); + }; + // Omitted rather than emitted as null: a consumer reads "the manifest entry + // recorded no usable version" off the key's absence, and the signal never + // carries a null for a field that was not known. + if (isValidSkillVersion(version)) properties.version = version; + emit(SIGNAL_REVOKED, properties); } // --------------------------------------------------------------------------- @@ -538,9 +548,19 @@ export function resolveFromStore(store: SkillStore, key: string, wantedVersion: return { error: `skill '${key}' failed integrity verification and was withheld`, reason: 'integrity_failure' }; } if (skill.key !== key) { + // `integrity_failure` rather than `absent`: content was delivered and its + // identity did not verify, which is the one token a caller is expected to + // fail closed on. Reporting `absent` would file a store that substitutes one + // skill for another in the bucket the same caller is invited to tolerate. It + // is not `wrong_version` either — that token names a version mismatch + // specifically, and there is deliberately no `wrong_key` to parallel it. + // + // This path reports the outcome reason only: `verifyRawSkill` has already + // passed, so no integrity signal is recorded and no `ld.skills.integrity_failure` + // record is logged. That asymmetry is deliberate and pinned by a test. return { error: `skill '${key}' is not available: the store answered under key '${skill.key}'`, - reason: 'wrong_version', + reason: 'integrity_failure', }; } if (wantedVersion !== null && skill.version !== wantedVersion) { diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index 17398f71..022be3c4 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -1162,7 +1162,10 @@ async function pruneOne( delete entries[relative]; - if (removedFromDisk) recordRevoked(key, isValidSkillVersion(version) ? version : null); + // The raw manifest values go through untouched: `recordRevoked` shape-checks + // and redacts both of them itself, so the signal omits a malformed version + // rather than recording the `null` a pre-check here would hand it. + if (removedFromDisk) recordRevoked(key, version); return createReconcileAction({ key, From 8c5dba47c2018f5ce3c5282cbfb454e0d4370a45 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 16 Sep 2026 15:42:29 -0400 Subject: [PATCH 28/55] =?UTF-8?q?fix(client)!:=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20the=20multi-version=20store=20model?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `InMemorySkillStore` implemented a single-version design the spec and the Python SDK have both moved past: one object per key, `getObject`'s `version` argument ignored outright, and `allObjects` spreading the key-indexed map. Three sites depend on that model being right, and they are one change. Port Python's `_versions` / `_loose` structure into the store. Several versions of one key coexist, because they coexist in a real payload: the newest version of every skill plus every version a variation pins. `getObject` answers a pin with exactly that version, an omitted version with the newest held, and — when nothing well-formed is filed under the key — the version-less entry, so an object too malformed to carry a usable version still reaches verification and is withheld *with a signal*. Reading it as absent instead would let a tampered object look like a deleted one, and would let prune delete the last known-good copy on disk. The backing maps are real `Map`s, not plain objects. Skill keys come off the wire and a plain object inherits names that are not keys: `put`-ting `__proto__` corrupted what every later lookup resolved to, and `getObject('skill', 'constructor')` on an *empty* store answered with the `Object` function — `constructor` satisfies the skill-key pattern, so it arrived as an ordinary lookup. Nothing unverified escaped to user code either way, so this is the store's own semantics rather than a content-integrity hole. `allObjects` builds a null-prototype record for the same reason: a loose object filed under `__proto__` would otherwise set the record's prototype and vanish from the listing. `allObjects` now returns one entry per `(key, version)`, so both whole-store consumers have to collapse. `newestByKey` does it, keeping objects too malformed to carry a usable key or version rather than filtering them — dropping them early shrinks the resolved set, which on the materialization path is indistinguishable from a revocation. `allSkills` uses it because a list holding two versions of one key is not a set of skills, and `writeSkills('*')` uses it because `//SKILL.md` is a single path: writing it twice in one run is not a duplicate report but a write race against itself, resolved by whichever version iteration reached last. One caller-visible consequence, pinned by a test: a pin the store does not hold now reports `absent` rather than `wrong_version`. The store honours the pin and answers `null`, which is an absence. `wrong_version` is for a store that answers *and answers wrongly*, which is what the equality check in `resolveFromStore` defends against — so the tests for it now drive a store shaped like Python's `_WrongVersionAnsweringStore` instead of leaning on the bundled store ignoring its own argument. Tests: several were vacuous by the spec's own definition because they seeded a single version, which cannot distinguish "newest" from "the only one". The `allObjects` test also asserted the spelling of keys the `SkillStore` contract explicitly disclaims; it now asserts the objects' own `key` fields. Nine of the new or repaired assertions fail against the previous implementation. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 4 +- packages/client/agents.md | 4 +- .../client/src/__tests__/skills-fs.test.ts | 23 ++ packages/client/src/__tests__/skills.test.ts | 206 +++++++++++++++++- packages/client/src/skills-core.ts | 44 ++++ packages/client/src/skills-fs.ts | 7 +- packages/client/src/skills.ts | 113 ++++++++-- 7 files changed, 367 insertions(+), 34 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 5c64d7e8..23fc7da8 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -323,9 +323,9 @@ example above, materializes only what the resolved variation actually asked for. | `getSkill(key, { version? })` | One verified skill. Omit `version` for the newest available. Resolves to `null` for not-found or a version mismatch; throws only when no store is configured. | | `getSkillResult(key, { version? })` | The same retrieval as `getSkill`, reported instead of collapsed. Resolves to `{ skill, reason, detail }`, where `reason` is one of `ok` / `absent` / `integrity_failure` / `store_unavailable` / `wrong_version`. Throws only when no store is configured — same as `getSkill`. See [fail closed on tampering](#fail-closed-on-tampering-getskillresult). | | `getSkills(refs)` | Batch form. Accepts `SkillReference` values and bare key strings (string = latest). Results follow input order; missing or unverifiable entries are omitted. | -| `allSkills()` | Every verified skill the store holds. | +| `allSkills()` | Every verified skill the store holds, newest version per key. | | `writeSkills(skills, root, options?)` | Materialize to `//SKILL.md`. Accepts `Skill` / `SkillReference` / key strings, or the literal `'*'`. Returns a `ReconcileReport`. Throws for a caller error — an unusable `root`, a bare string other than `'*'` — as distinct from the per-skill `error` actions in the report. | -| `InMemorySkillStore` | A `SkillStore` backed by a plain object. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Holds one object per key, so it answers a version pin with what it has and lets the accessor refuse a mismatch. `addListener` throws for any kind but `'skill'`. | +| `InMemorySkillStore` | A `SkillStore` backed by in-memory maps. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Several versions of one key coexist, as they do in a real payload: `getObject` answers a pin with exactly that version and an omitted version with the newest held, and `allObjects` returns one entry per `(key, version)` under keys opaque to callers. `addListener` throws for any kind but `'skill'`. | | `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | | `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object`, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | diff --git a/packages/client/agents.md b/packages/client/agents.md index 6ccf91ad..62435cbd 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -67,7 +67,7 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b **A hashless object is held, not dropped.** Verification withholds it with `missing_content_hash`; the transport's job is to make that loud (an error per object, a summary each time the held store becomes wholly hashless or its withheld set changes, `diagnostics.hashlessObjects`) rather than to work around it. Neither error repeats for a payload re-delivered unchanged, which matters most in polling mode, where the same payload arrives on every interval; `_warnedHashless` remembers what has been said and is capped, so a frequently versioned environment cannot accumulate an entry per version. Dropping it at the transport would report `absent` — indistinguishable from "no such skill" — and would let a prune delete the last known-good copy on disk. Never synthesize a hash from the delivered content: that certifies the content against itself and verifies nothing. -**`SkillObjectSet.snapshot` collapses to one object per key, keyed by the bare skill key.** `//SKILL.md` is a single path, so a whole-store consumer must see one object per key or a `'*'` reconcile writes the same path twice and `allSkills` returns two versions of one skill. The keys must be skill keys, not wire `key:version` keys, because `writeSkills('*')` derives its prune keep-set from them. `getObject` still resolves a pinned version out of the full set. +**`SkillObjectSet.snapshot` collapses to one object per key, keyed by the bare skill key.** `//SKILL.md` is a single path, so a whole-store consumer must see one object per key or a `'*'` reconcile writes the same path twice and `allSkills` returns two versions of one skill. Both consumers also collapse for themselves through `newestByKey`, because the seam admits any store — this is the transport holding up its end, not the only guard. The keys must be skill keys, not wire `key:version` keys, because `writeSkills('*')` derives its prune keep-set from them. `getObject` still resolves a pinned version out of the full set. **There is one network timeout, not two.** `readTimeoutMs` is applied through a `ReadDeadline` composed with the store's own abort signal, so connect, headers and each body read share it. Its default is per mode: `DEFAULT_POLL_TIMEOUT_MS` bounds a whole poll request, `DEFAULT_STREAM_READ_TIMEOUT_MS` bounds the gap between reads on a stream, and tripping it on a stream that has gone quiet *reconnects* — the `timeouts` suite measures the bound against a socket that accepts and never answers. Do not add a separate connect timeout. @@ -363,7 +363,7 @@ Two decisions here look like unfinished work and are not. Neither should be reve The two `integrity_failure` rows are **not** interchangeable, and the difference is the easy thing to get wrong. The `verifyRawSkill` row fires *inside* verification, so it records the `AgentControl Skill Integrity Failure` signal and writes the `ld.skills.integrity_failure` log record with one of the eight `IntegrityReasonCode` tokens. The key-mismatch row fires *after* verification has already passed, so it does neither — it is an outcome reason and nothing else, with no signal, no log record, and no `reason_code`. Do not "fix" that by adding an emission at the key check: the eight-token vocabulary does not cover it, so it would need a ninth token (`key_mismatch`) and a matching change in the Python SDK. A test asserts the silence in both directions. Adding an eighth row means answering "which of the five does a caller see?" before writing the code. `unavailable` stays a separate field rather than folding into `reason`: it is narrower, it is what suppresses pruning, and the two bottom rows above never reach an accessor at all. -- `wantedVersion` is passed **into** `SkillStore.getObject(kind, key, version)`, and the post-hoc `skill.version !== wantedVersion` check is kept anyway. The parameter is there because a store may hold several versions of one key and only the store can pick between them; the equality check is a **defense**, not the selection mechanism, because the store is untrusted. Removing either one is wrong: without the parameter a satisfiable pin gets reported as `wrong_version`, and without the check a lying store gets its answer through. `InMemorySkillStore` holds one object per key and so ignores the parameter by design — it answers with what it has rather than with `null`, so a pin it cannot satisfy reports `wrong_version` rather than `absent`. Giving it real multi-version semantics is a separate change that pulls in `allObjects` and `allSkills`. +- `wantedVersion` is passed **into** `SkillStore.getObject(kind, key, version)`, and the post-hoc `skill.version !== wantedVersion` check is kept anyway. The parameter is there because a store may hold several versions of one key and only the store can pick between them; the equality check is a **defense**, not the selection mechanism, because the store is untrusted. Removing either one is wrong: without the parameter a satisfiable pin gets reported as `wrong_version`, and without the check a lying store gets its answer through. Both shipped stores hold several versions of one key and honour the parameter: `getObject` answers a pin with exactly that version, and an omitted version with the newest held. A pin that misses while well-formed versions exist is `absent`; a pin that matches nothing well-formed falls through to the version-less entry, so a malformed object reaches verification and is withheld with a signal rather than reading as a deletion. - `getSkill`'s contract — "resolves to `null`, never rejects; throws only when no store is configured" — is **frozen**. It is documented in its JSDoc and in the README, and every existing caller treats that `null` as "no skill", so a reason must be added *alongside* it (as `getSkillResult` was) and never by changing what `getSkill` returns. `getSkillResult` is a projection of the same `Resolution` and shares the single throw; it records no telemetry and writes no log record of its own, because the integrity record already fired inside verification before the resolution returned and reporting it again would double-count one failure. `getSkills` and `allSkills` deliberately have no reporting equivalents yet. - This package has no logger abstraction — every `console.*` call site carries a `biome-ignore` saying so. Introducing a `logger` option is a package-wide API decision affecting unrelated call sites, not a skills change; until one exists, the integrity record has to be self-describing in the string it logs, which is why the event name appears both in the prefix and in the JSON. - Signal names, property keys, wire field names (`contentHash`), the manifest filename and format, and the exported constants are **identical strings** to the Python SDK. A polyglot fleet has to reconcile the same directory identically, so changing one of these is a cross-language breaking change. diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index eb4383c4..4cc7213e 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -291,6 +291,29 @@ describe('writeSkills basic writes', () => { expect(report.actions.filter((a) => a.action !== 'error')).toHaveLength(3); }); + it('"*" collapses several versions of one key to the newest', async () => { + // //SKILL.md is a single path, so two versions of one key is not + // a duplicate report but a write race against itself, resolved by whichever + // version iteration happened to reach last. A store holding one version per + // key cannot distinguish this from "write everything", so the two-version + // seed is the whole test. Newer seeded first, so insertion order cannot pass + // for ordering. + const newer = 'newer body\n'; + const store = new InMemorySkillStore(); + store.put(rawSkill('a', 5, newer)); + store.put(rawSkill('a', 2, 'older body\n')); + _setStore(store); + + const report = await writeSkills('*', root); + + expect(report.ok).toBe(true); + const written = report.actions.filter((a) => a.action !== 'error'); + expect(written).toHaveLength(1); + expect(written[0].key).toBe('a'); + expect(written[0].version).toBe(5); + expect(await readFile(path.join(root, 'a', SKILL_MD), 'utf-8')).toBe(newer); + }); + it('reports one action per requested skill — no silent skips', async () => { const report = await writeSkills([skill('a'), skill('b')], root); expect(report.actions.map((a) => a.key).sort()).toEqual(['a', 'b']); diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 6e3e19bc..1d4b0564 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -435,11 +435,130 @@ describe('InMemorySkillStore', () => { expect(store.getObject('skill', 'a')).toEqual(raw); }); - it('allObjects returns everything held, keyed by skill key', () => { + it("allObjects returns everything held, identified by each object's own key", () => { const store = new InMemorySkillStore(); store.put(rawSkill({ key: 'a' })); store.put(rawSkill({ key: 'b' })); - expect(Object.keys(store.allObjects('skill')).sort()).toEqual(['a', 'b']); + + const held = Object.values(store.allObjects('skill')); + + // Deliberately not `Object.keys`: the record's keys are opaque + // store-internal identifiers, and `SkillStore` says identity is read off + // each object's own `key` and `version`. A test that asserted the spelling + // would be pinning an implementation detail the interface disclaims. + expect(held).toHaveLength(2); + expect(held.map((raw) => raw.key).sort()).toEqual(['a', 'b']); + }); + + it('allObjects returns one entry per (key, version), not per key', () => { + const store = new InMemorySkillStore(); + store.put(rawSkill({ key: 'a', version: 2 })); + store.put(rawSkill({ key: 'a', version: 5 })); + + const held = Object.values(store.allObjects('skill')); + + expect(held).toHaveLength(2); + expect(held.map((raw) => raw.version).sort()).toEqual([2, 5]); + }); + + it('replaces an object put twice under the same (key, version)', () => { + const store = new InMemorySkillStore(); + store.put(rawSkill({ key: 'a', version: 2, content: 'first' })); + store.put(rawSkill({ key: 'a', version: 2, content: 'second' })); + + const held = Object.values(store.allObjects('skill')); + + expect(held).toHaveLength(1); + expect(held[0].content).toBe('second'); + }); + + it('getObject with no version answers with the newest version held', () => { + const store = new InMemorySkillStore(); + // Put the newer one first, so insertion order cannot pass for ordering. + store.put(rawSkill({ key: 'a', version: 5 })); + store.put(rawSkill({ key: 'a', version: 2 })); + + expect(store.getObject('skill', 'a')?.version).toBe(5); + expect(store.getObject('skill', 'a', null)?.version).toBe(5); + }); + + it('getObject with a version answers with exactly that version', () => { + const store = new InMemorySkillStore(); + store.put(rawSkill({ key: 'a', version: 2 })); + store.put(rawSkill({ key: 'a', version: 5 })); + + expect(store.getObject('skill', 'a', 2)?.version).toBe(2); + expect(store.getObject('skill', 'a', 5)?.version).toBe(5); + }); + + it('getObject returns null for a pin that misses while well-formed versions exist', () => { + // A plain miss. Answering it with some other version the store happens to + // hold would record an integrity failure for a skill whose integrity is not + // in question. + const store = new InMemorySkillStore(); + store.put(rawSkill({ key: 'a', version: 2 })); + store.put(rawSkill({ key: 'a', version: 5 })); + + expect(store.getObject('skill', 'a', 3)).toBeNull(); + }); + + it('serves a malformed object for a pinned request when nothing well-formed is held', () => { + // An object too malformed to carry a usable version is still served, so + // verification is what withholds it — with a signal. Reading it as simply + // absent would let a tampered object look like a deleted one, and would let + // a prune delete the last known-good copy on disk. + const store = new InMemorySkillStore(); + const malformed = rawSkill({ key: 'a', version: 'not-a-version' as unknown as number }); + store.put(malformed); + + expect(store.getObject('skill', 'a', 4)).toBe(malformed); + expect(store.getObject('skill', 'a')).toBe(malformed); + expect(Object.values(store.allObjects('skill'))).toEqual([malformed]); + }); + + it('cannot be reached through the prototype', () => { + // `constructor` satisfies the skill-key pattern, so it arrives as an + // ordinary lookup. Backed by a plain object, an empty store would answer it + // with the `Object` function. Asserted on the store's own answer rather than + // an accessor's: verification already refuses a function as a non-object, so + // this is about the store's semantics being right, not a content-integrity + // hole. + const empty = new InMemorySkillStore(); + expect(empty.getObject('skill', 'constructor')).toBeNull(); + expect(empty.getObject('skill', '__proto__')).toBeNull(); + expect(empty.getObject('skill', 'toString')).toBeNull(); + expect(Object.values(empty.allObjects('skill'))).toEqual([]); + }); + + it('cannot be corrupted by a put of an inherited name', () => { + // Backed by a plain object, this `put` writes the *prototype*, and every + // later lookup whose key happens to name a field of the planted object — + // `name` and `description` both satisfy the skill-key pattern — answers with + // that field's value instead of null. + const store = new InMemorySkillStore(); + const planted = rawSkill({ key: '__proto__', version: 1 }); + store.put(planted); + + expect(store.getObject('skill', 'name')).toBeNull(); + expect(store.getObject('skill', 'description')).toBeNull(); + expect(store.getObject('skill', 'content')).toBeNull(); + + store.put(rawSkill({ key: 'a', version: 1 })); + expect(store.getObject('skill', 'a')?.key).toBe('a'); + + // And the planted object is held under its own key rather than lost, so the + // listing does not silently shrink. + expect(store.getObject('skill', '__proto__')).toBe(planted); + expect(Object.values(store.allObjects('skill'))).toHaveLength(2); + }); + + it('cannot be corrupted by an inherited name in the constructor map', () => { + const planted = rawSkill({ key: '__proto__', version: 1 }); + const store = new InMemorySkillStore({ ['__proto__']: planted }); + + expect(store.getObject('skill', 'name')).toBeNull(); + expect(store.getObject('skill', 'description')).toBeNull(); + expect(store.getObject('skill', '__proto__')).toBe(planted); }); it('returns nothing for a kind other than skill', () => { @@ -687,7 +806,11 @@ describe('getSkill', () => { }); it('an omitted version means the newest available', async () => { + // Two versions, newer one seeded first: a store holding one version cannot + // distinguish "newest" from "the only one", and insertion order must not + // pass for ordering. store.put(rawSkill({ key: 'a', version: 7 })); + store.put(rawSkill({ key: 'a', version: 4 })); expect((await getSkill('a'))?.version).toBe(7); }); @@ -837,6 +960,42 @@ describe('allSkills', () => { expect((await allSkills()).map((s) => s.key)).toEqual(['good']); }); + it('returns the newest version per key', async () => { + // A list carrying two versions of one key is not a set of skills, and + // downstream `//SKILL.md` is a single path. The store holds both, + // so the collapse has to happen here. Newer seeded first, so insertion order + // cannot pass for ordering. + const store = new InMemorySkillStore(); + store.put(rawSkill({ key: 'a', version: 5 })); + store.put(rawSkill({ key: 'a', version: 2 })); + store.put(rawSkill({ key: 'b', version: 1 })); + _setStore(store); + + const found = await allSkills(); + + expect(found.filter((s) => s.key === 'a')).toHaveLength(1); + expect(found.find((s) => s.key === 'a')?.version).toBe(5); + expect(found.map((s) => s.key).sort()).toEqual(['a', 'b']); + }); + + it('keeps an unusable object in the set so verification withholds it with a signal', async () => { + // Filtering it here instead would silently shrink the resolved set, which on + // the materialization path is indistinguishable from a revocation and would + // let prune delete a live skill's last good copy. The recorded signal is + // what proves the object reached verification rather than being dropped. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + const store = new InMemorySkillStore(); + store.put(rawSkill({ key: 'good', version: 1 })); + store.put(rawSkill({ key: 'unusable', version: 'nope' as unknown as number })); + _setStore(store); + + const found = await allSkills(); + + expect(found.map((s) => s.key)).toEqual(['good']); + expect(emitter.signals(INTEGRITY_SIGNAL)).toHaveLength(1); + }); + it('returns an empty list for an empty store', async () => { _setStore(new InMemorySkillStore()); expect(await allSkills()).toEqual([]); @@ -1470,6 +1629,27 @@ describe('getSkillResult', () => { return store; } + /** + * A store that answers a pinned lookup with some other version. + * + * `wrong_version` is not reachable through `InMemorySkillStore`, which honours + * the pin and answers a miss with `null` — that is `absent`, correctly, because + * the store said it holds nothing for that pin. The reason exists for a store + * that *does* answer and answers with the wrong thing, so the test needs one: + * `resolveFromStore`'s equality check is a defense against an untrusted store, + * not the selection mechanism. + */ + function wrongVersionStore(answeredVersion = 99): SkillStore { + return { + getObject(_kind: string, key: string) { + return rawSkill({ key, version: answeredVersion }); + }, + allObjects() { + return {}; + }, + }; + } + /** * One store per reason token. The stores are shaped so each reaches a * different construction site in `resolveFromStore`, which is what makes the @@ -1484,11 +1664,7 @@ describe('getSkillResult', () => { () => getSkillResult('a'), ], ['store_unavailable', throwingStore, () => getSkillResult('a')], - [ - 'wrong_version', - () => storeHolding(rawSkill({ key: 'a', version: 3 })), - () => getSkillResult('a', { version: 2 }), - ], + ['wrong_version', () => wrongVersionStore(3), () => getSkillResult('a', { version: 2 })], ]; it.each(cases)('reports reason %s', async (reason, makeStore, run) => { @@ -1732,8 +1908,22 @@ describe('getSkillResult', () => { } }); + it('reports absent, not wrong_version, for a pin a multi-version store does not hold', async () => { + // The store honours the pin and answers `null`: it has told us it holds + // nothing at that version, which is an absence. `wrong_version` is for a + // store that answers *and answers wrongly* — the two are not the same + // report, and a caller tolerating `absent` should not be handed a + // tampering-shaped token for a version nobody published. + _setStore(storeHolding(rawSkill({ key: 'a', version: 2 }), rawSkill({ key: 'a', version: 5 }))); + + const outcome = await getSkillResult('a', { version: 3 }); + + expect(outcome.reason).toBe('absent'); + expect(outcome.skill).toBeNull(); + }); + it('detail names the key and both versions on a mismatch, and no path', async () => { - _setStore(storeHolding(rawSkill({ key: 'pdf-extraction', version: 3 }))); + _setStore(wrongVersionStore(3)); const { detail } = await getSkillResult('pdf-extraction', { version: 2 }); diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 4c61725f..631bea2e 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -479,6 +479,50 @@ export function allRawObjects(store: SkillStore): RawListing { return { objects: objects as Record, error: null }; } +/** One raw object, paired with the store key it was served under. */ +export type ServedObject = { + /** The store's own map key — opaque, and only a fallback identity. */ + readonly objectKey: string; + readonly raw: RawSkillObject; +}; + +/** + * One raw object per skill key — the highest version of each, paired with the + * store key it was served under. + * + * `allObjects` may hold several versions of one key, and both callers that + * consume the whole store want one skill per key: `allSkills` because a list + * holding two versions of one key is not a set of skills, and the `'*'` + * reconcile because `//SKILL.md` is a single path and writing it twice + * in one run is a bug rather than a policy. + * + * The store key is carried through rather than discarded because the reconcile + * attributes a failure to it when the object's own key is unusable. + * + * Objects too malformed to carry a usable key and version are **kept**, not + * dropped, so verification is what withholds them: a silently dropped object + * falls out of the requested set, and prune would then delete the last + * known-good copy already on disk. + */ +export function newestByKey(objects: Record): ServedObject[] { + // The winning version is carried beside the object rather than re-read off it: + // `RawSkillObject.version` is `unknown`, and narrowing it once at the point it + // was validated is what keeps the comparison from needing a cast. + const best = new Map(); + const unusable: ServedObject[] = []; + for (const [objectKey, raw] of Object.entries(objects)) { + const key = typeof raw === 'object' && raw !== null ? raw.key : undefined; + const version = typeof raw === 'object' && raw !== null ? raw.version : undefined; + if (!isValidSkillKey(key) || !isValidSkillVersion(version)) { + unusable.push({ objectKey, raw }); + continue; + } + const held = best.get(key); + if (held === undefined || version > held.version) best.set(key, { served: { objectKey, raw }, version }); + } + return [...[...best.values()].map(({ served }) => served), ...unusable]; +} + // --------------------------------------------------------------------------- // Resolution internals — shared with the materialization path // --------------------------------------------------------------------------- diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index 022be3c4..b2256a5f 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -39,6 +39,7 @@ import { getStore, isVerificationFailure, NO_STORE_MESSAGE, + newestByKey, type Resolution, recordMaterialized, recordRevoked, @@ -606,8 +607,12 @@ function resolveAll(deadline: number, onUnavailable: OnUnavailable): { requests: const { objects, error } = allRawObjects(store); if (error !== null) return unavailableRun(unavailable(error), onUnavailable); + // One object per key, at its newest version. `allObjects` may hold several + // versions of one key, and //SKILL.md is a single path — writing it + // twice in one run is not a duplicate report but a write race against itself, + // resolved by whichever version iteration happened to reach last. return { - requests: Object.entries(objects).map(([key, raw]) => pendingForRaw(key, raw)), + requests: newestByKey(objects).map(({ objectKey, raw }) => pendingForRaw(objectKey, raw)), incomplete: false, }; } diff --git a/packages/client/src/skills.ts b/packages/client/src/skills.ts index 9a7a1fa3..aa9870a8 100644 --- a/packages/client/src/skills.ts +++ b/packages/client/src/skills.ts @@ -21,6 +21,7 @@ import { allRawObjects, clearState, + newestByKey, referenceTarget, requireStore, resolveFromStore, @@ -65,23 +66,72 @@ export function _clearState(): void { } /** - * A skill store backed by a plain object. + * A skill store backed by in-memory maps. * * Ships for local development, tests, and bring-your-own-content injection. * Holds raw wire objects verbatim and performs no validation of its own — * verification belongs at the accessor boundary, where it applies to every store * equally. + * + * Several versions of one key coexist here, because they coexist in a real + * delivery payload: the newest version of every skill, plus every version a + * variation currently pins. `getObject` therefore selects on `(key, version)`, + * and an omitted version means "the newest held". + * + * An object whose `version` is not an integer >= 1 is still accepted and still + * served, under its key alone. Withholding it is verification's job, not the + * store's: a store that quietly refused it would make a malformed object + * indistinguishable from an absent one, and no integrity signal would be + * recorded. + * + * The backing maps are real `Map`s rather than plain objects. Skill keys come off + * the wire, and a plain object inherits names that are not keys: a `put` of + * `__proto__` would corrupt what every other key resolves to, and `constructor` + * on an empty store would answer with the `Object` function. Nothing unverified + * escapes to user code either way — verification rejects a function as a + * non-object — but the store's own answers have to be right. */ export class InMemorySkillStore implements SkillStore { - private readonly objects: Record; + /** Well-formed objects, `key` -> `version` -> object. */ + private readonly versions = new Map>(); + + /** Objects carrying no usable version, under their key alone. */ + private readonly loose = new Map(); + private readonly listeners = new Map unknown>>(); constructor(objects: Record = {}) { - this.objects = { ...objects }; + // Own enumerable entries only, so a map handed in with a `__proto__` entry + // is filed like any other rather than reaching through to the prototype. + for (const [objectKey, raw] of Object.entries(objects)) this.place(objectKey, raw); } /** - * Adds or replaces a raw skill object, keyed by its own `key` field. + * Files one raw object under its own identity, verbatim. + * + * `fallbackKey` is the map key it arrived under, used only when the object + * carries no string `key` of its own — which the constructor admits and `put` + * refuses. + */ + private place(fallbackKey: string, raw: RawSkillObject): void { + const own = typeof raw === 'object' && raw !== null ? raw.key : undefined; + const key = typeof own === 'string' ? own : fallbackKey; + const version = typeof raw === 'object' && raw !== null ? raw.version : undefined; + if (isValidSkillVersion(version)) { + const held = this.versions.get(key) ?? new Map(); + held.set(version, raw); + this.versions.set(key, held); + } else { + this.loose.set(key, raw); + } + } + + /** + * Adds or replaces a raw skill object, keyed by its own `key` and `version` + * fields. + * + * Putting a second version of a key keeps both; putting the same + * `(key, version)` twice replaces it. * * Notifies every skill-kind listener with the raw object as a single argument. * No validation happens here — verification belongs at the accessor boundary, @@ -91,33 +141,51 @@ export class InMemorySkillStore implements SkillStore { put(raw: RawSkillObject): void { const { key } = raw; if (typeof key !== 'string') throw new Error("a raw skill object must carry a string 'key'"); - this.objects[key] = raw; + this.place(key, raw); // A copy, so a listener that removes itself mid-notification does not // shift its neighbours out from under the iteration. for (const listener of [...(this.listeners.get(SKILL_OBJECT_KIND) ?? [])]) listener(raw); } /** - * The raw object held for `key`, whatever version it happens to be. - * - * The parameter is accepted because the `SkillStore` seam carries it — a real - * store may hold several versions of one key, and only the store can pick - * between them. This one holds a single object per key, so it answers with - * what it has and lets the accessor boundary refuse an answer that is not the - * version that was asked for. + * The raw object held for `key` at `version`, or the newest held when no + * version is asked for. * - * Deliberately not a filter: returning `null` for a pin this store cannot - * satisfy would make a version mismatch indistinguishable from an absence, - * which is the whole distinction the typed outcome exists to preserve. + * With nothing well-formed filed under the key, the version-less entry is all + * there is: it is served, and verification withholds it with a signal rather + * than it reading as simply absent. A pin that misses while well-formed + * versions *do* exist is a plain miss, and answering it with a leftover + * malformed object would record an integrity failure for a skill whose + * integrity is not in question. */ - getObject(kind: string, key: string, _version?: number | null): RawSkillObject | null { + getObject(kind: string, key: string, version?: number | null): RawSkillObject | null { if (kind !== SKILL_OBJECT_KIND) return null; - return this.objects[key] ?? null; + const held = this.versions.get(key); + if (held === undefined || held.size === 0) return this.loose.get(key) ?? null; + if (version !== undefined && version !== null) return held.get(version) ?? null; + return held.get(Math.max(...held.keys())) ?? null; } + /** + * Every object held, one entry per `(key, version)`. + * + * The record's keys are **opaque store-internal identifiers**, as `SkillStore` + * documents: identity is read from each object's own `key` and `version`. Do + * not parse them and do not assume one entry per skill key. + * + * Null-prototype, because a loose object can be filed under the key + * `__proto__` and assigning that name on a plain object would set its + * prototype and silently drop the entry — shrinking a listing, which reads + * downstream as a revocation. + */ allObjects(kind: string): Record { if (kind !== SKILL_OBJECT_KIND) return {}; - return { ...this.objects }; + const out: Record = Object.create(null); + for (const [key, held] of this.versions) { + for (const [version, raw] of held) out[`${key}:${version}`] = raw; + } + for (const [key, raw] of this.loose) out[key] = raw; + return out; } /** @@ -273,15 +341,18 @@ export async function getSkills(refs: ReadonlyArray): P /** * Retrieves every verified skill the store currently holds. * - * Skills that fail verification are omitted. Throws only when no skill store is - * configured. + * Where the store holds several versions of one key, the newest is the one + * returned. Skills that fail verification are omitted. Throws only when no skill + * store is configured. */ export async function allSkills(): Promise { const { objects, error } = allRawObjects(requireStore()); if (error !== null) return []; + // One entry per key at its newest version: `allObjects` may hold several + // versions of one key, and a list carrying two of them is not a set of skills. const skills: Skill[] = []; - for (const raw of Object.values(objects)) { + for (const { raw } of newestByKey(objects)) { const skill = verifyRawSkill(raw); if (skill) skills.push(skill); } From 8807c40ee441dd97724ad2f08d9b46e94ccbb5d8 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 18 Sep 2026 15:46:00 -0400 Subject: [PATCH 29/55] fix(client): stop reporting a withholding for a key that resolved MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `newestByKey` kept every object too malformed to carry a usable key and version, so verification rather than the collapse would withhold it. That is right when nothing else resolves the key — the key stays in the requested set and prune leaves the copy on disk alone — but wrong when another version of the same key resolved anyway. In that case `writeSkills('*')` reported both a `written` and an `error` action for one key, flipping `report.ok` to false for a skill that is correctly on disk, and claiming the copy there "was left alone" when the same run had just written it. `allSkills` recorded an integrity failure for the same key. Drop a malformed object only when its own skill key resolved from another version; keep it otherwise, which is the case the protection exists for. Ports the Python fix (python-ai-sdk 098455f) to TypeScript, so the two SDKs agree on the collapse again. Co-Authored-By: Claude Opus 5 --- .../client/src/__tests__/skills-fs.test.ts | 38 +++++++++++++++++++ packages/client/src/__tests__/skills.test.ts | 19 ++++++++++ packages/client/src/skills-core.ts | 22 ++++++++--- 3 files changed, 74 insertions(+), 5 deletions(-) diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index 4cc7213e..d8b95aa1 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -314,6 +314,44 @@ describe('writeSkills basic writes', () => { expect(await readFile(path.join(root, 'a', SKILL_MD), 'utf-8')).toBe(newer); }); + it('"*" reports no error for a key a malformed sibling did not stop writing', async () => { + // A malformed object beside a good version of the same key. The good one + // resolves and materializes, so the run succeeded for that key: reporting + // the sibling as well would flip `report.ok` for a skill that is correctly + // on disk, and claim the copy there "was left alone" when this very run had + // just written it. + const store = new InMemorySkillStore(); + store.put(rawSkill('a', 1)); + store.put(rawSkill('a', 'nope' as unknown as number)); + _setStore(store); + + const report = await writeSkills('*', root); + + expect(report.ok).toBe(true); + expect(report.actions.map((a) => [a.key, a.action])).toEqual([['a', 'written']]); + expect(await readFile(path.join(root, 'a', SKILL_MD), 'utf-8')).toBe(SKILL_BODY); + }); + + it('"*" still reports a key nothing could resolve, and prunes nothing', async () => { + // The converse, and the reason a malformed object is kept at all. No + // version of 'b' is usable, so it stays in the requested set: the failure is + // reported, and prune leaves the copy already on disk alone rather than + // reading the key as revoked. + const existing = await placeManaged(root, 'b', SKILL_BODY); + const store = new InMemorySkillStore(); + store.put(rawSkill('a', 1)); + store.put(rawSkill('b', 'nope' as unknown as number)); + _setStore(store); + + const report = await writeSkills('*', root); + + expect(report.ok).toBe(false); + expect(actionsByKey(report).a.action).toBe('written'); + expect(actionsByKey(report).b.action).toBe('error'); + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + expect(await readFile(existing, 'utf-8')).toBe(SKILL_BODY); + }); + it('reports one action per requested skill — no silent skips', async () => { const report = await writeSkills([skill('a'), skill('b')], root); expect(report.actions.map((a) => a.key).sort()).toEqual(['a', 'b']); diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 1d4b0564..0c9da15c 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -996,6 +996,25 @@ describe('allSkills', () => { expect(emitter.signals(INTEGRITY_SIGNAL)).toHaveLength(1); }); + it('records no signal for a key an unusable sibling did not stop resolving', async () => { + // The other side of the rule above. A store may hold a malformed object + // beside a well-formed version of the same key; the well-formed one + // resolves, so the key was never withheld and must not be reported as + // though it were. Keeping the sibling could only add an integrity failure + // for a skill whose integrity is not in question. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + const store = new InMemorySkillStore(); + store.put(rawSkill({ key: 'a', version: 1 })); + store.put(rawSkill({ key: 'a', version: 'nope' as unknown as number })); + _setStore(store); + + const found = await allSkills(); + + expect(found.map((s) => s.key)).toEqual(['a']); + expect(emitter.signals(INTEGRITY_SIGNAL)).toEqual([]); + }); + it('returns an empty list for an empty store', async () => { _setStore(new InMemorySkillStore()); expect(await allSkills()).toEqual([]); diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 631bea2e..58a4f8e6 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -499,10 +499,13 @@ export type ServedObject = { * The store key is carried through rather than discarded because the reconcile * attributes a failure to it when the object's own key is unusable. * - * Objects too malformed to carry a usable key and version are **kept**, not - * dropped, so verification is what withholds them: a silently dropped object - * falls out of the requested set, and prune would then delete the last - * known-good copy already on disk. + * An object too malformed to carry a usable key and version is **kept**, so + * verification is what withholds it: a silently dropped object falls out of the + * requested set, and prune would then delete the last known-good copy already on + * disk. The exception is an object whose skill key resolved anyway from another + * version — there the resolved object already holds the key in the requested + * set, so keeping the malformed one would only report a withholding for a key + * that in fact resolved. */ export function newestByKey(objects: Record): ServedObject[] { // The winning version is carried beside the object rather than re-read off it: @@ -520,7 +523,16 @@ export function newestByKey(objects: Record): ServedObje const held = best.get(key); if (held === undefined || version > held.version) best.set(key, { served: { objectKey, raw }, version }); } - return [...[...best.values()].map(({ served }) => served), ...unusable]; + // An unusable object whose own key resolved from another version is dropped + // here rather than passed on: the resolved object already holds that key in + // the requested set, so prune is already held off the copy on disk and the + // only thing the malformed sibling could still add is a withholding reported + // against a key that resolved. + const withheld = unusable.filter(({ raw }) => { + const key = typeof raw === 'object' && raw !== null ? raw.key : undefined; + return !(isValidSkillKey(key) && best.has(key)); + }); + return [...[...best.values()].map(({ served }) => served), ...withheld]; } // --------------------------------------------------------------------------- From 432d75c0c1725867e15b5c309d1b6d88fd069813 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 16 Sep 2026 16:22:14 -0400 Subject: [PATCH 30/55] =?UTF-8?q?fix(client)!:=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20narrow=20the=20filesystem=20destructive=20and=20DoS?= =?UTF-8?q?=20surface?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four defects on the materialization path, one reviewer mindset: bound what an attacker who can write inside the managed root can make this SDK read, write, or spend. Plus the two pieces of evidence the defenses were missing. **The adoption comparison read was unbounded.** `writeOne` reads `//SKILL.md` and compares its hash against the resolved content's. That read deliberately reaches files the manifest does *not* vouch for — it is the crash-recovery self-heal, and refusing to look is what wedges a skill whose manifest write was killed — so the file it opens may be one that was planted, at whatever size was chosen. `readRegularFile` now takes `maxBytes` and reads at most `maxBytes + 1`. The `+ 1` is load-bearing and must not be trimmed to `maxBytes`: a file that is the content *plus* trailing bytes would read back as exactly the content, hash equal, and be adopted despite not being current. Anything longer than the resolved content cannot match its digest, so one byte past the length is all that is needed to prove inequality. The manifest read stays unbounded — it is parsed rather than compared, no caller can predict its length, and it is the one file under the root this SDK writes itself. **The hashed bytes were not necessarily the written bytes.** `createSkill` freezes the wrapper, but a TypedArray's elements cannot be frozen, so a `Skill` shares its buffer with whoever built it — `types.ts` said so already. `verifiedBytes` hashed that buffer by reference, and on the write path the rename happens several `await`s later, with the existence check and the comparison read in between. A caller mutating `skill.content` inside that window wrote bytes nothing had verified, which defeats verify-then-write entirely. The content is now copied before it is hashed. `new Uint8Array(content)`, deliberately, and not `content.slice()`: `Buffer` is a `Uint8Array` subclass, a caller may legally hand one to `createSkill`, and `Buffer.prototype.slice` is the legacy spelling that returns a **view over the same memory**. `slice()` would have snapshotted a plain `Uint8Array` and silently failed to snapshot a `Buffer` — the fix present in the diff and absent in effect. Two of the three parameterized cases cover exactly that, including a pooled `Buffer` view, so `byteOffset` is covered as well as `byteLength`. Python needs none of this: its `bytes` are immutable. **The prune loop never checked its deadline.** It was checked once at the phase boundary, which bounds "the whole call" only in the letter — a long manifest is an unbounded number of `unlink` and `rmdir` calls afterwards, each able to block, so the call overran the timeout it promised by an amount the caller could not predict. Checked per entry now, as Python does. An exhausted entry is reported and the loop *continues* rather than breaking, so the report names every skill left in place instead of only the first. Stopping partway is safe by construction: the manifest is rewritten from what actually happened, so entries this run never reached stay listed and the next reconcile takes them — which the second test asserts rather than assumes. **The integrity log carried no structured record.** §3.24 requires the event twice over and says why neither half suffices: a default `console.error` transport shows only the message text, so a structured-only record is invisible under the setup most customers have, and severity cannot stand in because other store-failure paths in this module also log at error level. The sorted-key JSON stays in the message — the key order is what makes the line byte-identical across SDKs — and the same mapping now goes out as a second argument, which is the console counterpart to Python's `extra={"ld_skills": record}`. **Two pieces of missing evidence.** The manifest read had already moved to `readRegularFile` through the pinned root, but nothing proved the descriptor half: A.12 notes that the materialization-layer races "prove the pair and neither member", and an implementation that read the manifest by path would pass every other case in the root-swap file, since the writes and unlinks would still land in the real root and only the *decisions* would come from somebody else's directory. Two cases now fire a root swap from the manifest open. They deliberately do not assert that the run succeeds — `unsafePathReason` also catches a swapped root, so both runs refuse there, which is defense in depth working as documented. What isolates the manifest read is whose entries the run acted on, so the attacker's manifest names a key the real root never had and the assertions come in pairs: the real key present, the planted key absent from both the report and the rewritten manifest. The Linux fast path also owed a primitive-layer positive control. Its defense lives in the *addressing* of a call rather than in a check any suite can watch fail, so nothing established that `directoryAddress()` lands in the pinned inode rather than in whatever the directory's name resolves to at call time. Both primitives now have one: pin a handle, swap the directory for a symlink to an outside directory, operate through the address built from the handle, and assert the bytes landed in the moved-aside original with the outside directory untouched — not even a temp file, unlike the `lstat` floor, where the temp is created through the link before the check fires. Verified by mutation, each fix against its own tests: an unbounded read, a bound of `maxBytes`, a bare reference instead of a copy, `content.slice()` against the two `Buffer` cases, a deleted per-entry deadline check, a dropped structured argument, a by-path manifest read, and a `directoryAddress` that abandons the descriptor — every one of them turns its tests red. The capability-gated set grows from six tests to ten. All ten skip on macOS off the `SUPPORTS_DIR_FD` / `SUPPORTS_PROC_FD` probes and were run on Linux (Node 24, `/proc` mounted, both new files included): 1424 passed, 1 skipped. Co-Authored-By: Claude Opus 5 --- packages/client/src/__tests__/safe-fs.test.ts | 95 ++++- .../skills-fs-byte-integrity.test.ts | 354 ++++++++++++++++++ .../src/__tests__/skills-fs-root-swap.test.ts | 104 +++++ .../client/src/__tests__/skills-fs.test.ts | 133 +++++++ packages/client/src/__tests__/skills.test.ts | 59 ++- packages/client/src/skills-core.ts | 38 +- packages/client/src/skills-fs.ts | 66 +++- 7 files changed, 836 insertions(+), 13 deletions(-) create mode 100644 packages/client/src/__tests__/skills-fs-byte-integrity.test.ts diff --git a/packages/client/src/__tests__/safe-fs.test.ts b/packages/client/src/__tests__/safe-fs.test.ts index 672c2a92..15685594 100644 --- a/packages/client/src/__tests__/safe-fs.test.ts +++ b/packages/client/src/__tests__/safe-fs.test.ts @@ -36,7 +36,14 @@ vi.mock('node:fs/promises', async () => { }; }); -const { atomicWrite, openDirectoryNoFollow, openOrCreateDirectory, unlinkNoFollow } = await import('../safe-fs.js'); +const { + atomicWrite, + directoryAddress, + openDirectoryNoFollow, + openOrCreateDirectory, + SUPPORTS_PROC_FD, + unlinkNoFollow, +} = await import('../safe-fs.js'); let scratch: string; @@ -237,6 +244,92 @@ describe('pinned-directory identity re-check', () => { }); }); +/** + * The Linux fast path's own positive control. + * + * The block above tests the *floor* — deliberately, by passing a plain path, so it + * exercises the identity re-check on every platform. That leaves the fast path + * itself unproven by anything: on Linux the defense lives in how the call is + * *addressed* rather than in a check a test can observe failing, and the §3.23.2 + * swap races in `skills-fs.test.ts` and `skills-fs-root-swap.test.ts` go through + * the materialization layer, where the shared path check and the primitive's own + * `O_NOFOLLOW` both stand in the way — so they pass when either layer alone works + * and prove the pair rather than either member. + * + * So assert the property directly and at this layer: nothing else establishes that + * `directoryAddress()` resolves from the inode the handle is pinned to rather than + * from whatever the directory's *name* resolves to when the call is made. This is + * the positive mirror of the floor tests — there the contract is a refusal, here it + * is that the write lands in the right place despite the swap. + * + * Gated on `SUPPORTS_PROC_FD` because it tests that capability, not the floor; + * that makes it **Linux-only**, and a green macOS run is no evidence about it. + */ +describe.skipIf(!SUPPORTS_PROC_FD)('descriptor addressing on the /proc/self/fd fast path', () => { + it('atomicWrite lands in the pinned inode after the directory is swapped for a symlink', async () => { + const dir = path.join(scratch, 'skill'); + const outside = path.join(scratch, 'outside'); + await mkdir(dir); + await mkdir(outside); + + const handle = await openDirectoryNoFollow(dir); + try { + // Built *before* the swap, which is the point: the address holds the + // descriptor, so it keeps naming this inode no matter what happens to the + // name `dir`. + const address = directoryAddress(handle, dir); + expect(address).not.toBe(dir); + + const movedTo = `${dir}.real`; + await rename(dir, movedTo); + await symlink(outside, dir, 'dir'); + // The swap really is in place: `dir` now resolves outside. + expect((await lstat(dir)).isSymbolicLink()).toBe(true); + + await atomicWrite(address, 'SKILL.md', Buffer.from('body\n', 'utf-8'), handle); + + // Landed in the directory that was pinned, which is now `movedTo`. + expect(await readFile(path.join(movedTo, 'SKILL.md'), 'utf-8')).toBe('body\n'); + // And nothing at all reached the attacker's directory — not even a temp + // file, unlike the floor, where the temp is created through the link before + // the check fires. + expect(await readdir(outside)).toEqual([]); + } finally { + await handle.close(); + } + }); + + it('unlinkNoFollow removes from the pinned inode after the directory is swapped for a symlink', async () => { + // The destructive half. `unlink` never follows a trailing symlink, but it does + // resolve the directory above it — so a path-addressed unlink through a + // swapped directory removes the outside file. The victim is a real file with + // the managed name, so only the addressing can be what saves it. + const dir = path.join(scratch, 'skill'); + const outside = path.join(scratch, 'outside'); + await mkdir(dir); + await mkdir(outside); + await writeFile(path.join(dir, 'SKILL.md'), 'managed\n', 'utf-8'); + const victim = path.join(outside, 'SKILL.md'); + await writeFile(victim, 'victim content\n', 'utf-8'); + + const handle = await openDirectoryNoFollow(dir); + try { + const address = directoryAddress(handle, dir); + const movedTo = `${dir}.real`; + await rename(dir, movedTo); + await symlink(outside, dir, 'dir'); + + await unlinkNoFollow(address, 'SKILL.md', handle); + + // The managed file — the one that was pinned — is the one that went. + expect(await readdir(movedTo)).toEqual([]); + expect(await readFile(victim, 'utf-8')).toBe('victim content\n'); + } finally { + await handle.close(); + } + }); +}); + describe('unlinkNoFollow', () => { it('removes a real file', async () => { const dir = path.join(scratch, 'skill'); diff --git a/packages/client/src/__tests__/skills-fs-byte-integrity.test.ts b/packages/client/src/__tests__/skills-fs-byte-integrity.test.ts new file mode 100644 index 00000000..b1040ba6 --- /dev/null +++ b/packages/client/src/__tests__/skills-fs-byte-integrity.test.ts @@ -0,0 +1,354 @@ +/** + * Which bytes `writeSkills` reads, and which bytes it writes. + * + * Two guarantees that the materialization suite cannot observe, because both are + * about what happens *inside* a single reconcile rather than about its report: + * + * - the adoption comparison read is **bounded** at the resolved content's length + * plus one byte (§3.22), and + * - the bytes that were hashed are the bytes that get **written** (§3.21) — the + * content is snapshotted before hashing, so a caller mutating `skill.content` + * during the awaits in between cannot substitute bytes nothing verified. + * + * Both need `vi.mock('node:fs/promises')`, which is file-wide, so they live here + * rather than in `skills-fs.test.ts`: the first has to count the bytes a read + * actually consumed, and the second has to fire a mutation from inside the window + * between the hash and the write. This is the same reason + * `skills-fs-root-swap.test.ts` is a file of its own. + * + * Nothing here is platform-gated: both properties hold on every platform, and + * both tests run everywhere. + */ + +import { createHash } from 'node:crypto'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +// ─── The read-counting / mutation hook ─────────────────────────────────────── +// +// `open` is wrapped so the FileHandle it returns reports how much was read +// through it, and `mkdir` is wrapped so a test can act at a chosen point in the +// write path. Everything else `node:fs/promises` exports is the real thing. +// +// `reads` is keyed on the *basename*, because on Linux the implementation opens +// its targets through `/proc/self/fd//` rather than through a path +// anyone would recognize (see `SUPPORTS_PROC_FD`), so a key built from the full +// path would simply never match there and the assertions would pass vacuously. + +const hook = vi.hoisted(() => ({ + /** Bytes consumed per opened basename, however the read was issued. */ + reads: new Map(), + /** Which basenames were read with an unbounded whole-file `readFile`. */ + wholeFileReads: new Set(), + /** Fires on `mkdir` of this basename, or `null` when disarmed. */ + onMkdir: null as string | null, + fire: null as (() => void) | null, + fired: false, +})); + +vi.mock('node:fs/promises', async (importOriginal) => { + const actual = await importOriginal(); + + const count = (name: string, bytes: number): void => { + hook.reads.set(name, (hook.reads.get(name) ?? 0) + bytes); + }; + + return { + ...actual, + async mkdir(...args: unknown[]) { + if (hook.onMkdir !== null && typeof args[0] === 'string' && path.basename(args[0]) === hook.onMkdir) { + hook.fired = true; + hook.fire?.(); + } + return (actual.mkdir as (...a: unknown[]) => Promise)(...args); + }, + async open(...args: unknown[]) { + const handle = await (actual.open as (...a: unknown[]) => Promise)( + ...args, + ); + if (typeof args[0] !== 'string') return handle; + const name = path.basename(args[0]); + // A Proxy rather than a rebuilt object: a FileHandle carries internals the + // implementation uses (`fd`, `stat`, `close`), and only the two read paths + // need observing. + return new Proxy(handle, { + get(target, property, receiver) { + if (property === 'readFile') { + return async (...a: unknown[]) => { + const data = await (target.readFile as (...x: unknown[]) => Promise)(...a); + hook.wholeFileReads.add(name); + count(name, typeof data === 'string' ? Buffer.byteLength(data) : data.byteLength); + return data; + }; + } + if (property === 'read') { + return async (...a: unknown[]) => { + const result = await (target.read as (...x: unknown[]) => Promise<{ bytesRead: number }>)(...a); + count(name, result.bytesRead); + return result; + }; + } + const value = Reflect.get(target, property, receiver); + return typeof value === 'function' ? value.bind(target) : value; + }, + }); + }, + }; +}); + +const { mkdir, mkdtemp, readFile, realpath, rm, writeFile } = await import('node:fs/promises'); +const { _clearState } = await import('../skills.js'); +const { writeSkills } = await import('../skills-fs.js'); +const { createSkill } = await import('../types.js'); +const { fsOps } = await import('../safe-fs.js'); + +const SKILL_MD = 'SKILL.md'; +const SKILL_BODY = '---\nname: Test Skill\n---\nDo the thing.\n'; +const NEVER_FIRED = 'the hook never fired; the test proves nothing'; + +function hash(bytes: Uint8Array | string): string { + return createHash('sha256') + .update(typeof bytes === 'string' ? Buffer.from(bytes, 'utf-8') : bytes) + .digest('hex'); +} + +function disarm(): void { + hook.reads.clear(); + hook.wholeFileReads.clear(); + Object.assign(hook, { onMkdir: null, fire: null, fired: false }); +} + +let scratch: string; +let root: string; + +beforeEach(async () => { + _clearState(); + disarm(); + scratch = await realpath(await mkdtemp(path.join(tmpdir(), 'ld-ai-skills-bytes-'))); + root = path.join(scratch, 'skills'); + await mkdir(root); +}); + +afterEach(async () => { + vi.restoreAllMocks(); + disarm(); + _clearState(); + await rm(scratch, { recursive: true, force: true }); +}); + +// ─── The bounded comparison read ───────────────────────────────────────────── + +/** + * Adoption (§3.22) reads the file at `//SKILL.md` and compares its + * hash against the resolved content's. That read deliberately reaches files the + * manifest does **not** vouch for — that is the whole crash-recovery self-heal — + * so the file it opens may be one an attacker with write access to the root + * planted, at whatever size they chose. Unbounded, it pulls that file into memory + * before the comparison ever runs. + * + * The bound is `len(content) + 1`, and the `+ 1` is load-bearing rather than + * slack: a file that is the content *plus* trailing bytes would, under a bound of + * exactly `len(content)`, read back as exactly the content, hash equal, and be + * adopted despite not being current. The extra byte is what proves inequality. + */ +describe('the adoption comparison read is bounded', () => { + /** The post-crash state adoption exists for: a managed path, no manifest. */ + async function placeOrphaned(key: string, content: string | Uint8Array): Promise { + const target = path.join(root, key, SKILL_MD); + await mkdir(path.dirname(target), { recursive: true }); + await writeFile(target, content); + return target; + } + + function skill(key: string, content = SKILL_BODY) { + return createSkill({ key, version: 1, content: new TextEncoder().encode(content), contentHash: hash(content) }); + } + + it('reads at most one byte past the resolved content and refuses an over-large file', async () => { + // A megabyte where the resolved content is a few dozen bytes. The size is + // beside the point — what is asserted is that the read stopped at the bound, + // which is the property that holds for a multi-gigabyte file too. + const planted = Buffer.alloc(1024 * 1024, 0x41); + const target = await placeOrphaned('a', planted); + const encoded = new TextEncoder().encode(SKILL_BODY); + + const report = await writeSkills([skill('a')], root); + + // Refused — the bytes are not the resolved content, and no manifest entry + // vouches for the path. + expect(report.ok).toBe(false); + const action = report.actions.find((a) => a.key === 'a'); + expect(action?.action).toBe('error'); + expect(action?.error).toContain('does not record it as managed'); + // Refused *after* a successful comparison, not because the read blew up: + // "we could not look" and "we looked and the bytes are foreign" are + // deliberately different refusals, and only the second proves the bound was + // what stopped the read. + expect(action?.error).not.toContain('could not be read'); + + // The bound itself. `+ 1` and no more, and never the whole file. + expect(hook.reads.get(SKILL_MD)).toBe(encoded.byteLength + 1); + expect(hook.wholeFileReads.has(SKILL_MD)).toBe(false); + // And the planted file is untouched. + expect((await readFile(target)).byteLength).toBe(planted.byteLength); + }); + + it('refuses a file that is the resolved content plus trailing bytes', async () => { + // The case a bound of exactly `len(content)` would adopt: the first + // `len(content)` bytes *are* the resolved content, so a read truncated there + // hashes equal. Only the extra byte distinguishes them. + const trailing = `${SKILL_BODY}extra`; + const target = await placeOrphaned('a', trailing); + const renames: string[] = []; + const realRename = fsOps.rename.bind(fsOps); + vi.spyOn(fsOps, 'rename').mockImplementation(async (src: string, dst: string) => { + if (dst.endsWith(SKILL_MD)) renames.push(dst); + return realRename(src, dst); + }); + + const report = await writeSkills([skill('a')], root); + + expect(report.ok).toBe(false); + const action = report.actions.find((a) => a.key === 'a'); + expect(action?.action).toBe('error'); + expect(action?.error).toContain('does not record it as managed'); + // Not adopted: nothing was claimed into the manifest, and the file is as it + // was found. + expect(await readFile(target, 'utf-8')).toBe(trailing); + expect(renames).toEqual([]); + expect(JSON.parse(await readFile(path.join(root, '.launchdarkly-skills.json'), 'utf-8')).entries).toEqual({}); + }); + + it('still adopts an exact match, so the bound did not break the self-heal', async () => { + // The positive control for the two above: a bound that refused everything + // would satisfy both of them. Exactly `len(content)` bytes of the resolved + // content is the one file adoption is *for*. + await placeOrphaned('a', SKILL_BODY); + const encoded = new TextEncoder().encode(SKILL_BODY); + + const report = await writeSkills([skill('a')], root); + + expect(report.ok).toBe(true); + expect(report.actions.find((a) => a.key === 'a')?.action).toBe('skipped_current'); + // The read stops at EOF rather than at the bound, so it is one byte short of + // what the over-large case consumed — which is what "anything longer cannot + // match" means in practice. + expect(hook.reads.get(SKILL_MD)).toBe(encoded.byteLength); + }); + + it('reads the manifest unbounded, because its length is not predictable', async () => { + // The counterpart: `maxBytes` is opt-in, and the manifest does not opt in. + // It is parsed rather than compared, no caller can predict its length, and it + // is the one file under the root this SDK writes itself. + await writeSkills([skill('a')], root); + disarm(); + + await writeSkills([skill('a')], root); + + expect(hook.wholeFileReads.has('.launchdarkly-skills.json')).toBe(true); + }); +}); + +// ─── The bytes that were hashed are the bytes that get written ─────────────── + +/** + * `createSkill` freezes the wrapper, but a TypedArray's elements cannot be frozen + * — `Object.freeze` throws on an array-buffer view — so a `Skill` shares its + * buffer with whoever constructed it, and `types.ts` says so in terms. + * + * On the write path the hash is computed in `verifiedBytes` and the rename + * happens several `await`s later: the existence check and, when the target + * exists, the comparison read sit in between. So without a snapshot, a caller + * mutating `skill.content` inside that window writes bytes that were never + * hashed, and verify-then-write verifies nothing. + * + * This is a TypeScript-shaped hazard. Python's `bytes` are immutable, so the + * guarantee is free there and its test asserts a property rather than a guard. + */ +describe('content is snapshotted before it is hashed', () => { + /** + * Both buffer kinds a caller can legally hand to `createSkill`. + * + * `Buffer` is not a curiosity here, it is the trap: it extends `Uint8Array`, so + * it satisfies `Skill.content` without a cast, and `Buffer.prototype.slice` is + * the legacy spelling that returns a *view* rather than a copy. A snapshot + * written as `content.slice()` therefore protects a plain `Uint8Array` and + * silently fails to protect a `Buffer` — present in the diff, absent in effect. + * A pooled `Buffer` (a `subarray` into a larger allocation, which is what + * `Buffer.from(string)` hands out for small strings anyway) is the third shape, + * because a copy that respected `byteLength` but not `byteOffset` would pass the + * other two. + */ + const bufferKinds: Array<[string, (bytes: Uint8Array) => Uint8Array]> = [ + ['Uint8Array', (bytes) => Uint8Array.from(bytes)], + ['Buffer', (bytes) => Buffer.from(bytes)], + [ + 'pooled Buffer view', + (bytes) => { + const pool = Buffer.alloc(bytes.byteLength + 8, 0x7a); + pool.set(bytes, 4); + return pool.subarray(4, 4 + bytes.byteLength); + }, + ], + ]; + + it.each(bufferKinds)('a %s mutated between the hash and the write does not reach the file', async (_kind, make) => { + const original = new TextEncoder().encode(SKILL_BODY); + // The caller's own buffer, which `createSkill` will alias. + const mutable = make(original); + const contentHash = hash(original); + const skill = createSkill({ key: 'a', version: 1, content: mutable, contentHash }); + // The alias is real — otherwise this test is about nothing. + expect(skill.content).toBe(mutable); + + // `mkdir` of `/a` runs after `verifiedBytes` has hashed and before + // `atomicWrite` writes a byte, which is exactly the window. Fired from the + // filesystem hook rather than from implementation internals, so the test is + // not coupled to how the snapshot is taken. + const tampered = new TextEncoder().encode('---\nname: Tampered\n---\nDo something else.\n'); + Object.assign(hook, { + onMkdir: 'a', + fire: () => { + for (let i = 0; i < mutable.byteLength; i += 1) mutable[i] = tampered[i % tampered.byteLength]; + }, + fired: false, + }); + + const report = await writeSkills([skill], root); + + if (!hook.fired) throw new Error(NEVER_FIRED); + // The mutation landed on the caller's buffer, so the window was real. + expect(Buffer.from(mutable)).not.toEqual(Buffer.from(original)); + + expect(report.ok).toBe(true); + const action = report.actions.find((a) => a.key === 'a'); + expect(action?.action).toBe('written'); + + // The file holds the bytes that were hashed, not the bytes the buffer held + // when the write happened. + const onDisk = await readFile(path.join(root, 'a', SKILL_MD)); + expect(Buffer.from(onDisk)).toEqual(Buffer.from(original)); + // And the hash recorded for it is the hash *of those bytes* — the two halves + // agree, which is the whole claim. + const entries = JSON.parse(await readFile(path.join(root, '.launchdarkly-skills.json'), 'utf-8')).entries; + expect(entries[`a/${SKILL_MD}`].sha256).toBe(contentHash); + expect(hash(onDisk)).toBe(contentHash); + }); + + it('a buffer mutated before writeSkills is called fails verification instead', async () => { + // The mirror image, and the reason the snapshot is not a licence to ignore + // the caller: bytes that disagree with `contentHash` when `writeSkills` is + // *entered* are a verification failure, not something to snapshot and write. + const mutable = Uint8Array.from(new TextEncoder().encode(SKILL_BODY)); + const skill = createSkill({ key: 'a', version: 1, content: mutable, contentHash: hash(SKILL_BODY) }); + mutable[0] = 0x58; + + const report = await writeSkills([skill], root); + + expect(report.ok).toBe(false); + const action = report.actions.find((a) => a.key === 'a'); + expect(action?.action).toBe('error'); + expect(action?.error).toContain('failed verification'); + }); +}); diff --git a/packages/client/src/__tests__/skills-fs-root-swap.test.ts b/packages/client/src/__tests__/skills-fs-root-swap.test.ts index 5c7c1eb1..e3e145b0 100644 --- a/packages/client/src/__tests__/skills-fs-root-swap.test.ts +++ b/packages/client/src/__tests__/skills-fs-root-swap.test.ts @@ -175,6 +175,17 @@ let root: string; /** Where the real root ends up after the swap. */ const movedTo = (): string => `${root}.real`; +/** + * The `entries` map of the manifest in the root that was actually pinned. + * + * After a swap that is the moved-aside directory, which is where every + * descriptor-addressed operation — including the manifest rewrite — lands. + */ +async function realManifestEntries(): Promise> { + const raw = JSON.parse(await readFile(path.join(movedTo(), MANIFEST_NAME), 'utf-8')); + return raw.entries as Record; +} + /** Renames the managed root aside and leaves a symlink to `outside` in its place. */ async function swapRoot(outside: string): Promise { await rename(root, movedTo()); @@ -258,6 +269,99 @@ describe.skipIf(!SUPPORTS_PROC_FD)('writeSkills root swap races', () => { if (report.ok) expect(await exists(path.join(movedTo(), 'a', SKILL_MD))).toBe(false); }); + /** + * The manifest read, which is not itself a destructive step but decides every + * destructive step. + * + * It runs after the root is pinned and is addressed through the descriptor, so + * the property holds by construction — which is exactly why it is worth pinning + * rather than assuming. An implementation that read the manifest by path before + * pinning, or that re-derived `/` afterwards, passes every other + * case in this file: the writes and the unlinks would still be + * descriptor-addressed and would still land in the real root. Only the + * *decisions* would come from the attacker's directory, and `rewriteManifest` + * would then commit those decisions back over the real manifest — destroying + * the ownership record that protects the customer's files on every later run. + * + * **What these do not assert, and why.** Not that the run succeeds. A swapped + * root is also caught by `unsafePathReason`'s containment check — `realpath` of + * the descriptor-addressed skill directory lands under the moved-aside root, + * whose parent is no longer the validated root — so both runs below refuse at + * that layer, which is the correct fail-closed outcome and is defense in depth + * working as documented. Asserting "the write happened" would therefore be + * asserting the wrong thing, and would fail against a *correct* implementation. + * + * The observable that isolates the manifest read is **whose entries the run + * acted on**: the attacker's manifest names a key the real root has never heard + * of, so if theirs were read, that key would appear in the report (prune walks + * manifest entries) and in the rewritten manifest. Both assertions come in a + * pair — the real key present, the attacker's key absent — because either alone + * would pass against an implementation that read neither. + */ + /** The attacker's own manifest, naming a key the real root never managed. */ + async function plantForeignManifest(outside: string): Promise { + await mkdir(path.join(outside, 'zz'), { recursive: true }); + await writeFile(path.join(outside, 'zz', SKILL_MD), 'attacker planted\n', 'utf-8'); + await writeManifest(outside, { + manifestVersion: 1, + entries: { + [`zz/${SKILL_MD}`]: { + key: 'zz', + version: 1, + sha256: hash('attacker planted\n'), + writtenAt: '2026-08-14T19:00:00Z', + }, + }, + }); + } + + it('a root swapped at the manifest read reconciles from the real entries, not the attacker’s', async () => { + const outside = path.join(scratch, 'outside'); + await mkdir(outside); + await plantForeignManifest(outside); + await placeManaged(root, 'a', SKILL_BODY); + arm('open', MANIFEST_NAME, () => swapRoot(outside)); + + const report = await writeSkills([skill('a', 2, 'served update\n')], root); + + if (!race.fired) throw new Error(NEVER_FIRED); + // The real manifest's key is the one the run reasoned about... + expect(report.actions.map((action) => action.key)).toContain('a'); + // ...and the attacker's key never entered the run. Were their manifest read, + // `zz` is not in the requested set, so prune would have reported it. + expect(report.actions.some((action) => action.key === 'zz')).toBe(false); + + // The ownership record survives in the real root: still `a`, never `zz`. + const entries = await realManifestEntries(); + expect(Object.keys(entries)).toEqual([`a/${SKILL_MD}`]); + expect((entries[`a/${SKILL_MD}`] as { key: string }).key).toBe('a'); + // And nothing was done to the attacker's directory. + expect(await readFile(path.join(outside, 'zz', SKILL_MD), 'utf-8')).toBe('attacker planted\n'); + }); + + it('a root swapped at the manifest read prunes from the real entries, not the attacker’s', async () => { + // Same property on the destructive path, where reading the wrong manifest + // would aim the prune: `zz` is unrequested, so the attacker's entry is + // precisely what a prune driven by their manifest would go after. + const outside = path.join(scratch, 'outside'); + await mkdir(outside); + await plantForeignManifest(outside); + await placeManaged(root, 'a', SKILL_BODY); + arm('open', MANIFEST_NAME, () => swapRoot(outside)); + + const report = await writeSkills([], root); + + if (!race.fired) throw new Error(NEVER_FIRED); + expect(report.actions.map((action) => action.key)).toContain('a'); + expect(report.actions.some((action) => action.key === 'zz')).toBe(false); + + // The prune was refused by containment, so `a` stays listed — and `zz` was + // never listed. The next run, against an unswapped root, prunes `a`. + const entries = await realManifestEntries(); + expect(Object.keys(entries)).toEqual([`a/${SKILL_MD}`]); + expect(await readFile(path.join(outside, 'zz', SKILL_MD), 'utf-8')).toBe('attacker planted\n'); + }); + it('a root swapped before it is pinned is refused with a run-level error', async () => { // The one window pinning cannot cover, and the reason the pin is not the // whole story: `resolveRoot` validated the root's *name*, and the swap lands diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index d8b95aa1..c27be452 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -1966,6 +1966,139 @@ describe('writeSkills hostile manifest prune', () => { }); }); +// ─── The prune deadline ──────────────────────────────────────────────── + +/** + * The `timeout` bounds the whole call, and the prune loop is inside it. + * + * Checking the deadline once before pruning begins satisfies that in the letter + * only: a manifest with many entries is an unbounded number of `unlink` and + * `rmdir` calls after that check, each of which can block on a slow or hostile + * filesystem, so the call overruns the budget it promised by an amount the caller + * cannot predict. The check therefore belongs inside the loop, once per entry. + * + * Stopping partway is safe by construction rather than by luck, and the second + * test is what proves it: the manifest is rewritten from what actually happened, + * so an entry this run never reached stays listed and the next reconcile prunes + * it. That is also why an exhausted entry is reported and the loop *continues* — + * the report names every skill left in place, not just the first. + * + * `performance.now` is stubbed rather than slept through: the contract is about + * which side of the deadline each entry falls on, and a test that raced a real + * clock would assert that flakily instead of exactly. + */ +describe('writeSkills prune deadline', () => { + /** A fake monotonic clock, in the same units `performance.now` reports. */ + let nowMs: number; + + /** Installs the clock and hands back a knob to push it past the deadline. */ + function stubClock(): (byMs: number) => void { + nowMs = performance.now(); + vi.spyOn(performance, 'now').mockImplementation(() => nowMs); + return (byMs: number) => { + nowMs += byMs; + }; + } + + /** Four managed skills in one manifest, all files present on disk. */ + async function placeFour(): Promise { + const keys = ['a', 'b', 'c', 'd']; + const entries: Record = {}; + for (const key of keys) { + const target = path.join(root, key, SKILL_MD); + await mkdir(path.dirname(target), { recursive: true }); + await writeFile(target, SKILL_BODY, 'utf-8'); + entries[`${key}/${SKILL_MD}`] = manifestEntry(key, 1, SKILL_BODY); + } + await writeManifest(root, { manifestVersion: 1, entries }); + return keys; + } + + it('stops pruning when the deadline expires partway and reports what it left', async () => { + const keys = await placeFour(); + const advance = stubClock(); + // The first unlink burns the whole budget. Every entry after it is on the + // wrong side of the deadline. + const real = fsOps.unlink.bind(fsOps); + vi.spyOn(fsOps, 'unlink').mockImplementation(async (target: string) => { + await real(target); + advance(5_000); + }); + + const report = await writeSkills([], root, { timeout: 1 }); + + // Exactly one entry was pruned — the one whose check ran before the budget + // was gone — and the rest are reported rather than silently skipped. + const removed = report.actions.filter((a) => a.action === 'removed'); + expect(removed).toHaveLength(1); + const timedOut = report.errors.filter((a) => /timeout was exhausted/.test(a.error ?? '')); + expect(timedOut).toHaveLength(keys.length - 1); + // Every key is accounted for: the loop continued instead of breaking, so the + // report names all three it left in place. + expect(new Set([...removed, ...timedOut].map((a) => a.key))).toEqual(new Set(keys)); + expect(report.ok).toBe(false); + + // The unreached files are still there... + const survivors = keys.filter((key) => key !== removed[0].key); + for (const key of survivors) { + expect(await readFile(path.join(root, key, SKILL_MD), 'utf-8')).toBe(SKILL_BODY); + } + // ...and so are their manifest entries, which is what makes stopping safe: + // the rewrite records what happened, so the next run picks these up. + const entries = (await readManifest(root)).entries as Record; + expect(Object.keys(entries).sort()).toEqual(survivors.map((key) => `${key}/${SKILL_MD}`).sort()); + }); + + it('the next reconcile prunes what the expired one left behind', async () => { + // The other half of "stopping mid-prune is safe": the entries that survived + // are still entries, so a run with a budget finishes the job. Without this, + // the test above would also pass against an implementation that stopped and + // lost track. + await placeFour(); + const advance = stubClock(); + const real = fsOps.unlink.bind(fsOps); + const spy = vi.spyOn(fsOps, 'unlink').mockImplementation(async (target: string) => { + await real(target); + advance(5_000); + }); + + const first = await writeSkills([], root, { timeout: 1 }); + expect(first.actions.filter((a) => a.action === 'removed')).toHaveLength(1); + + // A fresh budget, and no more clock sabotage. + spy.mockRestore(); + nowMs = performance.now(); + vi.spyOn(performance, 'now').mockImplementation(() => nowMs); + + const second = await writeSkills([], root, { timeout: 10 }); + + expect(second.ok).toBe(true); + expect(second.actions.filter((a) => a.action === 'removed')).toHaveLength(3); + expect(await readManifest(root)).toMatchObject({ entries: {} }); + expect(await entryNames(root)).toEqual([MANIFEST_NAME]); + }); + + it('an already-expired deadline prunes nothing at all', async () => { + // The boundary case, and the one that says the check is not merely + // *somewhere* in the loop: with the budget gone before the first entry, the + // first entry is refused too. + const keys = await placeFour(); + const unlinked = recordUnlinks(); + + const report = await writeSkills([], root, { timeout: 0 }); + + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + expect(report.errors.filter((a) => /timeout was exhausted/.test(a.error ?? ''))).toHaveLength(keys.length); + // Not attempted, rather than attempted and failed. + expect(unlinked).toEqual([]); + for (const key of keys) { + expect(await readFile(path.join(root, key, SKILL_MD), 'utf-8')).toBe(SKILL_BODY); + } + const entries = (await readManifest(root)).entries as Record; + expect(Object.keys(entries).sort()).toEqual(keys.map((key) => `${key}/${SKILL_MD}`).sort()); + }); +}); + // ─── Telemetry seam (write half) ─────────────────────────────────────── describe('writeSkills telemetry', () => { diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 0c9da15c..7f06990a 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -1418,7 +1418,14 @@ describe('integrity-failure log record', () => { // text, and they cover the field set, not just the fact that something logged. const EVENT = 'ld.skills.integrity_failure'; - type LoggedRecord = { line: string; record: Record }; + type LoggedRecord = { + /** The message text — the first argument. */ + line: string; + /** The mapping parsed back out of the message text. */ + record: Record; + /** The structured attachment — every argument after the first. */ + rest: unknown[]; + }; async function logged(run: () => Promise): Promise { const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); @@ -1432,9 +1439,11 @@ describe('integrity-failure log record', () => { spy.mockRestore(); } return calls - .map(([first]) => String(first)) - .filter((line) => line.includes(EVENT)) - .map((line) => ({ line, record: JSON.parse(line.slice(line.indexOf('{'))) as Record })); + .filter(([first]) => String(first).includes(EVENT)) + .map(([first, ...rest]) => { + const line = String(first); + return { line, record: JSON.parse(line.slice(line.indexOf('{'))) as Record, rest }; + }); } const surrogate = 'hi \ud800 there'; @@ -1509,6 +1518,48 @@ describe('integrity-failure log record', () => { expect(line).toBe(`[LaunchDarkly] ${EVENT} ${JSON.stringify(record)}`); }); + it('emits the record twice over: in the message text and as structured data', async () => { + // Both forms are required, and neither is sufficient alone. A default + // `console.error` transport shows only the message text, so a + // structured-only record is invisible under the setup most customers have — + // and severity cannot stand in for it, because other store-failure paths in + // this module also log at error level. A structured pipeline, conversely, + // wants the mapping as *data* rather than as text it has to re-parse out of + // a prefixed line. Python's half of this is `extra={"ld_skills": record}`. + _setStore(new DictStore({ a: rawSkill({ key: 'a', version: 7, contentHash: 'd'.repeat(64) }) })); + + const [{ line, record, rest }] = await logged(() => getSkill('a')); + + // Form one: the event identity verbatim in the text, followed by the JSON. + expect(line.startsWith(`[LaunchDarkly] ${EVENT} `)).toBe(true); + // Form two: the same mapping, attached rather than serialized. + expect(rest).toHaveLength(1); + const attached = rest[0] as Record; + expect(typeof attached).toBe('object'); + expect(attached).not.toBeNull(); + // The *same* mapping — asserted field-for-field, because an attachment that + // had drifted from the text would defeat the point of having both. + expect(attached).toEqual(record); + // And key order agrees too, which is what keeps the two halves from + // diverging the day someone reorders one of them. + expect(Object.keys(attached)).toEqual(Object.keys(record)); + }); + + it('attaches the record on every reason_code, not just one path', async () => { + // The attachment is added at the single `console.error` call site, so one + // case would nearly prove it — but "nearly" is how a second call site gets + // added later without one. Sweep the vocabulary. + for (const [code, makeStore, run] of cases) { + _clearState(); + _setStore(makeStore()); + + const [{ record, rest }] = await logged(run); + + expect(rest, code).toHaveLength(1); + expect(rest[0], code).toEqual(record); + } + }); + it('orders keys alphabetically, so the JSON is byte-identical across SDKs', async () => { // Every SDK emits these keys sorted. Insertion order here is what makes the // outputs comparable with one parser and one alert rule. diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 58a4f8e6..52ab62bd 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -249,8 +249,18 @@ export function recordIntegrityFailure( record.skill_key = safeKey; if (isValidSkillVersion(extra.version)) record.version = extra.version; + // Emitted twice over, and both forms are required. The message text carries the + // event identity verbatim followed by the compact sorted-key JSON, which is the + // only form a plain `console.error` transport shows and what makes the line + // greppable and byte-comparable across SDKs. The same mapping goes out as a + // second argument, which is what a structured-console transport picks up as + // data rather than as text to re-parse — the counterpart to Python's + // `extra={"ld_skills": record}`. Neither alone is sufficient: a text-only + // record is invisible to a structured pipeline, and a structured-only one is + // invisible under the default setup, where severity cannot discriminate it from + // the other store-failure paths in this module that also log at error level. // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; integrity failures must be visible - console.error(`[LaunchDarkly] ${EVENT_INTEGRITY_FAILURE} ${JSON.stringify(record)}`); + console.error(`[LaunchDarkly] ${EVENT_INTEGRITY_FAILURE} ${JSON.stringify(record)}`, record); emit(SIGNAL_INTEGRITY_FAILURE, properties); } @@ -326,8 +336,18 @@ export function isVerificationFailure(result: VerifiedContent | VerificationFail * * The wire object delivers `content` as a JSON string; this is the one place it * is encoded to UTF-8 bytes, and everything downstream — the hash, the `Skill`, - * the file on disk — carries those bytes. A `Skill` already holds bytes, so the - * pre-write pass hands them straight in and they are hashed as-is. + * the file on disk — carries those bytes. A `Skill` already holds bytes, and they + * are **snapshotted** here before being hashed, which is load-bearing rather than + * defensive copying for its own sake: `createSkill` freezes the wrapper but a + * TypedArray's elements cannot be frozen, so a `Skill` shares its buffer with + * whoever constructed it (`types.ts` says so). On the write path the hash is + * computed here and the rename happens several `await`s later — the existence + * check and the comparison read sit in between — so without the copy a caller + * mutating `skill.content` during that window writes bytes that were never + * hashed, which defeats verify-then-write entirely. The snapshot is what makes + * "the bytes that were hashed are the bytes that get written" true rather than + * merely likely. Python gets the same guarantee for free, its `bytes` being + * immutable. * * Returns the verbatim bytes and their locally computed sha256, or a * human-readable reason — having already recorded the integrity signal, so the @@ -347,7 +367,17 @@ export function verifiedBytes( expectedHash: string, version: number, ): VerifiedContent | VerificationFailure { - const encoded = typeof content === 'string' ? new TextEncoder().encode(content) : content; + // A copy on the bytes path, not a bare reference: see the docblock. A string is + // already immutable and `encode` allocates, so that path needs no copy. + // + // `new Uint8Array(content)` rather than `content.slice()`, and the difference is + // not stylistic: `Buffer` is a `Uint8Array` subclass, a caller may perfectly + // legally hand one to `createSkill`, and `Buffer.prototype.slice` is the legacy + // spelling that returns a **view over the same memory** instead of a copy. So + // `slice()` would snapshot a plain `Uint8Array` and silently fail to snapshot a + // `Buffer` — the fix would look present and not be. The constructor copies for + // either, and normalizes a pooled `Buffer` view to a standalone buffer besides. + const encoded = typeof content === 'string' ? new TextEncoder().encode(content) : new Uint8Array(content); if (encoded.byteLength > MAX_SKILL_CONTENT_BYTES) { const reason = `content is ${encoded.byteLength} bytes, over the ${MAX_SKILL_CONTENT_BYTES} byte cap`; diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index b2256a5f..8948cb96 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -287,7 +287,7 @@ async function reconcile( // retrieval that failed, or a deadline that expired mid-write — means we do not // know what is still current. Either way, deleting would be a guess. if (prune && manifestError === null && !incomplete) { - actions.push(...(await pruneEntries(root, entries, new Set(requests.map((r) => r.key))))); + actions.push(...(await pruneEntries(root, entries, new Set(requests.map((r) => r.key)), deadline, timeout))); } if (manifestError === null) actions.push(...(await rewriteManifest(root, manifest, entries))); @@ -913,7 +913,13 @@ async function writeOne( // the skill is wedged until a human intervenes. let onDisk: Buffer; try { - onDisk = await readRegularFile(target); + // Bounded at the resolved content's length, because this read reaches + // *unmanaged* files — that is the whole point of the self-heal above — so + // the file it opens is one an attacker with write access to the root may + // have planted, at whatever size they chose. `readRegularFile` reads one + // byte past the bound, which is what keeps a longer file from comparing + // equal to a prefix of itself; see its docblock. + onDisk = await readRegularFile(target, encoded.byteLength); } catch (error) { // A read that failed must never become an overwrite: we do not know what // is on disk, so this is the fail-closed branch and not a fall-through to @@ -983,14 +989,43 @@ async function writeOne( * There is deliberately no `O_BINARY`: Node performs no CRLF translation on a * descriptor, so the bytes read back are already verbatim. * + * `maxBytes` bounds the read at `maxBytes + 1` bytes, and the `+ 1` is + * load-bearing rather than slack. The only consumer of a bounded read is the + * adoption comparison, which hashes what it gets: anything longer than the + * resolved content cannot match its digest, so one byte past the content's length + * is all that is needed to *prove* inequality — while bounding at exactly + * `maxBytes` would be a bug, because a file that is the content plus trailing + * bytes would read back as exactly the content, hash equal, and be adopted + * despite not being current. The bound matters because that read reaches + * arbitrary foreign files at a managed path rather than only files the manifest + * vouches for, so unbounded it pulls a planted multi-GB file into memory before + * the comparison ever runs. + * + * `undefined` reads to EOF, for the manifest — whose length no caller can predict + * and which is parsed rather than compared. The manifest is bounded differently, + * by being the one file under the root this SDK writes itself. + * * Throws for anything the caller must turn into a refusal. */ -async function readRegularFile(target: string): Promise { +async function readRegularFile(target: string, maxBytes?: number): Promise { const flags = fsConstants.O_RDONLY | (fsConstants.O_NOFOLLOW ?? 0) | (fsConstants.O_NONBLOCK ?? 0); const handle = await open(target, flags); try { if (!(await handle.stat()).isFile()) throw new Error('the path is not a regular file'); - return await handle.readFile(); + if (maxBytes === undefined) return await handle.readFile(); + + // Read into a buffer sized to the bound, so the file's own size never + // determines the allocation. Looped because a single `read` is permitted to + // return short; it stops at the bound or at EOF, whichever comes first. + const limit = maxBytes + 1; + const buffer = Buffer.alloc(limit); + let filled = 0; + while (filled < limit) { + const { bytesRead } = await handle.read(buffer, filled, limit - filled, filled); + if (bytesRead === 0) break; + filled += bytesRead; + } + return buffer.subarray(0, filled); } finally { await handle.close().catch(() => undefined); } @@ -1090,11 +1125,23 @@ function pruneError(key: string, message: string, version: unknown = null): Reco * This is also how revocation takes effect: a revoked skill is simply absent from * the resolved set, so the next reconcile removes it. There is deliberately no * opt-out. + * + * The deadline is checked **per entry**, not once before the loop. Checking once + * satisfies "the whole call is bounded" only in the letter: a long manifest is an + * unbounded number of `unlink` and `rmdir` calls after that check, each of which + * can block, so the call overruns the timeout it promised by an amount the caller + * cannot predict. Stopping partway is safe by construction — `rewriteManifest` + * rebuilds the manifest from what actually happened, so an entry this run never + * reached stays listed and the next reconcile prunes it. An exhausted entry is + * reported as an `error` and the loop continues rather than breaking, so the + * report names every skill that was left in place instead of only the first. */ async function pruneEntries( root: PinnedDirectory, entries: Record, requested: ReadonlySet, + deadline: number, + timeout: number, ): Promise { const actions: ReconcileAction[] = []; @@ -1104,6 +1151,17 @@ async function pruneEntries( const { key } = record; if (typeof key !== 'string' || requested.has(key)) continue; + if (performance.now() >= deadline) { + actions.push( + pruneError( + key, + `the ${timeout}s timeout was exhausted before '${relative}' could be pruned; it was left in place`, + record.version, + ), + ); + continue; + } + // Only a manifest path this SDK could have written is removable. if (keyRejectionReason(key) !== null || relative !== `${key}/${SKILL_FILENAME}`) { actions.push( From f74c412dd212efdb7fa44083cdcff6107b4d51a8 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 16 Sep 2026 16:57:47 -0400 Subject: [PATCH 31/55] =?UTF-8?q?fix(client)!:=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20test=20integrity,=20and=20the=20two=20guards=20the?= =?UTF-8?q?=20tests=20wanted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mostly a test change. Four suites were asserting less than they read as asserting, and filling those gaps turned up two production defects that only a test could have found — both called out below, since they are the only behaviour changes in the diff. **Three root-swap races were passing against an implementation that refuses every write.** `skills-fs-root-swap.test.ts` guarded its success assertions as `if (report.ok) expect(...)`, so all that survived unconditionally was the negative half: the outside directory is empty, the victim file is unchanged. Every one of those holds for a run that attempted nothing at all — which is exactly the regression the Linux `/proc/self/fd` fast path could introduce, and the fast path *is* the whole defense on Linux. In all three the swap fires after `unsafePathReason` has run, so containment is not what is under test; the descriptor addressing is, and it has to deliver the operation into the root that was pinned. Now asserted: `report.ok`, the file's bytes in the moved-aside real root, and on the prune side the `removed` action for the managed key rather than merely the outside file's survival. **The `watchSkills` coalescing test was vacuous twice over, and the obvious fix is not enough.** The twelve-skill payload committed *before* `watchSkills` attached its listener (`waitForSkills` → `_setStore` → `watchSkills`), and the fake endpoint then served a steady `304`, so no notification ever reached the watcher, `reconciles` stayed `0`, and `toBeLessThanOrEqual(2)` passed with the debouncing deleted. Both halves are fixed — the burst is queued after the watcher is up, the assertion is an equality, and the store's own counters now prove the endpoint really served the changes rather than nothing at all. The third half is not in the spec and mutation testing is what found it: `SkillWatcher.schedule` has a `pending` flag, so twelve objects arriving in *one synchronous* listener run collapse to one reconcile whether or not the debounce exists. A single twelve-object commit therefore still cannot discriminate. The burst is now twelve objects across **three commits a poll apart**: each lands after the previous reconcile has started and cleared `pending`, so only the debounce window can merge them. Deleting the debounce takes the counter to 2. **§3.26 was missing four required tests.** `onReconcile` was never passed anywhere in the suite, so nothing covered it firing for a delivery-triggered reconcile and *not* for the initial one, whose report is returned directly. Nor that a reconcile which throws leaves the watcher alive — driven through `onUnavailable: 'raise'` against a store that throws, the shortest real path to a rejecting reconcile. Nor that an invalid root raises **out of** `watchSkills` rather than surfacing in a background log. Nor the `catch { await watcher.close(); throw cause; }` path, ported from Python's `test_a_failed_initial_reconcile_leaves_no_listener_behind`: the listener is attached before the initial reconcile so a payload committing during its filesystem I/O is not lost, which means a failed reconcile has to detach on the way out — the caller is handed an exception, not a watcher to close. A.12's TypeScript-specific shape is pinned too: a `{ report, watcher }` object rather than a tuple, and `debounceMs` in milliseconds, asserted observably rather than by comment. **§3.25's two layering claims were unasserted.** That `skills-fdv2.ts` takes only the seam kind from the shared internals and nothing else from the feature, that nothing in the feature imports *it*, and that it adds no dependency beyond the standard library — all three read off the source text, since a leaf property cannot be observed at runtime. And that a full delivery cycle records no signal of its own: no recording emitter appeared anywhere in the file, and a payload carrying a hashless and a tampered object is exactly where an implementation would be tempted to emit one. **§3.24's two asymmetries.** The integrity signal's property set was asserted member by member but never as a *set*, so the four fields that belong only to the log record — `event`, `action`, `reason`, `reason_code` — could have leaked into it undetected. And nothing captured both surfaces from one failure: the record's fields are spread from the signal's rather than rebuilt, which is what keeps the two from disagreeing about which values are redacted and which fields are omitted, so both are now compared key by key on a redacting case and on an omitting one. **§3.22's missing shapes.** Prune suppression had one test, and it used a store that failed for *everything* — satisfiable by an implementation that gated pruning on "did we get anything at all". The spec-mandated shape is two references where one resolves and one fails, with a third manifest-listed, no-longer-requested skill that has to survive; and the same rule in its timed-out-mid-write shape, which additionally asserts the stale entry carries no action at all, separating the suppression gate from the per-entry deadline check inside the prune loop. Adoption gained the test that distinguishes a full claim from "tolerated once" — adopt, then reconcile an empty set and get `removed` — and §3.23.3's carve-out is now asserted against one setup, identical bytes adopted beside one-byte-different refused, which is what proves the check is byte equality rather than the missing manifest entry being ignored. --- The two production changes: **`skillRefs` dropped a malformed entry silently.** Python logs a warning per drop; this side did not. The silence is not cosmetic: the projection's output is what a caller hands `writeSkills`, and a shortened list is indistinguishable there from "that skill is no longer requested", so with `prune: true` — the default — a silently dropped entry *deletes the skill's files*. One `console.warn` per drop now, each naming the position and which field was unusable, mirroring Python's three messages. `packages/client/agents.md`'s Windows-reserved-names note said `skillRefs` "would silently drop the reference"; that sentence is now accurate. **`watchSkills` accepted a non-finite debounce.** `NaN < 0` is false, so `NaN` passed the guard, and `setTimeout(fn, NaN)` fires at 1 ms — collapsing the window to nothing and reconciling once per delivered object, which is the exact failure the debounce exists to prevent, arriving silently. `writeSkills` already guards its own `timeout` for `NaN`, so this was inconsistent within TypeScript as well as against the spec. Negative, `NaN`, and `Infinity` all raise now, and the message spells `NaN` as `NaN` rather than letting `JSON.stringify` render it as `null`. Also hoisted one module-scope `OVERSIZE` string in `skills.test.ts`, which had been allocating and hashing 10 MiB twice over for the two true over-cap cases. The root-swap de-vacuuming is the one item a local macOS run cannot speak to, so it was checked on Linux against the regression the guards were hiding — an implementation that fails closed on the fast path, modelled by un-skipping `assertUnswapped` for descriptor-addressed directories, where `lstat('/proc/self/fd/')` reports procfs's magic symlink rather than a directory and the identity check therefore throws on every operation. Under that mutation the three tests as they were **all six pass**; as they are now, the three fail. Nothing lands outside the root either way, which is the whole point: the negative assertions cannot tell the two implementations apart. Every other new or de-vacuumed assertion was verified by mutation: the debounce deleted, the `< 0` guard restored, the `skillRefs` warning removed, the four record-only fields leaked into the signal, the record's redaction dropped, a `null` written where a field should be omitted, the `!incomplete` prune gate removed, adoption reduced to a one-run exemption, the unmanaged-file refusal disabled, `onReconcile` fired for the initial reconcile, a throwing reconcile left to kill the watcher, the failed-initial detach removed, a signal emitted from the transport, and an import added in each layering direction. Each turns its own tests red. `tsc --noEmit`, `biome check`, and `vitest run` are clean on macOS: 1434 passed, 10 skipped. The ten skips are the capability-gated swap races, which is why the client suite was also run on Linux (Docker, Node 24.4.1, `/proc` mounted, the `SUPPORTS_PROC_FD` probe answering true): 914 passed, 1 skipped, with all ten executed. The one skip there is the `chmod 000` unreadable-file case, which `CAN_TEST_UNREADABLE` skips because the container runs as root. Co-Authored-By: Claude Opus 5 --- packages/client/agents.md | 2 +- .../client/src/__tests__/skills-fdv2.test.ts | 377 +++++++++++++++++- .../src/__tests__/skills-fs-root-swap.test.ts | 41 +- .../client/src/__tests__/skills-fs.test.ts | 148 +++++++ packages/client/src/__tests__/skills.test.ts | 146 ++++++- packages/client/src/skills-watch.ts | 12 +- packages/client/src/skills.ts | 32 +- 7 files changed, 726 insertions(+), 32 deletions(-) diff --git a/packages/client/agents.md b/packages/client/agents.md index 62435cbd..a2b8a91f 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -312,7 +312,7 @@ It is in **seconds**, defaulting to `10`. The signature is a cross-language cont Three specifics inside those two functions that a later contributor is most likely to widen: -- **The 22 Windows reserved device names** — `con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9` — are rejected by `keyRejectionReason`, unconditionally on every platform. Do not add a `process.platform === 'win32'` gate: a managed root written by a Linux container and read from a Windows host is an ordinary deployment, so the on-disk result must not depend on which OS ran the reconcile, and with no Windows CI runner in either repo a platform branch would be untestable. Do not move the check into `isValidSkillKey` or `SKILL_KEY_PATTERN` either. `parseAiConfig` fails closed on a bad `skills` entry, so a grammar-level rejection would invalidate the *whole* AI Config — model, provider, instructions, tools — for a customer who never touches Windows, and `skillRefs` would silently drop the reference, which lets `writeSkills` prune the skill's on-disk copy. That turns "fails to write on Windows" into "gets deleted on Linux". The 255-byte path-component bound lives in this layer for the same reason. The set is exactly the reserved names: `com0` and `lpt0` are not reserved, and no case folding or suffix stripping is needed because the key grammar admits no uppercase, no `.`, and no `$`. +- **The 22 Windows reserved device names** — `con`, `prn`, `aux`, `nul`, `com1`–`com9`, `lpt1`–`lpt9` — are rejected by `keyRejectionReason`, unconditionally on every platform. Do not add a `process.platform === 'win32'` gate: a managed root written by a Linux container and read from a Windows host is an ordinary deployment, so the on-disk result must not depend on which OS ran the reconcile, and with no Windows CI runner in either repo a platform branch would be untestable. Do not move the check into `isValidSkillKey` or `SKILL_KEY_PATTERN` either. `parseAiConfig` fails closed on a bad `skills` entry, so a grammar-level rejection would invalidate the *whole* AI Config — model, provider, instructions, tools — for a customer who never touches Windows, and `skillRefs` would drop the reference — with a warning, but still shortening the projection it hands `writeSkills`, which lets `prune` take the skill's on-disk copy with it. That turns "fails to write on Windows" into "gets deleted on Linux". The 255-byte path-component bound lives in this layer for the same reason. The set is exactly the reserved names: `com0` and `lpt0` are not reserved, and no case folding or suffix stripping is needed because the key grammar admits no uppercase, no `.`, and no `$`. - **The adoption rule is a narrowing of the clobber refusal, not a hole in it.** A file at a managed path with no manifest entry is adopted — recorded, and reported `skipped_current` — only when its on-disk sha256 equals the resolved content hash, which is what lets a reconcile killed between the content writes and the final manifest write heal itself instead of wedging those skills forever. Adopt on anything weaker than an exact hash match and the guarantee is gone. `skipped_current` is reused deliberately: adding a member to `ReconcileActionKind` would break every consumer with an exhaustive `switch`. - **`readRegularFile` is what makes that read safe**, and every part of it is load-bearing: `O_NONBLOCK`, because opening a FIFO with no writer blocks forever and would hang the reconcile along with the event loop; `O_NOFOLLOW`; and an `fstat` on the *handle* rather than a `stat` on the path, refusing anything that is not a regular file. A failed read is a refusal and never a fall-through to the write. There is no `O_BINARY` — Node does no CRLF translation — which is the one place this deliberately differs from the Python twin. diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 47cf7f05..394d0096 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -18,14 +18,22 @@ import { createHash } from 'node:crypto'; import { existsSync, readFileSync } from 'node:fs'; -import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'; import { createServer as createTcpServer, type Socket, type Server as TcpServer } from 'node:net'; import { tmpdir } from 'node:os'; import path from 'node:path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -import { _clearState, _setStore, allSkills, getSkill, getSkillResult, InMemorySkillStore } from '../skills.js'; +import { + _clearState, + _setEmitterForTesting, + _setStore, + allSkills, + getSkill, + getSkillResult, + InMemorySkillStore, +} from '../skills.js'; import { SKILL_OBJECT_KIND } from '../skills-core.js'; import { _warnedHashless, @@ -54,8 +62,8 @@ import { tombstoneFromDelete, } from '../skills-fdv2.js'; import { writeSkills } from '../skills-fs.js'; -import { watchSkills } from '../skills-watch.js'; -import type { RawSkillObject } from '../types.js'; +import { SkillWatcher, watchSkills } from '../skills-watch.js'; +import type { RawSkillObject, ReconcileReport } from '../types.js'; const SDK_KEY = 'sdk-00000000-0000-4000-8000-000000000000'; const SKILL_BODY = '---\nname: PDF Extraction\n---\nExtract text from PDFs.\n'; @@ -328,6 +336,14 @@ async function waitUntil(predicate: () => boolean, timeoutMs = 5000): Promise]> = []; + record(signal: string, properties: Record): void { + this.records.push([signal, properties]); + } +} + const logged = (spy: ReturnType): string => spy.mock.calls.map((call) => String(call[0])).join('\n'); const consoleErrors = (): string => logged(errorSpy); @@ -2224,23 +2240,78 @@ describe('watchSkills', () => { } }); - it('coalesces a burst of changes into few reconciles', async () => { - endpoint.queuePoll( - fullPayload(Array.from({ length: 12 }, (_, i) => ['put-object', putSkill(`skill-${i}`)] as [string, unknown])), - ); - endpoint.queuePoll([], { status: 304 }); - const store = pollStore(); + it('coalesces a burst of changes into exactly one reconcile', async () => { + // Three things make this test either evidence or theatre, and all three have + // to be right: + // + // 1. **The burst has to arrive after the watcher registers.** A payload + // committed before `watchSkills` attaches its listener notifies nobody, + // so the reconcile counter never leaves zero — and an upper bound + // (`<= 2`) then passes against an implementation with the debouncing + // deleted. So the seed payload is what boot waits for, and the burst is + // queued only once the watcher is up. + // 2. **The assertion has to be an equality, and it has to see movement.** + // Exactly one more reconcile than the initial one, and the counter has to + // have advanced at all; a run in which it stays at zero is not testing + // coalescing. + // 3. **The notifications have to be spread over time.** This one is not in + // the spec and it is what actually makes the test discriminate, verified + // by mutation: `SkillWatcher.schedule` also has a `pending` flag, so a + // burst that arrives in *one synchronous run* of the listener collapses + // to a single reconcile whether or not the debounce exists. Twelve + // objects in one commit are therefore not enough. Three commits arriving + // a poll apart are: each one lands after the previous reconcile would + // already have started and cleared `pending`, so only the debounce window + // can merge them. + endpoint.queuePoll(fullPayload([['put-object', putSkill('seed')]])); + const store = pollStore({ pollIntervalMs: 20 }); store.start(); - await store.waitForSkills(5000); + expect(await store.waitForSkills(5000)).toBe(true); _setStore(store); const root = path.join(await scratchRoot(), 'skills'); - const { watcher } = await watchSkills('*', root, { debounceMs: 50 }); + // Comfortably longer than the three commits take to arrive at a 20 ms poll + // interval — with enough margin that a loaded CI runner cannot push the + // third commit out of the window and turn a correct implementation red. + const { watcher } = await watchSkills('*', root, { debounceMs: 1000 }); try { - await new Promise((resolve) => setTimeout(resolve, 400)); - // Twelve objects committed in one payload fire twelve listener calls; - // without coalescing that is twelve reconciles of one root. - expect(watcher.reconciles).toBeLessThanOrEqual(2); + // The initial reconcile is the caller's own result and is excluded from + // the counter, so this is the baseline the equality is measured against. + expect(watcher.reconciles).toBe(0); + + // Twelve objects across three commits. Twelve, because a commit notifies + // once per *changed object* rather than once per commit (§3.25) — which is + // why the debounce exists at all rather than being a refinement of a + // per-commit notification. + for (let commit = 0; commit < 3; commit += 1) { + endpoint.queuePoll( + events( + ['server-intent', serverIntent('xfer-changes')], + ...Array.from( + { length: 4 }, + (_, i) => ['put-object', putSkill(`skill-${commit}-${i}`)] as [string, unknown], + ), + ['payload-transferred', transferred(`basis-${commit + 2}`)], + ), + ); + } + + // The endpoint really served the changes rather than a steady 304 for the + // duration — the other way this test goes vacuous. Asserted on the store's + // own counters, so "nothing was delivered" cannot read as "delivery was + // coalesced". + expect(await waitUntil(() => store.diagnostics.skillObjectsReceived >= 13, 10_000)).toBe(true); + expect(store.diagnostics.payloadsTransferred).toBeGreaterThanOrEqual(4); + + // The counter moved... + expect(await waitUntil(() => watcher.reconciles > 0, 10_000)).toBe(true); + // ...and then settled at exactly one. Waited out past a further whole + // window, so a second reconcile would have landed by now. + await new Promise((resolve) => setTimeout(resolve, 1500)); + expect(watcher.reconciles).toBe(1); + // And the one reconcile saw the settled state — the last commit's objects + // included — rather than a half-applied burst. + expect(await readFile(path.join(root, 'skill-2-3', 'SKILL.md'), 'utf8')).toBe(SKILL_BODY); } finally { await watcher.close(); } @@ -2344,6 +2415,190 @@ describe('watchSkills', () => { } }); + it('returns a named object, not a tuple, and takes its debounce in milliseconds', async () => { + // A.12 fixes both, and both are places a port from Python goes wrong + // silently: destructuring `[report, watcher]` off an object yields + // `undefined`s, and passing Python's seconds value to `debounceMs` sets a + // 0.5 ms window that looks like a timing bug rather than a unit bug. + const store = new InMemorySkillStore(); + store.put({ key: 'a', version: 1, content: 'body', contentHash: hash('body') }); + _setStore(store); + + const result = await watchSkills('*', path.join(await scratchRoot(), 'skills'), { + // Milliseconds. A whole second here, so the assertion below cannot be + // satisfied by a window that has already elapsed. + debounceMs: 1000, + }); + try { + expect(Object.keys(result).sort()).toEqual(['report', 'watcher']); + expect(Array.isArray(result)).toBe(false); + expect(result.report.actions.some((action) => action.action === 'written')).toBe(true); + expect(result.watcher).toBeInstanceOf(SkillWatcher); + + // The unit, observably: a change notified now has not reconciled a + // quarter of a second later, because the window is 1000 ms and not 1. + store.put({ key: 'a', version: 2, content: 'new body', contentHash: hash('new body') }); + await new Promise((resolve) => setTimeout(resolve, 250)); + expect(result.watcher.reconciles).toBe(0); + } finally { + await result.watcher.close(); + } + }); + + it('rejects a negative debounce and a non-finite one', async () => { + // `NaN` is the case a `< 0` guard misses — `NaN < 0` is false — and it is + // not benign: `setTimeout(fn, NaN)` fires at 1 ms, collapsing the window so + // that every delivered object reconciles and nothing coalesces at all. So + // the guard has to reject it the way `writeSkills` already rejects a + // non-numeric `timeout`. + const store = new InMemorySkillStore(); + _setStore(store); + const root = path.join(await scratchRoot(), 'skills'); + const listeners = (): unknown[] => + (store as unknown as { listeners: Map }).listeners.get(SKILL_OBJECT_KIND) ?? []; + + for (const debounceMs of [-1, Number.NaN, Number.POSITIVE_INFINITY]) { + await expect(watchSkills('*', root, { debounceMs })).rejects.toThrow(/debounceMs/); + } + // Refused before anything was registered or written: the validation runs + // ahead of the listener and ahead of the initial reconcile. + expect(listeners()).toEqual([]); + expect(existsSync(root)).toBe(false); + }); + + it('calls onReconcile for a delivery-triggered reconcile and not for the initial one', async () => { + // The initial report is returned directly, so delivering it through the + // callback as well would make a caller handle the same reconcile twice. + const store = new InMemorySkillStore(); + store.put({ key: 'a', version: 1, content: 'first', contentHash: hash('first') }); + _setStore(store); + + const seen: ReconcileReport[] = []; + const root = path.join(await scratchRoot(), 'skills'); + const { report, watcher } = await watchSkills('*', root, { + debounceMs: 20, + onReconcile: (each) => { + seen.push(each); + }, + }); + try { + // The initial reconcile did happen — it wrote the file — and it did not + // reach the callback. + expect(report.actions.some((action) => action.action === 'written')).toBe(true); + expect(await readFile(path.join(root, 'a', 'SKILL.md'), 'utf8')).toBe('first'); + expect(seen).toEqual([]); + + store.put({ key: 'a', version: 2, content: 'second', contentHash: hash('second') }); + + expect(await waitUntil(() => seen.length > 0, 10_000)).toBe(true); + expect(seen).toHaveLength(1); + // A real report for the re-reconcile, not the initial one handed over a + // second time. + expect(seen[0]).not.toBe(report); + expect(seen[0].actions.map((action) => [action.key, action.action])).toContainEqual(['a', 'updated']); + expect(seen[0].ok).toBe(true); + } finally { + await watcher.close(); + } + }); + + it('survives a reconcile that throws — the next notification still reconciles', async () => { + // A watcher that died on one bad run would silently stop pruning, which is + // strictly worse than a noisy one: revocations would keep arriving and + // nothing would act on them, and the process would look healthy. + // + // `onUnavailable: 'raise'` against a store that throws is the shortest real + // path to a *throwing* reconcile — every ordinary failure is reported as an + // `error` action instead. + const body = (content: string): RawSkillObject => ({ + key: 'a', + version: 1, + content, + contentHash: hash(content), + }); + let explode = false; + let notify: (() => void) | null = null; + _setStore({ + getObject: () => { + if (explode) throw new Error('transport failure'); + return body('first'); + }, + allObjects: () => ({ 'a:1': body('first') }), + addListener: (_kind: string, fn: () => void) => { + notify = fn; + }, + removeListener: () => { + notify = null; + }, + }); + + const root = path.join(await scratchRoot(), 'skills'); + const { watcher } = await watchSkills([{ key: 'a', version: 1 }], root, { + debounceMs: 10, + onUnavailable: 'raise', + }); + try { + expect(await readFile(path.join(root, 'a', 'SKILL.md'), 'utf8')).toBe('first'); + expect(notify).not.toBeNull(); + + explode = true; + (notify as unknown as () => void)(); + + // The failure is visible rather than swallowed, and the watcher says so. + expect(await waitUntil(() => /the watcher continues/.test(consoleErrors()), 10_000)).toBe(true); + // The throwing run does not count as a completed reconcile. + expect(watcher.reconciles).toBe(0); + + explode = false; + (notify as unknown as () => void)(); + + expect(await waitUntil(() => watcher.reconciles > 0, 10_000)).toBe(true); + } finally { + await watcher.close(); + } + }); + + it('raises an invalid root out of watchSkills rather than inside a worker', async () => { + // The initial reconcile runs on the caller's thread precisely so this is a + // rejected promise the caller can catch, exactly as `writeSkills` would give + // them — not a line in a background task's log that a caller cannot see and + // cannot react to. + const store = new InMemorySkillStore(); + _setStore(store); + const scratch = await scratchRoot(); + + // A root whose parent does not exist: never created recursively (§3.22). + await expect(watchSkills('*', path.join(scratch, 'a', 'b', 'c'))).rejects.toThrow(); + // And a root that is a file rather than a directory. + const file = path.join(scratch, 'file'); + await writeFile(file, '', 'utf8'); + await expect(watchSkills('*', file)).rejects.toThrow(); + + // Nothing was logged instead of thrown — the failure travelled one way only. + expect(consoleErrors()).toBe(''); + }); + + it('leaves no listener behind when the initial reconcile fails', async () => { + // The listener is registered *before* the initial reconcile so a payload + // committing during its filesystem I/O is not lost — which means a reconcile + // that throws has to detach on the way out. The caller is handed an + // exception, not a watcher to close, so nothing else can. + const store = new InMemorySkillStore(); + _setStore(store); + const listeners = (): unknown[] => + (store as unknown as { listeners: Map }).listeners.get(SKILL_OBJECT_KIND) ?? []; + const notADirectory = path.join(await scratchRoot(), 'file'); + await writeFile(notADirectory, '', 'utf8'); + + await expect(watchSkills('*', notADirectory)).rejects.toThrow(); + + expect(listeners()).toEqual([]); + // And the store holding no reference to it is the observable that matters: a + // later change must not reach a watcher nobody can close. + store.put({ key: 'a', version: 1, content: 'body', contentHash: hash('body') }); + expect(listeners()).toEqual([]); + }); + it('refuses a store with no addListener, loudly', async () => { _setStore({ getObject: () => null, @@ -2777,3 +3032,93 @@ describe('endpoints', () => { expect(urls[1]).toBe('https://stream.example.com/sdk/stream?basis=%28p%3Aa%3A1%29'); }); }); + +// ─── Layering, and the absence of telemetry ────────────────────────────────── + +describe('layering', () => { + /** `skills-fdv2.ts`'s own source text — the only way to assert a leaf. */ + const source = readFileSync(new URL('../skills-fdv2.ts', import.meta.url), 'utf8'); + + /** Every `from '...'` specifier in a source file, in order. */ + function importsOf(text: string): string[] { + return [...text.matchAll(/^\s*(?:import|export)\b[^;]*?from\s+'([^']+)'/gm)].map(([, specifier]) => specifier); + } + + it('adds no dependency beyond the standard library', () => { + // The package's sole runtime dependency is the OpenTelemetry API, and the + // transport is not allowed to add a second one. A `package.json` assertion + // would not catch it — a transitive dependency of another package resolves + // perfectly well from here. + const external = importsOf(source).filter((specifier) => !specifier.startsWith('.')); + expect(external.filter((specifier) => !specifier.startsWith('node:'))).toEqual([]); + }); + + it('imports the shared internals for the seam kind and nothing else from the feature', () => { + // The transport sits *below* the `SkillStore` interface: it produces raw + // wire objects and knows nothing about verification, the `Skill` type, or + // materialization. The seam kind is the one thing it legitimately needs from + // above, so that import is spelled out and the rest are named as forbidden. + const relative = [...new Set(importsOf(source).filter((specifier) => specifier.startsWith('.')))]; + // `types.js` is the package's value-type module, not a layer of the feature: + // the wire objects it produces have to be typed somehow. `skills-core.js` is + // the only feature module, and `skills.js` / `skills-fs.js` / + // `skills-watch.js` are absent by construction rather than by coincidence. + expect(relative.sort()).toEqual(['./skills-core.js', './types.js']); + + // And only the seam kind comes out of the shared internals — not the + // verification helpers, not the store state. + const fromCore = source.match(/import\s*\{([^}]*)\}\s*from\s*'\.\/skills-core\.js'/); + expect(fromCore).not.toBeNull(); + expect((fromCore as RegExpMatchArray)[1].split(',').map((name) => name.trim())).toEqual(['SKILL_OBJECT_KIND']); + }); + + it('is imported by nothing in the feature, so the layering cannot invert', () => { + // The package index re-exports it, which is the publication point rather + // than a layer — but no *module* of the feature may reach for it, or the + // accessors would start depending on one particular transport. + for (const module of ['skills.ts', 'skills-core.ts', 'skills-fs.ts', 'skills-watch.ts']) { + const text = readFileSync(new URL(`../${module}`, import.meta.url), 'utf8'); + expect(importsOf(text), module).not.toContain('./skills-fdv2.js'); + } + }); +}); + +describe('telemetry', () => { + it('records no signal of its own across a full delivery cycle', async () => { + // The three signals in the allowlist belong to verification and to + // materialization. The transport holds what arrived and counts what it + // ignored; it does not judge content, so it has nothing to report — in + // particular not for the hashless object below, which is verification's to + // withhold and not the transport's to flag. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + + endpoint.queuePoll( + fullPayload([ + ['put-object', putSkill('pdf-extraction')], + ['put-object', putSkill('hashless', { omitHash: true })], + ['put-object', putSkill('tampered', { contentHash: hash('something else') })], + ['put-object', putFlag()], + ]), + ); + endpoint.queuePoll( + events( + ['server-intent', serverIntent('xfer-changes')], + ['delete-object', deleteSkill('pdf-extraction')], + ['payload-transferred', transferred('basis-2')], + ), + ); + + const store = pollStore({ pollIntervalMs: 20 }); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(await waitUntil(() => store.diagnostics.objectsRevoked > 0, 10_000)).toBe(true); + // Serving is part of the cycle too, and it is the surface a naive + // implementation would verify on. + store.allObjects(SKILL_OBJECT_KIND); + store.getObject(SKILL_OBJECT_KIND, 'tampered'); + await store.close(); + + expect(emitter.records).toEqual([]); + }); +}); diff --git a/packages/client/src/__tests__/skills-fs-root-swap.test.ts b/packages/client/src/__tests__/skills-fs-root-swap.test.ts index e3e145b0..094ad3e2 100644 --- a/packages/client/src/__tests__/skills-fs-root-swap.test.ts +++ b/packages/client/src/__tests__/skills-fs-root-swap.test.ts @@ -24,10 +24,17 @@ * root's ancestors. * * Every test states the contract rather than the mechanism: nothing lands outside - * the root, no outside file is overwritten, no outside file is removed. The - * trigger matches both the path-based and descriptor-based spellings of the child - * path (see the hook below), so these same bodies fail against an unpinned root - * and pass against a pinned one — which is the only thing that makes them + * the root, no outside file is overwritten, no outside file is removed — **and**, + * for the three races where the swap lands after the containment check, that the + * operation went through in the root that was pinned. That second half is not + * decoration. The negative assertions alone are all satisfied by an + * implementation that refuses every write once anything has been swapped, which + * is precisely the regression the Linux fast path could introduce, so guarding + * them on `report.ok` would invert the test. + * + * The trigger matches both the path-based and descriptor-based spellings of the + * child path (see the hook below), so these same bodies fail against an unpinned + * root and pass against a pinned one — which is the only thing that makes them * evidence. * * In its own file because the swap has to land *before* the per-skill directory @@ -225,8 +232,16 @@ describe.skipIf(!SUPPORTS_PROC_FD)('writeSkills root swap races', () => { if (!race.fired) throw new Error(NEVER_FIRED); expect(await readdir(outside)).toEqual([]); - // Either the skill landed in the real root or the run says it did not. - if (report.ok) expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe(SKILL_BODY); + // Unconditionally, and that is the point: guarding this on `report.ok` would + // hand a pass to an implementation that refuses every write once anything + // has been swapped — which is exactly the regression the fast path could + // introduce, and which "the outside directory is empty" cannot distinguish + // from a write that landed correctly. The swap fires at the `mkdir`, after + // `unsafePathReason` has already run, so the containment check is not what + // is under test here: the descriptor addressing is, and it must deliver the + // file into the root that was validated. + expect(report.ok).toBe(true); + expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe(SKILL_BODY); }); it('a root swapped at the skill directory open cannot clobber an outside file', async () => { @@ -245,7 +260,11 @@ describe.skipIf(!SUPPORTS_PROC_FD)('writeSkills root swap races', () => { if (!race.fired) throw new Error(NEVER_FIRED); expect(await readFile(victim, 'utf-8')).toBe('precious\n'); - if (report.ok) expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe('served update\n'); + // Unconditional for the reason given above: "the victim is unchanged" holds + // for an implementation that wrote nothing at all, so the landing has to be + // asserted too. + expect(report.ok).toBe(true); + expect(await readFile(path.join(movedTo(), 'a', SKILL_MD), 'utf-8')).toBe('served update\n'); }); it('a root swapped at the prune cannot redirect the unlink', async () => { @@ -266,7 +285,13 @@ describe.skipIf(!SUPPORTS_PROC_FD)('writeSkills root swap races', () => { if (!race.fired) throw new Error(NEVER_FIRED); expect(await exists(victim)).toBe(true); expect(await readFile(victim, 'utf-8')).toBe('precious\n'); - if (report.ok) expect(await exists(path.join(movedTo(), 'a', SKILL_MD))).toBe(false); + // Unconditional, and on the destructive side it matters most: an + // implementation that pruned nothing satisfies "the outside file survives" + // perfectly. The managed file — the one this SDK owns — is the one that had + // to go, and the report has to say so. + expect(report.ok).toBe(true); + expect(report.actions.map((action) => [action.key, action.action])).toContainEqual(['a', 'removed']); + expect(await exists(path.join(movedTo(), 'a', SKILL_MD))).toBe(false); }); /** diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index c27be452..aa58e809 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -774,6 +774,101 @@ describe('writeSkills resilience', () => { expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); }); + it('does not prune when one of two references resolved and the other failed', async () => { + // The spec-mandated shape for the second suppression condition, and the one + // the whole-store-outage case above cannot stand in for: here the run *did* + // retrieve something, so an implementation that gated pruning on "did we get + // anything at all" passes that test and fails this one. One reference + // resolves, one does not, and a third manifest-listed skill that nobody + // asked for this run has to survive — because "revoked" and "did not arrive + // this time" are indistinguishable from here, and guessing is data loss. + const stale = path.join(root, 'stale', SKILL_MD); + await mkdir(path.dirname(stale), { recursive: true }); + await writeFile(stale, SKILL_BODY, 'utf-8'); + await writeManifest(root, { + manifestVersion: 1, + entries: { [`stale/${SKILL_MD}`]: manifestEntry('stale', 3, SKILL_BODY) }, + }); + + _setStore({ + getObject(_kind: string, key: string) { + if (key === 'resolves') return rawSkill('resolves'); + throw new Error('transport failure'); + }, + allObjects() { + return { resolves: rawSkill('resolves') }; + }, + }); + + const report = await writeSkills( + [ + { key: 'resolves', version: 1 }, + { key: 'fails', version: 1 }, + ], + root, + ); + + const byKey = actionsByKey(report); + // The run genuinely half-succeeded: one skill is on disk, the other is an + // error. That is what makes this shape different from a total outage. + expect(byKey.resolves.action).toBe('written'); + expect(await readFile(path.join(root, 'resolves', SKILL_MD), 'utf-8')).toBe(SKILL_BODY); + expect(byKey.fails.action).toBe('error'); + + // And the unrequested third skill was left alone, entry and all. + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + expect(byKey.stale).toBeUndefined(); + expect(await readFile(stale, 'utf-8')).toBe(SKILL_BODY); + const entries = (await readManifest(root)).entries as Record; + expect(entries[`stale/${SKILL_MD}`]).toMatchObject({ key: 'stale', version: 3 }); + }); + + it('does not prune when the deadline expired partway through the writes', async () => { + // The third shape of the same rule: retrieval was fine and the manifest was + // fine, but the budget ran out before every requested skill was written — so + // the run still does not know what is current. A caller must not read "no + // `removed` actions" as "nothing is stale". + const stale = path.join(root, 'stale', SKILL_MD); + await mkdir(path.dirname(stale), { recursive: true }); + await writeFile(stale, SKILL_BODY, 'utf-8'); + await writeManifest(root, { + manifestVersion: 1, + entries: { [`stale/${SKILL_MD}`]: manifestEntry('stale', 3, SKILL_BODY) }, + }); + + // A fake monotonic clock the first rename pushes past the deadline, so the + // second skill's write is on the wrong side of it. + let nowMs = performance.now(); + vi.spyOn(performance, 'now').mockImplementation(() => nowMs); + const real = fsOps.rename.bind(fsOps); + vi.spyOn(fsOps, 'rename').mockImplementation(async (src: string, dst: string) => { + const result = await real(src, dst); + if (dst.endsWith(SKILL_MD)) nowMs += 5_000; + return result; + }); + + const report = await writeSkills([skill('first'), skill('second')], root, { timeout: 1 }); + + const byKey = actionsByKey(report); + expect(byKey.first.action).toBe('written'); + // The injected expiry is what stopped the second write, rather than some + // unrelated refusal. + expect(byKey.second.action).toBe('error'); + expect(byKey.second.error).toMatch(/timeout was exhausted/); + + // So nothing was pruned, and the stale entry survives for the next run. + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + // Not *reached*, rather than reached and refused: the prune phase was + // suppressed outright, so `stale` carries no action at all. This is the + // assertion that separates the suppression gate from the per-entry deadline + // check inside the prune loop (§3.22 resilience), which would have reported + // a timeout error for the same entry and left the same file on disk. + expect(byKey.stale).toBeUndefined(); + expect(await readFile(stale, 'utf-8')).toBe(SKILL_BODY); + const entries = (await readManifest(root)).entries as Record; + expect(entries[`stale/${SKILL_MD}`]).toMatchObject({ key: 'stale', version: 3 }); + }); + it('an exhausted timeout behaves as unavailable', async () => { const store = new InMemorySkillStore(); store.put(rawSkill('a')); @@ -1287,6 +1382,38 @@ describe('writeSkills clobber protection', () => { expect(await readFile(target, 'utf-8')).toBe('user authored\n'); }); + it('distinguishes the adoption carve-out from an unmanaged file it must refuse', async () => { + // Adoption (§3.22) is the one carve-out in this row, and it does not weaken + // it: the only file ever claimed is one whose bytes *already are* the + // LaunchDarkly-resolved content. So the two have to be asserted against the + // same setup — the same root, the same absent manifest, the same + // `writeSkills` call — because byte equality is the entry condition, and an + // implementation that simply ignored the missing manifest entry would adopt + // both. This is the assertion that says the check is the bytes and not the + // absence. + const identical = path.join(root, 'same', SKILL_MD); + const divergent = path.join(root, 'differs', SKILL_MD); + await mkdir(path.dirname(identical), { recursive: true }); + await mkdir(path.dirname(divergent), { recursive: true }); + await writeFile(identical, SKILL_BODY, 'utf-8'); + // One byte longer, and otherwise the resolved content exactly. + await writeFile(divergent, `${SKILL_BODY} `, 'utf-8'); + expect(await exists(manifestPath(root))).toBe(false); + + const report = await writeSkills([skill('same'), skill('differs')], root); + + const byKey = actionsByKey(report); + // Identical bytes: adopted, and recorded as managed from now on. + expect(byKey.same.action).toBe('skipped_current'); + expect(await readFile(identical, 'utf-8')).toBe(SKILL_BODY); + const entries = (await readManifest(root)).entries as Record; + expect(entries[`same/${SKILL_MD}`]).toMatchObject({ key: 'same' }); + // One byte different: still refused, still untouched, still unclaimed. + expect(byKey.differs.action).toBe('error'); + expect(await readFile(divergent, 'utf-8')).toBe(`${SKILL_BODY} `); + expect(entries[`differs/${SKILL_MD}`]).toBeUndefined(); + }); + it('a manifest entry with a mismatched key does not authorize destruction', async () => { const target = path.join(root, 'a', SKILL_MD); await mkdir(path.dirname(target)); @@ -1379,6 +1506,27 @@ describe('writeSkills crash-mid-reconcile recovery', () => { expect(await readManifest(root)).toMatchObject({ entries: {} }); }); + it('an adopted file is prunable afterwards', async () => { + // Adoption is a full claim, not a one-run exemption — and this is the test + // that says so. Without it, "adopted" is indistinguishable from "tolerated + // once": an implementation that reported `skipped_current` without ever + // writing the manifest entry would pass every other case in this block, and + // then leave the file behind forever on the run that revoked it. + const target = await placeOrphaned('a', SKILL_BODY); + const first = await writeSkills([skill('a')], root); + expect(actionsByKey(first).a.action).toBe('skipped_current'); + + const second = await writeSkills([], root); + + // Removed exactly as a file this SDK wrote would have been: the action, the + // file, the now-empty directory, and the manifest entry. + expect(second.ok).toBe(true); + expect(actionsByKey(second).a.action).toBe('removed'); + expect(await exists(target)).toBe(false); + expect(await entryNames(root)).toEqual([MANIFEST_NAME]); + expect(await readManifest(root)).toMatchObject({ entries: {} }); + }); + it('adopts one skill and refuses another in the same run', async () => { await placeOrphaned('adopted', SKILL_BODY); const foreign = await placeOrphaned('foreign', 'user authored\n'); diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 7f06990a..44c154c2 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -96,6 +96,15 @@ function rawSkill(overrides: Partial & { key?: unknown } = {}): } as RawSkillObject; } +/** + * One over-cap string for the whole file. + * + * At 10 MiB this costs real time and memory to allocate and to hash, and §3.21 + * asks for one true over-cap case rather than one per test — so the cases that + * need it share this. + */ +const OVERSIZE = 'x'.repeat(MAX_SKILL_CONTENT_BYTES + 1); + function skill(content: Uint8Array = SKILL_BODY_BYTES, key = 'test-skill', version = 1): Skill { return createSkill({ key, version, content, contentHash: hash(content) }); } @@ -385,6 +394,60 @@ describe('skillRefs', () => { expect(skillRefs(undefined)).toEqual([]); }); + it('drops a malformed entry and logs one warning per drop', async () => { + // The silence is what makes this load-bearing rather than cosmetic. The + // projection's output is what a caller hands `writeSkills`, and a shortened + // list is indistinguishable there from "that skill is no longer requested" — + // so with `prune: true` (the default) a silently dropped entry *deletes the + // skill's files*. One warning per drop, each naming the position, is what + // lets an operator find the offending entry. + const spy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + try { + const refs = skillRefs({ + ...base, + skills: [ + { key: 'ok', version: 1 }, + // Not an object at all. + 'nope' as unknown as { key: string; version: number }, + // An invalid key. + { key: 'Not/A/Key', version: 1 }, + // An invalid version. + { key: 'also-ok', version: 0 }, + { key: 'also-ok', version: 2 }, + ], + }); + + // The usable entries survive, in order. + expect(refs).toEqual([ + { key: 'ok', version: 1 }, + { key: 'also-ok', version: 2 }, + ]); + + const lines = spy.mock.calls.map(([line]) => String(line)); + expect(lines).toHaveLength(3); + // Each names its own position, so three drops are three distinct reports + // rather than one summary an operator cannot act on. + expect(lines[0]).toContain('skills[1]'); + expect(lines[1]).toContain('skills[2]'); + expect(lines[2]).toContain('skills[3]'); + for (const line of lines) expect(line).toContain('dropped from the projection'); + } finally { + spy.mockRestore(); + } + }); + + it('warns about nothing when every entry is usable', async () => { + // The positive control: a suite in which the warning never fires at all + // cannot tell a per-drop warning from an unconditional one. + const spy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + try { + expect(skillRefs({ ...base, skills: [{ key: 'a', version: 1 }] })).toHaveLength(1); + expect(spy.mock.calls).toEqual([]); + } finally { + spy.mockRestore(); + } + }); + it('emits no telemetry', () => { const emitter = new RecordingEmitter(); _setEmitterForTesting(emitter); @@ -1096,9 +1159,8 @@ describe('integrity verification', () => { it('rejects content one byte over the size cap even when its hash matches', async () => { const emitter = new RecordingEmitter(); _setEmitterForTesting(emitter); - const oversize = 'x'.repeat(MAX_SKILL_CONTENT_BYTES + 1); const store = new InMemorySkillStore(); - store.put(rawSkill({ key: 'a', content: oversize })); + store.put(rawSkill({ key: 'a', content: OVERSIZE })); _setStore(store); expect(await getSkill('a')).toBeNull(); @@ -1291,6 +1353,33 @@ describe('telemetry seam, accessor half', () => { expect(props.expected_hash).toBe('b'.repeat(64)); expect(props.observed_hash).toBe(hash(SKILL_BODY)); expect(props.language).toBe('typescript'); + // Exactly those five and no more: the signal's property set is a documented + // allowlist, so the assertion is on the whole set rather than on each + // member. The five present is half a test. + expect(Object.keys(props).sort()).toEqual(['expected_hash', 'language', 'observed_hash', 'skill_key', 'version']); + }); + + it('keeps the four record-only fields out of the signal', async () => { + // The log record is the larger of the two surfaces and carries four fields + // the signal must not: a stable `event` identity, the `action` taken, the + // human-readable `reason`, and the machine-parseable `reason_code`. Letting + // them leak into the signal is the regression this guards — the signal's + // property set is an allowlist that does not grow, and the two surfaces are + // deliberately different sizes. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + _setStore(new DictStore({ a: rawSkill({ key: 'a', contentHash: 'b'.repeat(64) }) })); + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + try { + await getSkill('a'); + } finally { + spy.mockRestore(); + } + + const [props] = emitter.signals(INTEGRITY_SIGNAL); + for (const field of ['event', 'action', 'reason', 'reason_code']) { + expect(field in props, field).toBe(false); + } }); it('never puts the skill body in a signal', async () => { @@ -1475,7 +1564,7 @@ describe('integrity-failure log record', () => { ['missing_content', spoiled({}, ['content']), GET], ['missing_content_hash', spoiled({}, ['contentHash']), GET], ['not_utf8', spoiled({ content: surrogate, contentHash: hash(Buffer.from(surrogate, 'utf-8')) }), GET], - ['over_size_cap', spoiled({ content: 'x'.repeat(MAX_SKILL_CONTENT_BYTES + 1) }), GET], + ['over_size_cap', spoiled({ content: OVERSIZE }), GET], ['hash_mismatch', spoiled({ contentHash: 'd'.repeat(64) }), GET], ]; @@ -1659,6 +1748,57 @@ describe('integrity-failure log record', () => { expect(line).not.toContain(secret); }); + it('agrees with the signal on every key the two surfaces share', async () => { + // The record's fields are *spread* from the signal's rather than rebuilt, + // and that is the property worth pinning: two independently assembled + // mappings would drift, and the ones most likely to drift are exactly the + // ones that matter — which fields are redacted and which are omitted. So + // this captures both surfaces from one failure and compares them key by key. + // + // A *redacting* failure on purpose: `contentHash` is not a digest, so + // `expected_hash` goes through the replacement branch on its way into both + // surfaces. A well-formed failure would compare equal even against an + // implementation that rebuilt the record by hand. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + _setStore(new DictStore({ a: rawSkill({ key: 'a', version: 7, contentHash: 'not-a-digest' }) })); + + const [{ record }] = await logged(() => getSkill('a')); + const [props] = emitter.signals(INTEGRITY_SIGNAL); + + // Every key the signal carries is a key the record carries — the overlap is + // the signal's whole property set, not an incidental one or two. + expect(Object.keys(props).sort()).toEqual(['expected_hash', 'language', 'observed_hash', 'skill_key', 'version']); + for (const key of Object.keys(props)) expect(record[key], key).toEqual(props[key]); + // Including the redaction, which is the half a rebuilt record loses. + expect(props.expected_hash).toBe(''); + }); + + it('agrees with the signal on which fields are omitted, not only on values', async () => { + // The other half of "spread, not rebuilt": a field neither surface knows has + // to be missing from both. A record assembled separately is exactly where an + // explicit `null` or a stale default creeps in on one side only. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + const raw = rawSkill({ key: 'a' }); + // No content and an unusable version: nothing was hashed, so neither hash is + // known, and the version cannot be reported either. + delete raw.content; + raw.version = 0; + _setStore(new DictStore({ a: raw })); + + const [{ record }] = await logged(() => getSkill('a')); + const [props] = emitter.signals(INTEGRITY_SIGNAL); + + for (const key of ['version', 'expected_hash', 'observed_hash']) { + expect(key in props, `signal ${key}`).toBe(false); + expect(key in record, `record ${key}`).toBe(false); + } + // And what is left still agrees. + expect(Object.keys(props).sort()).toEqual(['language', 'skill_key']); + for (const key of Object.keys(props)) expect(record[key], key).toEqual(props[key]); + }); + it('is logged with no emitter configured — telemetry off is not detection off', async () => { // The reason this record exists: it is the whole detection story for a // customer whose telemetry is switched off, or who has no destination. diff --git a/packages/client/src/skills-watch.ts b/packages/client/src/skills-watch.ts index 1a1a0041..261c64c7 100644 --- a/packages/client/src/skills-watch.ts +++ b/packages/client/src/skills-watch.ts @@ -238,8 +238,18 @@ export async function watchSkills( ); } + // `NaN` is the case a `< 0` guard misses — `NaN < 0` is false — and it is not a + // harmless one: `setTimeout(fn, NaN)` fires at 1 ms, which collapses the + // coalescing window to nothing and reconciles once per *delivered object*. A + // twelve-skill payload would then run twelve reconciles of one root. Guarded + // the way `writeSkills` already guards its own `timeout`. const { debounceMs = DEFAULT_DEBOUNCE_MS, onReconcile, ...writeOptions } = options; - if (debounceMs < 0) throw new Error(`debounceMs must not be negative, got ${JSON.stringify(debounceMs)}`); + if (typeof debounceMs !== 'number' || !Number.isFinite(debounceMs) || debounceMs < 0) { + // `String` rather than `JSON.stringify` for the number case: the latter + // serializes `NaN` as `null`, which names the wrong mistake. + const shown = typeof debounceMs === 'number' ? String(debounceMs) : JSON.stringify(debounceMs); + throw new Error(`debounceMs must be a non-negative, finite number of milliseconds, got ${shown}`); + } const watcher = new SkillWatcher(store, skills, root, writeOptions, debounceMs, onReconcile); diff --git a/packages/client/src/skills.ts b/packages/client/src/skills.ts index aa9870a8..df3404f3 100644 --- a/packages/client/src/skills.ts +++ b/packages/client/src/skills.ts @@ -228,12 +228,31 @@ export class InMemorySkillStore implements SkillStore { // Reference discovery // --------------------------------------------------------------------------- +/** + * Warns that one `skills` entry was left out of the projection. + * + * A dropped entry cannot be silent. The projection's output is what a caller + * hands `writeSkills`, and a shortened list is indistinguishable there from "that + * skill is no longer requested" — so with `prune: true` (the default) a silently + * dropped entry *deletes the skill's files*. Naming the position is what lets an + * operator find the offending entry in a hand-built config. + */ +function warnDropped(index: number, why: string): void { + // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; a dropped skill reference must be visible + console.warn(`[LaunchDarkly] skills[${index}]${why}; it was dropped from the projection`); +} + /** * Projects a resolved AI Config's `skills` array into typed references. * * A pure projection — no network, no client, no store, no telemetry. Returns `[]` * when the config carries no skills. Compose it with the accessors for * per-context resolution: `await getSkills(skillRefs(config))`. + * + * A config that came through `parseAiConfig` never carries an invalid entry — + * parsing fails closed on one. A hand-built object can, and this re-validates + * rather than trusting that it did not, so every entry it cannot use is dropped + * and **logged**; see {@link warnDropped} for why the silence would matter. */ export function skillRefs(config: AiConfigRep | null | undefined): SkillReference[] { if (typeof config !== 'object' || config === null) return []; @@ -242,10 +261,17 @@ export function skillRefs(config: AiConfigRep | null | undefined): SkillReferenc if (!Array.isArray(raw)) return []; const refs: SkillReference[] = []; - for (const entry of raw) { - if (typeof entry !== 'object' || entry === null) continue; + for (const [index, entry] of raw.entries()) { + if (typeof entry !== 'object' || entry === null) { + warnDropped(index, ' is not a { key, version } object'); + continue; + } const { key, version } = entry as { key?: unknown; version?: unknown }; - if (isValidSkillKey(key) && isValidSkillVersion(version)) { + if (!isValidSkillKey(key)) { + warnDropped(index, '.key must be 1–256 characters matching /^[a-z0-9][a-z0-9-]*$/'); + } else if (!isValidSkillVersion(version)) { + warnDropped(index, '.version must be an integer >= 1'); + } else { refs.push(createSkillReference({ key, version })); } } From 49db83bd330d8564997d5c0e2c3945e8b83153da Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Thu, 17 Sep 2026 09:02:52 -0400 Subject: [PATCH 32/55] docs(client): regenerate the export block and correct the platform note MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "Public Exports" block had drifted from `src/index.ts` in both directions: it listed a `./tracking.js` re-export that `index.ts` does not have, and omitted the `content.js` set, the judge helpers, several `types.js` names, and the skills option and mode types. It is now generated from `index.ts` verbatim and in its order, with a note saying so, because a hand-curated copy of a 20-statement barrel drifts again. Pitfall 6 still described the pre-`/proc/self/fd` world — that `SUPPORTS_DIR_FD` is `false` and the swap-race tests are therefore skipped with the exposure "real and recorded, not fixed". Since 0a15b10 that is wrong on Linux: `SUPPORTS_PROC_FD` addressing closes the window there and the tests run. Records the two things a reader needs to keep straight — that the identity re-check is the macOS floor only and must not be attempted on the fast path, and that a green macOS run says nothing, because all ten of those tests execute on Linux CI alone. Also records the third test hook, the `vi.mock` of `mkdir`/`open` the root-swap races fire from, which is deliberately not part of `fsOps`. Co-Authored-By: Claude Opus 5 --- packages/client/agents.md | 118 ++++++++++++++++++++++++++++++-------- 1 file changed, 94 insertions(+), 24 deletions(-) diff --git a/packages/client/agents.md b/packages/client/agents.md index a2b8a91f..de23258d 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -85,35 +85,101 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b ## Public Exports (`src/index.ts`) +This block is **generated from `src/index.ts`, verbatim and in its order** — it is not a curated +summary. Regenerate it rather than hand-editing, or it drifts: it has previously listed a +`./tracking.js` re-export that `index.ts` does not have, while omitting a dozen names it does. + ```ts -export type { InspectConfigResult } from './lifecycle.js'; -export { getClient, initClient, inspectConfig, shutdown, shutdownTelemetry, waitForTelemetry } from './lifecycle.js'; -export type { LDContext, LDClientInterface, LDSingleKindContext, LDMultiKindContext, LDUser } from './types.js'; -export { config } from './client.js'; export type { AiConfigRep } from './client.js'; -export type { - Tool, VariationMeta as LDVariationMeta, ProviderHandler, ProviderSetupFn, ProviderResponse, - ConfigArgs, TrackData, - GraphTopology, GraphEdge, GraphNode, GraphDefinition, GraphOptions, GraphArgs, - HandlerStreamEvent, StreamEvent, ProviderGraphResponse, -} from './types.js'; -export { GraphTopologySchema, NativeTool, NATIVE_TOOL_KEY } from './types.js'; -export { Registry, globalRegistry, compose } from './registry.js'; -export { parseTemplate, parseJSONWithPossibleFences, createHandler } from './utils.js'; +export { config } from './client.js'; +export type { ContentCaptureOptions, SpanMessage, SpanMessagePart, ToolDefinitionInput } from './content.js'; +export { + langChainFinishReasons, + langChainSpanMessages, + setInputContentAttributes, + setOutputContentAttributes, + setToolCallContentAttributes, + setToolDefinitionAttributes, + textMessage, + toSemconvFinishReason, +} from './content.js'; export { graph, resolveGraph } from './graph.js'; -export { parseUsage, normalizeMode, parseAiConfig } from './tracking.js'; - -// Agent Skills -export { skillRefs, getSkill, getSkillResult, getSkills, allSkills, InMemorySkillStore } from './skills.js'; -export { writeSkills, SKILL_FILENAME, MANIFEST_FILENAME, MANIFEST_VERSION } from './skills-fs.js'; -export { FDv2SkillStore, DEFAULT_BASE_URI, DEFAULT_STREAM_URI } from './skills-fdv2.js'; -export { watchSkills, SkillWatcher, DEFAULT_DEBOUNCE_MS } from './skills-watch.js'; +export { buildJudgeTasks, runJudge } from './judges.js'; +export type { InspectConfigResult } from './lifecycle.js'; +export { getClient, initClient, inspectConfig, shutdown, shutdownTelemetry, waitForTelemetry } from './lifecycle.js'; +export { compose, globalRegistry, Registry } from './registry.js'; +export { allSkills, getSkill, getSkillResult, getSkills, InMemorySkillStore, skillRefs } from './skills.js'; +export type { FDv2Mode, FDv2SkillStoreOptions, StoreDiagnostics } from './skills-fdv2.js'; +export { DEFAULT_BASE_URI, DEFAULT_STREAM_URI, FDv2SkillStore } from './skills-fdv2.js'; export type { WriteSkillsOptions } from './skills-fs.js'; -export { createSkill, createSkillOutcome, createSkillReference } from './types.js'; +export { MANIFEST_FILENAME, MANIFEST_VERSION, SKILL_FILENAME, writeSkills } from './skills-fs.js'; +export type { WatchSkillsOptions } from './skills-watch.js'; +export { DEFAULT_DEBOUNCE_MS, SkillWatcher, watchSkills } from './skills-watch.js'; export type { - Skill, SkillOutcome, SkillOutcomeReason, SkillReference, SkillStore, RawSkillObject, - ReconcileAction, ReconcileActionKind, ReconcileReport, OnUnavailable, + ConfigArgs, + GraphArgs, + GraphDefinition, + GraphEdge, + GraphNode, + GraphOptions, + GraphTopology, + HandlerStreamEvent, + JudgeCallResult, + JudgeRunResult, + JudgeTask, + LDClientInterface, + LDContext, + LDMultiKindContext, + LDSingleKindContext, + LDUser, + Message, + OnUnavailable, + ProviderGraphResponse, + ProviderHandler, + ProviderResponse, + ProviderSetupFn, + RawSkillObject, + ReconcileAction, + ReconcileActionKind, + ReconcileReport, + RegistryInput, + RouteResult, + RunNodeOptions, + Skill, + SkillOutcome, + SkillOutcomeReason, + SkillReference, + SkillStore, + StreamEvent, + TokenUsage, + Tool, + ToolHandlerFn, + TrackData, + TraverseVisitor, + VariationMeta as LDVariationMeta, } from './types.js'; +export { + createSkill, + createSkillOutcome, + createSkillReference, + GraphTopologySchema, + NATIVE_TOOL_KEY, + NativeTool, +} from './types.js'; +export type { RunUsage, SpanUsage } from './utils.js'; +export { + addCachedTokensToInput, + collapseMessagesToInstructions, + createHandler, + createRunUsage, + endSpanOnce, + langChainSpanUsage, + parseJSONWithPossibleFences, + parseTemplate, + setLdSpanAttributes, + setModelIdentityAttributes, + setUsageSpanAttributes, +} from './utils.js'; ``` When adding a new export, add it here. Handler packages must never import from sub-paths (e.g. `@launchdarkly/ai-server/dist/client`). @@ -318,7 +384,11 @@ Three specifics inside those two functions that a later contributor is most like ### 6. Bypassing `fsOps` for a destructive filesystem call -`safe-fs.ts` routes the final rename and the managed-file unlink through the `fsOps` record so tests can intercept exactly those two operations. The orphaned-temp sweep goes through `unlinkNoFollow` for the same reason, and derives its filename pattern from `tempNamePattern` in `safe-fs.ts` rather than carrying a copy: that sweep is only entitled to unlink a file because the *name* identifies it as one this SDK created, so two spellings of the naming rule would eventually let it either miss orphans or remove something it did not write. Calling `fs.rename`/`fs.unlink` directly makes the operation invisible to the atomicity and "no operation was attempted" assertions, which then pass vacuously. Note also the limitation those tests document: Node exposes no `renameat`/`unlinkat`, so `SUPPORTS_DIR_FD` is `false` and the TOCTOU swap-race tests are skipped — the residual exposure is real and recorded, not fixed. +`safe-fs.ts` routes the final rename and the managed-file unlink through the `fsOps` record so tests can intercept exactly those two operations. The orphaned-temp sweep goes through `unlinkNoFollow` for the same reason, and derives its filename pattern from `tempNamePattern` in `safe-fs.ts` rather than carrying a copy: that sweep is only entitled to unlink a file because the *name* identifies it as one this SDK created, so two spellings of the naming rule would eventually let it either miss orphans or remove something it did not write. Calling `fs.rename`/`fs.unlink` directly makes the operation invisible to the atomicity and "no operation was attempted" assertions, which then pass vacuously. + +There is a **third** hook, and it is deliberately not part of `fsOps`: the root-swap races mock `mkdir` and `open` through `vi.mock('node:fs/promises')`, because they have to fire *before* the root is pinned. Widening `fsOps` to cover those would blur what a "no filesystem operation was attempted" assertion means, so leave them separate. + +On the platform bound: Node exposes no `renameat`/`unlinkat`, so `SUPPORTS_DIR_FD` is `false` on every release to date — but that is **not** the end of the story, and an older version of this note said it was. `SUPPORTS_PROC_FD` addresses children through `/proc/self/fd//`, which the kernel resolves from the inode the descriptor holds rather than from the name it was opened under. That **closes** the swap window on Linux, so the swap-race tests *run* there and are skipped only where neither capability is present. Two consequences worth keeping straight: the `(dev, ino)` identity re-check is the macOS floor only and must not be attempted on the fast path (`lstat` of `/proc/self/fd/` reports procfs's magic symlink, not the directory); and a green macOS test run is not evidence about any of this, since all ten of those tests skip locally and execute on Linux CI alone. ### 7. "Fixing" the platform bound, or adding a writability field to `ReconcileReport` From 7bfa88ec14c9d5e6ad23f592cea2adbef64efbb4 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 30 Sep 2026 12:52:28 -0400 Subject: [PATCH 33/55] =?UTF-8?q?fix(client)!:=20Agent=20Skills=20?= =?UTF-8?q?=E2=80=94=20close=20the=20readiness=20and=20attribution=20prune?= =?UTF-8?q?=20holes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two data-loss-class ports from the Python SDK (5a896cef, 7c070d0a), plus the manifest version gate that belongs with them. All three let a reconcile treat "could not retrieve" as "was revoked", and prune a customer's files over it. - A store still waiting for its first payload and an environment with no skills answer `allObjects` identically, and `writeSkills('*')` read the empty answer as "every skill was revoked". `SkillStore` gains an optional `isInitialized()`, `FDv2SkillStore` implements it as "a payload has committed", and the single retrieval gate in `skills-fs.ts` reports an uninitialized store unavailable — run incomplete, prune suppressed, the remedy (`waitForSkills` resolving true) in the message. - `resolveAll` returned `incomplete: false` unconditionally, so an object whose own `key` and map key were both unusable was reported run-level and then pruned in the same run. The run is now incomplete whenever a request carries the run-level key with no skill. - `loadManifest` accepted `manifestVersion` 0 and negatives as readable older schemas; the gate is now `[1, MANIFEST_VERSION]`. `availableStore` is now the single gate both the one-reference and the `'*'` path consult, so the two cannot disagree about what counts as "the store could not answer". `ScriptedRequester` in the FDv2 suite now honours the store's abort signal the way the real requester does, so a scripted promise that never settles still lets `close()` return. The readiness test needs a store that never receives a payload, and without this its `afterEach` close hangs. Co-Authored-By: Claude Fable 5.1 Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 12 +- .../client/src/__tests__/skills-fdv2.test.ts | 57 ++++- .../client/src/__tests__/skills-fs.test.ts | 203 +++++++++++++++++- packages/client/src/__tests__/skills.test.ts | 65 +++++- packages/client/src/skills-core.ts | 32 +++ packages/client/src/skills-fdv2.ts | 18 ++ packages/client/src/skills-fs.ts | 101 ++++++--- packages/client/src/types.ts | 17 ++ 8 files changed, 461 insertions(+), 44 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 23fc7da8..50df9a96 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -326,7 +326,7 @@ example above, materializes only what the resolved variation actually asked for. | `allSkills()` | Every verified skill the store holds, newest version per key. | | `writeSkills(skills, root, options?)` | Materialize to `//SKILL.md`. Accepts `Skill` / `SkillReference` / key strings, or the literal `'*'`. Returns a `ReconcileReport`. Throws for a caller error — an unusable `root`, a bare string other than `'*'` — as distinct from the per-skill `error` actions in the report. | | `InMemorySkillStore` | A `SkillStore` backed by in-memory maps. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Several versions of one key coexist, as they do in a real payload: `getObject` answers a pin with exactly that version and an omitted version with the newest held, and `allObjects` returns one entry per `(key, version)` under keys opaque to callers. `addListener` throws for any kind but `'skill'`. | -| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | +| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `isInitialized()`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | | `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object`, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | @@ -382,6 +382,10 @@ try { } ``` +**A reconcile that runs before delivery answers does not prune.** Through the `SkillStore` seam, a store still waiting for its first payload and an environment that holds no skills give the same answer — an empty one — and `writeSkills('*')` would otherwise read that as every skill having been revoked and delete the files it wrote on a previous run. `FDv2SkillStore` reports readiness through the optional `isInitialized()`, the same fact `waitForSkills` resolves to without the wait, so a reconcile against a store that has not received a payload reports the retrieval unavailable and leaves everything on disk alone. `report.ok` is `false` in that case, and the error names the remedy. A store that does not implement `isInitialized()` is treated as initialized, which is right for `InMemorySkillStore`. + +**`watchSkills` is one watcher per root.** It registers the store's change listener — so the configured store must implement the optional `addListener`, and `watchSkills` throws if it does not, rather than degrading to a one-shot reconcile — then runs `writeSkills` once and returns that report alongside a `SkillWatcher`. Every delivery change after that schedules a re-reconcile, coalesced over `debounceMs` (default `DEFAULT_DEBOUNCE_MS`, 500 ms) so a payload of forty objects runs one reconcile rather than forty; `onReconcile` receives each of those reports, and never the initial one. The watcher exposes `reconciles` (how many re-reconciles have completed), `notify` (the listener it registered, for tests) and `close()`, which detaches and awaits any reconcile in flight. Do not point two watchers at one root, and do not call `writeSkills` on a watched root yourself: a reconcile's contract is one root, one reconcile at a time, and two interleaved runs lose the loser's manifest entries. Note that `debounceMs` is in **milliseconds** while the `timeout` it sits beside in the same options bag is in **seconds**. + **`waitForSkills` orders boot against the first payload.** It resolves `true` once a payload has been committed, or once a `304` confirms the payload already held is the current one. It resolves `false` if the wait times out, the store is closed, or delivery stops for good — a fatal error such as an unauthorized key resolves it right away rather than at the timeout, so a boot gated on the return value does not proceed on a dead store. Neither a closed store nor one that has given up waits at all, whether the wait was already pending when it happened or started afterwards. Read `failed` to tell a store that gave up from one that timed out; a store you closed yourself leaves `failed` as `null`, because closing is not a failure. **`close()` is final.** It stops delivery and does not undo it: `start()` throws on a closed store rather than opening a second connection, since a store that looks live and is not is worse than one that plainly refuses. Retrieval needs no restart — a closed store keeps answering from the content it received — so construct a new store if you need delivery again. @@ -565,10 +569,14 @@ All types are re-exported from this package. Handler packages import them from h | `SkillReference` | A version-pinned pointer to a skill: `{ key, version }` | | `SkillOutcome` | What `getSkillResult()` resolves to: `{ skill, reason, detail }`. `skill` is non-null exactly when `reason` is `'ok'` | | `SkillOutcomeReason` | The closed set of retrieval outcomes: `'absent' \| 'integrity_failure' \| 'ok' \| 'store_unavailable' \| 'wrong_version'` | -| `SkillStore` | The structural seam skill content is retrieved through: `getObject(kind, key, version?)`, `allObjects`, optional `addListener` / `removeListener` | +| `SkillStore` | The structural seam skill content is retrieved through: `getObject(kind, key, version?)`, `allObjects`, optional `isInitialized()`, `addListener` / `removeListener`. A store without `isInitialized()` is treated as initialized. | | `RawSkillObject` | The wire shape a `SkillStore` serves, before verification. Every field is untrusted. | | `ReconcileReport` | The result of `writeSkills()`: `{ actions, ok, errors }` | | `ReconcileAction` | One outcome from a reconcile: `{ key, action, version, path, error }` | | `ReconcileActionKind` | The closed set of reconcile outcomes: `'written' \| 'updated' \| 'skipped_current' \| 'removed' \| 'error'` | | `OnUnavailable` | `'keep' \| 'raise'` — how `writeSkills` reacts to content it could not retrieve | | `WriteSkillsOptions` | Options accepted by `writeSkills()` (`prune`, `timeout` in seconds, `onUnavailable`) | +| `WatchSkillsOptions` | `WriteSkillsOptions` plus `debounceMs` (milliseconds) and `onReconcile` — what `watchSkills()` accepts | +| `FDv2Mode` | `'stream' \| 'poll'` — the `mode` option of `FDv2SkillStore` | +| `FDv2SkillStoreOptions` | Options accepted by the `FDv2SkillStore` constructor (`mode`, `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`) | +| `StoreDiagnostics` | The read-only counters `FDv2SkillStore.diagnostics` returns | diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 394d0096..a77c96a2 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -1407,21 +1407,33 @@ class ScriptedRequester implements Requester { constructor(private readonly outcomes: unknown[] = []) {} - private async next(): Promise> { + /** + * Honours the store's abort signal the way the real requester does: a + * scripted promise that never settles still lets `close()` return, since the + * store aborts the signal and awaits the delivery loop. + */ + private async next(signal: AbortSignal): Promise> { const outcome = this.outcomes.length > 0 ? this.outcomes.shift() : new RecoverableTransportError('x'); if (outcome instanceof Error) throw outcome; - if (outcome instanceof Promise) return outcome as Promise>; + if (outcome instanceof Promise) { + const aborted = new Promise((_, reject) => { + const onAbort = (): void => reject(signal.reason ?? new Error('aborted')); + if (signal.aborted) onAbort(); + else signal.addEventListener('abort', onAbort, { once: true }); + }); + return Promise.race([outcome as Promise>, aborted]); + } return outcome as Array<[string, unknown]>; } - async poll(basis: string | null, etag: string | null): Promise { + async poll(basis: string | null, etag: string | null, signal: AbortSignal): Promise { this.calls.push([basis, etag]); - return { notModified: false, events: await this.next(), etag: null }; + return { notModified: false, events: await this.next(signal), etag: null }; } - async stream(basis: string | null): Promise> { + async stream(basis: string | null, signal: AbortSignal): Promise> { this.calls.push([basis, null]); - const scripted = await this.next(); + const scripted = await this.next(signal); return (async function* () { yield* scripted; })(); @@ -2748,6 +2760,39 @@ describe('lifecycle', () => { expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); }); + it('isInitialized tracks the first payload, and stays true after close (§3.25)', async () => { + // The probe `writeSkills('*')` reads to decide whether it may prune. Before + // the first payload, an empty store and an environment with no skills are + // the same answer through `allObjects`; this is what tells them apart. + const silent = new FDv2SkillStore(SDK_KEY, { + mode: 'poll', + pollIntervalMs: 60_000, + requester: new ScriptedRequester([new Promise(() => {})]), + }); + openStores.push(silent); + expect(silent.isInitialized()).toBe(false); + silent.start(); + expect(await silent.waitForSkills(50)).toBe(false); + expect(silent.isInitialized()).toBe(false); + + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const delivering = pollStore(); + delivering.start(); + expect(await delivering.waitForSkills(5000)).toBe(true); + expect(delivering.isInitialized()).toBe(true); + await delivering.close(); + // Content outlives the connection, so the fact about it does too. + expect(delivering.isInitialized()).toBe(true); + }); + + it('a 304 counts as initialized — the payload held is confirmed current', async () => { + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(store.isInitialized()).toBe(true); + }); + it('times out waitForSkills rather than hanging', async () => { const store = new FDv2SkillStore(SDK_KEY, { mode: 'poll', diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index aa58e809..6aca1ef5 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -869,15 +869,26 @@ describe('writeSkills resilience', () => { expect(entries[`stale/${SKILL_MD}`]).toMatchObject({ key: 'stale', version: 3 }); }); - it('an exhausted timeout behaves as unavailable', async () => { + it('an exhausted timeout behaves as unavailable, and retrieves nothing', async () => { const store = new InMemorySkillStore(); store.put(rawSkill('a')); - _setStore(store); + // `timeout` bounds the whole call *including* retrieval: with the budget + // already spent, the store must not be consulted at all. + let lookups = 0; + _setStore( + new Proxy(store, { + get(target, property, receiver) { + if (property === 'getObject') lookups += 1; + return Reflect.get(target, property, receiver); + }, + }), + ); const calls = interceptRename(); const report = await writeSkills([{ key: 'a', version: 1 }], root, { timeout: 0 }); expect(report.ok).toBe(false); + expect(lookups).toBe(0); expect(calls).toEqual([]); expect(await exists(path.join(root, 'a', SKILL_MD))).toBe(false); }); @@ -1859,6 +1870,11 @@ const CORRUPT_MANIFESTS: Array<[string, unknown]> = [ ['version_not_int', { manifestVersion: '1', entries: {} }], ['future_version_live_entries', { manifestVersion: 2, entries: liveEntries() }], ['version_not_int_live_entries', { manifestVersion: '1', entries: liveEntries() }], + // §3.22: the version gate is bounded below as well as above. No release ever + // wrote a version under 1, so these are not older schemas this release can + // still read — they are schemas that never existed. + ['version_zero_live_entries', { manifestVersion: 0, entries: liveEntries() }], + ['version_negative_live_entries', { manifestVersion: -1, entries: liveEntries() }], ]; const LIVE_ENTRY_MANIFESTS = CORRUPT_MANIFESTS.filter(([name]) => name.endsWith('_live_entries')); @@ -1882,6 +1898,30 @@ describe('writeSkills corrupt manifest', () => { expect(await readFile(target, 'utf-8')).toBe(DIVERGENT_CONTENT); }); + it.each([0, -1])('treats manifestVersion %s as corrupt, not as an older readable schema', async (declared) => { + // Python parity: `test_manifest_version_below_one_is_corrupt`. Before the + // lower bound existed, `manifestVersion: 0` pruned a managed file and + // reported `ok` true — the destructive steps ran against entries of a + // schema that never existed. + const target = await placeManaged(root, 'a', SKILL_BODY); + await writeManifest(root, { + manifestVersion: declared, + entries: { [`a/${SKILL_MD}`]: manifestEntry('a', 1, SKILL_BODY) }, + }); + const unlinks = interceptUnlink(); + + const report = await writeSkills([], root); + + expect(report.ok).toBe(false); + const messages = errorMessages(report); + expect(messages.some((m) => m.includes('manifestVersion') && m.includes(MANIFEST_NAME))).toBe(true); + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + expect(unlinks).toEqual([]); + expect(await exists(target)).toBe(true); + // The manifest itself is left alone, as with any other corruption. + expect((await readManifest(root)).manifestVersion).toBe(declared); + }); + it('a run-level error carries the empty-key sentinel', async () => { // The empty string is public API surface: a caller grouping the report by // key has to know the sentinel exists. @@ -2519,6 +2559,165 @@ describe('writeSkills telemetry', () => { }); }); +// ─── Readiness: a store that has not heard yet must not authorize a prune ──── + +/** + * §3.21 "`isInitialized()` is the optional readiness half of the seam" and + * §3.22 "Pruning is suppressed whenever the run cannot tell what is still + * current" — the third of the four conditions. + * + * Through `allObjects` there is no difference between "this environment holds + * no skills" and "delivery has not answered yet": both are an empty result. + * `writeSkills('*')` reads the first as every skill having been revoked, so + * without the probe a reconcile racing a slow boot deletes every managed file + * and reports success. Mirrors Python's `TestUninitializedStore`. + */ +describe('writeSkills uninitialized store', () => { + /** A delivery store whose first payload has not arrived. */ + function waiting(initialized: boolean): SkillStore & { isInitialized(): boolean } { + return { + isInitialized: () => initialized, + getObject: () => null, + allObjects: () => ({}), + }; + } + + it('"*" does not prune before the first payload', async () => { + const existing = await placeManaged(root, 'a', SKILL_BODY); + const unlinks = interceptUnlink(); + _setStore(waiting(false)); + + const report = await writeSkills('*', root); + + expect(report.ok).toBe(false); + expect(await readFile(existing, 'utf-8')).toBe(SKILL_BODY); + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + expect(unlinks).toEqual([]); + // The entry survives, so a later reconcile still knows it owns the file. + expect((await readManifest(root)).entries).toHaveProperty(`a/${SKILL_MD}`); + // The error is run-level, is worded as a retrieval-unavailable failure, and + // names the remedy: wait for delivery. + const messages = errorMessages(report); + expect(messages.some((m) => m.includes('initial data'))).toBe(true); + expect(messages.some((m) => m.includes('skill retrieval unavailable'))).toBe(true); + expect(messages.some((m) => m.includes('waitForSkills'))).toBe(true); + expect(report.errors.every((a) => a.key === '')).toBe(true); + }); + + it('a reference resolve against an uninitialized store is unavailable too, and suppresses prune', async () => { + // The check lives in the single gate both forms go through, so the + // reference form reports the same unavailability and holds prune off the + // unrelated managed entry. + const stale = await placeManaged(root, 'stale', SKILL_BODY); + _setStore(waiting(false)); + + const report = await writeSkills([{ key: 'a', version: 1 }], root); + + expect(report.ok).toBe(false); + expect(actionsByKey(report).a.action).toBe('error'); + expect(actionsByKey(report).a.error).toContain('initial data'); + expect(actionsByKey(report).stale).toBeUndefined(); + expect(await readFile(stale, 'utf-8')).toBe(SKILL_BODY); + }); + + it('"*" throws in raise mode before the first payload', async () => { + _setStore(waiting(false)); + await expect(writeSkills('*', root, { onUnavailable: 'raise' })).rejects.toThrow(/initial data/); + }); + + it('an initialized store prunes as before — the positive control', async () => { + await placeManaged(root, 'a', SKILL_BODY); + _setStore(waiting(true)); + + const report = await writeSkills('*', root); + + expect(report.ok).toBe(true); + expect(actionsByKey(report).a.action).toBe('removed'); + expect(await exists(path.join(root, 'a'))).toBe(false); + }); + + it('a store without the probe is treated as initialized', async () => { + // `isInitialized` is optional; absent means initialized. A hand-populated + // store is never waiting for anything, so requiring the probe would break + // every InMemorySkillStore caller. + const store = new InMemorySkillStore(); + expect('isInitialized' in store).toBe(false); + _setStore(store); + await placeManaged(root, 'a', SKILL_BODY); + + const report = await writeSkills('*', root); + + expect(report.ok).toBe(true); + expect(actionsByKey(report).a.action).toBe('removed'); + }); + + it('a probe that throws counts as uninitialized, and warns', async () => { + // A store that cannot say whether it is ready does not get to delete. + const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); + const existing = await placeManaged(root, 'a', SKILL_BODY); + _setStore({ + isInitialized: () => { + throw new Error('cannot tell'); + }, + getObject: () => null, + allObjects: () => ({}), + }); + + const report = await writeSkills('*', root); + + expect(report.ok).toBe(false); + expect(await exists(existing)).toBe(true); + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + expect(warnSpy.mock.calls.map((c) => String(c[0])).join('\n')).toMatch(/isInitialized/); + }); +}); + +// ─── The "*" form's own hole: a withholding with no key to protect anything ── + +describe('writeSkills unattributed withholding', () => { + it('an unattributable failure never prunes (§3.22: the run is reported incomplete)', async () => { + // Every per-object failure the wildcard resolve can attribute to a key + // keeps that key in the requested set, which is what holds prune off the + // copy on disk. When neither the object's own `key` nor the store's map key + // is usable the failure is run-level, so there is no key to hold the + // on-disk copy with — reporting the run incomplete is the only thing left + // that stops prune reading an unreadable object as a revocation. + const seed = new InMemorySkillStore(); + seed.put(rawSkill('pdf-extraction')); + seed.put(rawSkill('other')); + _setStore(seed); + const first = await writeSkills('*', root); + expect(first.ok).toBe(true); + const existing = path.join(root, 'pdf-extraction', SKILL_MD); + + // The shipped store keys `allObjects` as `:`, which is never a + // valid skill key, so a mangled `key` field leaves no fallback. The good + // skill is still served alongside it. + const mangled: RawSkillObject = { + key: 'PDF Extraction!', + version: 1, + content: SKILL_BODY, + contentHash: hash(SKILL_BODY), + }; + _setStore({ + getObject: () => null, + allObjects: () => ({ 'other:1': rawSkill('other'), 'pdf-extraction:1': mangled }), + }); + const unlinks = interceptUnlink(); + + const report = await writeSkills('*', root); + + expect(report.ok).toBe(false); + expect(report.errors.some((a) => a.key === '' && /invalid key/.test(a.error ?? ''))).toBe(true); + expect(report.actions.filter((a) => a.action === 'removed')).toEqual([]); + expect(unlinks).toEqual([]); + expect(await readFile(existing, 'utf-8')).toBe(SKILL_BODY); + expect((await readManifest(root)).entries).toHaveProperty(`pdf-extraction/${SKILL_MD}`); + // The good skill still reconciled — the run is incomplete, not abandoned. + expect(actionsByKey(report).other.action).toBe('skipped_current'); + }); +}); + // ─── Cleanup guard ─────────────────────────────────────────────────────────── describe('test hygiene', () => { diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 44c154c2..35d45b74 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -332,6 +332,18 @@ describe('package exports', () => { expect(Object.keys(exhaustive).sort()).toEqual(['keep', 'raise']); }); + it('exports the delivery transport, the watcher, and their defaults from the package root', () => { + // §3.25 / §3.26: the transport and the watcher are root exports in both + // languages; the base-URI and debounce defaults are TypeScript-only root + // exports (A.12). + expect(typeof packageIndex.FDv2SkillStore).toBe('function'); + expect(typeof packageIndex.watchSkills).toBe('function'); + expect(typeof packageIndex.SkillWatcher).toBe('function'); + expect(packageIndex.DEFAULT_BASE_URI).toBe('https://sdk.launchdarkly.com'); + expect(packageIndex.DEFAULT_STREAM_URI).toBe('https://stream.launchdarkly.com'); + expect(packageIndex.DEFAULT_DEBOUNCE_MS).toBe(500); + }); + it('exports getSkillResult and the outcome factory from the package root', () => { expect(typeof packageIndex.getSkillResult).toBe('function'); expect(typeof packageIndex.createSkillOutcome).toBe('function'); @@ -491,6 +503,10 @@ describe('InMemorySkillStore', () => { expect(new InMemorySkillStore().getObject('skill', 'nope')).toBeNull(); }); + it('implements no isInitialized probe — a hand-populated store is never waiting (§3.21)', () => { + expect('isInitialized' in new InMemorySkillStore()).toBe(false); + }); + it('put then get', () => { const store = new InMemorySkillStore(); const raw = rawSkill({ key: 'a' }); @@ -1167,6 +1183,26 @@ describe('integrity verification', () => { expect(emitter.signals(INTEGRITY_SIGNAL)).toHaveLength(1); }); + it('reports over_size_cap, not not_utf8, for over-cap content that also carries a lone surrogate (§3.21)', async () => { + // The checks run in a fixed order — shape, size, encoding, hash — and size + // precedes encoding deliberately: running an encoding pass over a 10 MiB + // body before rejecting it for being 10 MiB is a DoS foothold. This is the + // one boundary where the order is observable. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + errorSpy.mockClear(); + const content = `\ud800${OVERSIZE}`; + _setStore(new DictStore({ a: rawSkill({ key: 'a', content, contentHash: hash(Buffer.from(content, 'utf-8')) }) })); + + expect(await getSkill('a')).toBeNull(); + + const logged = errorSpy.mock.calls.map((c) => String(c[0])).join('\n'); + expect(logged).toContain('"reason_code":"over_size_cap"'); + expect(logged).not.toContain('not_utf8'); + expect(emitter.signals(INTEGRITY_SIGNAL)).toHaveLength(1); + }); + it('accepts content at exactly the size cap', async () => { const atCap = 'x'.repeat(MAX_SKILL_CONTENT_BYTES); const store = new InMemorySkillStore(); @@ -1235,9 +1271,33 @@ describe('integrity verification', () => { expect(await getSkill('a')).toBeNull(); }); - it('rejects a non-object raw entry', async () => { + it('rejects a non-object raw entry — absent when pinned, not_an_object when listed (§3.21)', async () => { + // The asymmetry is the contract. A pinned lookup that comes back as a + // non-object is a broken store adapter answering "nothing", reported as + // `absent` with no signal and no log record — firing a tampering signal on + // every broken adapter would drown the real ones. The listing path hands + // every raw value straight to verification, which is where `not_an_object` + // fires. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + // `spyOn` on an already-spied method hands back the existing spy with its + // history, so clear it: only what *this* test logs may count. + const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + errorSpy.mockClear(); _setStore(new DictStore({ a: 'not an object' as unknown as RawSkillObject })); + expect(await getSkill('a')).toBeNull(); + const outcome = await getSkillResult('a'); + expect(outcome.reason).toBe('absent'); + expect(emitter.records).toEqual([]); + expect(errorSpy).not.toHaveBeenCalled(); + + expect(await allSkills()).toEqual([]); + const signals = emitter.signals(INTEGRITY_SIGNAL); + expect(signals).toHaveLength(1); + expect(signals[0].skill_key).toBe(''); + const logged = errorSpy.mock.calls.map((c) => String(c[0])).join('\n'); + expect(logged).toContain('"reason_code":"not_an_object"'); }); it('rejects an uppercase hash — hashes are lowercase hex', async () => { @@ -2140,6 +2200,9 @@ describe('getSkillResult', () => { expect(detail).toContain('pdf-extraction'); expect(detail).toContain('version 2'); expect(detail).toContain('version 3'); + // Safe to log: no path separator and no root path of any kind. + expect(detail).not.toMatch(/[\\/]/); + expect(detail).not.toContain(process.cwd()); }); }); diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 52ab62bd..2473deb4 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -179,6 +179,38 @@ export function requireStore(): SkillStore { return store; } +/** + * Whether `store` has received its initial data. + * + * `true` for a store that does not implement the optional `isInitialized`, + * since a hand-populated store is never waiting for anything. Probed rather + * than required on the seam for the reason `SkillStore` gives: a required + * member would reject every store without it. + * + * This is what keeps `writeSkills('*')` from reading a store that has not yet + * received a payload as an environment whose every skill was revoked. Retrieval + * through such a store is reported unavailable, which suppresses pruning — the + * same treatment a throwing store gets, and for the same reason: deleting a + * customer's files because content could not be retrieved would turn a slow + * boot into data loss. + * + * A probe that throws counts as not initialized. A store that cannot answer + * whether it is ready is not one to authorize deletions on. + */ +export function storeIsInitialized(store: SkillStore): boolean { + const probe = (store as { isInitialized?: unknown }).isInitialized; + if (typeof probe !== 'function') return true; + try { + return Boolean(probe.call(store)); + } catch (error) { + // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; a failing store must be visible + console.warn( + `[LaunchDarkly] The skill store's isInitialized() threw; treating the store as not yet initialized: ${storeThrew(error)}`, + ); + return false; + } +} + /** * Records one signal. Never throws into the calling operation — a broken emitter * must not be able to fail a retrieval or a reconcile. diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index ebad0f26..cb07e262 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1655,6 +1655,24 @@ export class FDv2SkillStore implements SkillStore { this.releaseWaiters(); } + /** + * Whether a payload has arrived, so reads reflect delivery rather than an + * empty store still waiting for its first one. + * + * The optional half of the `SkillStore` seam, and the same fact + * `waitForSkills` resolves to — without the wait. `writeSkills('*')` consults + * it so a reconcile that runs before delivery reports the retrieval + * unavailable rather than pruning every managed skill as though the + * environment had revoked it. + * + * Stays `true` once a payload has arrived, including after `close`: a closed + * store still answers from what it received, and a later reconcile against + * that content is a reconcile against real delivery. + */ + isInitialized(): boolean { + return this.firstPayload; + } + /** Why delivery stopped for good, or `null` while it is running. */ get failed(): string | null { return this.failedReason; diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index 8948cb96..e5c9fb1e 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -45,6 +45,7 @@ import { recordRevoked, referenceTarget, resolveFromStore, + storeIsInitialized, verifiedBytes, verifyRawSkill, } from './skills-core.js'; @@ -56,6 +57,7 @@ import type { ReconcileReport, Skill, SkillReference, + SkillStore, } from './types.js'; import { createReconcileAction, @@ -479,9 +481,9 @@ function isSkill(item: Skill | SkillReference | string): item is Skill { * Turns the caller's input into one request per skill. * * Also reports whether any retrieval was left incomplete — an absent store, a - * throwing store, or an exhausted timeout. That flag suppresses pruning: deleting - * managed files because retrieval failed would turn a transport outage into data - * loss. + * store still waiting for its initial data, a throwing store, or an exhausted + * timeout. That flag suppresses pruning: deleting managed files because + * retrieval failed would turn a transport outage into data loss. */ async function resolveRequests( skills: ReadonlyArray | '*', @@ -515,29 +517,58 @@ async function resolveRequests( return { requests, incomplete }; } +/** Retrieval could not happen at all; `error` is already worded as unavailable. */ +type RetrievalBlocked = { readonly blocked: string }; + +function isBlocked(result: SkillStore | RetrievalBlocked): result is RetrievalBlocked { + return 'blocked' in result; +} + /** - * Resolves one reference for the materialization path. + * The single gate that decides whether retrieval is available — for one + * reference and for the `'*'` form alike, so the two cannot disagree about + * what counts as "the store could not answer". + * + * Three conditions block it, and every one of them is `store_unavailable` + * rather than the store answering "no": an exhausted deadline, an absent store, + * and a store that has not received its initial data. Reporting any of them as + * an absence is what would let a prune delete working files over a non-answer. + * + * The readiness check is the one that is easy to leave out. A store still + * waiting for its first delivery answers every read with "nothing", which is + * indistinguishable through the seam from an environment that holds no skills — + * and the `'*'` form reads that as every skill having been revoked. Blocking + * here reports the run incomplete, which is what suppresses the prune. * - * Same core as the accessors, plus the two conditions only this path treats as - * data rather than as an exception: an exhausted deadline and an absent store. + * `subject` is what the message says could not be retrieved. */ -function resolveReference(key: string, wantedVersion: number | null, deadline: number): Resolution { - // Both of this function's own outcomes are `store_unavailable`: neither is the - // store answering "no". An exhausted deadline and an absent store are reasons - // the retrieval could not happen, and reporting either as an absence is what - // would let a prune delete working files over a non-answer. +function availableStore(deadline: number, subject: string): SkillStore | RetrievalBlocked { if (performance.now() >= deadline) { + return { blocked: unavailable(`the timeout was exhausted before ${subject} could be retrieved`) }; + } + const store = getStore(); + if (store === null) return { blocked: unavailable(NO_STORE_MESSAGE) }; + if (!storeIsInitialized(store)) { return { - error: unavailable(`the timeout was exhausted before '${key}' could be retrieved`), - reason: 'store_unavailable', - unavailable: true, + blocked: unavailable( + `the skill store has not received its initial data, so ${subject} could not be retrieved and nothing on ` + + 'disk was changed. Wait for delivery before reconciling: FDv2SkillStore.waitForSkills(timeoutMs) ' + + 'resolves true once the first payload has arrived.', + ), }; } + return store; +} - const store = getStore(); - if (store === null) { - return { error: unavailable(NO_STORE_MESSAGE), reason: 'store_unavailable', unavailable: true }; - } +/** + * Resolves one reference for the materialization path. + * + * Same core as the accessors, plus the conditions only this path treats as data + * rather than as an exception — see {@link availableStore}. + */ +function resolveReference(key: string, wantedVersion: number | null, deadline: number): Resolution { + const store = availableStore(deadline, `'${key}'`); + if (isBlocked(store)) return { error: store.blocked, reason: 'store_unavailable', unavailable: true }; const resolved = resolveFromStore(store, key, wantedVersion); if (resolved.unavailable && resolved.error) { @@ -581,6 +612,10 @@ function pendingForRaw(objectKey: string, raw: unknown): PendingWrite { const candidate = typeof raw === 'object' && raw !== null ? (raw as RawSkillObject).key : undefined; const key = isValidSkillKey(candidate) ? candidate : objectKey; if (!isValidSkillKey(key)) { + // Neither key is usable, so this failure cannot be attributed to a skill — + // the run-level sentinel is the honest report. `resolveAll` reads that + // sentinel back as an incomplete run, because a failure with no key cannot + // protect its copy on disk the way the branch below does. return { key: '', error: 'the skill store served an object under an invalid key; it was withheld' }; } return { @@ -591,15 +626,8 @@ function pendingForRaw(objectKey: string, raw: unknown): PendingWrite { /** Resolves the `'*'` form — everything the store currently holds. */ function resolveAll(deadline: number, onUnavailable: OnUnavailable): { requests: PendingWrite[]; incomplete: boolean } { - if (performance.now() >= deadline) { - return unavailableRun( - unavailable('the timeout was exhausted before the skill set could be retrieved'), - onUnavailable, - ); - } - - const store = getStore(); - if (store === null) return unavailableRun(unavailable(NO_STORE_MESSAGE), onUnavailable); + const store = availableStore(deadline, 'the skill set'); + if (isBlocked(store)) return unavailableRun(store.blocked, onUnavailable); // Deliberately not via allSkills(), which reports a throwing store as an empty // result — that would look like "every skill was revoked" and let prune delete @@ -611,10 +639,14 @@ function resolveAll(deadline: number, onUnavailable: OnUnavailable): { requests: // versions of one key, and //SKILL.md is a single path — writing it // twice in one run is not a duplicate report but a write race against itself, // resolved by whichever version iteration happened to reach last. - return { - requests: newestByKey(objects).map(({ objectKey, raw }) => pendingForRaw(objectKey, raw)), - incomplete: false, - }; + const requests = newestByKey(objects).map(({ objectKey, raw }) => pendingForRaw(objectKey, raw)); + // A withholding that could not be attributed to a key leaves the run + // incomplete. Every other failure keeps its key in the requested set, which is + // what holds prune off the copy on disk; a run-level failure has no key to do + // that with, so suppressing prune wholesale is the only thing left that stops + // an unreadable object reading as a revocation. + const unattributed = requests.some((request) => !request.skill && request.key === ''); + return { requests, incomplete: unattributed }; } // ------------------------------------------------------------------------- @@ -683,7 +715,7 @@ async function resolveRoot(root: string): Promise { * Loads the manifest. * * A manifest that cannot be read, cannot be parsed, is not an object, carries a - * `manifestVersion` this release does not understand, or has a malformed `entries` + * `manifestVersion` outside `[1, MANIFEST_VERSION]`, or has a malformed `entries` * map is **corrupt**. The caller then performs no destructive action and leaves the * file itself alone: rewriting it would destroy the only record of what the SDK * owns, and acting on a manifest we cannot read would mean guessing at which of @@ -736,7 +768,10 @@ async function loadManifest( const manifest = data as Record; const version = manifest.manifestVersion; - if (typeof version !== 'number' || !Number.isInteger(version) || version > MANIFEST_VERSION) { + // Bounded at both ends. No release ever wrote a version below 1, so 0 or a + // negative is not an older schema this release could still read — it is a + // schema that never existed, and acting on its entries would be a guess. + if (typeof version !== 'number' || !Number.isInteger(version) || version < 1 || version > MANIFEST_VERSION) { return { manifest: {}, error: `the skills manifest ${MANIFEST_FILENAME} declares manifestVersion ${JSON.stringify(version)}, which this SDK cannot read; refusing every destructive action`, diff --git a/packages/client/src/types.ts b/packages/client/src/types.ts index 76055ffd..72f3893c 100644 --- a/packages/client/src/types.ts +++ b/packages/client/src/types.ts @@ -275,6 +275,17 @@ export type RawSkillObject = { * only, and do throw. A watcher that silently never fires is indistinguishable * from one whose objects never changed. `removeListener` has no such constraint: * a kind holding no listeners is simply nothing to remove. + * * `isInitialized` is the optional readiness half of the seam, probed the same + * way `addListener` is. It reports whether the store has received its initial + * data — for a delivery transport, whether a payload has arrived yet. A store + * that does not implement it is treated as initialized, which is correct for + * one populated by hand (`InMemorySkillStore` implements nothing here), and a + * probe that throws counts as *not* initialized. It matters because "the store + * holds nothing" and "the store has not heard yet" are the same answer through + * `allObjects`, and `writeSkills('*')` reads the first as "every skill was + * revoked" — so a reconcile against an uninitialized store reports retrieval + * unavailable and prunes nothing. `FDv2SkillStore` implements it as "a payload + * has committed", the same fact `waitForSkills` resolves to without the wait. * * Everything a store serves is untrusted input. The transport is not part of the * trust boundary — key, version, size, and content hash are revalidated at the @@ -296,6 +307,12 @@ export type SkillStore = { allObjects(kind: string): Record; addListener?(kind: string, fn: (raw: RawSkillObject) => unknown): void; removeListener?(kind: string, fn: (raw: RawSkillObject) => unknown): void; + /** + * Whether the store has received its initial data. Optional; absent means + * initialized. See the type's docblock for why the materialization path + * consults it before it may prune. + */ + isInitialized?(): boolean; }; /** Builds a frozen {@link SkillReference}. */ From 43ed09356449234e7bd35202ee67b3c1c057e97c Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Thu, 17 Sep 2026 15:58:19 -0400 Subject: [PATCH 34/55] test(client): pin the key-mismatch detection surface (failing) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tests for TESTING.md §3.21/§3.24 as just reconciled: a store answering under a different key records the `ld.skills.integrity_failure` log record with `reason_code: key_mismatch` and records no product signal. Splits the old test in two. The outcome half is unchanged and still passes; the detection half asserted the *old* contract — that neither surface fires — and is replaced by the new one. `key_mismatch` joins the `cases` table as the ninth token, which pulls it into the existing sweeps for free: `attaches the record on every reason_code` and `never emits a null or an empty value in any record` both now exercise it. Adds key-order coverage, since `served_key` has to land in its alphabetical place for the line to stay byte-comparable across SDKs, and a redaction guard called directly — unreachable today, because verification accepts the served key before this path runs, but that is a property of the call order rather than of the recorder. Also pins the listing path, where there is deliberately no mismatch to report: an object listed under a disagreeing map key is filed under its own key with neither surface firing. That one passes already; it is a regression guard against a "fix" that would break every multi-version store. 7 failing: no record is emitted yet and `recordKeyMismatch` does not exist. The no-signal half already passes — the implementation is additive. Co-Authored-By: Claude Opus 5 --- packages/client/src/__tests__/skills.test.ts | 206 +++++++++++++++++-- 1 file changed, 186 insertions(+), 20 deletions(-) diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 35d45b74..8994ab1a 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -908,7 +908,7 @@ describe('getSkill', () => { expect(await getSkill('nope')).toBeNull(); }); - it('withholds a store answering under a different key as integrity_failure, silently', async () => { + it('withholds a store answering under a different key as integrity_failure', async () => { // The key needs the same post-fetch defense the version already has. // Identity is read off the object itself, and the store is untrusted. An // answer served under a different key would otherwise be handed back under @@ -921,12 +921,6 @@ describe('getSkill', () => { // substituting store filed under it would be invisible to a caller // branching on the token. Pinned here because the choice is not recoverable // from the message. - // - // The silence is the other half, and it is asserted in both directions: the - // check runs *after* verifyRawSkill has already passed, so neither §3.24 - // detection surface fires. A recorded signal or a logged record here would - // mean the check had migrated into verification — a real change, not a - // cosmetic one, since it would need a ninth reason_code to go with it. const aliasing: SkillStore = { getObject() { return rawSkill({ key: 'other-key' }); @@ -937,20 +931,13 @@ describe('getSkill', () => { }; _setStore(aliasing); - const emitter = new RecordingEmitter(); - _setEmitterForTesting(emitter); - const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); - let errorCalls: unknown[][] = []; let collapsed: Skill | null; let outcome: Awaited>; try { collapsed = await getSkill('asked-for'); outcome = await getSkillResult('asked-for'); } finally { - // Read the calls out before restoring: `mockRestore` also resets the - // recorded history, so a read afterwards sees nothing. - errorCalls = [...spy.mock.calls]; spy.mockRestore(); } @@ -961,10 +948,79 @@ describe('getSkill', () => { // only that it is present and carries no skill body. expect(outcome.detail).toBeTruthy(); expect(outcome.detail).not.toContain('Do the thing.'); + }); + + it('records the key mismatch on the log surface but not the product signal', async () => { + // The asymmetry is the contract, and it is one-directional, so it is + // asserted in both directions here. + // + // The log record fires because a substituting store is a genuine tampering + // indicator and the record is the customer-owned detection path — the only + // one that works with telemetry off. Reusing the `ld.skills.integrity_failure` + // event identity is deliberate: a customer's existing SIEM rule catches this + // case without being rewritten, and `reason_code` is what distinguishes it. + // + // The product signal stays out of it because the overwhelmingly common cause + // of a key mismatch is not an attacker but a broken store adapter — a stale + // cache entry, a colliding key, a wrong index lookup — and LaunchDarkly's + // own counter must not fill up with customers' adapter bugs. That is the + // same false positive the pinned-non-object path refuses for the same reason. + const aliasing: SkillStore = { + getObject() { + return rawSkill({ key: 'served-key' }); + }, + allObjects() { + return {}; + }, + }; + _setStore(aliasing); + + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let errorCalls: unknown[][] = []; + try { + await getSkill('asked-for'); + } finally { + // Read the calls out before restoring: `mockRestore` also resets the + // recorded history, so a read afterwards sees nothing. + errorCalls = [...spy.mock.calls]; + spy.mockRestore(); + } + // The signal surface saw nothing at all — not merely no integrity signal. expect(emitter.records).toEqual([]); - const lines = errorCalls.map(([first]) => String(first)); - expect(lines.filter((line) => line.includes('ld.skills.integrity_failure'))).toEqual([]); + + const lines = errorCalls.filter(([first]) => String(first).includes('ld.skills.integrity_failure')); + expect(lines).toHaveLength(1); + const [first, ...rest] = lines[0]; + const line = String(first); + const record = JSON.parse(line.slice(line.indexOf('{'))) as Record; + + expect(record.reason_code).toBe('key_mismatch'); + expect(record.event).toBe('ld.skills.integrity_failure'); + expect(record.action).toBe('withheld'); + expect(record.language).toBe('typescript'); + expect(typeof record.reason).toBe('string'); + + // Both keys are named, and `skill_key` keeps the meaning it has on every + // other record — the key the *caller asked for* — so a rule grouping by it + // still works. The key the store actually answered under is the datum that + // makes a broken adapter diagnosable, so it is a parseable field rather + // than prose buried in `reason`. + expect(record.skill_key).toBe('asked-for'); + expect(record.served_key).toBe('served-key'); + + // Verification passed, so there is no hash disagreement to report and the + // two hash fields stay absent rather than being emitted as null. + expect('expected_hash' in record).toBe(false); + expect('observed_hash' in record).toBe(false); + + // The structured attachment is required alongside the text, same as every + // other record. + expect(rest).toHaveLength(1); + expect(rest[0]).toEqual(record); }); }); @@ -1057,6 +1113,38 @@ describe('allSkills', () => { expect(found.map((s) => s.key).sort()).toEqual(['a', 'b']); }); + it('files a listed object under its own key, with no key_mismatch for a disagreeing map key', async () => { + // The counterpart to the pinned path's `key_mismatch`, and the asymmetry is + // deliberate rather than a gap. A listing carries no requested key, so there + // is nothing for the object's key to disagree *with*: identity comes off the + // object, and the store's map key is used for one thing only — attributing a + // failure when the object's own key is unusable. + // + // The seam never promised a map key spells a skill key, either: a store + // holding several versions of one key has reason to spell it `key:version`, + // which is exactly what FDv2SkillStore does. Pinned because the asymmetry + // with the pinned path is surprising enough to invite a "fix" that would + // break every multi-version store. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + _setStore(new DictStore({ 'filed-under-this': rawSkill({ key: 'its-own-key' }) })); + + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let errorCalls: unknown[][] = []; + let found: Skill[]; + try { + found = await allSkills(); + } finally { + errorCalls = [...spy.mock.calls]; + spy.mockRestore(); + } + + expect(found.map((s) => s.key)).toEqual(['its-own-key']); + // Neither surface fires: nothing failed. + expect(emitter.records).toEqual([]); + expect(errorCalls.filter(([first]) => String(first).includes('ld.skills.integrity_failure'))).toEqual([]); + }); + it('keeps an unusable object in the set so verification withholds it with a signal', async () => { // Filtering it here instead would silently shrink the resolved set, which on // the materialization path is indistinguishable from a revocation and would @@ -1626,6 +1714,13 @@ describe('integrity-failure log record', () => { ['not_utf8', spoiled({ content: surrogate, contentHash: hash(Buffer.from(surrogate, 'utf-8')) }), GET], ['over_size_cap', spoiled({ content: OVERSIZE }), GET], ['hash_mismatch', spoiled({ contentHash: 'd'.repeat(64) }), GET], + // The ninth is the odd one out: it is not a verification failure, so it + // comes from `recordKeyMismatch` rather than from a call site inside + // `verifyRawSkill`. It is in the same vocabulary anyway, because a + // customer's detection rule cares that integrity failed and not about which + // layer noticed. A store that serves a perfectly valid object under the + // wrong key reaches it: everything `verifyRawSkill` checks passes. + ['key_mismatch', spoiled({ key: 'served-under-this' }), GET], ]; it.each(cases)('logs one record carrying reason_code %s', async (code, makeStore, run) => { @@ -1643,14 +1738,15 @@ describe('integrity-failure log record', () => { }); it('covers the whole reason_code vocabulary and nothing else', () => { - // The eight tokens are one per call site of `recordIntegrityFailure`, and - // every language implementation emits the same eight. A ninth in one SDK - // only is the - // regression this test exists to catch. + // Eight of the nine are one per call site of `recordIntegrityFailure`; the + // ninth, `key_mismatch`, comes from `recordKeyMismatch` at the retrieval + // boundary. Every language implementation emits the same nine, so a tenth + // in one SDK only is the regression this test exists to catch. expect(cases.map(([code]) => code).sort()).toEqual([ 'hash_mismatch', 'invalid_key', 'invalid_version', + 'key_mismatch', 'missing_content', 'missing_content_hash', 'not_an_object', @@ -1659,6 +1755,30 @@ describe('integrity-failure log record', () => { ]); }); + it('fires the record without the signal for key_mismatch, and never the reverse', async () => { + // The one-directional exception to "the two surfaces agree". Swept across + // the whole vocabulary rather than asserted on the one case, because what + // makes it safe is that *only* this code is signal-free: an implementation + // that dropped the signal for some other code would satisfy a single-case + // assertion. + for (const [code, makeStore, run] of cases) { + _clearState(); + _setStore(makeStore()); + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + + const records = await logged(run); + expect(records, code).toHaveLength(1); + + const signals = emitter.signals(INTEGRITY_SIGNAL); + if (code === 'key_mismatch') { + expect(signals, code).toHaveLength(0); + } else { + expect(signals, code).toHaveLength(1); + } + } + }); + it('logs the event name and nothing but the record, so a grep finds it', async () => { _setStore(new DictStore({ a: rawSkill({ key: 'a', contentHash: 'd'.repeat(64) }) })); @@ -1729,6 +1849,52 @@ describe('integrity-failure log record', () => { ]); }); + it('sorts the key_mismatch record too, with served_key in its alphabetical place', async () => { + // `served_key` is the one record-only field beyond the four, and it is built + // on a separate path — so the sort that makes every other record + // byte-comparable across SDKs has to be asserted here independently rather + // than assumed from the case above. + _setStore(new DictStore({ a: rawSkill({ key: 'served-under-this' }) })); + + const [{ record }] = await logged(() => getSkill('a')); + + const keys = Object.keys(record); + expect(keys).toEqual([...keys].sort()); + expect(keys).toEqual([ + 'action', + 'event', + 'language', + 'reason', + 'reason_code', + 'served_key', + 'skill_key', + 'version', + ]); + }); + + it('redacts a hostile served_key', async () => { + // Unreachable today — verification accepts the served key before this path + // runs, so it is well-formed by construction. Asserted anyway, because that + // is a property of the current call order rather than of the recorder, and + // the guard is what keeps a future reordering from publishing a body here. + const body = 'UNIQUE-SERVED-BODY/../x'; + const { recordKeyMismatch } = await import('../skills-core.js'); + + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let calls: unknown[][] = []; + try { + recordKeyMismatch('asked-for', body); + } finally { + calls = [...spy.mock.calls]; + spy.mockRestore(); + } + + const line = String(calls[0][0]); + const record = JSON.parse(line.slice(line.indexOf('{'))) as Record; + expect(record.served_key).toBe(''); + expect(line).not.toContain(body); + }); + it('redacts a hostile key, body and all', async () => { // The key comes off the wire, so it gets the same shape-check-then-redact // treatment as the signal — the body must not reach the log line either. From 75df51f3c2a85c94dfcee6ede8906db97b138c71 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Thu, 17 Sep 2026 16:05:36 -0400 Subject: [PATCH 35/55] fix(client): record the key mismatch on the log surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the §3.21/§3.24 decision the previous commit's tests pin: a store answering under a different key now writes the `ld.skills.integrity_failure` log record with `reason_code: key_mismatch`, and still records no product signal. Adds `recordKeyMismatch` beside `recordIntegrityFailure`, so the single-emission-site rule still holds by reading one module, and widens `IntegrityReasonCode` to nine tokens. The check stays where it was, at the retrieval boundary: `verifyRawSkill` is unary and this comparison is relational, so moving it inside would mean an optional expected-key parameter that silently disables the check when a caller omits it. The record carries both keys. `skill_key` keeps the meaning it has everywhere else — the key requested — and the key the store answered under goes in `served_key`, the one record-only field beyond the four, since it is what makes a broken adapter diagnosable and prose in `reason` is not parseable. No hash fields and no `version`: verification passed, so neither the hashes nor the version is what disqualified the answer, and reporting a served version beside a requested key would mix two frames in one record. Docs corrected in the same change, and one of them was actively wrong rather than merely stale: `AGENTS.md`'s outcome table told contributors not to "fix" the silence by adding an emission at the key check. Both the README `reason_code` table and the record field table gain rows, the outcome table's `integrity_failure` row no longer claims no record is written, and the alerting guidance says to suspect a custom store adapter before tampering. Verified the emitted JSON is byte-identical to the Python SDK's for the same input, modulo `language`. 979 tests passing, tsc and biome clean. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 14 ++-- packages/client/agents.md | 10 ++- packages/client/src/__tests__/skills.test.ts | 16 ++-- packages/client/src/skills-core.ts | 88 ++++++++++++++++++-- 4 files changed, 103 insertions(+), 25 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 50df9a96..9f609eb1 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -424,11 +424,12 @@ The line is a `[LaunchDarkly] ` prefix, the event name, a space, and a single JS |---|---|---| | `event` | yes | `ld.skills.integrity_failure`. | | `action` | yes | `withheld` — what the SDK did about it. Content that fails verification is never returned and never written to disk. | -| `skill_key` | yes | The skill key, or `` when the key itself failed validation. Keys arrive from the wire, so one is never echoed verbatim. | +| `skill_key` | yes | The skill key **requested**, or `` when the key itself failed validation. Keys arrive from the wire, so one is never echoed verbatim. | | `reason_code` | yes | A stable token from the closed vocabulary below. Alert on this field, not on `reason`. | | `reason` | yes | Human-readable detail, including byte counts. Wording may change between releases. | | `language` | yes | `typescript`. Distinguishes SDKs in a polyglot fleet; the Python SDK emits the same record with `python`. | -| `version` | no | The skill version. Omitted when the delivered version was not an integer >= 1. | +| `served_key` | no | Only on `key_mismatch`: the key the store actually answered under, with the same redaction as `skill_key`. Omitted on every other failure mode. | +| `version` | no | The skill version. Omitted when the delivered version was not an integer >= 1, and on `key_mismatch`. | | `expected_hash` | no | The `contentHash` delivered with the content, or `` when it was not 64 lowercase hex characters. Omitted when the failure happened before any hash was read. | | `observed_hash` | no | The sha256 this SDK computed locally. Omitted when the failure happened before hashing. | @@ -444,8 +445,11 @@ Optional fields are **omitted, never null** — the absence of `observed_hash` m | `not_utf8` | The content has no UTF-8 encoding (a lone surrogate), so there are no bytes LaunchDarkly could have hashed. | | `over_size_cap` | The content exceeds the internal `MAX_SKILL_CONTENT_BYTES` cap, so it is inauthentic whatever it hashes to. The reason string names the bound. | | `hash_mismatch` | The content does not hash to the `contentHash` delivered alongside it. | +| `key_mismatch` | The store answered under a different key than the one requested. Carries an extra `served_key` field naming the key it answered under, and — uniquely — records **no** `AgentControl Skill Integrity Failure` signal. | -These eight tokens are the whole vocabulary, and the Python SDK emits the same eight for the same conditions — including identical JSON key order — so one parser and one alert rule cover a polyglot fleet. +These nine tokens are the whole vocabulary, and the Python SDK emits the same nine for the same conditions — including identical JSON key order — so one parser and one alert rule cover a polyglot fleet. + +`key_mismatch` is the only code that reaches this log record without also reaching the product signal. It is decided after verification has passed, and its usual cause is a bug in a custom `SkillStore` adapter — a stale cache entry, a colliding key, a wrong index lookup — rather than tampering, so it does not inflate LaunchDarkly's own integrity counter. It still reaches this record, because a store substituting one skill for another is worth seeing, and your rule on `ld.skills.integrity_failure` catches it without modification. #### Fail closed on tampering: `getSkillResult` @@ -482,7 +486,7 @@ switch (outcome.reason) { |---|---|---| | `ok` | the skill | Retrieved and verified. | | `absent` | `null` | The store holds nothing under that key. Not configured, not yet delivered, or revoked. | -| `integrity_failure` | `null` | Content was delivered and its identity did not verify, so it was withheld. Two cases reach this token: content that failed hash/size/shape verification, for which the `ld.skills.integrity_failure` record above was also written; and a store that answered under a **different key** than the one requested, which is reported here but records no signal and writes no log record (the check runs after verification has already passed). This is the outcome to fail closed on. | +| `integrity_failure` | `null` | Content was delivered and its identity did not verify, so it was withheld. Two cases reach this token, and both write the `ld.skills.integrity_failure` record above: content that failed hash/size/shape verification, which also records the product signal; and a store that answered under a **different key** than the one requested, which writes the record with `reason_code: key_mismatch` and a `served_key` field but records **no** signal (the check runs after verification has already passed, and a mismatch is usually a store-adapter bug rather than tampering). This is the outcome to fail closed on. | | `wrong_version` | `null` | A version was pinned and the store answered with a different one. Only a version mismatch — a key mismatch is `integrity_failure`, and there is deliberately no `wrong_key`. | | `store_unavailable` | `null` | The store threw. Nothing was retrieved, so nothing is known either way. | @@ -492,7 +496,7 @@ switch (outcome.reason) { These five tokens are the whole vocabulary, and the Python SDK publishes the same five for the same conditions. There is deliberately no batch equivalent: `getSkills` and `allSkills` still omit entries they could not return, so call `getSkillResult` per key where the outcome matters. -**`hash_mismatch` deserves a page, not a dashboard.** The other codes are consistent with a malformed store, a bad deployment, or a truncated response. `hash_mismatch` means content and its declared digest disagree, which is the shape of active tampering with skill delivery — in transit, in a cache, or in whatever backs your `SkillStore`. If you serve skill content only from LaunchDarkly, `over_size_cap` and `not_utf8` warrant alerts on the same reasoning: neither should ever occur. +**`hash_mismatch` deserves a page, not a dashboard.** The other codes are consistent with a malformed store, a bad deployment, or a truncated response. `hash_mismatch` means content and its declared digest disagree, which is the shape of active tampering with skill delivery — in transit, in a cache, or in whatever backs your `SkillStore`. If you serve skill content only from LaunchDarkly, `over_size_cap` and `not_utf8` warrant alerts on the same reasoning: neither should ever occur. `key_mismatch` belongs in the same tier as `hash_mismatch` **if** you use the SDK's own `FDv2SkillStore` and nothing else, since it then also means delivery served the wrong skill; behind a custom store adapter, suspect the adapter first. #### Privilege separation: the agent must not be able to rewrite its own skills diff --git a/packages/client/agents.md b/packages/client/agents.md index de23258d..829cb4b2 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -416,8 +416,8 @@ Two decisions here look like unfinished work and are not. Neither should be reve - Skills telemetry goes through the private emitter seam in `skills-core.ts` and nowhere else. **Never** call `client.track()` for a skills operation: these are LaunchDarkly product-analytics signals, not customer analytics, and `track()` would require an LD context, spend the customer's event volume, and land in their data export. Exactly three signal names exist (`AgentControl Skill Integrity Failure`, `AgentControl Skill Materialized`, `AgentControl Skill Revoked Received`) and they are an allowlist, not a floor — in particular, `AgentControl Skill SDK Reference Returned` and `AgentControl Skill Content Retrieved` must **never** be emitted. Every signal is constructed by a `record*` function in `skills-core.ts`, so the allowlist is enforceable by reading one section of one file. - No filesystem paths and no skill content in telemetry — hashes and byte counts only. Paths belong in the `ReconcileReport`, which is user-facing API. - The `ld.skills.integrity_failure` log record is a **documented customer-facing contract**, not a debug line. It is constructed in exactly one place — `recordIntegrityFailure` — and written to `console.error` on every failure, unconditionally, because it is the only detection surface a customer with telemetry off has and the only one that can exist where no telemetry destination is reachable. Its keys are inserted in **alphabetical order** so `JSON.stringify` is byte-identical to the Python SDK's `json.dumps(record, sort_keys=True, separators=(",", ":"))` for the same failure, modulo `language`; do not reorder them. Absent optional fields are omitted, never emitted as `null`. Renaming the event, renaming or dropping a field, or emitting a null is a breaking change to a security control — the README documents this record for customers to alert on. -- `reason_code` is a **closed eight-token vocabulary**, one token per call site of `recordIntegrityFailure`: `hash_mismatch`, `invalid_key`, `invalid_version`, `missing_content`, `missing_content_hash`, `not_an_object`, `not_utf8`, `over_size_cap`. A ninth token means adding it in the Python SDK, in the README's `reason_code` table, and in the vocabulary test in `src/__tests__/skills.test.ts`, in the same change — a token that exists on one side only silently breaks a customer's cross-language alert rule. `reason_code` deliberately stays **out** of the telemetry properties: the three signals' property keys are the allowlist above, and the vocabulary belongs to the customer-owned detection path rather than to LaunchDarkly's counter. -- `SkillOutcomeReason` is a **closed five-token vocabulary** and the public half of what `resolveFromStore` decides: `absent`, `integrity_failure`, `ok`, `store_unavailable`, `wrong_version`. It is published by `getSkillResult` as API, customers branch on it, and the Python SDK publishes the same five for the same conditions — so a sixth token is a cross-language change, and it means adding it in the Python SDK, in the README's `reason` table, and in the union-exhaustiveness test in `src/__tests__/skills.test.ts`, in the same change. Note this is a **different and coarser** vocabulary from `IntegrityReasonCode`: the eight integrity tokens say *which check failed* and go to the operator through the log record; the five outcome tokens say *what the caller got* and are the actionable programmatic surface. `verifyRawSkill` returns `null` and does not surface which integrity token fired, deliberately — plumbing the finer token into `SkillOutcome` would change that function's return type for a detail the operator already has. +- `reason_code` is a **closed nine-token vocabulary**: `hash_mismatch`, `invalid_key`, `invalid_version`, `key_mismatch`, `missing_content`, `missing_content_hash`, `not_an_object`, `not_utf8`, `over_size_cap`. Eight are one token per call site of `recordIntegrityFailure`; `key_mismatch` is the exception, coming from `recordKeyMismatch` at the retrieval boundary and firing the log record without the signal (see the outcome table below). A tenth token means adding it in the Python SDK, in the README's `reason_code` table, and in the vocabulary test in `src/__tests__/skills.test.ts`, in the same change — a token that exists on one side only silently breaks a customer's cross-language alert rule. `reason_code` deliberately stays **out** of the telemetry properties: the three signals' property keys are the allowlist above, and the vocabulary belongs to the customer-owned detection path rather than to LaunchDarkly's counter. +- `SkillOutcomeReason` is a **closed five-token vocabulary** and the public half of what `resolveFromStore` decides: `absent`, `integrity_failure`, `ok`, `store_unavailable`, `wrong_version`. It is published by `getSkillResult` as API, customers branch on it, and the Python SDK publishes the same five for the same conditions — so a sixth token is a cross-language change, and it means adding it in the Python SDK, in the README's `reason` table, and in the union-exhaustiveness test in `src/__tests__/skills.test.ts`, in the same change. Note this is a **different and coarser** vocabulary from `IntegrityReasonCode`: the nine integrity tokens say *which check failed* and go to the operator through the log record; the five outcome tokens say *what the caller got* and are the actionable programmatic surface. `verifyRawSkill` returns `null` and does not surface which integrity token fired, deliberately — plumbing the finer token into `SkillOutcome` would change that function's return type for a detail the operator already has. - `Resolution.reason` is set **explicitly at every construction site**, never derived from `Resolution.error`. Pattern-matching a prose string to decide what a customer's fail-closed branch sees is the fragility the typed outcome exists to remove, and the field is required precisely so a new internal outcome has to pick a public token rather than inherit `'absent'` by omission. The mapping today: | Where the resolution is built | `reason` | @@ -430,7 +430,11 @@ Two decisions here look like unfinished work and are not. Neither should be reve | `resolveFromStore` — success | `ok` | | `skills-fs.ts` `resolveReference` — deadline exhausted, or no store configured (both set `unavailable`) | `store_unavailable` | - The two `integrity_failure` rows are **not** interchangeable, and the difference is the easy thing to get wrong. The `verifyRawSkill` row fires *inside* verification, so it records the `AgentControl Skill Integrity Failure` signal and writes the `ld.skills.integrity_failure` log record with one of the eight `IntegrityReasonCode` tokens. The key-mismatch row fires *after* verification has already passed, so it does neither — it is an outcome reason and nothing else, with no signal, no log record, and no `reason_code`. Do not "fix" that by adding an emission at the key check: the eight-token vocabulary does not cover it, so it would need a ninth token (`key_mismatch`) and a matching change in the Python SDK. A test asserts the silence in both directions. + The two `integrity_failure` rows are **not** interchangeable, and the difference is the easy thing to get wrong. The `verifyRawSkill` row fires *inside* verification, so it records the `AgentControl Skill Integrity Failure` signal **and** writes the `ld.skills.integrity_failure` log record. The key-mismatch row fires *after* verification has passed, so it goes through `recordKeyMismatch` instead and writes **the log record only** — `reason_code: key_mismatch`, plus a `served_key` field no other record carries, and no product signal. + + That asymmetry is the decision, and it is pinned by tests in both directions. The record fires because a substituting store is worth seeing and the record is the customer-owned detection path; reusing the event identity means an existing SIEM rule catches it with the code as the discriminator. The signal does not, because the usual cause is a broken custom store adapter rather than an attacker, and LaunchDarkly's own counter must not fill with customers' adapter bugs — the same false positive the `raw` is-not-an-object row above refuses by reading as `absent`. Do not "fix" either half: emitting the signal here, or dropping the record, each breaks a test that says why. + + Keep the check where it is, too. `verifyRawSkill` is unary and decides whether one object is internally self-consistent; a key mismatch is relational. Threading an expected key into it would make the parameter optional at every call site, and a check that silently vanishes when a caller forgets an argument is worse than one at the single site where both values are in hand. Adding an eighth row means answering "which of the five does a caller see?" before writing the code. `unavailable` stays a separate field rather than folding into `reason`: it is narrower, it is what suppresses pruning, and the two bottom rows above never reach an accessor at all. - `wantedVersion` is passed **into** `SkillStore.getObject(kind, key, version)`, and the post-hoc `skill.version !== wantedVersion` check is kept anyway. The parameter is there because a store may hold several versions of one key and only the store can pick between them; the equality check is a **defense**, not the selection mechanism, because the store is untrusted. Removing either one is wrong: without the parameter a satisfiable pin gets reported as `wrong_version`, and without the check a lying store gets its answer through. Both shipped stores hold several versions of one key and honour the parameter: `getObject` answers a pin with exactly that version, and an omitted version with the newest held. A pin that misses while well-formed versions exist is `absent`; a pin that matches nothing well-formed falls through to the version-less entry, so a malformed object reaches verification and is withheld with a signal rather than reading as a deletion. diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 8994ab1a..531978ac 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -1860,16 +1860,12 @@ describe('integrity-failure log record', () => { const keys = Object.keys(record); expect(keys).toEqual([...keys].sort()); - expect(keys).toEqual([ - 'action', - 'event', - 'language', - 'reason', - 'reason_code', - 'served_key', - 'skill_key', - 'version', - ]); + // No `version`, and that is deliberate rather than an omission: the served + // object's version verified fine and is not what disqualified the answer. + // Reporting it beside a `skill_key` that means the *requested* key would + // also mix the two frames in one record, and `served_key` already says what + // came back. + expect(keys).toEqual(['action', 'event', 'language', 'reason', 'reason_code', 'served_key', 'skill_key']); }); it('redacts a hostile served_key', async () => { diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 2473deb4..471f4cb0 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -73,18 +73,26 @@ const SIGNAL_REVOKED = 'AgentControl Skill Revoked Received'; const EVENT_INTEGRITY_FAILURE = 'ld.skills.integrity_failure'; /** - * Why a skill was withheld: a closed vocabulary with exactly one token per call - * site of `recordIntegrityFailure`. + * Why a skill was withheld: a closed vocabulary of nine tokens. * * Customers alert on these tokens, and every language implementation emits the - * same eight for the same conditions, so a polyglot fleet writes one detection - * rule rather than two. A ninth token is a cross-SDK change — add it everywhere, + * same nine for the same conditions, so a polyglot fleet writes one detection + * rule rather than two. A tenth token is a cross-SDK change — add it everywhere, * or not at all. + * + * Eight of the nine are one per call site of `recordIntegrityFailure`, decided + * inside `verifyRawSkill` over a single object, and they fire **both** detection + * surfaces. `key_mismatch` is the exception on both counts: it comes from + * `recordKeyMismatch` at the retrieval boundary, after verification has already + * passed, and it fires the log record only. It shares this vocabulary anyway + * because a customer's detection rule cares that integrity failed, not about + * which layer noticed — see `recordKeyMismatch` for why the signal stays out. */ export type IntegrityReasonCode = | 'hash_mismatch' | 'invalid_key' | 'invalid_version' + | 'key_mismatch' | 'missing_content' | 'missing_content_hash' | 'not_an_object' @@ -296,6 +304,68 @@ export function recordIntegrityFailure( emit(SIGNAL_INTEGRITY_FAILURE, properties); } +/** + * Records a store answering under a key other than the one requested. + * + * **Log record only — no product signal.** This is the one integrity failure + * that fires one surface rather than both, and the asymmetry is the decision + * rather than an oversight. + * + * The record fires because a substituting store is a genuine tampering + * indicator, and the record is the customer-owned detection path — the only one + * that works when telemetry is opt-out or the instance has no telemetry + * destination at all. It reuses `EVENT_INTEGRITY_FAILURE` deliberately: that + * string is a documented compatibility surface a customer's SIEM matches on, so + * reusing it means an existing rule catches this case without being rewritten, + * with `reason_code` distinguishing it. + * + * The signal stays out because the overwhelmingly common cause of a key mismatch + * is not an attacker but a **broken store adapter** — a stale cache entry, a + * colliding key, a wrong index lookup. Counting those as integrity failures in + * LaunchDarkly's own product counter is the same false positive + * `resolveFromStore` already refuses when a pinned `getObject` answers with a + * non-object: it reads that as `absent` rather than inventing a tampering + * signal from a merely broken adapter. + * + * Lives here, beside `recordIntegrityFailure`, so the single-emission-site rule + * still holds by reading one module. + * + * Both keys are shape-checked and redacted on the same rule as every other key + * that reaches a surface. `served` cannot actually be hostile on the path that + * calls this — `verifyRawSkill` accepted it first — but that is a property of + * the current call order rather than of this function, and the check is what + * stops a future reordering from publishing a body here. + */ +export function recordKeyMismatch(requested: unknown, served: unknown): void { + const record: Record = { + action: 'withheld', + event: EVENT_INTEGRITY_FAILURE, + language: LANGUAGE, + // Named apart from the eight so a reader of the line can tell the retrieval + // boundary from a verification failure without consulting the spec. + reason: 'the skill store answered under a different key than the one requested', + reason_code: 'key_mismatch' satisfies IntegrityReasonCode, + // The key the store answered under: the one datum that makes a broken + // adapter diagnosable, so it is a parseable field rather than prose buried + // in `reason`. Record-only, and never added to the signal's allowlist. + served_key: isValidSkillKey(served) ? served : '', + // `skill_key` keeps the meaning it has on every other record — the key the + // *caller asked for* — so a rule that groups by it keeps working. + skill_key: isValidSkillKey(requested) ? requested : '', + }; + // No `expected_hash` or `observed_hash`: verification passed, so there is no + // hash disagreement to report, and an absent field stays absent rather than + // being emitted as null. No `version` either — the object's version verified + // fine and is not what disqualified the answer. + // + // Keys above are in alphabetical order, as in `recordIntegrityFailure`, so the + // emitted line is byte-identical across SDKs for the same input. Do not + // reorder. + // + // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; integrity failures must be visible + console.error(`[LaunchDarkly] ${EVENT_INTEGRITY_FAILURE} ${JSON.stringify(record)}`, record); +} + /** * Records a materialization. Deliberately carries no `target_path` and no * filesystem path of any kind — the same reasoning that keeps the skill body out @@ -673,9 +743,13 @@ export function resolveFromStore(store: SkillStore, key: string, wantedVersion: // is not `wrong_version` either — that token names a version mismatch // specifically, and there is deliberately no `wrong_key` to parallel it. // - // This path reports the outcome reason only: `verifyRawSkill` has already - // passed, so no integrity signal is recorded and no `ld.skills.integrity_failure` - // record is logged. That asymmetry is deliberate and pinned by a test. + // Records the log surface but not the product signal. `verifyRawSkill` has + // already passed, so this is not a verification failure and does not go + // through `recordIntegrityFailure`; see `recordKeyMismatch` for why the two + // surfaces part company here. The asymmetry is pinned by a test in both + // directions, because an implementation that emitted the signal too would + // look correct from every other angle. + recordKeyMismatch(key, skill.key); return { error: `skill '${key}' is not available: the store answered under key '${skill.key}'`, reason: 'integrity_failure', From 191fcdfe7e168e0d012d15ee6631d5e06b0d7e19 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 30 Sep 2026 12:54:04 -0400 Subject: [PATCH 36/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20align=20the=20FDv2=20protocol=20reader=20with=20the=20spec?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five spec contradictions in the protocol and store layers, aligned with TESTING.md §3.25. Each one made the transport report something that had not happened. - `objectsRevoked` counted every `delete-object`, including a tombstone for a key the store never held. It now counts only a delete that removed something; the tombstone is still reported, just not counted. - A `none` intent completing a foreign payload adopted its basis, so the next request resumed from a point this store had never reached. - The consecutive-failure counter reset on any parsed `server-intent`. A parsed `xfer-*` intent is a promise, not a delivery: a server that announces a transfer and drops before committing could keep the counter at zero forever. The counter now resets on a committed payload or a `none` intent only. - Objects arriving under an intent code this reader cannot apply were silently dropped. They are counted as ignored and warned about once per intent. - Hashless-report memory was module-global, so two stores in one process cross-talked. It is now held per `ProtocolReader`. Plus the input and lifecycle guards the same section names: `pollIntervalMs` and `waitForSkills(timeoutMs)` reject NaN and Infinity — `setTimeout(fn, NaN)` fires at once, which would read as a timeout rather than a wrong wait — a stream interrupted by `close()` no longer counts as a success, and a listener whose async callback rejects is logged like a throwing one instead of surfacing as an unhandled rejection. Co-Authored-By: Claude Fable 5.1 Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 2 +- packages/client/agents.md | 8 +- .../client/src/__tests__/skills-fdv2.test.ts | 252 +++++++++++++- packages/client/src/skills-fdv2.ts | 325 +++++++++++------- 4 files changed, 449 insertions(+), 138 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 9f609eb1..bb49b468 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -328,7 +328,7 @@ example above, materializes only what the resolved variation actually asked for. | `InMemorySkillStore` | A `SkillStore` backed by in-memory maps. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Several versions of one key coexist, as they do in a real payload: `getObject` answers a pin with exactly that version and an omitted version with the newest held, and `allObjects` returns one entry per `(key, version)` under keys opaque to callers. `addListener` throws for any kind but `'skill'`. | | `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `isInitialized()`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | -| `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object`, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | +| `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object` that removed something, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation, and a tombstone for a key never held is reported but not counted), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | | `createSkillOutcome(init)` | Build a frozen `SkillOutcome`. Exported for tests and for wrapping your own retrieval in the same shape. | | `SKILL_FILENAME` | `'SKILL.md'`. | diff --git a/packages/client/agents.md b/packages/client/agents.md index 829cb4b2..d1ee9470 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -59,13 +59,13 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b **Polling and streaming have different default hosts.** `GET /sdk/poll` is served from `DEFAULT_BASE_URI` (`sdk.launchdarkly.com`) and `GET /sdk/stream` from `DEFAULT_STREAM_URI` (`stream.launchdarkly.com`), matching the base server-side SDK. Sending the stream request to the polling host fails against a real environment, and the fake endpoint cannot catch it because it serves both from one origin. `baseUri` given alone therefore applies to both, since a relay or private instance usually serves both from one host; only the LaunchDarkly defaults split them, and `streamUri` overrides the stream origin on its own. -**Changes commit at `payload-transferred`, not as objects arrive.** A payload version is the unit of consistency: a half-applied full transfer would publish a state the server never described, and would briefly empty the store — which, with pruning on, is the difference between a reconcile and deleting a customer's skill files. An interrupted transfer therefore leaves last known good intact, and listeners fire once per commit. +**Changes commit at `payload-transferred`, not as objects arrive.** A payload version is the unit of consistency: a half-applied full transfer would publish a state the server never described, and would briefly empty the store — which, with pruning on, is the difference between a reconcile and deleting a customer's skill files. An interrupted transfer therefore leaves last known good intact, and listeners fire at commit — once per changed object, all of them at `payload-transferred` (which is why `watchSkills` debounces: a per-object notification does not coalesce a payload on its own). **`objectsRevoked` counts revocations, not tombstones.** A full transfer revokes by omission, so `payloadTransferred` diffs the committed set against the pending one and pushes the departures into `changes` as tombstones at `(key, version)` granularity — a listener that reads versions needs both halves of a version move. The *counter* is coarser on purpose: only a key the payload dropped altogether counts, because a key surviving under a new version was never revoked, and this is a field operators alert on. `keysFullyRevoked` is where the two granularities part. **The first payload intent is read, and is assumed to be the skill payload.** Delivery provides one payload per credential and the protocol requires a client to ignore all but the first payload intent, so `payloads[0]` is both what arrives and what the protocol says to read. The cost of that assumption is that an `xfer-full` for somebody *else's* payload would start an empty pending set, and the next `payload-transferred` would publish it — every skill reported revoked, and with pruning on, a customer's files deleted. `ProtocolReader` therefore learns which payload skills arrive on, from the intent's `id` or from the `(p::)` selector, and declines to apply a transfer of any other: once at warning level, counted in `diagnostics.payloadsIgnored`, holding last known good. A transfer that names no payload is applied, since one-payload delivery is the common case. The residual is the first transfer of a connection — before a skill has arrived there is nothing to compare against — which is what the separate warning on a multi-payload intent is for. -**A hashless object is held, not dropped.** Verification withholds it with `missing_content_hash`; the transport's job is to make that loud (an error per object, a summary each time the held store becomes wholly hashless or its withheld set changes, `diagnostics.hashlessObjects`) rather than to work around it. Neither error repeats for a payload re-delivered unchanged, which matters most in polling mode, where the same payload arrives on every interval; `_warnedHashless` remembers what has been said and is capped, so a frequently versioned environment cannot accumulate an entry per version. Dropping it at the transport would report `absent` — indistinguishable from "no such skill" — and would let a prune delete the last known-good copy on disk. Never synthesize a hash from the delivered content: that certifies the content against itself and verifies nothing. +**A hashless object is held, not dropped.** Verification withholds it with `missing_content_hash`; the transport's job is to make that loud (an error per object, a summary each time the held store becomes wholly hashless or its withheld set changes, `diagnostics.hashlessObjects`) rather than to work around it. Neither error repeats for a payload re-delivered unchanged, which matters most in polling mode, where the same payload arrives on every interval; each `ProtocolReader` holds its own `HashlessMemory` of what has been said (exposed to tests as `_warnedHashless`), per store rather than module-scoped so two stores in one process do not suppress each other's reports, and capped so a frequently versioned environment cannot accumulate an entry per version. Dropping it at the transport would report `absent` — indistinguishable from "no such skill" — and would let a prune delete the last known-good copy on disk. Never synthesize a hash from the delivered content: that certifies the content against itself and verifies nothing. **`SkillObjectSet.snapshot` collapses to one object per key, keyed by the bare skill key.** `//SKILL.md` is a single path, so a whole-store consumer must see one object per key or a `'*'` reconcile writes the same path twice and `allSkills` returns two versions of one skill. Both consumers also collapse for themselves through `newestByKey`, because the seam admits any store — this is the transport holding up its end, not the only guard. The keys must be skill keys, not wire `key:version` keys, because `writeSkills('*')` derives its prune keep-set from them. `getObject` still resolves a pinned version out of the full set. @@ -73,9 +73,9 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b **A body read that fails is recoverable; a protocol-reader or dispatch error is not.** `iterSse` wraps the read itself — a reset, a truncated chunk, the read deadline — as `RecoverableTransportError`, because a live stream dies mid-body far more often than it refuses to open, and the delivery loop reads anything else as a bug and stops for the process lifetime. Whatever the consumer's loop body throws while the generator is suspended at a `yield` passes through untouched, so a bug still surfaces as one. Every retry delay is clamped to `maxBackoffMs` and floored at `initialBackoffMs` — a server asking for no delay with `Retry-After: 0` would otherwise reconnect in a loop and burn the whole retry bound in milliseconds, and a blank `Retry-After` must read as "no delay given" rather than as zero, because `Number("")` is `0` and finite. -**What resets the failure counter, and what escapes it.** The counter resets on any sign of a working server — a `server-intent` that parsed, or a payload that committed — not when a connection returns: a stream only ever ends by being dropped, so resetting on return would count every healthy, server-recycled connection as a failure. Waiting for a *commit* is not enough either, because a reconnect whose basis is already current is answered with the `none` intent and commits nothing, which would expire an environment whose skills never change. A non-catastrophic `goodbye` is exempt from the counter, but only on a connection that had reached a working server, which is what `reachedServer` tracks per attempt: a server that says goodbye having sent no intent delivered nothing, and exempting it would let it reconnect without limit and without any of it reaching `diagnostics` or `failed`. Keep both halves — the reset and the `reachedServer` qualifier — or one of those two failures comes back. +**What resets the failure counter, and what escapes it.** The counter resets on a completed exchange only — a payload that committed, or a `none` intent — not when a connection returns: a stream only ever ends by being dropped, so resetting on return would count every healthy, server-recycled connection as a failure. Waiting for a *commit* alone is not enough either, because a reconnect whose basis is already current is answered with the `none` intent and commits nothing, which would expire an environment whose skills never change. But a parsed `xfer-full` or `xfer-changes` intent does **not** reset it: an announced transfer is a promise, not a delivery, and a server that announces one and drops before `payload-transferred`, every time, has delivered nothing — counting the announcement as health would retry it forever at the initial backoff (the `gives up on a server that announces a transfer and drops` test pins this). A non-catastrophic `goodbye` is exempt from the counter, but only on a connection that had completed such an exchange, which is what `reachedServer` tracks per attempt: a server that says goodbye having sent no intent delivered nothing, and exempting it would let it reconnect without limit and without any of it reaching `diagnostics` or `failed`. Keep both halves — the reset and the `reachedServer` qualifier — or one of those two failures comes back. -**HTTP 400 is recoverable exactly once.** The `basis` selector and the `If-None-Match` etag are the only client state in the request, and a selector the server no longer accepts is the one rejection the adapter can act on — so a 400 arrives as `StaleRequestStateError`, the state is dropped, and a full transfer is requested. A 400 for a request carrying neither is the request itself being refused, and is fatal. Do not make 400 unconditionally fatal again: that strands delivery for the process lifetime on a stale selector. 405, 406, 414 and 501 stay fatal, since none of them is about client state. +**HTTP 400 is recoverable exactly once, and only when there was state to drop.** The `basis` selector and the `If-None-Match` etag are the only client state in the request, and a selector the server no longer accepts is the one rejection the adapter can act on — so a 400 on a request that carried either arrives as `StaleRequestStateError`, the state is dropped, and a full transfer is requested from scratch; a second 400 is fatal. A 400 for a request carrying neither is the request itself being refused: retrying would send the byte-identical request, so it is fatal at once with no retry. Do not make 400 unconditionally fatal again: that strands delivery for the process lifetime on a stale selector. 405, 406, 414 and 501 stay fatal, since none of them is about client state. **`close` aborts the signal, it does not just set a flag.** The delivery task spends its life awaiting a stream read, and a flag it never checks would leave a healthy stream running until the process exited. The store's signal is the parent of every `ReadDeadline`, so one abort reaches a pending connect and a pending read alike, and a store closing is passed through rather than reported as a failure. Every backoff timer, deadline timer and the `waitForSkills` timer are `unref`ed for the same reason: a background store must not be why `node` stays up. diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index a77c96a2..c5eb8675 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -36,7 +36,6 @@ import { } from '../skills.js'; import { SKILL_OBJECT_KIND } from '../skills-core.js'; import { - _warnedHashless, backoffDelayMs, classifyStatus, DEFAULT_BASE_URI, @@ -282,8 +281,6 @@ beforeEach(async () => { openStores = []; tempRoots = []; _clearState(); - // The hashless-object error is deduped per process; per test here. - _warnedHashless.clear(); warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}); errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); }); @@ -538,6 +535,36 @@ describe('protocol reader', () => { expect(outcomes.at(-1)?.basis).toBe('basis-1'); }); + it('counts objects arriving under an unknown intent code as ignored, and warns once per intent', () => { + // A future intent code is neither a full nor a changes transfer, so its + // objects cannot be applied without guessing — and guessing could empty the + // store. They are dropped, but visibly: counted under `objectsIgnored`, with + // one warning per intent rather than one per object. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill('kept')]])); + const before = warnSpy.mock.calls.length; + drive( + reader, + events( + ['server-intent', serverIntent('xfer-future')], + ['put-object', putSkill('a')], + ['put-object', putSkill('b')], + ['delete-object', deleteSkill('kept')], + ['payload-transferred', transferred('basis-2')], + ), + ); + expect(reader.diagnostics.objectsIgnored).toBe(3); + expect(reader.diagnostics.skillObjectsReceived).toBe(1); + expect(warnSpy.mock.calls.length - before).toBe(1); + expect(logged(warnSpy)).toContain('xfer-future'); + expect(held.get('kept', null)).not.toBeNull(); + expect(held.size).toBe(1); + // A fresh intent announcement warns afresh. + drive(reader, events(['server-intent', serverIntent('xfer-future')], ['put-object', putSkill('c')])); + expect(warnSpy.mock.calls.length - before).toBe(2); + }); + it('shows nothing before payload-transferred', () => { // A payload version is the unit of consistency; half of one is not a state. const held = new SkillObjectSet(); @@ -603,6 +630,52 @@ describe('protocol reader', () => { expect(reader.diagnostics.objectsRevoked).toBe(1); }); + it('reports but does not count a delete-object for a key it never held (§3.25)', () => { + // `objectsRevoked` is read precisely when somebody is working out whether a + // revocation landed, so a tombstone for nothing must not inflate it. The + // tombstone still reaches listeners through `changes`. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill('kept')]])); + const outcomes = drive( + reader, + events( + ['server-intent', serverIntent('xfer-changes')], + ['delete-object', deleteSkill('never-held')], + ['payload-transferred', transferred('basis-2')], + ), + ); + expect(reader.diagnostics.objectsRevoked).toBe(0); + expect((outcomes.at(-1)?.changes ?? []).map((raw) => ({ key: raw.key, version: raw.version }))).toEqual([ + { key: 'never-held', version: 3 }, + ]); + expect(held.size).toBe(1); + }); + + it('counts a delete-object that actually removed something — the other half', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive( + reader, + fullPayload([ + ['put-object', putSkill('a')], + ['put-object', putSkill('b')], + ]), + ); + drive( + reader, + events( + ['server-intent', serverIntent('xfer-changes')], + ['delete-object', deleteSkill('a')], + ['delete-object', deleteSkill('a')], + ['payload-transferred', transferred('basis-2')], + ), + ); + // The second delete of the same object removed nothing, so one, not two. + expect(reader.diagnostics.objectsRevoked).toBe(1); + expect(held.size).toBe(1); + }); + it('notifies a delete with a tombstone carrying no content', () => { const held = new SkillObjectSet(); const reader = new ProtocolReader(held); @@ -935,6 +1008,23 @@ describe('payload identity', () => { expect(held.get('a', null)).toBeNull(); }); + it('does not adopt the basis of a foreign payload announced with the none intent (§3.25)', () => { + // A `none` intent builds no pending set, and the foreign check must not be + // gated on one: the transfer that follows still names a payload, and + // adopting its selector would resume the next connection from someone + // else's payload with every diagnostic reading healthy. + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill()]], 'basis-skills')); + const outcomes = drive( + reader, + events(['server-intent', serverIntent('none', 'env-flags')], ['payload-transferred', transferred('basis-flags')]), + ); + expect(outcomes.at(-1)?.basis).toBeNull(); + expect(reader.diagnostics.payloadsIgnored).toBe(1); + expect(held.get('pdf-extraction', null)).not.toBeNull(); + }); + it('does not adopt the selector of a payload it declined', () => { // Ignoring a foreign payload's contents while adopting its resume point // would ask the next poll or stream to resume from someone else's payload: @@ -1710,6 +1800,25 @@ describe('failure handling', () => { expect(store.failed).toContain('gave up after 4 consecutive failures'); }); + it('gives up on a server that announces a transfer and drops before committing, every time (§3.25)', async () => { + // An `xfer-full` intent is a promise, not a delivery. A server that sends + // one and drops before `payload-transferred` has delivered nothing, and a + // store that counted the announcement as health would retry it forever at + // the initial backoff. Only a committed payload or a `none` intent resets + // the row of failures. + const outcomes: unknown[] = []; + for (let i = 0; i < 10; i += 1) { + outcomes.push(asPairs(events(['server-intent', serverIntent('xfer-full')], ['put-object', putSkill()]))); + } + const requester = new ScriptedRequester(outcomes); + const store = scriptedStreamStore(requester, { maxConsecutiveFailures: 3 }); + store.start(); + expect(await waitUntil(() => store.failed !== null, 5000)).toBe(true); + expect(store.failed).toContain('gave up after 4 consecutive failures'); + expect(requester.calls).toHaveLength(4); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).toBeNull(); + }); + it('honours a Retry-After header off the wire', async () => { endpoint.queuePoll([], { status: 429, retryAfter: '1' }); endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); @@ -2054,15 +2163,28 @@ describe('the missing contentHash', () => { ), ); } - expect(_warnedHashless.size).toBeLessThan(600); + expect(reader._warnedHashless.size).toBeLessThan(600); }); it('forgets what it remembers once everything held verifies', () => { const reader = new ProtocolReader(new SkillObjectSet()); drive(reader, fullPayload([['put-object', putSkill('a', { omitHash: true })]])); - expect(_warnedHashless.size).toBeGreaterThan(0); + expect(reader._warnedHashless.size).toBeGreaterThan(0); drive(reader, fullPayload([['put-object', putSkill('a')]], 'basis-2')); - expect(_warnedHashless.size).toBe(0); + expect(reader._warnedHashless.size).toBe(0); + }); + + it('remembers per reader, so two stores in one process do not cross-talk', () => { + // The dedupe memory is held by the reader, not the module: a second store + // in the same process reporting the same hashless object is a second + // deployment problem, and must be told about it. + const first = new ProtocolReader(new SkillObjectSet()); + drive(first, fullPayload([['put-object', putSkill('a', { omitHash: true })]])); + const before = errorSpy.mock.calls.length; + const second = new ProtocolReader(new SkillObjectSet()); + drive(second, fullPayload([['put-object', putSkill('a', { omitHash: true })]])); + expect(errorSpy.mock.calls.length).toBeGreaterThan(before); + expect(first._warnedHashless).not.toBe(second._warnedHashless); }); it('still reports each hashless object in a payload separately', () => { @@ -3080,6 +3202,124 @@ describe('endpoints', () => { // ─── Layering, and the absence of telemetry ────────────────────────────────── +// ─── Transport contract assertions the spec names (§3.25) ──────────────────── + +describe('transport contract', () => { + const source = readFileSync(new URL('../skills-fdv2.ts', import.meta.url), 'utf8'); + + it('holds the wire kind as its own declaration, not an alias of the seam kind', () => { + // One is a wire value LaunchDarkly owns, the other an SDK seam. They are + // equal today; a change to either must be a deliberate change to that one. + expect(source).toMatch(/export const FDV2_OBJECT_KIND = 'skill';/); + expect(source).not.toMatch(/FDV2_OBJECT_KIND\s*=\s*SKILL_OBJECT_KIND/); + expect(FDV2_OBJECT_KIND).toBe(SKILL_OBJECT_KIND); + }); + it('a goodbye on a store holding committed content keeps it', () => { + const held = new SkillObjectSet(); + const reader = new ProtocolReader(held); + drive(reader, fullPayload([['put-object', putSkill()]])); + const outcome = reader.handle('goodbye', { reason: 'recycle', silent: true, catastrophe: false }); + expect(outcome.disconnect).toBeTruthy(); + expect(held.get('pdf-extraction', null)).not.toBeNull(); + // An in-flight transfer is abandoned too, without touching what was committed. + drive(reader, events(['server-intent', serverIntent('xfer-full')], ['put-object', putSkill('other')])); + reader.handle('goodbye', { reason: 'recycle', silent: true }); + expect(held.size).toBe(1); + }); + it('holds a tampered object while the accessor reports integrity_failure', async () => { + // Verification is the accessor's job, not the transport's: the store keeps + // what arrived, and the reported outcome is what tells a caller to fail closed. + endpoint.queuePoll(fullPayload([['put-object', putSkill('tampered', { contentHash: hash('something else') })]])); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + _setStore(store); + expect(store.getObject(SKILL_OBJECT_KIND, 'tampered')).not.toBeNull(); + const outcome = await getSkillResult('tampered'); + expect(outcome.reason).toBe('integrity_failure'); + expect(outcome.skill).toBeNull(); + expect(store.getObject(SKILL_OBJECT_KIND, 'tampered')).not.toBeNull(); + }); + it.each([ + 0, + -1, + Number.NaN, + Number.POSITIVE_INFINITY, + ])('rejects a non-positive or non-finite pollIntervalMs (%s)', (value) => { + // `NaN` is the case a `<= 0` guard misses. + expect(() => new FDv2SkillStore(SDK_KEY, { mode: 'poll', pollIntervalMs: value })).toThrow(/pollIntervalMs/); + }); + it.each([ + Number.NaN, + Number.NEGATIVE_INFINITY, + -1, + ])('waitForSkills rejects a non-finite or negative timeoutMs (%s)', async (value) => { + const store = pollStore(); + await expect(store.waitForSkills(value)).rejects.toThrow(/timeoutMs/); + }); + it('a stream interrupted by close does not count as a success', async () => { + // `streamOnce` returning because the signal aborted is the store closing, + // not the server answering — so the row of failures must not be reset by it. + let release: (() => void) | null = null; + const parked: Requester = { + poll() { + throw new Error('not a polling double'); + }, + async stream(_basis, signal) { + return (async function* () { + yield ['heart-beat', {}] as [string, unknown]; + await new Promise((resolve) => { + release = resolve; + signal.addEventListener('abort', () => resolve(), { once: true }); + }); + // One more event after the abort, so the loop observes the signal. + yield ['heart-beat', {}] as [string, unknown]; + })(); + }, + }; + const requester = new ScriptedRequester([new RecoverableTransportError('x'), parked as never]); + const wrapped: Requester = { + poll: (b, e, s) => requester.poll(b, e, s), + stream: async (b, s) => { + if (requester.calls.length === 0) return requester.stream(b, s); + requester.calls.push([b, null]); + return parked.stream(b, s); + }, + }; + const store = scriptedStreamStore(wrapped, { maxConsecutiveFailures: 5 }); + store.start(); + expect(await waitUntil(() => release !== null, 5000)).toBe(true); + expect(store.diagnostics.connectionFailures).toBe(1); + await store.close(); + expect(store.diagnostics.connectionFailures).toBe(1); + }); +}); +describe('listeners', () => { + it('logs an async listener whose promise rejects rather than leaving it unhandled', async () => { + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + const unhandled: unknown[] = []; + const onUnhandled = (reason: unknown): void => { + unhandled.push(reason); + }; + process.on('unhandledRejection', onUnhandled); + try { + store.addListener(SKILL_OBJECT_KIND, async () => { + throw new Error('async listener exploded'); + }); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(await waitUntil(() => /async listener exploded/.test(consoleErrors()), 5000)).toBe(true); + expect(consoleErrors()).toContain('delivery continues'); + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(unhandled).toEqual([]); + expect(store.failed).toBeNull(); + } finally { + process.off('unhandledRejection', onUnhandled); + } + }); +}); + describe('layering', () => { /** `skills-fdv2.ts`'s own source text — the only way to assert a leaf. */ const source = readFileSync(new URL('../skills-fdv2.ts', import.meta.url), 'utf8'); diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index cb07e262..028b9919 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -247,11 +247,13 @@ export type StoreDiagnostics = { */ readonly hashlessObjects: number; /** - * Recoverable transport failures in a row. Back to zero on any sign of a - * working server — a parsed `server-intent` or a committed payload — and not - * raised by a connection the server closes normally after reaching that - * point, which is how a long-lived stream is recycled. A connection closed - * without ever getting there delivered nothing, and does raise it. + * Recoverable transport failures in a row. Back to zero on a completed + * exchange — a committed payload, or a `none` intent saying the payload held + * is current — and not raised by a connection the server closes normally + * after reaching that point, which is how a long-lived stream is recycled. A + * parsed transfer intent alone does not reset it: an announced transfer that + * never commits delivered nothing. A connection closed without ever getting + * there delivered nothing either, and does raise it. */ readonly connectionFailures: number; /** The most recent transport error, if any. Human-readable; do not parse. */ @@ -263,99 +265,103 @@ const HASHLESS_ADVICE = "'missing_content_hash' and its content will not resolve. Contact LaunchDarkly support."; /** - * What has already been reported hashless: one entry per `(key, version)`, plus - * one describing the store-wide summary last spoken. - * - * Module-scoped so each error is one per object per process rather than one per - * re-delivered payload, which matters most in polling mode — the same payload - * arrives on every interval. - */ -export const _warnedHashless = new Set(); - -/** - * Ceiling on remembered reports, so a process whose skills are versioned often - * cannot accumulate an entry per version indefinitely. Oldest out first; an - * evicted object can be reported a second time, which is the cheaper of the two - * failure modes. + * Ceiling on remembered hashless reports, so a process whose skills are + * versioned often cannot accumulate an entry per version indefinitely. Oldest + * out first; an evicted object can be reported a second time, which is the + * cheaper of the two failure modes. */ const HASHLESS_MEMORY_LIMIT = 512; /** Distinguishes the store-wide summary's entry from a per-object one. */ const SUMMARY_MARKER = '\u0000summary\u0000'; -function rememberHashless(entry: string): void { - // Insertion-ordered iteration makes the first entry the oldest. - while (_warnedHashless.size >= HASHLESS_MEMORY_LIMIT) { - const oldest = _warnedHashless.values().next().value; - if (oldest === undefined) break; - _warnedHashless.delete(oldest); - } - _warnedHashless.add(entry); -} - /** - * One error per `(key, version)` whose envelope had no `contentHash`. + * What one reader has already reported hashless: one entry per + * `(key, version)`, plus one describing the store-wide summary last spoken. * - * At error level rather than warn, and per object rather than once per process, - * because this is the difference between a broken deployment and an - * empty-by-design one. + * Held **per `ProtocolReader`** — so per store — rather than at module scope. + * Each error is then one per object per store rather than one per re-delivered + * payload, which matters most in polling mode where the same payload arrives on + * every interval; and two stores in one process do not share a memory, so a + * second store seeing the same broken object is told about it too. */ -function warnHashless(raw: RawSkillObject): void { - const identity = `${String(raw.key)}:${String(raw.version)}`; - if (_warnedHashless.has(identity)) return; - rememberHashless(identity); - error( - `Skill '${String(raw.key)}' version ${String(raw.version)} arrived without a contentHash and will be ` + - `withheld. ${HASHLESS_ADVICE}`, - ); -} +class HashlessMemory { + readonly seen = new Set(); + + private remember(entry: string): void { + // Insertion-ordered iteration makes the first entry the oldest. + while (this.seen.size >= HASHLESS_MEMORY_LIMIT) { + const oldest = this.seen.values().next().value; + if (oldest === undefined) break; + this.seen.delete(oldest); + } + this.seen.add(entry); + } + + /** + * One error per `(key, version)` whose envelope had no `contentHash`. + * + * At error level rather than warn, and per object rather than once per store, + * because this is the difference between a broken deployment and an + * empty-by-design one. + */ + warnHashless(raw: RawSkillObject): void { + const identity = `${String(raw.key)}:${String(raw.version)}`; + if (this.seen.has(identity)) return; + this.remember(identity); + error( + `Skill '${String(raw.key)}' version ${String(raw.version)} arrived without a contentHash and will be ` + + `withheld. ${HASHLESS_ADVICE}`, + ); + } -/** Forgets the summary, so a relapse after a recovery is reported afresh. */ -function forgetHashlessSummary(): void { - for (const entry of _warnedHashless) { - if (entry.startsWith(SUMMARY_MARKER)) _warnedHashless.delete(entry); + /** Forgets the summary, so a relapse after a recovery is reported afresh. */ + private forgetSummary(): void { + for (const entry of this.seen) { + if (entry.startsWith(SUMMARY_MARKER)) this.seen.delete(entry); + } } -} -/** - * One error per *distinct* committed store in which nothing held can possibly - * verify. - * - * Fires at delivery time, so the condition is visible in a process that boots, - * materializes nothing, and exits — which is the shape a skills deployment fails - * in. The accessor boundary's own withholding summary only speaks once a caller - * asks. - * - * Spoken when the condition becomes true and whenever the hashless objects - * change, and not again for a store that has not moved: an unchanging payload - * re-delivered on every poll describes one problem, not one per interval. A - * store that recovers and relapses is reported again. - */ -function warnIfNothingCanVerify(held: RawSkillObject[]): void { - const hashless = held.filter((raw) => typeof raw.contentHash !== 'string'); - if (hashless.length === 0) { - // Everything held verifies, so there is nothing outstanding to remember. - _warnedHashless.clear(); - return; - } - if (hashless.length !== held.length) { - forgetHashlessSummary(); - return; - } - - const summary = - SUMMARY_MARKER + - hashless - .map((raw) => `${String(raw.key)}:${String(raw.version)}`) - .sort() - .join('\u0000'); - if (_warnedHashless.has(summary)) return; - forgetHashlessSummary(); - rememberHashless(summary); - error( - `All ${held.length} skill object(s) in the delivered payload arrived without a contentHash. No skill content ` + - `will resolve from this store. ${HASHLESS_ADVICE}`, - ); + /** + * One error per *distinct* committed store in which nothing held can possibly + * verify. + * + * Fires at delivery time, so the condition is visible in a process that + * boots, materializes nothing, and exits — which is the shape a skills + * deployment fails in. The accessor boundary's own withholding summary only + * speaks once a caller asks. + * + * Spoken when the condition becomes true and whenever the hashless objects + * change, and not again for a store that has not moved: an unchanging payload + * re-delivered on every poll describes one problem, not one per interval. A + * store that recovers and relapses is reported again. + */ + warnIfNothingCanVerify(held: RawSkillObject[]): void { + const hashless = held.filter((raw) => typeof raw.contentHash !== 'string'); + if (hashless.length === 0) { + // Everything held verifies, so there is nothing outstanding to remember. + this.seen.clear(); + return; + } + if (hashless.length !== held.length) { + this.forgetSummary(); + return; + } + + const summary = + SUMMARY_MARKER + + hashless + .map((raw) => `${String(raw.key)}:${String(raw.version)}`) + .sort() + .join('\u0000'); + if (this.seen.has(summary)) return; + this.forgetSummary(); + this.remember(summary); + error( + `All ${held.length} skill object(s) in the delivered payload arrived without a contentHash. No skill ` + + `content will resolve from this store. ${HASHLESS_ADVICE}`, + ); + } } // --------------------------------------------------------------------------- @@ -649,9 +655,11 @@ export type TransferOutcome = { fatal?: string | null; disconnect?: string | null; /** - * Set by an event that proves the connection reached a working server: a - * `server-intent` that parsed. True even for the `none` intent, which commits - * nothing. + * Set by an event that proves the connection reached a working server and + * completed an exchange with it: the `none` intent, which commits nothing + * because the payload held is already current. A transfer intent does not + * set it — an announced transfer that never commits is not health, and the + * commit itself is reported through `committed`. */ healthy?: boolean; /** @@ -730,7 +738,8 @@ function freshDiagnostics(): MutableDiagnostics { * version is the unit of consistency: applying half of one would publish a state * the server never described, and on a full transfer it would briefly empty the * store — which, with pruning on, is the difference between a reconcile and - * deleting a customer's skill files. Listeners therefore fire once per commit. + * deleting a customer's skill files. Listeners therefore fire at commit — once + * per changed object, all of them at `payload-transferred`. * * **The first payload intent is read, and is assumed to be the skill payload**, * as the protocol requires. Because an `xfer-full` for a different payload would @@ -741,7 +750,14 @@ function freshDiagnostics(): MutableDiagnostics { */ export class ProtocolReader { readonly diagnostics = freshDiagnostics(); + private readonly hashless = new HashlessMemory(); + /** What this reader has reported hashless. Exposed for tests; not API. */ + readonly _warnedHashless: Set = this.hashless.seen; private intent: string | null = null; + // Whether this intent's unknown code has been warned about. Reset per + // `server-intent`, so the warning is once per announcement rather than once + // per object or once per reader. + private warnedUnknownIntent = false; private pending: SkillObjectSet | null = null; private changes: RawSkillObject[] = []; // The payload the current intent describes, and the payload skills have @@ -790,6 +806,7 @@ export class ProtocolReader { this.intentPayloadId = payloadIdOf(first); this.changes = []; this.skillsInPayload = 0; + this.warnedUnknownIntent = false; if (intent === INTENT_TRANSFER_FULL) { // A fresh set: the payload about to arrive replaces everything held. Built // alongside the live set rather than in place, so an interrupted transfer @@ -805,9 +822,14 @@ export class ProtocolReader { // reason an unknown kind is: guessing could empty the store. this.pending = null; } - // An intent that parsed means the connection reached a working server, even - // when the intent is `none` and no payload will follow. - return { healthy: true }; + // Only the `none` intent is a completed exchange: the payload held is + // current, and nothing more will follow, so it is the one sign of health a + // connection that commits nothing can give. An `xfer-full` or + // `xfer-changes` intent is a promise, not a delivery — a server that + // announces a transfer and drops before `payload-transferred`, every time, + // has delivered nothing, and counting the announcement as health would + // retry it forever at the initial backoff. + return { healthy: intent === INTENT_TRANSFER_NONE }; } private target(): SkillObjectSet | null { @@ -819,6 +841,27 @@ export class ProtocolReader { return this.pending; } + /** + * A skill object arrived under an intent this reader cannot apply — a future + * intent code, or `none`, under which no objects should arrive at all. + * + * Dropped rather than guessed at, since guessing could empty the store; but + * dropped **visibly**: counted under `objectsIgnored`, with one warning per + * intent announcement so a payload of many objects is one line, not many. + */ + private ignoreUnderUnknownIntent(): TransferOutcome { + this.diagnostics.objectsIgnored += 1; + if (!this.warnedUnknownIntent) { + this.warnedUnknownIntent = true; + warn( + `Skill objects arrived under FDv2 intent code ${JSON.stringify(this.intent)}, which this SDK cannot ` + + 'apply; they are being ignored and counted under objectsIgnored. The skills held are unchanged. If ' + + 'this persists, update the SDK.', + ); + } + return {}; + } + private putObject(data: unknown): TransferOutcome { if (!isSkillEvent(data)) { this.diagnostics.objectsIgnored += 1; @@ -826,7 +869,7 @@ export class ProtocolReader { } if (this.pending === null && this.intent === null) this.intent = INTENT_TRANSFER_CHANGES; const target = this.target(); - if (target === null) return {}; + if (target === null) return this.ignoreUnderUnknownIntent(); const raw = seamObjectFromPut(data as Record); if (raw === null) return {}; @@ -836,7 +879,7 @@ export class ProtocolReader { this.skillsInPayload += 1; if (typeof raw.contentHash !== 'string') { this.diagnostics.hashlessObjects += 1; - warnHashless(raw); + this.hashless.warnHashless(raw); } return {}; } @@ -848,12 +891,15 @@ export class ProtocolReader { } if (this.pending === null && this.intent === null) this.intent = INTENT_TRANSFER_CHANGES; const target = this.target(); - if (target === null) return {}; + if (target === null) return this.ignoreUnderUnknownIntent(); const tombstone = tombstoneFromDelete(data as Record); if (tombstone === null) return {}; - target.delete(tombstone); - this.diagnostics.objectsRevoked += 1; + // Counted only when the delete removed something. A tombstone for a key + // never held is still reported to listeners below, but `objectsRevoked` is + // read precisely when somebody is working out whether a revocation landed, + // and a delete of nothing would inflate the one number that answers that. + if (target.delete(tombstone).length > 0) this.diagnostics.objectsRevoked += 1; // A revocation identifies the payload as ours just as a put does. this.skillsInPayload += 1; // A tombstone, not a skill object: it carries identity and no content, so a @@ -866,7 +912,10 @@ export class ProtocolReader { private payloadTransferred(data: unknown): TransferOutcome { const state = (data as { state?: unknown } | null)?.state; const payloadId = this.intentPayloadId ?? payloadIdFromSelector(state); - const foreign = this.pending !== null && this.isForeignPayload(payloadId); + // Regardless of whether a pending set exists: a `none` intent builds none, + // and the transfer that completes it still names a payload whose selector + // must not become the resume point if it is not the payload skills arrive on. + const foreign = this.isForeignPayload(payloadId); if (foreign) { this.warnForeignPayload(payloadId); this.diagnostics.payloadsIgnored += 1; @@ -882,7 +931,7 @@ export class ProtocolReader { this.diagnostics.objectsRevoked += keysFullyRevoked(revoked, this.pending); } this.committed.replaceWith(this.pending); - warnIfNothingCanVerify(this.committed.allRaw()); + this.hashless.warnIfNothingCanVerify(this.committed.allRaw()); if (this.skillsInPayload > 0 && payloadId !== null) { // Learnt, not configured: nothing below the seam is told which payload // is which, so the payload that carried a skill put or revocation is @@ -1447,11 +1496,13 @@ export type FDv2SkillStoreOptions = { /** * Bounds the retry loop. On exceeding it the transport stops, logs an error, * and the store keeps serving last known good rather than pretending to be - * live — `failed` reports it. Only failures in a row count: reaching a working - * server resets the count, and a connection the server closes normally after - * that is exempt, so a stream being recycled never approaches the bound. A - * connection closed before any of that is a failure like any other, which is - * what bounds a server that does nothing but close connections. + * live — `failed` reports it. Only failures in a row count: a committed + * payload or a `none` intent resets the count, and a connection the server + * closes normally after that is exempt, so a stream being recycled never + * approaches the bound. A connection closed before either — including one + * that announced a transfer and dropped before committing it — is a failure + * like any other, which is what bounds a server that does nothing but close + * connections. */ readonly maxConsecutiveFailures?: number; /** Replaces the built-in `fetch` transport. Intended for testing. */ @@ -1517,15 +1568,17 @@ export class FDv2SkillStore implements SkillStore { private closed = false; private firstPayload = false; private readonly firstPayloadWaiters: Array<() => void> = []; - // Recoverable failures in a row, cleared by any sign of a working server: a - // `server-intent` that parsed, or a payload that committed. Not cleared when a - // connection returns, because a stream never returns normally — it only ends - // by being dropped, which is a failure, or by a goodbye, which is a failure - // only when the connection saying it never reached a working server. + // Recoverable failures in a row, cleared by a completed exchange with a + // working server: a payload that committed, or a `none` intent. Not cleared + // by a transfer intent that never commits, and not cleared when a connection + // returns, because a stream never returns normally — it only ends by being + // dropped, which is a failure, or by a goodbye, which is a failure only when + // the connection saying it never completed an exchange. private failures = 0; - // Whether the connection now open has shown a sign of a working server. Reset - // per attempt: it is what tells a stream being recycled from one that says - // goodbye having delivered nothing, and only the former escapes the bound. + // Whether the connection now open has completed an exchange with a working + // server. Reset per attempt: it is what tells a stream being recycled from + // one that says goodbye having delivered nothing, and only the former escapes + // the bound. private reachedServer = false; constructor(sdkKey: string, options: FDv2SkillStoreOptions = {}) { @@ -1535,8 +1588,9 @@ export class FDv2SkillStore implements SkillStore { throw new Error(`mode must be 'stream' or 'poll', got ${JSON.stringify(options.mode)}`); } this.pollIntervalMs = options.pollIntervalMs ?? 30_000; - if (this.pollIntervalMs <= 0) { - throw new Error(`pollIntervalMs must be positive, got ${JSON.stringify(options.pollIntervalMs)}`); + // `NaN` passes a bare `<= 0` guard, and `setTimeout(fn, NaN)` fires at once. + if (typeof this.pollIntervalMs !== 'number' || !Number.isFinite(this.pollIntervalMs) || this.pollIntervalMs <= 0) { + throw new Error(`pollIntervalMs must be a positive, finite number, got ${String(options.pollIntervalMs)}`); } const readTimeoutMs = options.readTimeoutMs ?? (this.mode === 'stream' ? DEFAULT_STREAM_READ_TIMEOUT_MS : DEFAULT_POLL_TIMEOUT_MS); @@ -1621,6 +1675,13 @@ export class FDv2SkillStore implements SkillStore { * answers `true` after close, matching what it will still serve. */ waitForSkills(timeoutMs = 10_000): Promise { + if (typeof timeoutMs !== 'number' || !Number.isFinite(timeoutMs) || timeoutMs < 0) { + // `setTimeout(fn, NaN)` fires at once, which would read as "timed out" — + // a wrong answer rather than a wrong wait. + return Promise.reject( + new Error(`waitForSkills timeoutMs must be a non-negative, finite number, got ${String(timeoutMs)}`), + ); + } if (this.firstPayload) return Promise.resolve(true); // A closed store, and delivery that has already stopped for good, both have // no payload left to wait for: answer now rather than after the timeout. @@ -1696,7 +1757,7 @@ export class FDv2SkillStore implements SkillStore { } /** - * Registers `fn` to be called once per committed change. + * Registers `fn` to be called for each changed object, at commit. * * Fires **once per changed object at payload-transferred**, not as objects * stream in, so a listener never observes a half-applied transfer. @@ -1751,16 +1812,23 @@ export class FDv2SkillStore implements SkillStore { // A copy, so a listener removed mid-commit does not shift its neighbours // out from under the iteration. const listeners = [...(this.listeners.get(SKILL_OBJECT_KIND) ?? [])]; + const report = (cause: unknown): void => { + error( + `A skill store change listener threw; delivery continues: ${ + cause instanceof Error ? cause.message : String(cause) + }`, + ); + }; for (const raw of changes) { for (const listener of listeners) { try { - listener(raw); + const result = listener(raw); + // An `async` listener rejects later rather than throwing now. Caught + // and logged the same way, so it cannot become an unhandled rejection + // — which in Node is a process-level event, not a delivery one. + if (result instanceof Promise) result.catch(report); } catch (cause) { - error( - `A skill store change listener threw; delivery continues: ${ - cause instanceof Error ? cause.message : String(cause) - }`, - ); + report(cause); } } } @@ -1774,9 +1842,11 @@ export class FDv2SkillStore implements SkillStore { this.reachedServer = false; if (this.mode === 'stream') await this.streamOnce(signal); else await this.pollOnce(signal); + // A return because the store is closing is not the server answering. + if (signal.aborted) return; // A poll that returned is a current answer even when it committed // nothing (HTTP 304). A stream never returns normally; its successes - // are counted in `apply`, at each intent and each commit. + // are counted in `apply`, at each `none` intent and each commit. this.recordSuccess(); } catch (cause) { if (signal.aborted) return; @@ -1857,10 +1927,11 @@ export class FDv2SkillStore implements SkillStore { private apply(name: string, data: unknown): TransferOutcome { const outcome = this.reader.handle(name, data); - // Any sign of a working server breaks the row of consecutive failures. A - // reconnect whose basis is already current is answered with the `none` - // intent and commits nothing, so waiting for a commit would leave an - // unchanging environment counting healthy connections against its bound. + // A completed exchange breaks the row of consecutive failures. A reconnect + // whose basis is already current is answered with the `none` intent and + // commits nothing, so waiting for a commit alone would leave an unchanging + // environment counting healthy connections against its bound — but an + // `xfer-*` intent is only a promise, so it does not count until it commits. if (outcome.healthy || outcome.committed) this.reachedServer = true; if (outcome.healthy) this.recordSuccess(); if (outcome.committed) { From 288e30214c3217ca04fbed71f4a8483bee7fab6b Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Thu, 17 Sep 2026 13:38:20 -0400 Subject: [PATCH 37/55] fix(client): keep the stale-selector repair out of the retry budget An HTTP 400 for a request carrying a `basis` selector or an etag means that state may be one the server no longer accepts, so the transport drops it and asks for a full transfer once before treating the status as fatal. That one request was not guaranteed to go out: it was prepared and then measured against the consecutive-failure bound like any other retry, so a 400 arriving on a budget an outage had already spent gave up while holding the only request known to repair it, and delivery stopped for the process lifetime over state the store had just dropped. The repair is now exempt from the bound. It cannot unbound the loop: the repaired request carries no state, so a second 400 is fatal on its own, and any other failure after it meets a budget still over the bound. Co-Authored-By: Claude Opus 5 --- .../client/src/__tests__/skills-fdv2.test.ts | 32 +++++++++++++++++++ packages/client/src/skills-fdv2.ts | 19 ++++++++--- 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index c5eb8675..138f5257 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -1886,6 +1886,38 @@ describe('failure handling', () => { expect(retry.ifNoneMatch).toBeUndefined(); }); + it('asks from scratch after a 400 even on a budget an outage has spent', async () => { + // The one repair available does not compete with the retry bound. A 400 + // arriving on a spent budget would otherwise give up while holding the one + // request known to fix it, and delivery would stop for the process lifetime + // over state the store was about to drop. + endpoint.queuePoll(fullPayload([['put-object', putSkill('first')]], 'basis-1'), { etag: 'etag-1' }); + endpoint.queuePoll([], { status: 500 }); + endpoint.queuePoll([], { status: 400 }); + endpoint.queuePoll(fullPayload([['put-object', putSkill('second')]], 'basis-2')); + const store = pollStore({ maxConsecutiveFailures: 1 }); + store.start(); + expect(await waitUntil(() => store.getObject(SKILL_OBJECT_KIND, 'second') !== null)).toBe(true); + expect(store.failed).toBeNull(); + // The repair went out from scratch rather than never going out at all. + const repair = endpoint.requests[3]; + expect(repair.query.basis).toBeUndefined(); + expect(repair.ifNoneMatch).toBeUndefined(); + }); + + it('meets the spent budget on a non-400 after the repair', async () => { + // The exemption is for the repair, not for the run that follows it. + endpoint.queuePoll(fullPayload([['put-object', putSkill()]], 'basis-1')); + endpoint.queuePoll([], { status: 500 }); + endpoint.queuePoll([], { status: 400 }); + endpoint.queuePoll([], { status: 500 }); + const store = pollStore({ maxConsecutiveFailures: 1 }); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toContain('gave up after 3 consecutive failures'); + expect(endpoint.requests).toHaveLength(4); + }); + it('stops on a 400 for a request that carried no basis', async () => { // Nothing left to drop: the request was already the from-scratch one, so // the endpoint is refusing the request itself. diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 028b9919..68b8ab58 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1499,10 +1499,11 @@ export type FDv2SkillStoreOptions = { * live — `failed` reports it. Only failures in a row count: a committed * payload or a `none` intent resets the count, and a connection the server * closes normally after that is exempt, so a stream being recycled never - * approaches the bound. A connection closed before either — including one - * that announced a transfer and dropped before committing it — is a failure - * like any other, which is what bounds a server that does nothing but close - * connections. + * approaches the bound, as is the one request built from nothing after a + * stale selector is refused. A connection closed before either — including + * one that announced a transfer and dropped before committing it — is a + * failure like any other, which is what bounds a server that does nothing + * but close connections. */ readonly maxConsecutiveFailures?: number; /** Replaces the built-in `fetch` transport. Intended for testing. */ @@ -1858,6 +1859,7 @@ export class FDv2SkillStore implements SkillStore { this.giveUp(`unexpected error in skill delivery: ${cause instanceof Error ? cause.message : String(cause)}`); return; } + let repairingState = false; if (cause instanceof StaleRequestStateError) { // The selector and etag are the only client state in the request, so // a rejection of a request carrying neither is the request itself @@ -1870,6 +1872,7 @@ export class FDv2SkillStore implements SkillStore { } this.basis = null; this.etag = null; + repairingState = true; } // A connection the server closed while serving it normally ended // without being a failure — see `dispatch` for which ones qualify. It @@ -1881,7 +1884,13 @@ export class FDv2SkillStore implements SkillStore { this.failures += 1; this.reader.diagnostics.connectionFailures = this.failures; this.reader.diagnostics.lastError = cause.message; - if (this.failures > this.maxConsecutiveFailures) { + // The one request built from nothing after a stale selector is + // refused is exempt from the bound, so an outage that has already + // spent the budget cannot swallow the one repair available. It + // cannot unbound the loop either: the repaired request carries no + // state, so a second 400 is fatal on its own, and any other failure + // after it meets a budget still over the bound. + if (this.failures > this.maxConsecutiveFailures && !repairingState) { this.giveUp(`gave up after ${this.failures} consecutive failures; last error: ${cause.message}`); return; } From 860b759ba9d0f3120447a35b7f0473e275536248 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 18 Sep 2026 16:32:01 -0400 Subject: [PATCH 38/55] fix(client): pair the poll etag with the basis it was issued against MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pollOnce` held one `etag` with no record of which request produced it, and adopted it before the body was applied. Two consequences: An ETag validates one representation of one resource, and the `basis` selector is part of the request that names it. Once a payload moved the basis on, the etag from the response before it was still offered, so a server validating it would be answering the question the store had stopped asking. Held as a pair now, and offered only while the pair holds — js-core keys its etag cache by request URL for the same reason. The cost is one unconditional request after each commit, which was never going to be a 304. A body that broke off partway — an `error` or `goodbye` after an announced transfer — left the payload it described unapplied, but its etag was already stored. The next `304` then reported that store as current and reset the failure count, where the 200 it replaced would have been retried and eventually given up on. Adopted after the body is applied in full. Co-Authored-By: Claude Opus 5 --- .../client/src/__tests__/skills-fdv2.test.ts | 36 ++++++++++++++++--- packages/client/src/skills-fdv2.ts | 33 ++++++++++++++--- 2 files changed, 60 insertions(+), 9 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 138f5257..5b02b5aa 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -1250,14 +1250,42 @@ describe('polling against the endpoint', () => { ]); }); - it('returns an ETag as If-None-Match', async () => { - endpoint.queuePoll(fullPayload([['put-object', putSkill()]]), { etag: 'W/"v1"' }); + it('returns an ETag as If-None-Match for the basis it was issued against', async () => { + // An ETag validates one representation of one resource, and the basis is + // part of the request that names it. `W/"v1"` answers the request that + // carried no basis at all, so it is not offered once the payload it came + // with moved the basis on; `W/"v2"` answers a request from `basis-1`, which + // is still the question being asked, so it is. + endpoint.queuePoll(fullPayload([['put-object', putSkill()]], 'basis-1'), { etag: 'W/"v1"' }); + endpoint.queuePoll(events(['server-intent', serverIntent('none')]), { etag: 'W/"v2"' }); endpoint.queuePoll([], { status: 304 }); const store = pollStore(); store.start(); await store.waitForSkills(5000); - expect(await waitUntil(() => endpoint.requests.length >= 2)).toBe(true); - expect(endpoint.requests[1].ifNoneMatch).toBe('W/"v1"'); + expect(await waitUntil(() => endpoint.requests.length >= 3)).toBe(true); + expect(endpoint.requests.slice(0, 3).map((r) => r.query.basis)).toEqual([undefined, 'basis-1', 'basis-1']); + expect(endpoint.requests.slice(0, 3).map((r) => r.ifNoneMatch)).toEqual([undefined, undefined, 'W/"v2"']); + }); + + it('does not offer the etag of a body it never applied', async () => { + // The body announced a transfer and then broke off, so the payload it + // described was never committed. Offering its etag would invite a `304` + // that reports a store still missing that payload as current and healthy — + // and unlike the 200 it replaces, a 304 carries nothing to notice that on. + endpoint.queuePoll( + events( + ['server-intent', serverIntent('xfer-full')], + ['put-object', putSkill()], + ['error', { reason: 'cut off mid-payload' }], + ), + { etag: 'W/"v1"' }, + ); + endpoint.queuePoll(fullPayload([['put-object', putSkill()]], 'basis-1'), { etag: 'W/"v2"' }); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(endpoint.requests[1].ifNoneMatch).toBeUndefined(); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); }); it('keeps held content across a 304', async () => { diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 68b8ab58..6c6e7fcd 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1559,6 +1559,15 @@ export class FDv2SkillStore implements SkillStore { private basis: string | null = null; private etag: string | null = null; + /** + * The basis the current `etag` was issued against. + * + * An ETag validates one representation of one resource, and the basis is part + * of the request that names it. Holding the pair is what lets `pollOnce` tell + * an etag that still answers the question it is about to ask from one that + * answers a question it has stopped asking. + */ + private etagBasis: string | null = null; private controller: AbortController | null = null; private loop: Promise | null = null; private failedReason: string | null = null; @@ -1872,6 +1881,7 @@ export class FDv2SkillStore implements SkillStore { } this.basis = null; this.etag = null; + this.etagBasis = null; repairingState = true; } // A connection the server closed while serving it normally ended @@ -1966,19 +1976,32 @@ export class FDv2SkillStore implements SkillStore { } private async pollOnce(signal: AbortSignal): Promise { - const result = await this.requester.poll(this.basis, this.etag, signal); - this.etag = result.etag; + const basis = this.basis; + // Only while the pair still holds. The basis is part of the request, so an + // etag issued before the basis moved validates a payload we have stopped + // asking for, and a server that answered it `304` would be answering the + // previous question. One unconditional request after each commit is the + // whole cost: a payload that changed was never going to be a 304 anyway. + const etag = this.etagBasis === basis ? this.etag : null; + const result = await this.requester.poll(basis, etag, signal); if (result.notModified) { // A 304 is a successful, current answer: the payload we hold is the payload - // the server has. It counts as a first payload so a boot that reconnects - // with a cached basis is not blocked on a transfer the server has no reason - // to send. + // the server has, because the etag that asked for it was issued for a body + // this store applied in full. It counts as a first payload so a boot that + // reconnects with a cached basis is not blocked on a transfer the server + // has no reason to send. this.markFirstPayload(); return; } for (const [name, data] of result.events) { this.dispatch(this.apply(name, data)); } + // Adopted only once the whole body has been applied. A body that threw + // partway — an `error` or `goodbye` after an announced transfer — left the + // payload it described unapplied, and keeping its etag would let the next + // `304` report a store that is missing that payload as current and healthy. + this.etag = result.etag; + this.etagBasis = basis; } private async streamOnce(signal: AbortSignal): Promise { From bd5e4dae41f8f26491464baf619a27ce070f1faf Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 30 Sep 2026 12:55:12 -0400 Subject: [PATCH 39/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20bound=20the=20SSE=20decoder=20per=20event?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The decoder held an unterminated line, and the accumulated `data:` lines of one event, without any limit: a server — or anything able to answer as one — could stream bytes that never framed and grow the buffer until the process died. Both are now bounded at 1 MiB per event (`MAX_SSE_EVENT_CHARS`) and exceeding it fails recoverably, so the store retries rather than aborting delivery for good. Co-Authored-By: Claude Fable 5.1 Co-Authored-By: Claude Opus 5 --- .../client/src/__tests__/skills-fdv2.test.ts | 90 +++++++++++++++++++ packages/client/src/skills-fdv2.ts | 34 ++++++- 2 files changed, 122 insertions(+), 2 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 5b02b5aa..5586e617 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -165,6 +165,9 @@ class FakeFDv2Endpoint { readonly requests: RecordedRequest[] = []; holdStreamOpen = false; dropStreams = false; + /** When set alongside `holdStreamOpen`, a `heart-beat` is sent on this interval. */ + heartbeatMs: number | null = null; + private readonly heartbeats = new Set>(); private readonly polls: Array<{ status: number; events: WireEvent[]; @@ -255,12 +258,22 @@ class FakeFDv2Endpoint { // Held so a test can assert on the store's state without racing the // reconnect path; released on `close`. this.held.add(res); + if (this.heartbeatMs !== null) { + const timer = setInterval(() => res.write('event: heart-beat\ndata: {}\n\n'), this.heartbeatMs); + this.heartbeats.add(timer); + res.on('close', () => { + clearInterval(timer); + this.heartbeats.delete(timer); + }); + } return; } res.end(); } async close(): Promise { + for (const timer of this.heartbeats) clearInterval(timer); + this.heartbeats.clear(); for (const res of this.held) res.end(); this.held.clear(); await new Promise((resolve) => this.server.close(() => resolve())); @@ -1397,6 +1410,40 @@ describe('SSE framing', () => { it('dispatches a named block with no data as a null payload', async () => { expect(await framed('event: heart-beat\n\n')).toEqual([['heart-beat', null]]); }); + + it('bounds an unterminated line and fails recoverably rather than buffering it without limit', async () => { + // A server (or a proxy) that never sends a newline would otherwise grow the + // line buffer until the process ran out of memory. Recoverable, so the + // connection is dropped and retried rather than the store giving up. + const chunks = new ReadableStream({ + start(controller) { + const chunk = new TextEncoder().encode('x'.repeat(256 * 1024)); + for (let i = 0; i < 6; i += 1) controller.enqueue(chunk); + controller.close(); + }, + }); + const drain = async (): Promise => { + for await (const _ of iterSse(chunks)) { + // nothing is expected to dispatch + } + }; + await expect(drain()).rejects.toBeInstanceOf(RecoverableTransportError); + }); + + it('bounds the accumulated data lines of one event the same way', async () => { + const line = `data: ${'y'.repeat(128 * 1024)}\n`; + const drain = async (): Promise => { + for await (const _ of iterSse(sseBody(`event: put-object\n${line.repeat(10)}`))) { + // nothing is expected to dispatch + } + }; + await expect(drain()).rejects.toBeInstanceOf(RecoverableTransportError); + }); + + it('still dispatches an event well inside the bound', async () => { + const big = JSON.stringify({ content: 'z'.repeat(64 * 1024) }); + expect(await framed(`event: put-object\ndata: ${big}\n\n`)).toEqual([['put-object', JSON.parse(big)]]); + }); }); describe('streaming against the endpoint', () => { @@ -3274,6 +3321,10 @@ describe('transport contract', () => { expect(source).not.toMatch(/FDV2_OBJECT_KIND\s*=\s*SKILL_OBJECT_KIND/); expect(FDV2_OBJECT_KIND).toBe(SKILL_OBJECT_KIND); }); + + it('the 401 message names the SDK key', () => { + expect(classifyStatus(401).message).toMatch(/SDK key/); + }); it('a goodbye on a store holding committed content keeps it', () => { const held = new SkillObjectSet(); const reader = new ProtocolReader(held); @@ -3286,6 +3337,18 @@ describe('transport contract', () => { reader.handle('goodbye', { reason: 'recycle', silent: true }); expect(held.size).toBe(1); }); + + it('bound exhaustion logs the error and keeps serving last known good, in one test', async () => { + const requester = new ScriptedRequester([asPairs(fullPayload([['put-object', putSkill()]]))]); + const store = scriptedStreamStore(requester, { maxConsecutiveFailures: 2 }); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(await waitUntil(() => store.failed !== null, 5000)).toBe(true); + expect(consoleErrors()).toMatch(/will not retry/); + expect(consoleErrors()).toMatch(/gave up after 3 consecutive failures/); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + expect(store.allObjects(SKILL_OBJECT_KIND)).toHaveProperty('pdf-extraction'); + }); it('holds a tampered object while the accessor reports integrity_failure', async () => { // Verification is the accessor's job, not the transport's: the store keeps // what arrived, and the reported outcome is what tells a caller to fail closed. @@ -3309,6 +3372,16 @@ describe('transport contract', () => { // `NaN` is the case a `<= 0` guard misses. expect(() => new FDv2SkillStore(SDK_KEY, { mode: 'poll', pollIntervalMs: value })).toThrow(/pollIntervalMs/); }); + + it('rejects a non-string credential', () => { + expect(() => new FDv2SkillStore(42 as never)).toThrow(/server-side SDK key/); + expect(() => new FDv2SkillStore(undefined as never)).toThrow(/server-side SDK key/); + }); + + it('refusal messages for mobile and client-side credentials say skills are server-side', () => { + expect(() => new FDv2SkillStore('mob-00000000-0000-4000-8000-000000000000')).toThrow(/server-side/); + expect(() => new FDv2SkillStore('0123456789abcdef01234567')).toThrow(/server-side/); + }); it.each([ Number.NaN, Number.NEGATIVE_INFINITY, @@ -3317,6 +3390,23 @@ describe('transport contract', () => { const store = pollStore(); await expect(store.waitForSkills(value)).rejects.toThrow(/timeoutMs/); }); + + it('an idle stream carrying heartbeats stays connected past what the read timeout alone would allow', async () => { + // The read deadline bounds the gap between reads, not the connection's + // life. Heartbeats well inside `readTimeoutMs` keep one connection open for + // several multiples of it, with no reconnect. + endpoint.holdStreamOpen = true; + endpoint.heartbeatMs = 20; + endpoint.queueStream(fullPayload([['put-object', putSkill()]])); + const store = streamStore({ readTimeoutMs: 100 }); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + await new Promise((resolve) => setTimeout(resolve, 450)); + expect(endpoint.requests).toHaveLength(1); + expect(store.failed).toBeNull(); + expect(store.diagnostics.connectionFailures).toBe(0); + expect(store.diagnostics.lastError).toBeNull(); + }); it('a stream interrupted by close does not count as a success', async () => { // `streamOnce` returning because the signal aborted is the store closing, // not the server answering — so the row of failures must not be reset by it. diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 6c6e7fcd..6f77ebd3 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1229,12 +1229,25 @@ function readFailure(cause: unknown, what: string, deadline: ReadDeadline | unde return new RecoverableTransportError(`${what} failed: ${cause instanceof Error ? cause.message : String(cause)}`); } +/** + * Bound on what the SSE decoder will hold for one event: the unterminated tail + * of the current line plus the accumulated `data:` lines, in UTF-16 code units. + * + * A skill object is a few kilobytes and the cap on content is 10 MiB after + * decoding, so a legitimate event is well inside this; a server or proxy that + * never sends a newline, or one event whose data never ends, would otherwise + * grow memory without limit. Exceeding it is a recoverable transport failure, + * so the connection is dropped and retried rather than the store giving up. + */ +export const MAX_SSE_EVENT_CHARS = 1024 * 1024; + /** * Decodes an SSE byte stream into `[event name, data]` pairs. * * Minimal on purpose — this consumes one LaunchDarkly endpoint, not the whole * spec: `event:`/`data:` fields, multi-line `data` joined with newlines, a blank - * line dispatching, and `:` comments skipped. + * line dispatching, and `:` comments skipped. Bounded by + * {@link MAX_SSE_EVENT_CHARS} per event. * * Only the read itself is wrapped as recoverable (see {@link readFailure}). * Whatever the consumer's loop body throws while this generator is suspended at @@ -1250,6 +1263,9 @@ export async function* iterSse( let buffer = ''; let name: string | null = null; let dataLines: string[] = []; + let dataChars = 0; + + const overBound = (): boolean => buffer.length + dataChars > MAX_SSE_EVENT_CHARS; // Every block that ends clears the buffered fields, whether or not it turns // into an event: a block with no `event:` field is the default `message` @@ -1260,6 +1276,7 @@ export async function* iterSse( const payload = dataLines.join('\n'); name = null; dataLines = []; + dataChars = 0; if (eventName === null) return null; if (payload === '') return [eventName, null]; try { @@ -1282,6 +1299,11 @@ export async function* iterSse( const { done, value } = chunk; if (done) break; buffer += decoder.decode(value, { stream: true }); + if (overBound()) { + throw new RecoverableTransportError( + `the FDv2 stream sent more than ${MAX_SSE_EVENT_CHARS} characters without completing an event`, + ); + } let newline = buffer.indexOf('\n'); while (newline !== -1) { const line = buffer.slice(0, newline).replace(/\r$/, ''); @@ -1295,7 +1317,15 @@ export async function* iterSse( let value2 = colon === -1 ? '' : line.slice(colon + 1); if (value2.startsWith(' ')) value2 = value2.slice(1); if (field === 'event') name = value2; - else if (field === 'data') dataLines.push(value2); + else if (field === 'data') { + dataLines.push(value2); + dataChars += value2.length + 1; + if (overBound()) { + throw new RecoverableTransportError( + `the FDv2 stream sent more than ${MAX_SSE_EVENT_CHARS} characters of data for one event`, + ); + } + } } newline = buffer.indexOf('\n'); } From 37fccfd474d2372d8252e51e8d2fdd64c9f1aa12 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Fri, 18 Sep 2026 15:24:35 -0400 Subject: [PATCH 40/55] fix(client): bound the transport per response, not per read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SSE decoder measured one predicate, `buffer.length + dataChars`, and checked it immediately after appending a read — before any line had been split. `buffer` at that moment is the whole unparsed remainder, so a read carrying several finished events was measured as though it were one oversized event and rejected. The rejection is deterministic, so every reconnect met it again until `maxConsecutiveFailures` was spent and delivery gave up for the process lifetime, frozen on last known good. The same conflation made the per-event check dead code: at every read size, from 16 KiB to a whole body in one read, the buffer branch fired first. The test that claimed to pin the accumulated `data:` accounting passed through the tail branch instead, and would have passed with that check deleted. They are two quantities and are now measured as two. The accumulated `data:` total is bounded on its own as each line lands. The unterminated tail is bounded once the split loop has drained every complete line in the read — at which point the tail really is the current event's next line, and adding the two is the honest measure of what one event holds. Two adjacent holes, found while confirming the above and fixed here because they are the same bound: - The bound was 1 MiB while verification caps content at 10 MiB, and content rides inline in the `put-object` envelope. A legitimate skill over ~1 MiB was therefore undeliverable through exactly the retry exhaustion above. It is now 64 MiB, the number Python already uses, and the docstring records why it has to clear the content cap and why the units cannot match Python's byte count exactly. - Polling read its body with `response.text()`, which is no bound at all: the allocation has happened by the time anything could measure it. It now reads through the same bound incrementally, matching Python's `_read_bounded`, and abandons a body as soon as it crosses. Reachability, for the record: `FetchRequester` feeds `iterSse` a `fetch` body, and undici caps its chunks at 64 KiB (16 KiB through gzip) however much the server wrote at once, so the multi-event read was not reachable through the shipped transport. The 1 MiB bound was. Each change has a test that fails without it: the burst test rejects a read past the bound whose events are each well inside it, the content test carries 11 MiB through the decoder, and the poll tests count reads to show the body is abandoned early rather than read whole and refused. Co-Authored-By: Claude Opus 5 --- .../client/src/__tests__/skills-fdv2.test.ts | 158 +++++++++++++++--- packages/client/src/skills-fdv2.ts | 111 ++++++++++-- 2 files changed, 231 insertions(+), 38 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 5586e617..3bf8785b 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -49,6 +49,7 @@ import { FetchRequester, isSkillEvent, iterSse, + MAX_RESPONSE_CHARS, type PollResult, ProtocolReader, RecoverableTransportError, @@ -1411,33 +1412,82 @@ describe('SSE framing', () => { expect(await framed('event: heart-beat\n\n')).toEqual([['heart-beat', null]]); }); - it('bounds an unterminated line and fails recoverably rather than buffering it without limit', async () => { - // A server (or a proxy) that never sends a newline would otherwise grow the - // line buffer until the process ran out of memory. Recoverable, so the - // connection is dropped and retried rather than the store giving up. - const chunks = new ReadableStream({ + /** + * Feeds `iterSse` the same text in reads of a fixed size, so a test can say + * which quantity the bound is being pushed past. `sseBody` hands the whole + * body over as one read, which is the one shape a real `fetch` never produces: + * undici caps its chunks at 64 KiB (16 KiB through gzip) however much the + * server wrote at once. + */ + const chunkedBody = (text: string, chunk: number): ReadableStream => + new ReadableStream({ start(controller) { - const chunk = new TextEncoder().encode('x'.repeat(256 * 1024)); - for (let i = 0; i < 6; i += 1) controller.enqueue(chunk); + const encoder = new TextEncoder(); + for (let i = 0; i < text.length; i += chunk) controller.enqueue(encoder.encode(text.slice(i, i + chunk))); controller.close(); }, }); - const drain = async (): Promise => { - for await (const _ of iterSse(chunks)) { - // nothing is expected to dispatch - } - }; - await expect(drain()).rejects.toBeInstanceOf(RecoverableTransportError); + + /** Enqueues exactly the reads given, so a test can place the read boundaries. */ + const readsOf = (...reads: string[]): ReadableStream => + new ReadableStream({ + start(controller) { + const encoder = new TextEncoder(); + for (const read of reads) controller.enqueue(encoder.encode(read)); + controller.close(); + }, + }); + + const drained = async (body: ReadableStream): Promise> => { + const seen: Array<[string, unknown]> = []; + for await (const event of iterSse(body)) seen.push(event); + return seen; + }; + + it('bounds an unterminated line and fails recoverably rather than buffering it without limit', async () => { + // A server (or a proxy) that never sends a newline would otherwise grow the + // line buffer until the process ran out of memory. Recoverable, so the + // connection is dropped and retried rather than the store giving up. + const newlineless = 'x'.repeat(MAX_RESPONSE_CHARS + 1024); + await expect(drained(chunkedBody(newlineless, 1024 * 1024))).rejects.toBeInstanceOf(RecoverableTransportError); }); it('bounds the accumulated data lines of one event the same way', async () => { - const line = `data: ${'y'.repeat(128 * 1024)}\n`; - const drain = async (): Promise => { - for await (const _ of iterSse(sseBody(`event: put-object\n${line.repeat(10)}`))) { - // nothing is expected to dispatch - } - }; - await expect(drain()).rejects.toBeInstanceOf(RecoverableTransportError); + // Every read here ends on a line boundary, so no tail is ever carried and the + // accumulated `data:` total is the only quantity that grows. That is what + // makes this the per-event check rather than the tail check: handed the whole + // body in one read, or in reads that straddle the lines, the tail crosses + // first and this would pass without the per-event accounting it pins. + const line = `data: ${'y'.repeat(1024 * 1024)}\n`; + const dataPerLine = line.length - 'data: '.length; + const lines = Math.ceil(MAX_RESPONSE_CHARS / dataPerLine) + 1; + const body = readsOf('event: put-object\n', ...Array.from({ length: lines }, () => line)); + await expect(drained(body)).rejects.toThrow(/characters of data for one event/); + }); + + it('accepts a read that delivered many finished events at once', async () => { + // The bound is per event, and the two quantities it measures must not be + // summed while a read is still being split: a burst whose events are each + // well inside the bound is not one oversized event, however much of it + // arrived together. Sized past the bound in total and nowhere near it per + // event, which is the only shape that tells the two apart. + const one = (i: number) => + `event: put-object\ndata: ${JSON.stringify({ i, pad: 'p'.repeat(4 * 1024 * 1024) })}\n\n`; + const count = Math.ceil(MAX_RESPONSE_CHARS / (4 * 1024 * 1024)) + 1; + const burst = Array.from({ length: count }, (_, i) => one(i)).join(''); + expect(burst.length).toBeGreaterThan(MAX_RESPONSE_CHARS); + expect(await drained(sseBody(burst))).toHaveLength(count); + }); + + it('accepts one event larger than the cap verification enforces on content', async () => { + // Content rides inline in the envelope, so the transport bound has to clear + // the 10 MiB content cap: a skill this size is verification's business to + // accept or withhold, and must reach it rather than being dropped as a + // framing failure and retried into the failure budget. + const big = JSON.stringify({ content: 'z'.repeat(11 * 1024 * 1024) }); + expect(big.length).toBeLessThan(MAX_RESPONSE_CHARS); + const framedEvents = await drained(chunkedBody(`event: put-object\ndata: ${big}\n\n`, 64 * 1024)); + expect(framedEvents).toEqual([['put-object', JSON.parse(big)]]); }); it('still dispatches an event well inside the bound', async () => { @@ -3305,6 +3355,74 @@ describe('endpoints', () => { expect(urls[0]).toBe('https://sdk.example.com/sdk/poll?basis=%28p%3Aa%3A1%29'); expect(urls[1]).toBe('https://stream.example.com/sdk/stream?basis=%28p%3Aa%3A1%29'); }); + + /** + * An oversized poll body, served in `chunk`-sized reads and counting how many + * were pulled. Finite on purpose, at a little past the bound: an unbounded + * reader then fails the read count rather than running the worker out of + * memory, so a regression here reads as an assertion and not as a crash. + */ + const oversizedPollBody = (chunk: number): { body: ReadableStream; reads: () => number } => { + const total = Math.ceil(MAX_RESPONSE_CHARS / chunk) + 8; + let reads = 0; + return { + reads: () => reads, + body: new ReadableStream({ + pull(controller) { + if (reads >= total) { + controller.close(); + return; + } + reads += 1; + controller.enqueue(new TextEncoder().encode('x'.repeat(chunk))); + }, + }), + }; + }; + + it('bounds a poll body rather than buffering whatever the server sends', async () => { + // `response.text()` would materialize the whole body and leave it to be + // measured after, which is no bound at all — the allocation has already + // happened. The read count is what pins that it stops early rather than + // reading to the end and rejecting the result. + const chunk = 1024 * 1024; + const { body, reads } = oversizedPollBody(chunk); + vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(body, { status: 200 })); + const requester = new FetchRequester(SDK_KEY, 'https://sdk.example.com', 1000); + await expect(requester.poll(null, null, new AbortController().signal)).rejects.toBeInstanceOf( + RecoverableTransportError, + ); + expect(reads()).toBeLessThanOrEqual(MAX_RESPONSE_CHARS / chunk + 2); + }); + + it('says nothing was applied when a poll body crosses the bound', async () => { + const { body } = oversizedPollBody(4 * 1024 * 1024); + vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(body, { status: 200 })); + const requester = new FetchRequester(SDK_KEY, 'https://sdk.example.com', 1000); + await expect(requester.poll(null, null, new AbortController().signal)).rejects.toThrow( + /exceeded the \d+ character transport bound.*nothing from it was applied/, + ); + }); + + it('reassembles a poll body that arrives across several reads', async () => { + // The bounded read decodes incrementally, so a multi-byte character split + // across two reads must not be mangled into replacement characters — that + // would corrupt content the hash is checked against. + const payload = JSON.stringify({ events: [{ event: 'put-object', data: { note: 'café — naïve' } }] }); + const bytes = new TextEncoder().encode(payload); + const split = bytes.indexOf(0xc3) + 1; // mid-sequence, inside 'é' + const body = new ReadableStream({ + start(controller) { + controller.enqueue(bytes.slice(0, split)); + controller.enqueue(bytes.slice(split)); + controller.close(); + }, + }); + vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(body, { status: 200 })); + const requester = new FetchRequester(SDK_KEY, 'https://sdk.example.com', 1000); + const result = await requester.poll(null, null, new AbortController().signal); + expect(result.events).toEqual([['put-object', { note: 'café — naïve' }]]); + }); }); // ─── Layering, and the absence of telemetry ────────────────────────────────── diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 6f77ebd3..98761f9a 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1122,6 +1122,52 @@ export type PollResult = { readonly etag: string | null; }; +/** + * Reads a whole poll body, holding no more than `limit` UTF-16 code units of it. + * + * Read in chunks rather than all at once so a body that is never going to be + * accepted is abandoned as soon as it crosses the bound, instead of being + * buffered whole by `response.text()` and measured after — which is no bound at + * all, because by then the allocation has already happened. + * + * Deliberately does not touch the caller's {@link ReadDeadline}. In `'poll'` + * mode that deadline bounds the whole request, body included; touching it per + * read would silently turn it into the per-read gap that `'stream'` mode wants + * and polling does not. + */ +async function readBoundedText(body: ReadableStream, limit: number): Promise { + const reader = body.getReader(); + const decoder = new TextDecoder(); + let text = ''; + try { + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + text += decoder.decode(value, { stream: true }); + if (text.length > limit) { + throw new RecoverableTransportError( + `polling response exceeded the ${limit} character transport bound ` + + `(at least ${text.length} received); nothing from it was applied`, + ); + } + } + // Flushes a truncated multi-byte sequence at the very end of the body as + // U+FFFD rather than dropping it, so a cut-short body fails in `JSON.parse` + // as the malformed payload it is. + return text + decoder.decode(); + } finally { + // Same contract as `iterSse`: cancelling closes the connection underneath, + // so a body abandoned at the bound does not leave a socket open behind the + // retry. Best effort — the stream may already be errored or closed. + reader.cancel().catch(() => {}); + try { + reader.releaseLock(); + } catch { + // Already released, or the stream errored. Nothing to recover. + } + } +} + /** * Unwraps `{"events": [...]}`. * @@ -1230,16 +1276,34 @@ function readFailure(cause: unknown, what: string, deadline: ReadDeadline | unde } /** - * Bound on what the SSE decoder will hold for one event: the unterminated tail - * of the current line plus the accumulated `data:` lines, in UTF-16 code units. + * The most the transport will hold in memory from one response, in UTF-16 code + * units: one whole poll body, or one streamed event — the unterminated tail of + * its current line plus its accumulated `data:` lines. + * + * A memory backstop, not a content limit. Verification caps each skill's content + * at 10 MiB (`MAX_SKILL_CONTENT_BYTES`) in `skills-core`, and content rides + * inline in the `put-object` envelope, so this bound has to sit *above* that one + * or a legitimate large skill becomes undeliverable: the decoder would reject + * it, the rejection is deterministic, and every retried connection would meet it + * again until the failure budget ran out and delivery gave up for the process + * lifetime. It is set far above any payload LaunchDarkly legitimately serves — + * the two bound different things and move independently. * - * A skill object is a few kilobytes and the cap on content is 10 MiB after - * decoding, so a legitimate event is well inside this; a server or proxy that - * never sends a newline, or one event whose data never ends, would otherwise - * grow memory without limit. Exceeding it is a recoverable transport failure, - * so the connection is dropped and retried rather than the store giving up. + * What it is really for is the unbounded case: a server or proxy that never sends + * a newline, one event whose data never ends, a poll body with no end in sight. + * Each of those would otherwise grow memory without limit. Crossing the bound is + * a recoverable transport failure, so the connection is dropped and retried and + * nothing from the payload in flight is committed, rather than the store giving + * up. + * + * The same number as Python's `MAX_RESPONSE_BYTES`, deliberately, so the two + * SDKs document one bound. The units are not identical and cannot be: Python + * counts bytes off the socket, this counts UTF-16 code units after decoding, so + * non-ASCII content is measured slightly differently either side. Acceptable in + * a backstop this far above real payloads; it would not be in a limit either SDK + * enforced as a contract. */ -export const MAX_SSE_EVENT_CHARS = 1024 * 1024; +export const MAX_RESPONSE_CHARS = 64 * 1024 * 1024; /** * Decodes an SSE byte stream into `[event name, data]` pairs. @@ -1247,7 +1311,7 @@ export const MAX_SSE_EVENT_CHARS = 1024 * 1024; * Minimal on purpose — this consumes one LaunchDarkly endpoint, not the whole * spec: `event:`/`data:` fields, multi-line `data` joined with newlines, a blank * line dispatching, and `:` comments skipped. Bounded by - * {@link MAX_SSE_EVENT_CHARS} per event. + * {@link MAX_RESPONSE_CHARS} per event. * * Only the read itself is wrapped as recoverable (see {@link readFailure}). * Whatever the consumer's loop body throws while this generator is suspended at @@ -1265,7 +1329,14 @@ export async function* iterSse( let dataLines: string[] = []; let dataChars = 0; - const overBound = (): boolean => buffer.length + dataChars > MAX_SSE_EVENT_CHARS; + // Two distinct quantities, deliberately not summed into one predicate. The + // data already accumulated for the current event is bounded on its own; the + // unterminated tail is bounded only once every complete line in the read has + // been consumed, at which point the tail is the current event's next line and + // adding the two is the honest measure of what one event holds. Summing them + // mid-split would measure whole finished events still waiting to be parsed. + const dataOverBound = (): boolean => dataChars > MAX_RESPONSE_CHARS; + const tailOverBound = (): boolean => buffer.length + dataChars > MAX_RESPONSE_CHARS; // Every block that ends clears the buffered fields, whether or not it turns // into an event: a block with no `event:` field is the default `message` @@ -1299,11 +1370,6 @@ export async function* iterSse( const { done, value } = chunk; if (done) break; buffer += decoder.decode(value, { stream: true }); - if (overBound()) { - throw new RecoverableTransportError( - `the FDv2 stream sent more than ${MAX_SSE_EVENT_CHARS} characters without completing an event`, - ); - } let newline = buffer.indexOf('\n'); while (newline !== -1) { const line = buffer.slice(0, newline).replace(/\r$/, ''); @@ -1320,15 +1386,20 @@ export async function* iterSse( else if (field === 'data') { dataLines.push(value2); dataChars += value2.length + 1; - if (overBound()) { + if (dataOverBound()) { throw new RecoverableTransportError( - `the FDv2 stream sent more than ${MAX_SSE_EVENT_CHARS} characters of data for one event`, + `the FDv2 stream sent more than ${MAX_RESPONSE_CHARS} characters of data for one event`, ); } } } newline = buffer.indexOf('\n'); } + if (tailOverBound()) { + throw new RecoverableTransportError( + `the FDv2 stream sent more than ${MAX_RESPONSE_CHARS} characters without completing an event`, + ); + } } } finally { deadline?.clear(); @@ -1402,9 +1473,13 @@ export class FetchRequester implements Requester { const response = await fetch(this.url(this.baseUri, POLL_PATH, basis), { headers, signal: deadline.signal }); if (response.status === 304) return { notModified: true, events: [], etag }; if (!response.ok) throw classifyStatus(response.status, response.headers); + // A 200 with no body at all is not a payload; `decodePollBody` rejects the + // empty string as the malformed response it is, under the same recoverable + // error as any other unusable body. + const body = response.body === null ? '' : await readBoundedText(response.body, MAX_RESPONSE_CHARS); return { notModified: false, - events: decodePollBody(await response.text()), + events: decodePollBody(body), etag: response.headers.get('ETag') ?? etag, }; } catch (cause) { From ae858c98f21cbc631c0425b94fd262c5bd8d6372 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 22 Sep 2026 15:18:41 -0400 Subject: [PATCH 41/55] fix(client): refuse cleartext base URIs and redirects in FDv2SkillStore Port the two FDv2 transport rules the Python AI SDK already enforces, so a deployment note or detection rule written against one SDK holds for the other. HTTPS only. `baseUri` and `streamUri` must be `https://` with a host; the constructor throws otherwise, whether or not a `Requester` was injected. The one exception is plain `http://` to a loopback host (`localhost`, `127.0.0.1`, `::1`), which never leaves the machine and is what a local test double listens on. The message explains that every request carries the server-side SDK key in `Authorization`, so cleartext is refused, and names the defaults. Redirects are never followed. Both fetches now pass `redirect: 'manual'`; `fetch`'s default `'follow'` copies `Authorization` onto the redirected request, so a 3xx from a proxy or a misconfigured private instance would hand the key to whatever host `Location` names. Any 3xx (304 excepted, and an `opaqueredirect` included) is routed through `classifyStatus` as a `FatalTransportError`, so `failed`, diagnostics and last-known-good behave exactly as for 401/403. A 304 on the poll path is settled before the redirect check and keeps its not-modified handling. Tests cover the scheme rules (refusal of http:// to a public or private host, of a non-URI and of a host-less URI; acceptance of the three loopback forms and of https://) and redirects on both /sdk/poll and /sdk/stream: fatal, not retried, reported by `failed`, committed content still served, and a second in-process host standing for `Location` sees no request at all, so no `Authorization` header. README, agents.md and the option JSDoc state both rules in the Python README's wording. Co-Authored-By: Claude Fable 5.1 --- packages/client/README.md | 7 +- packages/client/agents.md | 2 + .../client/src/__tests__/skills-fdv2.test.ts | 211 +++++++++++++++++- packages/client/src/skills-fdv2.ts | 122 +++++++++- 4 files changed, 336 insertions(+), 6 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index bb49b468..c61a19cf 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -329,6 +329,7 @@ example above, materializes only what the resolved variation actually asked for. | `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `isInitialized()`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | | `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object` that removed something, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation, and a tombstone for a key never held is reported but not counted), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | +| `DEFAULT_BASE_URI` / `DEFAULT_STREAM_URI` | `'https://sdk.launchdarkly.com'` and `'https://stream.launchdarkly.com'` — where `GET /sdk/poll` and `GET /sdk/stream` go by default. Overrides must be `https://` (plain `http://` only to a loopback host), and redirects from either are refused as a fatal failure. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | | `createSkillOutcome(init)` | Build a frozen `SkillOutcome`. Exported for tests and for wrapping your own retrieval in the same shape. | | `SKILL_FILENAME` | `'SKILL.md'`. | @@ -396,13 +397,17 @@ try { **Server-side only.** Skills are for server-side agent runtimes and skill content is customer-confidential. A mobile key (`mob-…`) or a client-side environment ID throws from the constructor. +**The SDK key goes only where you pointed it.** `baseUri` and `streamUri` must each be `https://` (plain `http://` is refused, except to a loopback host for a local test double), and redirects are never followed, so a 3xx from a proxy or a misconfigured private instance stops delivery rather than forwarding the key to whatever host the `Location` header names. + +**Polling and streaming have separate hosts.** LaunchDarkly serves `/sdk/poll` from `https://sdk.launchdarkly.com` and `/sdk/stream` from `https://stream.launchdarkly.com`, so the defaults are a pair. Pass `baseUri` on its own and it applies to both — what a relay or a private instance serving both endpoints from one host needs — or pass `streamUri` as well to override them independently. + **Streaming is the default, and it is what makes revocation fast.** A `delete-object` reaches a live stream in seconds; with `mode: 'poll'` it arrives within one `pollIntervalMs`. Paired with `watchSkills`, a revoked skill's `SKILL.md` leaves the disk without a restart. During an outage the store keeps serving the last content it received and `writeSkills`' default `onUnavailable: 'keep'` leaves managed files alone — an outage must not read as "everything was revoked". **One network timeout, and its default depends on the mode.** `readTimeoutMs` bounds every step of a request, connecting included. In `mode: 'poll'` it bounds the whole request and defaults to 10 seconds; in `mode: 'stream'` it bounds each wait for the next bytes and defaults to 300 seconds, well beyond LaunchDarkly's heartbeat interval. A stream that goes quiet past it reconnects rather than hanging, and a stream that dies mid-body — a reset, a truncated chunk — reconnects the same way. Every delay between retries, including one the server asks for with `Retry-After`, is capped at `maxBackoffMs`. **The connection also carries your flags.** A client cannot request only the skill payload, so a skills-enabled environment delivers flag and segment objects on the same connection. They are skipped, not evaluated — this store does no evaluation of any kind — and `diagnostics.objectsIgnored` counts them. -> **Beta caveats, worth knowing before you deploy.** Payload signing does not exist on this channel yet, so delivery is TLS-only and the content hash establishes self-consistency, not origin authenticity. The FDv2 protocol is opt-in per account: without it the endpoints return HTTP 403, which the store reports as a fatal error explaining what to do. `ld-relay` does not speak the FDv2 endpoints, so relay-only deployments cannot receive skills. +> **Beta caveats, worth knowing before you deploy.** Payload signing does not exist on this channel yet, so delivery is TLS-only — the store refuses a cleartext `baseUri`/`streamUri` and refuses to follow a redirect — and the content hash establishes self-consistency, not origin authenticity. The FDv2 protocol is opt-in per account: without it the endpoints return HTTP 403, which the store reports as a fatal error explaining what to do. `ld-relay` does not speak the FDv2 endpoints, so relay-only deployments cannot receive skills. **If every skill comes back empty, check `diagnostics.hashlessObjects`.** Verification requires `contentHash` on the delivered object and withholds anything without one, so a nonzero count there means hashless objects have arrived — skills are being withheld rather than the environment having none. It is a cumulative count of what the transport has seen, not the current size of the withheld set. The store logs an error per hashless object, and a summary whenever the store it holds becomes wholly hashless or the withheld objects change — both naming the reason, and neither repeated for a payload re-delivered unchanged. There is deliberately no fallback that skips verification: a hash the SDK computed from the content it was handed would certify the content against itself. diff --git a/packages/client/agents.md b/packages/client/agents.md index d1ee9470..1b088e26 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -75,6 +75,8 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b **What resets the failure counter, and what escapes it.** The counter resets on a completed exchange only — a payload that committed, or a `none` intent — not when a connection returns: a stream only ever ends by being dropped, so resetting on return would count every healthy, server-recycled connection as a failure. Waiting for a *commit* alone is not enough either, because a reconnect whose basis is already current is answered with the `none` intent and commits nothing, which would expire an environment whose skills never change. But a parsed `xfer-full` or `xfer-changes` intent does **not** reset it: an announced transfer is a promise, not a delivery, and a server that announces one and drops before `payload-transferred`, every time, has delivered nothing — counting the announcement as health would retry it forever at the initial backoff (the `gives up on a server that announces a transfer and drops` test pins this). A non-catastrophic `goodbye` is exempt from the counter, but only on a connection that had completed such an exchange, which is what `reachedServer` tracks per attempt: a server that says goodbye having sent no intent delivered nothing, and exempting it would let it reconnect without limit and without any of it reaching `diagnostics` or `failed`. Keep both halves — the reset and the `reachedServer` qualifier — or one of those two failures comes back. +**The SDK key goes only where it was pointed, and that is enforced in two places.** `requireHttpsUri` runs in the `FDv2SkillStore` constructor on both `baseUri` and `streamUri` (after the lone-`baseUri` resolution, and whether or not a `Requester` was injected): `https://` with a host, or plain `http://` to a loopback host only (`localhost`, `127.0.0.1`, `::1` — the test doubles listen there), and anything else throws with a message naming `https://` and the defaults. Then both fetches in `FetchRequester` are sent with `redirect: 'manual'`, and any 3xx — same host included, and an `opaqueredirect` if the runtime produces one — is a `FatalTransportError` through `classifyStatus`, so `failed`, diagnostics and last-known-good behave exactly as for 401/403. The reason is the same for both: every request carries the server-side SDK key in `Authorization`, and `fetch`'s default `'follow'` copies it onto whatever host `Location` names. A 304 is not a redirect and is settled before the redirect check. The Python SDK enforces the identical pair with the same wording, and a deployment note or detection rule written against one must hold for the other — change both or neither. + **HTTP 400 is recoverable exactly once, and only when there was state to drop.** The `basis` selector and the `If-None-Match` etag are the only client state in the request, and a selector the server no longer accepts is the one rejection the adapter can act on — so a 400 on a request that carried either arrives as `StaleRequestStateError`, the state is dropped, and a full transfer is requested from scratch; a second 400 is fatal. A 400 for a request carrying neither is the request itself being refused: retrying would send the byte-identical request, so it is fatal at once with no retry. Do not make 400 unconditionally fatal again: that strands delivery for the process lifetime on a stale selector. 405, 406, 414 and 501 stay fatal, since none of them is about client state. **`close` aborts the signal, it does not just set a flag.** The delivery task spends its life awaiting a stream read, and a flag it never checks would leave a healthy stream running until the process exited. The store's signal is the parent of every `ReadDeadline`, so one abort reaches a pending connect and a pending read alike, and a store closing is passed through rather than reported as a failure. Every backoff timer, deadline timer and the `waitForSkills` timer are `unref`ed for the same reason: a background store must not be why `node` stays up. diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 3bf8785b..854879ac 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -43,6 +43,7 @@ import { DEFAULT_STREAM_READ_TIMEOUT_MS, DEFAULT_STREAM_URI, decodePollBody, + FatalTransportError, FDV2_KEY_DELIMITER, FDV2_OBJECT_KIND, FDv2SkillStore, @@ -166,6 +167,8 @@ class FakeFDv2Endpoint { readonly requests: RecordedRequest[] = []; holdStreamOpen = false; dropStreams = false; + /** When set, `/sdk/stream` answers 307 with this `Location` instead of a body. */ + redirectStreamTo: string | null = null; /** When set alongside `holdStreamOpen`, a `heart-beat` is sent on this interval. */ heartbeatMs: number | null = null; private readonly heartbeats = new Set>(); @@ -174,6 +177,7 @@ class FakeFDv2Endpoint { events: WireEvent[]; etag?: string; retryAfter?: string; + location?: string; }> = []; private readonly streams: WireEvent[][] = []; private readonly held = new Set(); @@ -193,13 +197,14 @@ class FakeFDv2Endpoint { queuePoll( payloadEvents: WireEvent[] = [], - extra: { status?: number; etag?: string; retryAfter?: string } = {}, + extra: { status?: number; etag?: string; retryAfter?: string; location?: string } = {}, ): void { this.polls.push({ status: extra.status ?? 200, events: payloadEvents, etag: extra.etag, retryAfter: extra.retryAfter, + location: extra.location, }); } @@ -231,6 +236,7 @@ class FakeFDv2Endpoint { // Sent even when blank: a proxy that emits an empty `Retry-After` is a case // the store has to survive, so the fake has to be able to produce one. if (queued.retryAfter !== undefined) headers['Retry-After'] = queued.retryAfter; + if (queued.location !== undefined) headers.Location = queued.location; if (queued.status === 200) { const body = JSON.stringify({ events: queued.events }); res.writeHead(200, { ...headers, 'Content-Type': 'application/json' }); @@ -242,6 +248,11 @@ class FakeFDv2Endpoint { } private serveStream(res: ServerResponse): void { + if (this.redirectStreamTo !== null) { + res.writeHead(307, { Location: this.redirectStreamTo }); + res.end(); + return; + } const payloadEvents = this.streams.shift() ?? []; res.writeHead(200, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache' }); const body = payloadEvents @@ -284,6 +295,8 @@ class FakeFDv2Endpoint { // ─── Fixtures ──────────────────────────────────────────────────────────────── let endpoint: FakeFDv2Endpoint; +/** Extra hosts a test opened, to stand for wherever a `Location` header points. */ +let extraEndpoints: FakeFDv2Endpoint[]; let openStores: FDv2SkillStore[]; let tempRoots: string[]; let warnSpy: ReturnType; @@ -292,6 +305,7 @@ let errorSpy: ReturnType; beforeEach(async () => { endpoint = new FakeFDv2Endpoint(); await endpoint.listen(); + extraEndpoints = []; openStores = []; tempRoots = []; _clearState(); @@ -302,6 +316,7 @@ beforeEach(async () => { afterEach(async () => { for (const store of openStores) await store.close(); await endpoint.close(); + for (const extra of extraEndpoints) await extra.close(); for (const root of tempRoots) await rm(root, { recursive: true, force: true }); _clearState(); vi.restoreAllMocks(); @@ -332,6 +347,14 @@ function streamStore(options: Record = {}): FDv2SkillStore { return store; } +/** A second host, to stand for wherever a `Location` header points. */ +async function secondEndpoint(): Promise { + const second = new FakeFDv2Endpoint(); + await second.listen(); + extraEndpoints.push(second); + return second; +} + async function scratchRoot(): Promise { const root = await mkdtemp(path.join(tmpdir(), 'ld-skills-fdv2-')); tempRoots.push(root); @@ -3561,6 +3584,192 @@ describe('transport contract', () => { await store.close(); expect(store.diagnostics.connectionFailures).toBe(1); }); + + // Every request carries the SDK key in `Authorization`, so the base URI is + // `https://` only. Plain `http://` is allowed to a loopback host and nowhere + // else: that is what this suite's own endpoints listen on, and it never + // leaves the machine. + describe('base URI scheme', () => { + it('refuses a plain http:// baseUri, naming https://', () => { + expect(() => new FDv2SkillStore(SDK_KEY, { baseUri: 'http://example.com' })).toThrow(/cleartext/); + expect(() => new FDv2SkillStore(SDK_KEY, { baseUri: 'http://sdk.launchdarkly.com' })).toThrow(/https:\/\//); + }); + + it('refuses a plain http:// baseUri even with a requester injected', () => { + // The check is on the store, not on the socket it happens to open. + const requester = new ScriptedRequester([]); + expect(() => new FDv2SkillStore(SDK_KEY, { baseUri: 'http://relay.internal:8030', requester })).toThrow( + /cleartext/, + ); + }); + + it.each([ + 'http://localhost:8030', + 'http://127.0.0.1:8030', + 'http://[::1]:8030', + 'http://LOCALHOST/', + ])('allows plain http:// to a loopback host (%s)', (uri) => { + expect(requesterOf(new FDv2SkillStore(SDK_KEY, { baseUri: uri })).baseUri).toBe(uri.replace(/\/+$/, '')); + expect(requesterOf(new FDv2SkillStore(SDK_KEY, { streamUri: uri })).streamUri).toBe(uri.replace(/\/+$/, '')); + }); + + it('a private address is not loopback', () => { + // Only the machine itself is exempt; the LAN is not. + expect(() => new FDv2SkillStore(SDK_KEY, { baseUri: 'http://10.0.0.5:8030' })).toThrow(/cleartext/); + }); + + it.each([ + '', + ' ', + 'sdk.launchdarkly.com', + 'ftp://sdk.launchdarkly.com', + 'https://', + ])('refuses anything but an https:// URL with a host (%j)', (uri) => { + expect(() => new FDv2SkillStore(SDK_KEY, { baseUri: uri })).toThrow(/https:\/\//); + expect(() => new FDv2SkillStore(SDK_KEY, { streamUri: uri })).toThrow(/https:\/\//); + }); + + it('refuses a non-string URI', () => { + expect(() => new FDv2SkillStore(SDK_KEY, { baseUri: 42 as never })).toThrow(/https:\/\/.*none was given/); + }); + + it('accepts https://, and the defaults', () => { + expect(requesterOf(new FDv2SkillStore(SDK_KEY, { baseUri: 'https://sdk.example.com/' })).baseUri).toBe( + 'https://sdk.example.com', + ); + expect(requesterOf(new FDv2SkillStore(SDK_KEY)).baseUri).toBe(DEFAULT_BASE_URI); + }); + + it('refuses a plain http:// streamUri by name', () => { + // The streaming host is checked too, and the message names it. + expect( + () => + new FDv2SkillStore(SDK_KEY, { baseUri: 'https://sdk.example.com', streamUri: 'http://stream.example.com' }), + ).toThrow(/streamUri.*cleartext/); + }); + + it('the refusal names the defaults', () => { + expect(() => new FDv2SkillStore(SDK_KEY, { baseUri: 'http://example.com' })).toThrow( + /https:\/\/sdk\.launchdarkly\.com.*https:\/\/stream\.launchdarkly\.com/, + ); + }); + }); + + // `fetch`'s default `redirect: 'follow'` copies every request header onto + // the redirected request, `Authorization` included. Both requests are sent + // with `redirect: 'manual'` instead, so a 3xx is a fatal, non-retried failure + // and the SDK key never reaches the host `Location` names. + describe('redirects are refused', () => { + it.each([301, 302, 307, 308])('a poll redirect (%i) is fatal and not followed', async (status) => { + const second = await secondEndpoint(); + endpoint.queuePoll([], { status, location: `${second.baseUri}/sdk/poll` }); + const requester = new FetchRequester(SDK_KEY, endpoint.baseUri, 5000); + const failure = await requester.poll(null, null, new AbortController().signal).then( + () => null, + (cause: unknown) => cause, + ); + expect(failure).toBeInstanceOf(FatalTransportError); + expect((failure as Error).message).toContain(String(status)); + expect((failure as Error).message).toMatch(/redirect/i); + expect((failure as Error).message).toContain('not followed'); + expect(endpoint.requests).toHaveLength(1); + expect(second.requests).toEqual([]); + }); + + it('a stream redirect is fatal and not followed', async () => { + const second = await secondEndpoint(); + endpoint.redirectStreamTo = `${second.baseUri}/sdk/stream`; + const requester = new FetchRequester(SDK_KEY, endpoint.baseUri, 5000); + await expect(requester.stream(null, new AbortController().signal)).rejects.toThrow(FatalTransportError); + await expect(requester.stream(null, new AbortController().signal)).rejects.toThrow(/307.*redirect/); + expect(second.requests).toEqual([]); + }); + + it('a same-host redirect is refused too', async () => { + // The endpoints do not redirect, so there is nothing legitimate to follow. + endpoint.queuePoll([], { status: 302, location: `${endpoint.baseUri}/sdk/poll` }); + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const requester = new FetchRequester(SDK_KEY, endpoint.baseUri, 5000); + await expect(requester.poll(null, null, new AbortController().signal)).rejects.toThrow(FatalTransportError); + expect(endpoint.requests).toHaveLength(1); + }); + + it('routes a 3xx through classifyStatus as fatal, and 304 not at all', () => { + for (const status of [300, 301, 302, 303, 307, 308]) { + expect(classifyStatus(status)).toBeInstanceOf(FatalTransportError); + expect(classifyStatus(status).message).toMatch(/never forwarded/); + } + expect(classifyStatus(304)).not.toBeInstanceOf(FatalTransportError); + }); + + it('the SDK key never reaches the second host', async () => { + // End to end through the store: the redirect stops delivery for good, + // with no retry spent on it, and the second host sees no request at all — + // so no `Authorization` header, since that is what following would have + // forwarded. + const second = await secondEndpoint(); + endpoint.queuePoll([], { status: 301, location: `${second.baseUri}/sdk/poll` }); + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore(); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(await store.waitForSkills(5000)).toBe(false); + expect(store.failed).toContain('301'); + expect(store.failed).toContain('never forwarded'); + expect(consoleErrors()).toContain('redirect'); + expect(store.diagnostics.connectionFailures).toBe(0); + expect(endpoint.requests).toHaveLength(1); + expect(endpoint.requests[0]?.authorization).toBe(SDK_KEY); + expect(second.requests.map((r) => r.authorization)).toEqual([]); + }); + + it('a redirect in stream mode stops delivery', async () => { + const second = await secondEndpoint(); + endpoint.redirectStreamTo = `${second.baseUri}/sdk/stream`; + const store = streamStore(); + store.start(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toContain('307'); + expect(endpoint.requests).toHaveLength(1); + expect(second.requests).toEqual([]); + }); + + it('a redirect after a committed payload keeps serving last known good', async () => { + const second = await secondEndpoint(); + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + endpoint.queuePoll([], { status: 302, location: `${second.baseUri}/sdk/poll` }); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toContain('302'); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + expect(second.requests).toEqual([]); + }); + + it('a redirect with no Location is still fatal', async () => { + endpoint.queuePoll([], { status: 302 }); + const requester = new FetchRequester(SDK_KEY, endpoint.baseUri, 5000); + await expect(requester.poll(null, null, new AbortController().signal)).rejects.toThrow(FatalTransportError); + }); + + it('a 304 is not a redirect', async () => { + // The refusal must leave the poll's not-modified path exactly as it was. + endpoint.queuePoll([], { status: 304 }); + const requester = new FetchRequester(SDK_KEY, endpoint.baseUri, 5000); + const result = await requester.poll(null, 'etag-1', new AbortController().signal); + expect(result.notModified).toBe(true); + expect(result.etag).toBe('etag-1'); + expect(result.events).toEqual([]); + }); + + it('both fetches are sent with redirect: manual', () => { + // Pinned at the source, since the endpoint doubles cannot tell 'manual' + // from a runtime that happened not to follow. + expect(source.match(/^\s+redirect: 'manual',$/gm)).toHaveLength(2); + expect(source).not.toMatch(/redirect: 'follow'/); + }); + }); }); describe('listeners', () => { it('logs an async listener whose promise rejects rather than leaving it unhandled', async () => { diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 98761f9a..ae79ed4f 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -192,6 +192,52 @@ export function requireServerSideCredential(sdkKey: unknown): string { return key; } +/** The only hosts a plain `http://` URI may name: a local test double. */ +const LOOPBACK_HOSTS: ReadonlySet = new Set(['localhost', '127.0.0.1', '::1']); + +/** + * Refuses a URI that would send the SDK key in cleartext. + * + * Every request carries the environment's server-side SDK key in + * `Authorization`, so the transport is `https://` only. The one exception is + * `http://` to a loopback host (`localhost`, `127.0.0.1`, `::1`), which never + * leaves the machine and is what a local test double listens on. Throws rather + * than warns, for the same reason the credential check does: a store that would + * leak its key should not exist. + * + * Returns the URI trimmed. The check is on the store, not on the socket it + * happens to open, so it runs whether or not a `Requester` was injected. + */ +export function requireHttpsUri(uri: unknown, option: 'baseUri' | 'streamUri' = 'baseUri'): string { + if (typeof uri !== 'string' || uri.trim() === '') { + throw new Error(`FDv2SkillStore requires an https:// URI for ${option}; none was given.`); + } + const trimmed = uri.trim(); + let parsed: URL | null = null; + try { + parsed = new URL(trimmed); + } catch { + parsed = null; + } + // WHATWG `hostname` keeps the brackets on an IPv6 literal; the loopback list + // names the address, as the Python SDK's does. + const hostname = parsed?.hostname.replace(/^\[(.*)\]$/, '$1') ?? ''; + if (parsed?.protocol === 'https:' && hostname !== '') return trimmed; + if (parsed?.protocol === 'http:' && LOOPBACK_HOSTS.has(hostname)) return trimmed; + if (parsed?.protocol === 'http:') { + throw new Error( + `FDv2SkillStore refuses ${option} ${JSON.stringify(trimmed)}: a plain http:// URI would send the server-side ` + + 'SDK key in cleartext. Use https:// (the defaults are https://sdk.launchdarkly.com for polling and ' + + 'https://stream.launchdarkly.com for streaming). Plain http:// is allowed only for a loopback host ' + + '(localhost, 127.0.0.1, ::1) serving a local test double.', + ); + } + throw new Error( + `FDv2SkillStore refuses ${option} ${JSON.stringify(trimmed)}: expected an https:// URI with a host, such as ` + + 'https://sdk.launchdarkly.com.', + ); +} + // --------------------------------------------------------------------------- // Diagnostics — and the contentHash gap in particular // --------------------------------------------------------------------------- @@ -1089,6 +1135,37 @@ export function retryAfterMs(headers: Headers | null | undefined): number | null return Math.max(0, seconds * 1000); } +/** + * The fatal error for a 3xx. Both fetches are sent with `redirect: 'manual'`, + * because the default `'follow'` copies every request header onto the + * redirected request, `Authorization` included, so a 3xx from a proxy or a + * misconfigured private instance would hand the SDK key to whatever host + * `Location` names. Same-host redirects are refused too: the endpoints this + * module calls do not redirect, and a 304 is not a redirect and never reaches + * here. + */ +function redirectRefused(status: string): FatalTransportError { + return new FatalTransportError( + `LaunchDarkly returned ${status}, a redirect. Redirects are not followed, so the SDK key is never forwarded to a ` + + 'host other than the base URI. The SDK-facing FDv2 endpoints do not redirect; check the base URI, and any proxy ' + + 'in between, for the address being redirected to.', + ); +} + +/** + * The error for a response `fetch` answered with `redirect: 'manual'`, or + * `null` when it was not a redirect. A runtime that withholds the status of a + * redirect answers with an `opaqueredirect` response instead; that is still a + * redirect and still refused. + */ +function refusedRedirect(response: Response): FatalTransportError | null { + if (response.type === 'opaqueredirect') return redirectRefused('a 3xx status'); + if (response.status >= 300 && response.status < 400 && response.status !== 304) { + return classifyStatus(response.status, response.headers); + } + return null; +} + /** Turns an HTTP error status into the right error type. */ export function classifyStatus(status: number, headers?: Headers | null): Error { if (status === 401) { @@ -1098,6 +1175,7 @@ export function classifyStatus(status: number, headers?: Headers | null): Error ); } if (status === 403) return new FatalTransportError(`LaunchDarkly returned HTTP 403. ${FORBIDDEN_ADVICE}`); + if (status >= 300 && status < 400 && status !== 304) return redirectRefused(`HTTP ${status}`); if (status === 404) { return new FatalTransportError( 'LaunchDarkly returned HTTP 404 for the FDv2 endpoint. Check the base URI, and that this instance serves ' + @@ -1470,8 +1548,18 @@ export class FetchRequester implements Requester { // headers and body together. const deadline = readDeadline(signal, this.readTimeoutMs); try { - const response = await fetch(this.url(this.baseUri, POLL_PATH, basis), { headers, signal: deadline.signal }); + // `redirect: 'manual'`: a 3xx comes back as itself and is refused below, + // rather than being followed with the SDK key attached. + const response = await fetch(this.url(this.baseUri, POLL_PATH, basis), { + headers, + signal: deadline.signal, + redirect: 'manual', + }); + // A 304 is a current answer, not a redirect; it is settled before either + // check below can see it. if (response.status === 304) return { notModified: true, events: [], etag }; + const redirect = refusedRedirect(response); + if (redirect) throw redirect; if (!response.ok) throw classifyStatus(response.status, response.headers); // A 200 with no body at all is not a payload; `decodePollBody` rejects the // empty string as the malformed response it is, under the same recoverable @@ -1503,12 +1591,21 @@ export class FetchRequester implements Requester { const deadline = readDeadline(signal, this.readTimeoutMs); let response: Response; try { - response = await fetch(this.url(this.streamUri, STREAM_PATH, basis), { headers, signal: deadline.signal }); + response = await fetch(this.url(this.streamUri, STREAM_PATH, basis), { + headers, + signal: deadline.signal, + redirect: 'manual', + }); } catch (cause) { deadline.clear(); throw readFailure(cause, 'streaming request', deadline); } + const redirect = refusedRedirect(response); + if (redirect) { + deadline.clear(); + throw redirect; + } if (!response.ok) { deadline.clear(); throw classifyStatus(response.status, response.headers); @@ -1574,12 +1671,20 @@ export type FDv2SkillStoreOptions = { * Origin for `GET /sdk/poll`. Default {@link DEFAULT_BASE_URI}. When given * without `streamUri`, it is used for streaming too, which is what a relay or * private instance serving both endpoints from one host needs. + * + * Must be `https://`; the constructor throws otherwise, because every request + * carries the server-side SDK key in `Authorization`. Plain `http://` is + * accepted only for a loopback host (`localhost`, `127.0.0.1`, `::1`) serving + * a local test double. Redirects from it are never followed. */ readonly baseUri?: string; /** * Origin for `GET /sdk/stream`. Default {@link DEFAULT_STREAM_URI}, or * `baseUri` when that is given, since LaunchDarkly serves streaming from a * separate host but a relay or private instance usually does not. + * + * Held to the same rule as `baseUri`: `https://`, or plain `http://` to a + * loopback host only, and redirects from it are never followed. */ readonly streamUri?: string; readonly pollIntervalMs?: number; @@ -1635,6 +1740,12 @@ export type FDv2SkillStoreOptions = { * content is customer-confidential. A mobile key or a client-side environment ID * throws from the constructor. * + * **The SDK key goes only where it was pointed.** `baseUri` and `streamUri` must + * each be `https://` — plain `http://` is refused except to a loopback host, for + * local test doubles — and redirects are never followed, so a 3xx from a proxy + * or a private instance is a fatal failure rather than a request carrying the + * key to whatever host `Location` named. + * * **Delivery is in the background; retrieval is not.** A background task owns * the connection and fills memory, and `getObject` only ever reads what has * already arrived. A process that calls `getSkill` immediately after `start()` @@ -1715,10 +1826,13 @@ export class FDv2SkillStore implements SkillStore { this.initialBackoffMs = options.initialBackoffMs ?? 1_000; this.maxBackoffMs = options.maxBackoffMs ?? 30_000; this.maxConsecutiveFailures = options.maxConsecutiveFailures ?? 10; - const baseUri = options.baseUri ?? DEFAULT_BASE_URI; + const baseUri = requireHttpsUri(options.baseUri ?? DEFAULT_BASE_URI); // A custom `baseUri` alone means one host serves both endpoints; only the // LaunchDarkly defaults split them. - const streamUri = options.streamUri ?? (options.baseUri === undefined ? DEFAULT_STREAM_URI : baseUri); + const streamUri = requireHttpsUri( + options.streamUri ?? (options.baseUri === undefined ? DEFAULT_STREAM_URI : baseUri), + 'streamUri', + ); this.requester = options.requester ?? new FetchRequester(key, baseUri, readTimeoutMs, streamUri); } From 6baa20b50fe666f1d91ef41b62c7c63a659f8fec Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 30 Sep 2026 12:56:27 -0400 Subject: [PATCH 42/55] =?UTF-8?q?fix(client):=20Agent=20Skills=20=E2=80=94?= =?UTF-8?q?=20watcher,=20safe-fs=20and=20BYOC=20robustness?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The robustness remainder of the review: four independent fixes in files the rest of this stack does not touch. - `SkillWatcher.close()` marks itself closed and disarms the timer before detaching, so a change arriving mid-close cannot schedule a reconcile against a detached watcher. A `removeListener` that throws is logged rather than thrown, and an async `onReconcile` rejection is awaited and logged rather than left unhandled. - `safe-fs`: `isDescriptorAddressed` is exact equality against the pinned handle's address and throws on any other procfs shape, rather than matching a prefix. `atomicWrite` and `unlinkNoFollow` refuse a `name` that is not a single path component. On the `lstat` floor the pinned directory's identity is re-checked before the temp open too, not only before the rename. - `writeSkills` validates the request shape before creating the root, so a bare key string where an array was meant does not leave a freshly created root directory behind as a side effect of failing. A hostile key echoed into a `ReconcileAction.error` is truncated to 32 characters and quoted with `JSON.stringify`, so a 100 KB "key" cannot become a 100 KB error string. - The BYOC `initClient` overload passes its options bag through to telemetry setup instead of `{}`, so `otlpEndpoint`, `serviceName` and `environment` mean the same thing on both overloads. Docs: README documents the `watchSkills` surface, `SkillWatcher` and `DEFAULT_DEBOUNCE_MS`; agents.md gains the `debounceMs`/`timeout` unit pitfall and states the revocation bound without the watcher. `initClient`'s JSDoc now states the `skillStore` every-call asymmetry (§3.9). Co-Authored-By: Claude Fable 5.1 Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 23 +++- packages/client/agents.md | 18 ++- .../client/src/__tests__/lifecycle.test.ts | 17 +++ packages/client/src/__tests__/safe-fs.test.ts | 66 ++++++++++- .../client/src/__tests__/skills-fdv2.test.ts | 104 +++++++++++++++++- .../client/src/__tests__/skills-fs.test.ts | 78 ++++++++++++- packages/client/src/lifecycle.ts | 19 +++- packages/client/src/safe-fs.ts | 59 ++++++++-- packages/client/src/skills-fs.ts | 30 ++++- packages/client/src/skills-watch.ts | 37 +++++-- 10 files changed, 413 insertions(+), 38 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index c61a19cf..403bdfdb 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -326,10 +326,12 @@ example above, materializes only what the resolved variation actually asked for. | `allSkills()` | Every verified skill the store holds, newest version per key. | | `writeSkills(skills, root, options?)` | Materialize to `//SKILL.md`. Accepts `Skill` / `SkillReference` / key strings, or the literal `'*'`. Returns a `ReconcileReport`. Throws for a caller error — an unusable `root`, a bare string other than `'*'` — as distinct from the per-skill `error` actions in the report. | | `InMemorySkillStore` | A `SkillStore` backed by in-memory maps. `put(raw)`, `getObject(kind, key, version?)`, `allObjects(kind)`, `addListener(kind, fn)`, `removeListener(kind, fn)`. Several versions of one key coexist, as they do in a real payload: `getObject` answers a pin with exactly that version and an omitted version with the newest held, and `allObjects` returns one entry per `(key, version)` under keys opaque to callers. `addListener` throws for any kind but `'skill'`. | -| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `isInitialized()`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options: `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | -| `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }`; `await watcher.close()` when done, which also detaches the watcher from the store. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | +| `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `isInitialized()`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options (`FDv2SkillStoreOptions`): `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | +| `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }` — the initial reconcile's report and a `SkillWatcher`; `await watcher.close()` when done, which also detaches the watcher from the store. Options (`WatchSkillsOptions`): everything `writeSkills` takes, plus `debounceMs` (milliseconds, default `DEFAULT_DEBOUNCE_MS`) and `onReconcile`, called with each delivery-triggered report. Requires a store that implements `addListener`; one watcher per root. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | +| `SkillWatcher` | What `watchSkills` returns alongside the report: `reconciles` (re-reconciles completed, excluding the initial one), `notify` (the registered change listener), `close()` (idempotent; detaches and awaits any reconcile in flight). | | `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object` that removed something, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation, and a tombstone for a key never held is reported but not counted), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | | `DEFAULT_BASE_URI` / `DEFAULT_STREAM_URI` | `'https://sdk.launchdarkly.com'` and `'https://stream.launchdarkly.com'` — where `GET /sdk/poll` and `GET /sdk/stream` go by default. Overrides must be `https://` (plain `http://` only to a loopback host), and redirects from either are refused as a fatal failure. | +| `DEFAULT_DEBOUNCE_MS` | `500` — the `watchSkills` coalescing window in milliseconds. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | | `createSkillOutcome(init)` | Build a frozen `SkillOutcome`. Exported for tests and for wrapping your own retrieval in the same shape. | | `SKILL_FILENAME` | `'SKILL.md'`. | @@ -370,11 +372,20 @@ So on those two platforms, write permission on the managed root **and on its anc import { FDv2SkillStore, initClient, watchSkills } from '@launchdarkly/ai-server'; const store = new FDv2SkillStore(process.env.LD_SDK_KEY!).start(); -await store.waitForSkills(10_000); +if (!(await store.waitForSkills(10_000))) { + // No payload arrived. Reconciling now would find an empty store; see below. + console.warn(`skill delivery has not answered yet: ${store.failed ?? 'still waiting'}`); +} await initClient({ skillStore: store }); -// Materialize now, and re-materialize whenever delivery changes. -const { report, watcher } = await watchSkills('*', '.claude/skills'); +// Materialize now, and re-materialize whenever delivery changes. The report is +// the initial reconcile's; `onReconcile` sees the delivery-triggered ones. +const { report, watcher } = await watchSkills('*', '.claude/skills', { + debounceMs: 500, // the default: how long a burst of changes waits before one reconcile runs + onReconcile: (next) => { + if (!next.ok) for (const action of next.errors) console.error(`skill ${action.key}: ${action.error}`); + }, +}); try { // ... } finally { @@ -403,6 +414,8 @@ try { **Streaming is the default, and it is what makes revocation fast.** A `delete-object` reaches a live stream in seconds; with `mode: 'poll'` it arrives within one `pollIntervalMs`. Paired with `watchSkills`, a revoked skill's `SKILL.md` leaves the disk without a restart. During an outage the store keeps serving the last content it received and `writeSkills`' default `onUnavailable: 'keep'` leaves managed files alone — an outage must not read as "everything was revoked". +**Without the watcher, the revocation bound is process lifetime.** A deployment that calls `writeSkills` once at boot and never runs `watchSkills` reconciles exactly once, so a skill revoked in LaunchDarkly after boot stays on disk — and in the agent's context — until the process reconciles again. For such a deployment, a restart (or an explicit re-run of `writeSkills`) is the incident-response action when a skill must be pulled immediately. Neither path closes the already-loaded window: content an agent has already read stays in that conversation regardless, and no layer of this SDK can recall it. + **One network timeout, and its default depends on the mode.** `readTimeoutMs` bounds every step of a request, connecting included. In `mode: 'poll'` it bounds the whole request and defaults to 10 seconds; in `mode: 'stream'` it bounds each wait for the next bytes and defaults to 300 seconds, well beyond LaunchDarkly's heartbeat interval. A stream that goes quiet past it reconnects rather than hanging, and a stream that dies mid-body — a reset, a truncated chunk — reconnects the same way. Every delay between retries, including one the server asks for with `Retry-After`, is capped at `maxBackoffMs`. **The connection also carries your flags.** A client cannot request only the skill payload, so a skills-enabled environment delivers flag and segment objects on the same connection. They are skipped, not evaluated — this store does no evaluation of any kind — and `diagnostics.objectsIgnored` counts them. diff --git a/packages/client/agents.md b/packages/client/agents.md index 1b088e26..07d446cb 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -320,7 +320,7 @@ When `enabled` is `false`, `config` is always `null`. When `enabled` is `true` b - **Lazy initialization.** Importing the package does not initialize the LD client. The first API call that needs LaunchDarkly (`extractVariation`, graph resolution, etc.) calls `initClient()` internally, provided `LD_SDK_KEY` is set. - **Explicit initialization — Node SDK path.** `initClient(options?)` dynamically imports `@launchdarkly/node-server-sdk` at runtime (optional peer dep). If the package is not installed it throws with a clear message. -- **Explicit initialization — BYOC path.** `initClient(client, options?)` accepts any pre-initialized object that satisfies `LDClientInterface` — this is the path for Vercel, Cloudflare, or other edge runtimes whose SDK has different init semantics. No `@launchdarkly/node-server-sdk` is required. The optional second argument carries the same options bag as the other overload, which is how a BYOC caller configures `skillStore`. +- **Explicit initialization — BYOC path.** `initClient(client, options?)` accepts any pre-initialized object that satisfies `LDClientInterface` — this is the path for Vercel, Cloudflare, or other edge runtimes whose SDK has different init semantics. No `@launchdarkly/node-server-sdk` is required. The optional second argument carries the same options bag as the other overload and is passed through to telemetry setup whole, so `otlpEndpoint`, `serviceName` and `environment` mean the same thing here — and it is how a BYOC caller configures `skillStore`. - **`skillStore` is the one option applied on every call.** Every other option is ignored once the client singleton exists. `skillStore` is applied *before* the idempotency check, so a client that initialized lazily — or without a store — can be given one afterwards. A nullish `skillStore` never clears a configured store; only `shutdown()` does that, and it clears the skills state unconditionally, ahead of its own early return, because that state can exist without a client. - **Return value.** `initClient()` returns `Promise`. Callers that don't need the instance may discard the return value — this is a non-breaking change from the previous `Promise` signature. - `getClient()` throws if `initClient()` has not resolved — any code that calls `getClient()` directly must ensure initialization has occurred. @@ -374,6 +374,22 @@ Tier 0, so the runtime surface is deliberately tiny: two hard dependencies, and It is in **seconds**, defaulting to `10`. The signature is a cross-language contract that must match the Python SDK exactly, so the usual TypeScript `timeoutMs` instinct is wrong here. +### 4a. Passing `debounceMs` and `timeout` in the same unit + +`WatchSkillsOptions` is `WriteSkillsOptions` plus two fields, so one options bag carries `debounceMs` in **milliseconds** (the TypeScript convention) beside `timeout` in **seconds** (the cross-language contract from pitfall 4). `{ debounceMs: 500, timeout: 10 }` is the sane pair; `{ debounceMs: 0.5, timeout: 10_000 }` is a 1 ms coalescing window and a nearly three-hour reconcile budget. Both are guarded for sign and finiteness, not for plausibility. + +### 4b. Reading "no `removed` action" as "nothing is stale" + +Prune is **suppressed** — not merely empty — whenever the run cannot tell what is still current: an incomplete retrieval (a reference that did not resolve, a store that threw, an exhausted timeout), a store whose `isInitialized()` answers `false` (delivery has not sent a payload yet), a withholding that could not be attributed to a key (the `'*'` form's run-level `error`), or a corrupt manifest. In every one of those cases the report carries an `error` action and `ok` is `false`, and nothing has been pruned — including skills that genuinely were revoked. A caller that wants to know whether revocation has taken effect reads `ok` and `errors` first, not the absence of `removed`. + +### 4c. Two watchers on one root, or `writeSkills` on a watched root + +A reconcile's contract is one root, one reconcile at a time: two interleaved runs read the same manifest, each writes it back from its own picture, and the loser's entries vanish while the files it wrote stay on disk unmanaged. `SkillWatcher` chains its own reconciles so they never overlap, but it cannot see a second watcher on the same root or a caller's own `writeSkills` against it. Do neither. + +### 4d. Expecting revocation to reach a boot-only `writeSkills` deployment + +Without `watchSkills`, the revocation bound is process lifetime: a skill revoked after boot stays on disk until the process reconciles again, so a restart (or an explicit re-run of `writeSkills`) is the incident-response action — and content an agent has already read into a conversation is out of reach at this layer either way. + ### 5. Relaxing a path or manifest check in `skills-fs.ts` `keyRejectionReason` and `unsafePathReason` are shared by the write and prune paths precisely so the two cannot disagree about which paths this SDK may destroy, and both are **non-relaxable**. The same goes for the manifest rules: a destructive operation is permitted only on a path the manifest lists under a matching key, and a corrupt manifest suppresses every destructive action. Each of these has a dedicated abuse-case test in `src/__tests__/skills-fs.test.ts`; if one starts failing, the defense is what changed, not the test. diff --git a/packages/client/src/__tests__/lifecycle.test.ts b/packages/client/src/__tests__/lifecycle.test.ts index df2cdbcb..7327caa6 100644 --- a/packages/client/src/__tests__/lifecycle.test.ts +++ b/packages/client/src/__tests__/lifecycle.test.ts @@ -247,6 +247,23 @@ describe('lifecycle', () => { expect(mockTracerProviderRegister).toHaveBeenCalled(); }); + it('passes the BYOC overload options through to telemetry setup', async () => { + // The second argument is the same options bag as the other overload, so + // `otlpEndpoint` must reach the exporter rather than being dropped on the + // floor while only `skillStore` is read off it. + const byocClient = { + variation: vi.fn(), + track: vi.fn(), + flush: vi.fn().mockResolvedValue(undefined), + close: vi.fn().mockResolvedValue(undefined), + }; + const { initClient } = await import('../lifecycle.js'); + await initClient(byocClient, { otlpEndpoint: 'https://otlp.example.test/' }); + expect(mockOTLPTraceExporter).toHaveBeenCalledWith( + expect.objectContaining({ url: 'https://otlp.example.test/v1/traces' }), + ); + }); + it('is idempotent — calls init only once when called twice', async () => { const mockClient = makeMockClient(); mockLdInit.mockReturnValue(mockClient); diff --git a/packages/client/src/__tests__/safe-fs.test.ts b/packages/client/src/__tests__/safe-fs.test.ts index 15685594..63d4fdfc 100644 --- a/packages/client/src/__tests__/safe-fs.test.ts +++ b/packages/client/src/__tests__/safe-fs.test.ts @@ -39,6 +39,7 @@ vi.mock('node:fs/promises', async () => { const { atomicWrite, directoryAddress, + isDescriptorAddressed, openDirectoryNoFollow, openOrCreateDirectory, SUPPORTS_PROC_FD, @@ -206,11 +207,11 @@ describe('pinned-directory identity re-check', () => { await swapForSymlink(dir, outside); await expect(atomicWrite(dir, 'SKILL.md', Buffer.from('body\n', 'utf-8'), handle)).rejects.toThrow(/replaced/i); - // The temp file *is* created through the swapped link before the check fires - // (only the final component gets O_NOFOLLOW), so this proves the rename is - // refused and the temp cleaned up — not that the outside directory was never - // touched at all. + // The identity is re-checked *before* the temp file is opened as well as + // before the rename, so nothing — not even a temp file — is written + // through the swapped link. The outside directory was never touched. expect(await readdir(outside)).toEqual([]); + expect(tempOpens()).toEqual([]); } finally { await handle.close(); } @@ -330,6 +331,63 @@ describe.skipIf(!SUPPORTS_PROC_FD)('descriptor addressing on the /proc/self/fd f }); }); +describe('isDescriptorAddressed', () => { + // Pure string logic, ungated: it decides whether the identity re-check is + // skipped, so a lookalike must never be mistaken for the descriptor address. + it('accepts exactly the address built for this handle, and nothing else on procfs', async () => { + const handle = await openDirectoryNoFollow(scratch); + try { + expect(isDescriptorAddressed(`/proc/self/fd/${handle.fd}`, handle)).toBe(true); + expect(isDescriptorAddressed(scratch, handle)).toBe(false); + expect(isDescriptorAddressed('/tmp/proc/self/fd/3', handle)).toBe(false); + for (const shape of [ + '/proc/self/fd', + '/proc/self/fd/', + `/proc/self/fd/${handle.fd}/`, + `/proc/self/fd/${handle.fd}/child`, + `/proc/self/fd/${handle.fd + 1}`, + '/proc/self/fd/../../etc', + ]) { + expect(() => isDescriptorAddressed(shape, handle), shape).toThrow(/descriptor/i); + } + } finally { + await handle.close(); + } + }); +}); + +describe('single path component names', () => { + // `name` is the one argument that becomes part of a path this module writes + // or unlinks; a caller passing `../x` would be addressing outside the pinned + // directory. The check is structural rather than trusting every caller. + const badNames = ['', '.', '..', 'a/b', '../SKILL.md', 'SKILL.md/', '/SKILL.md']; + + it.each(badNames)('atomicWrite refuses name %j', async (name) => { + const dir = path.join(scratch, 'skill'); + await mkdir(dir); + const handle = await openDirectoryNoFollow(dir); + try { + await expect(atomicWrite(dir, name, Buffer.from('body\n'), handle)).rejects.toThrow(/single path component/); + } finally { + await handle.close(); + } + expect(tempOpens()).toEqual([]); + }); + + it.each(badNames)('unlinkNoFollow refuses name %j', async (name) => { + const dir = path.join(scratch, 'skill'); + await mkdir(dir); + await writeFile(path.join(dir, 'SKILL.md'), 'body\n', 'utf-8'); + const handle = await openDirectoryNoFollow(dir); + try { + await expect(unlinkNoFollow(dir, name, handle)).rejects.toThrow(/single path component/); + } finally { + await handle.close(); + } + expect(await readdir(dir)).toEqual(['SKILL.md']); + }); +}); + describe('unlinkNoFollow', () => { it('removes a real file', async () => { const dir = path.join(scratch, 'skill'); diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 854879ac..2f9a6bbf 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -2918,7 +2918,103 @@ describe('watchSkills', () => { getObject: () => null, allObjects: () => ({}), }); - await expect(watchSkills('*', await scratchRoot())).rejects.toThrow(/addListener/); + // The message names both remedies: the one-shot reconcile, and the store + // that does implement the listener half of the seam. + await expect(watchSkills('*', await scratchRoot())).rejects.toThrow(/writeSkills[\s\S]*FDv2SkillStore/); + }); + + it('passes prune and timeout straight through to writeSkills (§3.26)', async () => { + // Driven behaviourally rather than by spying on the import: a `prune: false` + // that reached `writeSkills` leaves a stale managed skill alone on the + // initial reconcile *and* on a re-reconcile, and a `timeout: 0` that reached + // it exhausts before retrieval — both are §3.22 outcomes only `writeSkills` + // produces. + const seed = new InMemorySkillStore(); + seed.put({ key: 'a', version: 1, content: 'first', contentHash: hash('first') }); + seed.put({ key: 'stale', version: 1, content: 'old', contentHash: hash('old') }); + _setStore(seed); + const root = path.join(await scratchRoot(), 'skills'); + expect((await writeSkills('*', root)).ok).toBe(true); + // Now a store that no longer holds `stale`: with pruning on it would be removed. + const store = new InMemorySkillStore(); + store.put({ key: 'a', version: 1, content: 'first', contentHash: hash('first') }); + _setStore(store); + + const { report, watcher } = await watchSkills('*', root, { debounceMs: 10, prune: false }); + try { + expect(report.ok).toBe(true); + expect(report.actions.some((a) => a.action === 'removed')).toBe(false); + expect(await readFile(path.join(root, 'stale', 'SKILL.md'), 'utf8')).toBe('old'); + + store.put({ key: 'a', version: 2, content: 'second', contentHash: hash('second') }); + expect(await waitUntil(() => watcher.reconciles === 1, 10_000)).toBe(true); + expect(await readFile(path.join(root, 'stale', 'SKILL.md'), 'utf8')).toBe('old'); + } finally { + await watcher.close(); + } + + const timed = await watchSkills([{ key: 'a', version: 2 }], root, { debounceMs: 10, timeout: 0 }); + try { + expect(timed.report.ok).toBe(false); + expect(timed.report.errors.map((a) => a.error).join('\n')).toMatch(/timeout was exhausted/); + } finally { + await timed.watcher.close(); + } + }); + + it('close() survives a store whose removeListener throws, and stays closed', async () => { + // Detaching is best effort: a store that cannot detach must not leave the + // watcher half-closed with its timer armed, and a second close is a no-op. + let notify: (() => void) | null = null; + _setStore({ + getObject: () => null, + allObjects: () => ({}), + addListener: (_kind: string, fn: () => void) => { + notify = fn; + }, + removeListener: () => { + throw new Error('cannot detach'); + }, + }); + const { watcher } = await watchSkills('*', path.join(await scratchRoot(), 'skills'), { debounceMs: 10 }); + await watcher.close(); + expect(consoleErrors()).toContain('cannot detach'); + const errorsAfterClose = errorSpy.mock.calls.length; + await watcher.close(); + expect(errorSpy.mock.calls.length).toBe(errorsAfterClose); + // Closed: a notification that still reaches it schedules nothing. + (notify as unknown as () => void)(); + await new Promise((resolve) => setTimeout(resolve, 60)); + expect(watcher.reconciles).toBe(0); + }); + + it('logs an async onReconcile that rejects, and the next commit still reconciles', async () => { + // `onReconcile` may be async. A rejection must be caught and logged like a + // synchronous throw — not left as an unhandled rejection — and must not + // stop the watcher. + const store = new InMemorySkillStore(); + store.put({ key: 'a', version: 1, content: 'first', contentHash: hash('first') }); + _setStore(store); + let calls = 0; + const root = path.join(await scratchRoot(), 'skills'); + const { watcher } = await watchSkills('*', root, { + debounceMs: 10, + onReconcile: async () => { + calls += 1; + if (calls === 1) throw new Error('async callback exploded'); + }, + }); + try { + store.put({ key: 'a', version: 2, content: 'second', contentHash: hash('second') }); + expect(await waitUntil(() => /async callback exploded/.test(consoleErrors()), 10_000)).toBe(true); + expect(consoleErrors()).toContain('callback threw'); + store.put({ key: 'a', version: 3, content: 'third', contentHash: hash('third') }); + expect(await waitUntil(() => watcher.reconciles === 2, 10_000)).toBe(true); + expect(calls).toBe(2); + expect(await readFile(path.join(root, 'a', 'SKILL.md'), 'utf8')).toBe('third'); + } finally { + await watcher.close(); + } }); it('throws when no store is configured', async () => { @@ -3466,6 +3562,7 @@ describe('transport contract', () => { it('the 401 message names the SDK key', () => { expect(classifyStatus(401).message).toMatch(/SDK key/); }); + it('a goodbye on a store holding committed content keeps it', () => { const held = new SkillObjectSet(); const reader = new ProtocolReader(held); @@ -3490,6 +3587,7 @@ describe('transport contract', () => { expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); expect(store.allObjects(SKILL_OBJECT_KIND)).toHaveProperty('pdf-extraction'); }); + it('holds a tampered object while the accessor reports integrity_failure', async () => { // Verification is the accessor's job, not the transport's: the store keeps // what arrived, and the reported outcome is what tells a caller to fail closed. @@ -3504,6 +3602,7 @@ describe('transport contract', () => { expect(outcome.skill).toBeNull(); expect(store.getObject(SKILL_OBJECT_KIND, 'tampered')).not.toBeNull(); }); + it.each([ 0, -1, @@ -3523,6 +3622,7 @@ describe('transport contract', () => { expect(() => new FDv2SkillStore('mob-00000000-0000-4000-8000-000000000000')).toThrow(/server-side/); expect(() => new FDv2SkillStore('0123456789abcdef01234567')).toThrow(/server-side/); }); + it.each([ Number.NaN, Number.NEGATIVE_INFINITY, @@ -3548,6 +3648,7 @@ describe('transport contract', () => { expect(store.diagnostics.connectionFailures).toBe(0); expect(store.diagnostics.lastError).toBeNull(); }); + it('a stream interrupted by close does not count as a success', async () => { // `streamOnce` returning because the signal aborted is the store closing, // not the server answering — so the row of failures must not be reset by it. @@ -3771,6 +3872,7 @@ describe('transport contract', () => { }); }); }); + describe('listeners', () => { it('logs an async listener whose promise rejects rather than leaving it unhandled', async () => { endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index 6aca1ef5..3ed7918d 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -45,6 +45,9 @@ const MANIFEST_NAME = '.launchdarkly-skills.json'; const SKILL_BODY = '---\nname: Test Skill\n---\nDo the thing.\n'; const INJECTED = 'simulated crash between write and rename'; +/** One over-cap string for the whole file: 10 MiB costs real time to allocate and hash. */ +const OVERSIZE = 'x'.repeat(MAX_SKILL_CONTENT_BYTES + 1); + const INTEGRITY_SIGNAL = 'AgentControl Skill Integrity Failure'; const MATERIALIZED_SIGNAL = 'AgentControl Skill Materialized'; const REVOKED_SIGNAL = 'AgentControl Skill Revoked Received'; @@ -634,8 +637,19 @@ describe('writeSkills bare-string guard', () => { await expect(writeSkills([skill('a')], root, { onUnavailable: 'explode' as 'keep' })).rejects.toThrow(); }); - it('rejects a negative timeout', async () => { - await expect(writeSkills([skill('a')], root, { timeout: -1 })).rejects.toThrow(); + it('rejects a negative timeout as a caller value error — the same class an unusable root raises', async () => { + const error = await writeSkills([skill('a')], root, { timeout: -1 }).catch((e: unknown) => e); + expect(error).toBeInstanceOf(Error); + expect(error).not.toBeInstanceOf(TypeError); + expect((error as Error).message).toContain('timeout'); + }); + + it('validates the request shape before creating the root directory', async () => { + // A bare string is refused before `resolveRoot` runs, so a mistaken call + // does not leave an empty directory behind as a side effect of failing. + const absent = path.join(scratch, 'never-created'); + await expect(writeSkills('pdf-extraction' as unknown as Skill[], absent)).rejects.toThrow(/"\*"/); + expect(await exists(absent)).toBe(false); }); }); @@ -1040,8 +1054,29 @@ describe('writeSkills verify-then-write', () => { expect(emitter.signals(INTEGRITY_SIGNAL)).toHaveLength(1); }); + it('truncates a hostile key echoed into the error to 32 characters', async () => { + // A key is attacker-reachable input, and a rejected one is echoed into + // `ReconcileAction.error`. The byte-limit branch already truncates; the + // grammar branch must too, or a 100 KB "key" becomes a 100 KB error string. + const hostile = 'A'.repeat(300); + const bad = createSkill({ + key: hostile, + version: 1, + content: new TextEncoder().encode('x'), + contentHash: hash('x'), + }); + + const report = await writeSkills([bad], root); + + expect(report.ok).toBe(false); + const message = report.errors[0].error ?? ''; + expect(message).toContain('A'.repeat(32)); + expect(message).not.toContain('A'.repeat(33)); + expect(message).toContain('...'); + }); + it('rejects an oversize Skill', async () => { - const oversize = 'x'.repeat(MAX_SKILL_CONTENT_BYTES + 1); + const oversize = OVERSIZE; const bad = createSkill({ key: 'a', version: 1, @@ -1517,6 +1552,41 @@ describe('writeSkills crash-mid-reconcile recovery', () => { expect(await readManifest(root)).toMatchObject({ entries: {} }); }); + it('bounds the adoption comparison read at content.byteLength + 1 (§3.22)', async () => { + // The read reaches arbitrary foreign files at a managed path, so a planted + // multi-GB file must not be pulled into memory before the hash is compared. + // One byte past the resolved content's length is all that is needed to + // prove inequality. + const planted = Buffer.byteLength(SKILL_BODY, 'utf-8') + 2 * 1024 * 1024; + const target = await placeOrphaned('a', `${SKILL_BODY}${'p'.repeat(2 * 1024 * 1024)}`); + const { open } = await import('node:fs/promises'); + const probe = await open(target, 'r'); + const prototype = Object.getPrototypeOf(probe) as { read: (...args: unknown[]) => Promise<{ bytesRead: number }> }; + await probe.close(); + const consumed: number[] = []; + const realRead = prototype.read; + const spy = vi.spyOn(prototype, 'read').mockImplementation(async function (this: unknown, ...args: unknown[]) { + const result = await realRead.apply(this, args); + consumed.push(result.bytesRead); + return result; + }); + + const report = await writeSkills([skill('a')], root); + + // `mockRestore` also resets the recorded history, so read it out first. + const reads = spy.mock.calls.length; + spy.mockRestore(); + expect(report.ok).toBe(false); + expect(actionsByKey(report).a.action).toBe('error'); + expect(actionsByKey(report).a.error).toMatch(/does not record it as managed/); + expect(reads).toBeGreaterThan(0); + const total = consumed.reduce((sum, n) => sum + n, 0); + expect(total).toBeGreaterThan(0); + expect(total).toBeLessThanOrEqual(Buffer.byteLength(SKILL_BODY, 'utf-8') + 1); + // Refused and untouched. + expect((await stat(target)).size).toBe(planted); + }); + it('an adopted file is prunable afterwards', async () => { // Adoption is a full claim, not a one-run exemption — and this is the test // that says so. Without it, "adopted" is indistinguishable from "tolerated @@ -2534,7 +2604,7 @@ describe('writeSkills telemetry', () => { // case reachable from both layers with the expected hash in hand throughout. const emitter = new RecordingEmitter(); _setEmitterForTesting(emitter); - const oversize = 'x'.repeat(MAX_SKILL_CONTENT_BYTES + 1); + const oversize = OVERSIZE; const contentHash = hash(oversize); // Layer 1 — the accessor boundary. diff --git a/packages/client/src/lifecycle.ts b/packages/client/src/lifecycle.ts index 99cb1278..7eab11a9 100644 --- a/packages/client/src/lifecycle.ts +++ b/packages/client/src/lifecycle.ts @@ -218,7 +218,15 @@ function isLDClient(value: unknown): value is LDClientInterface { * * **Overload 2 — pre-initialized client (edge / custom runtimes):** * Pass an already-initialized `LDClientInterface`-compatible client (e.g. from - * `@launchdarkly/vercel-server-sdk`) to bypass the Node SDK entirely. + * `@launchdarkly/vercel-server-sdk`) to bypass the Node SDK entirely. The + * optional second argument is the same options bag as the first overload. + * + * Idempotent for the client singleton: a second call returns the existing + * client and every other option is ignored — with one deliberate exception. + * **`skillStore` is applied on every call, before the idempotency check**, so a + * client that was lazily auto-initialized, or initialized without a store, can + * be given one afterwards with `initClient({ skillStore: store })`. A nullish + * `skillStore` never clears a configured store; `shutdown()` does that. * * Both overloads return the client instance for further customization. */ @@ -243,9 +251,12 @@ export async function initClient( if (isLDClient(optionsOrClient)) { // Pre-initialized client path (edge / custom runtimes). // Still run telemetry setup so OTel traces work regardless of which - // LD SDK is providing the client. SDK key is optional here — it's only - // used for the highlight.project_id resource attribute. - await setupTelemetry({}, process.env.LD_SDK_KEY ?? ''); + // LD SDK is providing the client, and with the caller's options — the + // second argument is the same bag as the other overload, so `otlpEndpoint`, + // `serviceName` and `environment` mean the same thing here. SDK key is + // optional here — it's only used for the highlight.project_id resource + // attribute. + await setupTelemetry(clientOptions ?? {}, clientOptions?.sdkKey ?? process.env.LD_SDK_KEY ?? ''); singleton.client = optionsOrClient; singleton.initPromise = Promise.resolve(optionsOrClient); return optionsOrClient; diff --git a/packages/client/src/safe-fs.ts b/packages/client/src/safe-fs.ts index e097f614..b7b583a1 100644 --- a/packages/client/src/safe-fs.ts +++ b/packages/client/src/safe-fs.ts @@ -75,9 +75,12 @@ const AT_FAMILY = ['renameat', 'unlinkat', 'openat'] as const; * `unlinkat`, or `openat`, and `FileHandle` has no `rename` or `unlink` — a * descriptor can be held, but nothing destructive can be addressed relative to * it. So this is `false` on every Node release to date, and a residual exposure - * follows: an attacker with write permission on the managed root can still swap a - * validated directory for a symlink between the identity check below and the - * path-based operation. + * follows **off {@link SUPPORTS_PROC_FD}** — that is, everywhere but Linux with + * procfs mounted: an attacker with write permission on the managed root can + * still swap a validated directory for a symlink between the identity check + * below and the path-based operation. On the procfs fast path the kernel + * resolves from the pinned inode and that window is closed; the module docblock + * lays out which platform gets which. * * Probed rather than hardcoded so the descriptor-relative path can be added behind * it if Node ever ships the family. @@ -155,9 +158,41 @@ export function directoryAddress(handle: FileHandle, realPath: string): string { return SUPPORTS_PROC_FD ? `${PROC_SELF_FD}/${handle.fd}` : realPath; } -/** Whether `directory` is a descriptor address rather than an ordinary path. */ -function isDescriptorAddressed(directory: string): boolean { - return directory === PROC_SELF_FD || directory.startsWith(`${PROC_SELF_FD}/`); +/** + * Whether `directory` is the descriptor address {@link directoryAddress} built + * for `handle` — exactly `/proc/self/fd/` — rather than an ordinary + * path. + * + * Exact equality, deliberately. This decides whether the identity re-check in + * {@link assertUnswapped} is *skipped*, so anything that merely looks like a + * procfs address — a different descriptor's, a child under it, a trailing + * slash, `..` traversal — must not be taken for one. Such a string cannot come + * from this module's own addressing, so it is a caller error and throws rather + * than quietly running the check (which would fail on procfs's magic symlink + * anyway) or quietly skipping it. Exported for its test; not API. + */ +export function isDescriptorAddressed(directory: string, handle: FileHandle): boolean { + if (directory === `${PROC_SELF_FD}/${handle.fd}`) return true; + if (directory === PROC_SELF_FD || directory.startsWith(`${PROC_SELF_FD}/`)) { + throw new Error( + `a descriptor address must be exactly ${PROC_SELF_FD}/ for the pinned handle; got ${JSON.stringify(directory)}`, + ); + } + return false; +} + +/** + * Refuses a `name` that is not a single path component. + * + * `name` becomes the final component of a path this module writes or unlinks + * under a pinned directory. A separator, `.` or `..` would address somewhere + * else — up or out of the pinned inode — and every caller in this package + * passes a literal, so a violation is a bug rather than an input to tolerate. + */ +function assertSingleComponent(name: string): void { + if (name === '' || name === '.' || name === '..' || name.includes('/') || name.includes('\\')) { + throw new Error(`name must be a single path component, got ${JSON.stringify(name)}`); + } } /** @@ -251,7 +286,7 @@ export async function openOrCreateDirectory(directory: string): Promise { - if (isDescriptorAddressed(directory)) return; + if (isDescriptorAddressed(directory, handle)) return; const pinned = await identityOf(handle); const onDisk = await lstat(directory, { bigint: true }); if (!onDisk.isDirectory() || Number(onDisk.dev) !== pinned.dev || onDisk.ino !== pinned.ino) { @@ -293,6 +328,12 @@ export function tempNamePattern(target: string): RegExp { * rename itself survives a crash. Mode is set on the *handle* rather than the * path, so it cannot be redirected by anything swapping the temp path underneath * us, and it is independent of the process umask. + * + * On the `lstat` floor the pinned directory's identity is re-checked **twice**: + * once before the temp file is opened, so content is never written through a + * directory name that has already been swapped for a link, and once again + * before the rename. Neither closes the window (see {@link assertUnswapped}); + * the first narrows what an attacker who wins the race gets to see written. */ export async function atomicWrite( directory: string, @@ -300,9 +341,12 @@ export async function atomicWrite( data: Uint8Array, pinned: FileHandle, ): Promise { + assertSingleComponent(name); const target = path.join(directory, name); const flags = fsConstants.O_WRONLY | fsConstants.O_CREAT | fsConstants.O_EXCL | (fsConstants.O_NOFOLLOW ?? 0); + await assertUnswapped(directory, pinned); + let temp = ''; let handle: FileHandle | null = null; for (let attempt = 0; attempt < TEMP_NAME_ATTEMPTS; attempt += 1) { @@ -352,6 +396,7 @@ export async function atomicWrite( * can intercept it. */ export async function unlinkNoFollow(directory: string, name: string, pinned: FileHandle): Promise { + assertSingleComponent(name); const target = path.join(directory, name); if ((await lstat(target)).isSymbolicLink()) throw new Error('the target file is a symlink'); await assertUnswapped(directory, pinned); diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index e5c9fb1e..52ffc4d8 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -224,6 +224,13 @@ export async function writeSkills( throw new Error(`timeout must be a non-negative number of seconds, got ${JSON.stringify(timeout)}`); } + // The request shape is validated before the root is resolved, so a mistaken + // call — a bare key string where an array was meant — does not leave a freshly + // created root directory behind as a side effect of failing. + if (typeof skills === 'string' && skills !== '*') { + throw new Error(`writeSkills takes an array of skills or the literal "*"; got ${JSON.stringify(skills)}`); + } + const deadline = performance.now() + timeout * 1000; const rootPath = await resolveRoot(root); @@ -354,7 +361,7 @@ async function writeAll( createReconcileAction({ key: request.key, action: 'error', - error: request.error ?? `skill '${request.key}' could not be resolved`, + error: request.error ?? `skill ${shownKey(request.key)} could not be resolved`, }), ); continue; @@ -491,6 +498,8 @@ async function resolveRequests( onUnavailable: OnUnavailable, ): Promise<{ requests: PendingWrite[]; incomplete: boolean }> { if (typeof skills === 'string') { + // `writeSkills` has already refused any string but '*', before the root + // was resolved; this is the same guard as a type narrowing. if (skills !== '*') { throw new Error(`writeSkills takes an array of skills or the literal "*"; got ${JSON.stringify(skills)}`); } @@ -793,6 +802,21 @@ async function loadManifest( // Per-skill reconcile // ------------------------------------------------------------------------- +/** How much of an attacker-reachable key an error message echoes. */ +const SHOWN_KEY_CHARS = 32; + +/** + * A key as it may appear in an error message: truncated to + * {@link SHOWN_KEY_CHARS}, because a key is attacker-reachable input and a + * rejected one is echoed into `ReconcileAction.error` — a 100 KB "key" must not + * become a 100 KB error string. Quoted with `JSON.stringify` so control + * characters are escaped rather than written into the log. + */ +function shownKey(key: unknown): string { + if (typeof key !== 'string') return JSON.stringify(key) ?? String(key); + return key.length > SHOWN_KEY_CHARS ? `${JSON.stringify(key.slice(0, SHOWN_KEY_CHARS))}...` : JSON.stringify(key); +} + /** * Why `key` must not become a directory name under the managed root, or `null`. * @@ -804,14 +828,14 @@ async function loadManifest( */ function keyRejectionReason(key: unknown): string | null { if (!isValidSkillKey(key)) { - return `${JSON.stringify(key)} is not a valid skill key (^[a-z0-9][a-z0-9-]*$, at most ${SKILL_KEY_MAX_LENGTH} characters)`; + return `${shownKey(key)} is not a valid skill key (^[a-z0-9][a-z0-9-]*$, at most ${SKILL_KEY_MAX_LENGTH} characters)`; } // The data model allows 256 characters; no mainstream filesystem allows a // 256-byte path component. Catch it here so it is a reported action rather than // an ENAMETOOLONG thrown from the first stat in the caller. const keyBytes = Buffer.byteLength(key, 'utf-8'); if (keyBytes > MAX_PATH_COMPONENT_BYTES) { - return `skill key '${key.slice(0, 32)}...' is ${keyBytes} bytes, over the ${MAX_PATH_COMPONENT_BYTES}-byte limit for a single directory name`; + return `skill key ${shownKey(key)} is ${keyBytes} bytes, over the ${MAX_PATH_COMPONENT_BYTES}-byte limit for a single directory name`; } // Same argument one step further: the grammar admits names Windows resolves as // devices instead of as paths. This layer is the right place for it — see diff --git a/packages/client/src/skills-watch.ts b/packages/client/src/skills-watch.ts index 261c64c7..dac52744 100644 --- a/packages/client/src/skills-watch.ts +++ b/packages/client/src/skills-watch.ts @@ -33,7 +33,12 @@ export const DEFAULT_DEBOUNCE_MS = 500; export type WatchSkillsOptions = WriteSkillsOptions & { /** Coalescing window, in **milliseconds**. Default {@link DEFAULT_DEBOUNCE_MS}. */ debounceMs?: number; - /** Called with each re-reconcile's report. Exceptions are logged, not thrown. */ + /** + * Called with each re-reconcile's report — the ones delivery triggers, not + * the initial reconcile, whose report `watchSkills` returns directly. May be + * `async`; a throw or a rejection is logged, not thrown, and does not stop + * the watcher. + */ onReconcile?: (report: ReconcileReport) => unknown; }; @@ -142,7 +147,9 @@ export class SkillWatcher { this.completed += 1; if (this.onReconcile) { try { - this.onReconcile(report); + // Awaited inside the try, so an `async` callback that rejects is logged + // like a synchronous throw rather than left as an unhandled rejection. + await Promise.resolve(this.onReconcile(report)); } catch (cause) { error(`A watchSkills callback threw: ${cause instanceof Error ? cause.message : String(cause)}`); } @@ -167,25 +174,37 @@ export class SkillWatcher { * case the manifest format has to recover from — worth avoiding when we control * the timing. * - * Detaches `notify` from the store first, so no further change reaches a - * watcher that is shutting down and the store no longer holds a reference to - * it. A store without the optional `removeListener` is left as it is rather - * than failing the close. + * Marks the watcher closed and disarms the timer *before* detaching, so a + * `removeListener` that throws cannot leave a watcher that is half-closed with + * a reconcile still scheduled. Detaching is best effort: a failure is logged, + * not thrown, and a store without the optional `removeListener` is left as it + * is rather than failing the close. Either way no further change reaches a + * watcher that is shutting down, because `notify` checks `closed` first. */ async close(): Promise { - this.detach(); + if (this.closed) { + await this.running; + return; + } this.closed = true; if (this.timer !== null) { clearTimeout(this.timer); this.timer = null; } + this.detach(); await this.running; } private detach(): void { - if (this.closed) return; - if (typeof this.store.removeListener === 'function') { + if (typeof this.store.removeListener !== 'function') return; + try { this.store.removeListener(SKILL_OBJECT_KIND, this.notify); + } catch (cause) { + error( + `The skill store's removeListener threw while a watcher was closing; the watcher is closed regardless: ${ + cause instanceof Error ? cause.message : String(cause) + }`, + ); } } } From 3b8dcd002a0a60df407c35222971e9eb35a30eff Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Thu, 24 Sep 2026 14:33:23 -0400 Subject: [PATCH 43/55] feat(client): declare ?kinds=agent-skill, and treat a 422 as "no skills yet" Delivery now narrows a connection to the payload kinds it declares and defaults to flags (launchdarkly/streamer#4730), so the store has to ask for the agent-skill payload or receive the environment's flags and no skills at all. The declaration goes on every request, before any basis exists as well as alongside one: it selects what the connection is served rather than describing what the store already holds. It also fixes something that was already wrong. A skill-enabled environment assigns two payloads, so the reader has been warning about the second and reading only the first intent, and the flag payload was re-downloaded and discarded on every reconnect because its basis was never adopted. Declaring one kind makes the connection single-payload, which is the shape the reader is built for. The 422 that comes with it is the interesting half. It is the answer when the credential is assigned no agent-skill payload, which is every project where no skill has ever been created -- gonfalon creates that row with the first skill and never lazily. As an ordinary recoverable failure it would spend maxConsecutiveFailures and then report "gave up after N consecutive failures: LaunchDarkly returned HTTP 422" for an ordinary configuration; as a fatal one, the skill created a minute later would never arrive without a process restart. So it is its own class: NoSkillPayloadError, which reuses the existing `expected` flag to stay off connectionFailures, lastError, failed and the per-attempt warning, is said once per store, counted under the new payloadUnavailable diagnostic, and retried at maxBackoffMs indefinitely. The retry is at the cap because `failures` deliberately never moves, so the exponential schedule would otherwise sit at the initial delay forever. Seven tests, each of which fails with the source reverted: the declaration on both endpoints and on both hosts, the two kind constants held apart by source text, the 422's classification, that it never stops delivery and never counts, that it waits the cap and not the initial delay, that a skill arriving after it is picked up, and that it leaves the store uninitialized so a wildcard reconcile prunes nothing. The fake endpoint gained a standing default status, since "every request is answered 422" is not something a queue can express. Also corrects docs that described the payload as classified `generic`, and the `mv` rationale: delivery overrides the requested model version for any non-flagging payload rather than refusing the connection over it. Gate: tsc --noEmit clean, biome check clean, vitest run 1024 passed / 10 skipped in packages/client (the ten are the capability-gated TOCTOU tests), and every workspace's suite green from the root. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 4 +- packages/client/agents.md | 10 +- .../client/src/__tests__/skills-fdv2.test.ts | 160 ++++++++++++++++-- packages/client/src/skills-fdv2.ts | 140 ++++++++++++--- 4 files changed, 274 insertions(+), 40 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 403bdfdb..b401a0b9 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -329,7 +329,7 @@ example above, materializes only what the resolved variation actually asked for. | `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `isInitialized()`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options (`FDv2SkillStoreOptions`): `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }` — the initial reconcile's report and a `SkillWatcher`; `await watcher.close()` when done, which also detaches the watcher from the store. Options (`WatchSkillsOptions`): everything `writeSkills` takes, plus `debounceMs` (milliseconds, default `DEFAULT_DEBOUNCE_MS`) and `onReconcile`, called with each delivery-triggered report. Requires a store that implements `addListener`; one watcher per root. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | | `SkillWatcher` | What `watchSkills` returns alongside the report: `reconciles` (re-reconciles completed, excluding the initial one), `notify` (the registered change listener), `close()` (idempotent; detaches and awaits any reconcile in flight). | -| `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object` that removed something, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation, and a tombstone for a key never held is reported but not counted), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | +| `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object` that removed something, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation, and a tombstone for a key never held is reported but not counted), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `payloadUnavailable` (requests LaunchDarkly answered with "no payload of the kind you asked for" — see below), `lastError`. | | `DEFAULT_BASE_URI` / `DEFAULT_STREAM_URI` | `'https://sdk.launchdarkly.com'` and `'https://stream.launchdarkly.com'` — where `GET /sdk/poll` and `GET /sdk/stream` go by default. Overrides must be `https://` (plain `http://` only to a loopback host), and redirects from either are refused as a fatal failure. | | `DEFAULT_DEBOUNCE_MS` | `500` — the `watchSkills` coalescing window in milliseconds. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | @@ -404,6 +404,8 @@ try { **`addListener` observes skill changes only.** The store notifies changes to `'skill'` objects; flag and segment objects on the same connection are skipped, never dispatched. A listener registered under any other kind therefore throws rather than being accepted and silently never firing. `removeListener` tolerates any kind, so a consumer that detaches on close can do so unconditionally. `watchSkills` is the intended consumer of both. +**A project with no skills yet is a waiting state, not a failure.** Every request declares the payload it wants (`kinds=agent-skill`), and LaunchDarkly answers HTTP 422 when the environment has no such payload — which is the case until somebody creates the first skill in the project. The store says so once, keeps asking at its backoff cap, and picks up that first skill without a restart. `failed` stays `null` throughout, `connectionFailures` stays at zero, and `diagnostics.payloadUnavailable` counts the answers: nonzero and rising beside an empty store means "there is nothing to deliver", not "delivery is broken". + **Nothing above the store changes.** The accessors, integrity verification, and `writeSkills` are transport-agnostic: they see raw objects through the `SkillStore` seam and cannot tell which store produced them. Everything documented above about verification and reconcile semantics applies unchanged. **Server-side only.** Skills are for server-side agent runtimes and skill content is customer-confidential. A mobile key (`mob-…`) or a client-side environment ID throws from the constructor. diff --git a/packages/client/agents.md b/packages/client/agents.md index 07d446cb..3b0249d0 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -51,11 +51,15 @@ The Agent Skills modules are a deliberate split, and the dependencies run **one "object":{"contentType":"text/markdown","content":"…","contentHash":"…","name":"…"}} ``` -The `3` after the delimiter is what a `{key, version}` reference pins and what becomes the seam's `version`, under the seam key `pdf-extraction`. `version` (42) is the version of the *payload* the object arrived in — it moves when anything in the environment moves, including a flag with nothing to do with skills. Reading it as the skill's version fails **silently**: the object verifies, the hash matches, and the caller gets content under a version number that means nothing. There is no separate field for the skill's version: the agent-skill payload is a *generic* payload, and generic objects carry only `key`, `kind`, `version` and `object`, exactly like a flag. `splitWireKey` is the only place the wire key is read, `seamObjectFromPut` and `tombstoneFromDelete` both go through it, and the `version translation` suite asserts the translation in both directions. A wire key that will not split cleanly is *held*, not dropped — version-less, or with the offending text as its version — so verification withholds it with `invalid_version` under a key the caller recognises; only a key with nothing before the delimiter is dropped, since there is no identity to hold it under. +The `3` after the delimiter is what a `{key, version}` reference pins and what becomes the seam's `version`, under the seam key `pdf-extraction`. `version` (42) is the version of the *payload* the object arrived in — it moves when anything in the environment moves, including a flag with nothing to do with skills. Reading it as the skill's version fails **silently**: the object verifies, the hash matches, and the caller gets content under a version number that means nothing. There is no separate field for the skill's version: the agent-skill payload is opaque to delivery, and its objects carry only `key`, `kind`, `version` and `object`, exactly like a flag. `splitWireKey` is the only place the wire key is read, `seamObjectFromPut` and `tombstoneFromDelete` both go through it, and the `version translation` suite asserts the translation in both directions. A wire key that will not split cleanly is *held*, not dropped — version-less, or with the offending text as its version — so verification withholds it with `invalid_version` under a key the caller recognises; only a key with nothing before the delimiter is dropped, since there is no identity to hold it under. -**Skills are identified by `kind === 'skill'`; everything else is ignored, not rejected.** Object kinds on the SDK-facing channel are open strings, and the agent-skill payload is classified `generic`, so a skill arrives under the kind its producer registered — the bare category name — not under a broader wrapper kind with a narrowing field. An environment's payload assignment carries its flag payload alongside its agent-skill payload, so flag and segment objects arrive as a matter of course. Throwing on an unrecognised kind is the unknown-kind reconnect loop this feature must not reproduce — a flag-delivery outage caused by a skills rollout. +**Skills are identified by `kind === 'skill'`; everything else is ignored, not rejected.** Object kinds on the SDK-facing channel are open strings, so a skill arrives under the kind its producer registered — the bare category name — not under a broader wrapper kind with a narrowing field. The `kinds` declaration below means a flag or segment object should no longer arrive at all, but the skip stays and stays tested: throwing on an unrecognised kind is the unknown-kind reconnect loop this feature must not reproduce — a flag-delivery outage caused by a skills rollout — and an object nobody expected belongs outside the skill set rather than in it. -**No `mv` parameter, deliberately.** It selects the *flag* data model, the connection rejects any value but the flag default, and the generic agent-skill payload is served regardless of it. Sending `mv=1` — the skill payload's own model version — gets the whole connection refused. +**Every request declares `kinds=agent-skill`, and that is load-bearing.** Delivery narrows a connection to the payload kinds it declares and defaults to flags, so a request without it is served the environment's flag payload and no skills at all — the store would run, report healthy, and hold nothing. It also makes the connection carry exactly one payload, which is the shape `ProtocolReader` is built for: without it, a skill-enabled environment assigns two, and the reader warns about the second and reads only the first intent. `FDV2_PAYLOAD_KIND` is the payload's kind and `FDV2_OBJECT_KIND` the kind of the objects inside it — two different strings, held apart on purpose, and asserted apart by source text so an alias cannot pass. + +**No `mv` parameter, deliberately.** It selects the *flag* data model, and delivery overrides whatever a request asks for with the payload's own default for any non-flagging payload, so sending it would state a preference that is ignored. + +**HTTP 422 is not a failure.** It is the answer to that declaration when the credential is assigned no agent-skill payload, which is every project in which no skill has ever been created — gonfalon creates that payload row with the first skill and never lazily. So `classifyStatus` maps it to `NoSkillPayloadError`, which the delivery loop treats as its own case: counted under `diagnostics.payloadUnavailable`, logged once per store, retried at `maxBackoffMs` indefinitely, and (through the existing `expected` flag) kept off `connectionFailures`, `lastError` and `failed`. Neither obvious classification is right — as a recoverable failure it spends `maxConsecutiveFailures` and then gives up permanently on an ordinary configuration; as a fatal one, the skill created a minute later never arrives without a process restart. The retry is at the *cap* rather than the initial backoff because `failures` deliberately never moves, so the exponential schedule would sit at the initial delay forever. **Polling and streaming have different default hosts.** `GET /sdk/poll` is served from `DEFAULT_BASE_URI` (`sdk.launchdarkly.com`) and `GET /sdk/stream` from `DEFAULT_STREAM_URI` (`stream.launchdarkly.com`), matching the base server-side SDK. Sending the stream request to the polling host fails against a real environment, and the fake endpoint cannot catch it because it serves both from one origin. `baseUri` given alone therefore applies to both, since a relay or private instance usually serves both from one host; only the LaunchDarkly defaults split them, and `streamUri` overrides the stream origin on its own. diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 2f9a6bbf..079203fd 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -18,7 +18,7 @@ import { createHash } from 'node:crypto'; import { existsSync, readFileSync } from 'node:fs'; -import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'; import { createServer as createTcpServer, type Socket, type Server as TcpServer } from 'node:net'; import { tmpdir } from 'node:os'; @@ -46,11 +46,13 @@ import { FatalTransportError, FDV2_KEY_DELIMITER, FDV2_OBJECT_KIND, + FDV2_PAYLOAD_KIND, FDv2SkillStore, FetchRequester, isSkillEvent, iterSse, MAX_RESPONSE_CHARS, + NoSkillPayloadError, type PollResult, ProtocolReader, RecoverableTransportError, @@ -165,6 +167,13 @@ type RecordedRequest = { */ class FakeFDv2Endpoint { readonly requests: RecordedRequest[] = []; + /** + * What a poll is answered with once the queued script runs out. 304 — + * "nothing has changed" — is the right default for a healthy environment, but + * a standing 422 is what an environment with no skill payload answers *every* + * request with, and a queue cannot express "every". + */ + defaultPollStatus = 304; holdStreamOpen = false; dropStreams = false; /** When set, `/sdk/stream` answers 307 with this `Location` instead of a body. */ @@ -230,7 +239,7 @@ class FakeFDv2Endpoint { } private servePoll(res: ServerResponse): void { - const queued = this.polls.shift() ?? { status: 304, events: [] }; + const queued = this.polls.shift() ?? { status: this.defaultPollStatus, events: [] }; const headers: Record = {}; if (queued.etag) headers.ETag = queued.etag; // Sent even when blank: a proxy that emits an empty `Retry-After` is a case @@ -1218,10 +1227,10 @@ describe('polling against the endpoint', () => { }); it('sends the SDK key and no data model version', async () => { - // No `mv`: that parameter selects the *flag* data model, the connection - // rejects any value but the flag default, and the generic agent-skill - // payload is served regardless of it. Sending `mv=1` — the skill payload's - // own model version — gets the whole connection refused. + // No `mv`: that parameter selects the *flag* data model, and delivery + // overrides whatever a request asks for with the payload's own default for + // any non-flagging payload. Sending it would state a preference that is + // ignored, so the store states none. endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); const store = pollStore(); store.start(); @@ -1231,6 +1240,24 @@ describe('polling against the endpoint', () => { expect('mv' in endpoint.requests[0].query).toBe(false); }); + it('declares the agent-skill payload kind on every request', async () => { + // Delivery narrows a connection to the kinds it declares and defaults to + // flags, so a request without this is served the environment's flag payload + // and no skills at all. It is on the first request as well as the ones + // after it: the declaration selects what the connection is served rather + // than describing what it already holds, so there is no state to wait on. + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + endpoint.queuePoll([], { status: 304 }); + const store = pollStore(); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(await waitUntil(() => endpoint.requests.length >= 2)).toBe(true); + expect(endpoint.requests[0].query.kinds).toBe('agent-skill'); + expect(endpoint.requests[0].query.basis).toBeUndefined(); + expect(endpoint.requests[1].query.kinds).toBe('agent-skill'); + expect(endpoint.requests[1].query.basis).toBe('basis-1'); + }); + it('sends no basis on the first request', async () => { endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); const store = pollStore(); @@ -1539,6 +1566,15 @@ describe('streaming against the endpoint', () => { expect(endpoint.requests[0].accept).toBe('text/event-stream'); }); + it('declares the agent-skill payload kind on the stream request too', async () => { + endpoint.holdStreamOpen = true; + endpoint.queueStream(fullPayload([['put-object', putSkill()]])); + const store = streamStore(); + store.start(); + await store.waitForSkills(5000); + expect(endpoint.requests[0].query.kinds).toBe('agent-skill'); + }); + it('applies a streamed revocation without a restart', async () => { endpoint.holdStreamOpen = true; endpoint.queueStream([ @@ -2136,6 +2172,89 @@ describe('failure handling', () => { expect(classifyStatus(400)).toBeInstanceOf(StaleRequestStateError); expect(classifyStatus(400)).toBeInstanceOf(RecoverableTransportError); expect(classifyStatus(400).message).toContain('base URI'); + // 422 is the third class: recoverable enough to be retried, but its own + // type so the loop can keep it off the failure budget. + expect(classifyStatus(422)).toBeInstanceOf(NoSkillPayloadError); + expect(classifyStatus(422)).toBeInstanceOf(RecoverableTransportError); + expect(classifyStatus(422)).not.toBeInstanceOf(FatalTransportError); + // The message explains the state rather than reciting the status: this is + // the line a user reads when their skills never show up. + expect(classifyStatus(422).message).toContain('no Agent Skills payload'); + }); + + it('never stops delivery on a 422, and never counts it as a failure', async () => { + // A 422 means the credential is assigned no agent-skill payload, which is + // every project in which no skill has ever been created. Counting it as a + // failure would spend the budget and report an ordinary configuration as + // "gave up after N consecutive failures"; so it is counted, said once, and + // retried for as long as the store is open. + endpoint.defaultPollStatus = 422; + const store = pollStore({ maxConsecutiveFailures: 1 }); + store.start(); + // Well past a bound of one, and still asking. + expect(await waitUntil(() => store.diagnostics.payloadUnavailable >= 4)).toBe(true); + expect(store.failed).toBeNull(); + // None of it reads as a failure, because none of it is one. + expect(store.diagnostics.connectionFailures).toBe(0); + expect(store.diagnostics.lastError).toBeNull(); + // Said once, not once per attempt. + expect(logged(warnSpy).match(/Skill delivery is idle/g)).toHaveLength(1); + expect(logged(warnSpy)).toMatch(/no Agent Skills payload/); + expect(logged(warnSpy)).not.toMatch(/Skill delivery failed/); + expect(consoleErrors()).not.toMatch(/gave up/); + }); + + it('waits the backoff cap rather than the initial delay before asking again', async () => { + // The exponential schedule is a function of the failure count, which this + // case deliberately never advances — so reusing it would hold a store + // waiting for its first skill at the *initial* delay forever, reconnecting + // as fast as a fresh connection retries. Asserted by counting requests in a + // window rather than by timing one: at the 5ms initial delay this window + // holds dozens of them, at the cap it holds the one. + endpoint.defaultPollStatus = 422; + const store = pollStore({ initialBackoffMs: 5, maxBackoffMs: 3000, pollIntervalMs: 5 }); + store.start(); + expect(await waitUntil(() => store.diagnostics.payloadUnavailable >= 1)).toBe(true); + await new Promise((resolve) => setTimeout(resolve, 250)); + expect(endpoint.requests).toHaveLength(1); + }); + + it('picks up a skill payload that appears after a 422', async () => { + // The reason this is not fatal: a skill created after the store started is + // delivered to the store that is already running, with nothing restarted. + endpoint.queuePoll([], { status: 422 }); + endpoint.queuePoll([], { status: 422 }); + endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); + const store = pollStore({ maxConsecutiveFailures: 1 }); + store.start(); + expect(await store.waitForSkills(5000)).toBe(true); + expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); + // Both readings, in the order they happened. + expect(store.diagnostics.payloadUnavailable).toBe(2); + expect(store.diagnostics.payloadsTransferred).toBe(1); + expect(store.failed).toBeNull(); + }); + + it('leaves the store uninitialized on a 422, so a wildcard reconcile prunes nothing', async () => { + // "LaunchDarkly has no skill payload for this environment" and "this + // environment's every skill was revoked" are the two readings of an empty + // answer, and only the second may delete a customer's files. A 422 commits + // no payload, so the readiness probe stays false and the prune is withheld. + const root = await scratchRoot(); + const stale = path.join(root, 'left-behind'); + await mkdir(stale, { recursive: true }); + await writeFile(path.join(stale, 'SKILL.md'), 'not ours to delete', 'utf8'); + + endpoint.defaultPollStatus = 422; + const store = pollStore({ maxConsecutiveFailures: 1 }); + store.start(); + expect(await waitUntil(() => store.diagnostics.payloadUnavailable >= 3)).toBe(true); + expect(store.isInitialized()).toBe(false); + expect(await store.waitForSkills(100)).toBe(false); + _setStore(store); + const report = await writeSkills('*', root); + expect(existsSync(path.join(stale, 'SKILL.md'))).toBe(true); + expect(report.actions.some((action) => action.action === 'removed')).toBe(false); }); it('makes backoff exponential and capped', () => { @@ -3436,15 +3555,15 @@ describe('endpoints', () => { expect(requester.baseUri).toBe(DEFAULT_BASE_URI); expect(requester.streamUri).toBe(DEFAULT_STREAM_URI); const { poll, stream } = await requestedUrls(requester); - expect(poll).toBe('https://sdk.launchdarkly.com/sdk/poll'); - expect(stream).toBe('https://stream.launchdarkly.com/sdk/stream'); + expect(poll).toBe('https://sdk.launchdarkly.com/sdk/poll?kinds=agent-skill'); + expect(stream).toBe('https://stream.launchdarkly.com/sdk/stream?kinds=agent-skill'); }); it('sends both endpoints to a custom baseUri when no streamUri is given', async () => { const requester = requesterOf(new FDv2SkillStore(SDK_KEY, { baseUri: 'https://relay.example.com/' })); const { poll, stream } = await requestedUrls(requester); - expect(poll).toBe('https://relay.example.com/sdk/poll'); - expect(stream).toBe('https://relay.example.com/sdk/stream'); + expect(poll).toBe('https://relay.example.com/sdk/poll?kinds=agent-skill'); + expect(stream).toBe('https://relay.example.com/sdk/stream?kinds=agent-skill'); }); it('lets streamUri differ from baseUri', async () => { @@ -3452,8 +3571,8 @@ describe('endpoints', () => { new FDv2SkillStore(SDK_KEY, { baseUri: 'https://sdk.example.com', streamUri: 'https://stream.example.com/' }), ); const { poll, stream } = await requestedUrls(requester); - expect(poll).toBe('https://sdk.example.com/sdk/poll'); - expect(stream).toBe('https://stream.example.com/sdk/stream'); + expect(poll).toBe('https://sdk.example.com/sdk/poll?kinds=agent-skill'); + expect(stream).toBe('https://stream.example.com/sdk/stream?kinds=agent-skill'); }); it('keeps the default poll host when only streamUri is given', () => { @@ -3471,8 +3590,8 @@ describe('endpoints', () => { }); await requester.poll('(p:a:1)', null, new AbortController().signal); await expect(requester.stream('(p:a:1)', new AbortController().signal)).rejects.toThrow(); - expect(urls[0]).toBe('https://sdk.example.com/sdk/poll?basis=%28p%3Aa%3A1%29'); - expect(urls[1]).toBe('https://stream.example.com/sdk/stream?basis=%28p%3Aa%3A1%29'); + expect(urls[0]).toBe('https://sdk.example.com/sdk/poll?kinds=agent-skill&basis=%28p%3Aa%3A1%29'); + expect(urls[1]).toBe('https://stream.example.com/sdk/stream?kinds=agent-skill&basis=%28p%3Aa%3A1%29'); }); /** @@ -3559,6 +3678,19 @@ describe('transport contract', () => { expect(FDV2_OBJECT_KIND).toBe(SKILL_OBJECT_KIND); }); + it('holds the payload kind apart from the object kind, by source text', () => { + // Two different things, and neither derived from the other: the store asks + // for a payload of kind `agent-skill` and reads objects of kind `skill` out + // of it. Held apart so a rename of either cannot silently move the other, + // and asserted against the source for the same reason the seam-kind test + // above is — an alias would satisfy the value assertions on its own. + expect(source).toMatch(/export const FDV2_PAYLOAD_KIND = 'agent-skill';/); + expect(source).not.toMatch(/FDV2_PAYLOAD_KIND\s*=\s*FDV2_OBJECT_KIND/); + expect(source).not.toMatch(/FDV2_OBJECT_KIND\s*=\s*FDV2_PAYLOAD_KIND/); + expect(FDV2_PAYLOAD_KIND).toBe('agent-skill'); + expect(FDV2_PAYLOAD_KIND).not.toBe(FDV2_OBJECT_KIND); + }); + it('the 401 message names the SDK key', () => { expect(classifyStatus(401).message).toMatch(/SDK key/); }); diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index ae79ed4f..9daf9b91 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -45,15 +45,34 @@ import { isValidSkillVersion } from './types.js'; /** * The FDv2 `kind` skills are delivered under. * - * Object kinds on the SDK-facing channel are open strings: the agent-skill - * payload is classified `generic` and every object in it carries the kind its - * producer registered, which for skills is the bare category name. Delivery - * lower-cases the kind, so an exact comparison is the whole test. The kind - * happens to equal `SKILL_OBJECT_KIND` today; they are still separate constants, - * because one is a wire value LaunchDarkly owns and the other is an SDK seam. + * Object kinds on the SDK-facing channel are open strings: every object in the + * agent-skill payload carries the kind its producer registered, which for skills + * is the bare category name. Delivery lower-cases the kind, so an exact + * comparison is the whole test. The kind happens to equal `SKILL_OBJECT_KIND` + * today; they are still separate constants, because one is a wire value + * LaunchDarkly owns and the other is an SDK seam. + * + * Not to be confused with {@link FDV2_PAYLOAD_KIND}: this is the kind of the + * *objects*, that is the kind of the *payload* they arrive in. */ export const FDV2_OBJECT_KIND = 'skill'; +/** + * The kind of the FDv2 payload skills are delivered in, declared on every + * request as `?kinds=`. + * + * Delivery narrows a connection to the payload kinds it declares and defaults to + * flags, so this is not an optimisation: a request that omits it receives the + * environment's flag payload and no skills at all. Declaring it is also what + * makes the connection carry exactly one payload — the shape + * {@link ProtocolReader} is built for — since a skill-enabled environment + * assigns both the flag payload and this one. + * + * The wire accepts a comma-separated list, but this store wants the skill + * payload and nothing else, so it declares this one kind alone. + */ +export const FDV2_PAYLOAD_KIND = 'agent-skill'; + /** * What separates a skill's key from its version inside the object's wire `key`. * @@ -302,6 +321,19 @@ export type StoreDiagnostics = { * there delivered nothing either, and does raise it. */ readonly connectionFailures: number; + /** + * Requests LaunchDarkly answered with "no payload of the kind you asked for" + * (HTTP 422) — the answer for a project in which no skill has ever been + * created. + * + * It is deliberately not a `connectionFailures`: nothing is wrong, there is + * nothing to deliver. Nonzero and rising alongside an empty store is the + * difference between "this environment has no skills" and "delivery is + * broken", which is the pair this whole type exists to separate. It does not + * reset, so a store that was empty and then received its first payload reads + * as both, in the order it happened. + */ + readonly payloadUnavailable: number; /** The most recent transport error, if any. Human-readable; do not parse. */ readonly lastError: string | null; }; @@ -769,6 +801,7 @@ function freshDiagnostics(): MutableDiagnostics { payloadsIgnored: 0, hashlessObjects: 0, connectionFailures: 0, + payloadUnavailable: 0, lastError: null, }; } @@ -1112,6 +1145,31 @@ export class RecoverableTransportError extends Error { */ export class StaleRequestStateError extends RecoverableTransportError {} +/** + * An HTTP 422: delivery has no payload of the kind this store declared. + * + * That is the answer for every environment whose project has never had a skill, + * because the agent-skill payload is created with the first one and the + * declaration then matches nothing the credential is assigned. So it is neither + * a failure nor fatal, and is classed as neither: + * + * - counting it as a failure would spend `maxConsecutiveFailures` and then give + * up permanently — reported as "gave up after N consecutive failures" — on a + * configuration that is merely waiting for its first skill; + * - treating it as fatal would mean the skill created a minute later never + * arrives, because nothing reopens delivery short of a process restart. + * + * `expected` keeps it off `connectionFailures`, `lastError` and the per-attempt + * warning; the loop reports it once, counts it under + * `diagnostics.payloadUnavailable`, and retries at the backoff cap for as long + * as the store is open. + */ +export class NoSkillPayloadError extends RecoverableTransportError { + constructor(message: string) { + super(message, null, true); + } +} + const REQUEST_ADVICE = 'The request this adapter sent was not understood. It carries only the SDK key and, after the first payload, ' + "a 'basis' selector, so check the base URI and that the endpoint speaks FDv2."; @@ -1186,6 +1244,16 @@ export function classifyStatus(status: number, headers?: Headers | null): Error // be one the server no longer accepts. Recoverable so the selector can be // dropped and a full transfer requested; fatal once that has been tried. if (status === 400) return new StaleRequestStateError(`LaunchDarkly returned HTTP 400. ${REQUEST_ADVICE}`); + // Not "retrying will not fix it" and not a failure either: the payload this + // store asks for does not exist yet. Creating the first skill in the project + // is what fixes it, and delivery keeps asking until then. + if (status === 422) { + return new NoSkillPayloadError( + 'LaunchDarkly has no Agent Skills payload for this environment (HTTP 422). This is what it answers until the ' + + 'first skill is created in this project, so delivery keeps asking and picks one up without a restart. If ' + + 'this environment does have skills, check that this SDK key belongs to it.', + ); + } if ([405, 406, 414, 501].includes(status)) { return new FatalTransportError( `LaunchDarkly returned HTTP ${status}, which retrying will not fix. ${REQUEST_ADVICE}`, @@ -1524,16 +1592,22 @@ export class FetchRequester implements Requester { } /** - * The request URL: the path, plus `basis` once a payload has committed. + * The request URL: the path, the payload kind this store accepts, and `basis` + * once a payload has committed. + * + * `kinds` is on every request, including the first one, because it selects + * what the connection is served rather than describing what it already holds + * (see {@link FDV2_PAYLOAD_KIND}). * * Deliberately no `mv` (data model version). That parameter selects the *flag* - * data model and the connection rejects any value but the flag default; the - * agent-skill payload is generic, is served regardless of it, and has no model - * version of its own to ask for. + * data model; delivery overrides whatever a request asks for with the + * payload's own default for any non-flagging payload, so sending it would + * state a preference that is ignored. */ private url(origin: string, path: string, basis: string | null): string { - if (!basis) return `${origin}${path}`; - return `${origin}${path}?${new URLSearchParams({ basis }).toString()}`; + const params = new URLSearchParams({ kinds: FDV2_PAYLOAD_KIND }); + if (basis) params.set('basis', basis); + return `${origin}${path}?${params.toString()}`; } /** @@ -1806,6 +1880,10 @@ export class FDv2SkillStore implements SkillStore { // one that says goodbye having delivered nothing, and only the former escapes // the bound. private reachedServer = false; + // Whether the "no skill payload for this environment" line has been said. Once + // per store, not once per attempt: the condition persists until somebody + // creates a skill, and delivery keeps asking the whole time. + private warnedNoSkillPayload = false; constructor(sdkKey: string, options: FDv2SkillStoreOptions = {}) { const key = requireServerSideCredential(sdkKey); @@ -2103,6 +2181,16 @@ export class FDv2SkillStore implements SkillStore { this.etagBasis = null; repairingState = true; } + if (cause instanceof NoSkillPayloadError) { + this.reader.diagnostics.payloadUnavailable += 1; + if (!this.warnedNoSkillPayload) { + this.warnedNoSkillPayload = true; + warn( + `Skill delivery is idle: ${cause.message} Retrying every ${Math.round(this.maxBackoffMs)}ms; this is ` + + 'the only time it will be said.', + ); + } + } // A connection the server closed while serving it normally ended // without being a failure — see `dispatch` for which ones qualify. It // reconnects like one, but it neither counts against @@ -2125,16 +2213,24 @@ export class FDv2SkillStore implements SkillStore { } } const requested = cause.retryAfterMs; - const delay = Math.min( - requested !== null && Number.isFinite(requested) - ? // A server asking for no delay still gets one: honouring - // `Retry-After: 0` literally would reconnect in a loop and burn - // the whole retry bound in milliseconds. - Math.max(requested, this.initialBackoffMs) - : backoffDelayMs(this.failures, this.initialBackoffMs, this.maxBackoffMs), - // `Retry-After` is a request and `maxBackoffMs` is a promise. - this.maxBackoffMs, - ); + const delay = + cause instanceof NoSkillPayloadError + ? // Not backing off from a failure, waiting for somebody to create a + // skill. `failures` never moved, so the exponential schedule would + // hold this at the *initial* delay forever — the cap is both the + // cheapest place to sit and the one that does not depend on a + // counter this case deliberately leaves alone. + this.maxBackoffMs + : Math.min( + requested !== null && Number.isFinite(requested) + ? // A server asking for no delay still gets one: honouring + // `Retry-After: 0` literally would reconnect in a loop and burn + // the whole retry bound in milliseconds. + Math.max(requested, this.initialBackoffMs) + : backoffDelayMs(this.failures, this.initialBackoffMs, this.maxBackoffMs), + // `Retry-After` is a request and `maxBackoffMs` is a promise. + this.maxBackoffMs, + ); if (!cause.expected) { warn(`Skill delivery failed (${cause.message}); retrying in ${Math.round(delay)}ms`); } From ec7fc2f8f5ae71e50b4ef98b965e46c13489544e Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Thu, 24 Sep 2026 15:28:36 -0400 Subject: [PATCH 44/55] docs(client): say the 422's cause once, on the type that names it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mirrors the Python trim (launchdarkly/python-ai-sdk#109): the same sentence had been repeated at every site that touches the 422, so NoSkillPayloadError now owns the explanation and the others state only what is local to them — the diagnostic links the type and keeps the "not a connectionFailures" distinction, the loop keeps why it waits the cap, and classifyStatus keeps nothing, since the type it returns and the message it builds already say it twice over. No behaviour change, and the user-facing 422 message is untouched. 12 lines of comment removed. Co-Authored-By: Claude Opus 5 --- packages/client/src/skills-fdv2.ts | 50 ++++++++++++------------------ 1 file changed, 19 insertions(+), 31 deletions(-) diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 9daf9b91..47338348 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -53,7 +53,7 @@ import { isValidSkillVersion } from './types.js'; * LaunchDarkly owns and the other is an SDK seam. * * Not to be confused with {@link FDV2_PAYLOAD_KIND}: this is the kind of the - * *objects*, that is the kind of the *payload* they arrive in. + * *objects*, that one the kind of the *payload* they arrive in. */ export const FDV2_OBJECT_KIND = 'skill'; @@ -322,16 +322,13 @@ export type StoreDiagnostics = { */ readonly connectionFailures: number; /** - * Requests LaunchDarkly answered with "no payload of the kind you asked for" - * (HTTP 422) — the answer for a project in which no skill has ever been - * created. + * Requests answered with "no payload of the kind you asked for" + * ({@link NoSkillPayloadError}). Cumulative, and never reset. * * It is deliberately not a `connectionFailures`: nothing is wrong, there is * nothing to deliver. Nonzero and rising alongside an empty store is the * difference between "this environment has no skills" and "delivery is - * broken", which is the pair this whole type exists to separate. It does not - * reset, so a store that was empty and then received its first payload reads - * as both, in the order it happened. + * broken", which is the pair this whole type exists to separate. */ readonly payloadUnavailable: number; /** The most recent transport error, if any. Human-readable; do not parse. */ @@ -1148,21 +1145,18 @@ export class StaleRequestStateError extends RecoverableTransportError {} /** * An HTTP 422: delivery has no payload of the kind this store declared. * - * That is the answer for every environment whose project has never had a skill, - * because the agent-skill payload is created with the first one and the - * declaration then matches nothing the credential is assigned. So it is neither - * a failure nor fatal, and is classed as neither: + * That is the answer for every project in which no skill has ever been created, + * since the agent-skill payload row is created with the first one. Neither of + * the two obvious classifications is right, which is why this is its own class: * - * - counting it as a failure would spend `maxConsecutiveFailures` and then give - * up permanently — reported as "gave up after N consecutive failures" — on a - * configuration that is merely waiting for its first skill; - * - treating it as fatal would mean the skill created a minute later never - * arrives, because nothing reopens delivery short of a process restart. + * - as a failure it would spend `maxConsecutiveFailures` and then give up + * permanently — "gave up after N consecutive failures" — on a configuration + * that is merely waiting for its first skill; + * - as fatal, the skill created a minute later would never arrive, because + * nothing reopens delivery short of a process restart. * - * `expected` keeps it off `connectionFailures`, `lastError` and the per-attempt - * warning; the loop reports it once, counts it under - * `diagnostics.payloadUnavailable`, and retries at the backoff cap for as long - * as the store is open. + * `expected` is what keeps it off `connectionFailures`, `lastError` and the + * per-attempt warning; the rest is in the delivery loop. */ export class NoSkillPayloadError extends RecoverableTransportError { constructor(message: string) { @@ -1244,9 +1238,6 @@ export function classifyStatus(status: number, headers?: Headers | null): Error // be one the server no longer accepts. Recoverable so the selector can be // dropped and a full transfer requested; fatal once that has been tried. if (status === 400) return new StaleRequestStateError(`LaunchDarkly returned HTTP 400. ${REQUEST_ADVICE}`); - // Not "retrying will not fix it" and not a failure either: the payload this - // store asks for does not exist yet. Creating the first skill in the project - // is what fixes it, and delivery keeps asking until then. if (status === 422) { return new NoSkillPayloadError( 'LaunchDarkly has no Agent Skills payload for this environment (HTTP 422). This is what it answers until the ' + @@ -1880,9 +1871,8 @@ export class FDv2SkillStore implements SkillStore { // one that says goodbye having delivered nothing, and only the former escapes // the bound. private reachedServer = false; - // Whether the "no skill payload for this environment" line has been said. Once - // per store, not once per attempt: the condition persists until somebody - // creates a skill, and delivery keeps asking the whole time. + // Said once per store rather than once per attempt: the condition holds until + // somebody creates a skill, and delivery keeps asking throughout. private warnedNoSkillPayload = false; constructor(sdkKey: string, options: FDv2SkillStoreOptions = {}) { @@ -2215,11 +2205,9 @@ export class FDv2SkillStore implements SkillStore { const requested = cause.retryAfterMs; const delay = cause instanceof NoSkillPayloadError - ? // Not backing off from a failure, waiting for somebody to create a - // skill. `failures` never moved, so the exponential schedule would - // hold this at the *initial* delay forever — the cap is both the - // cheapest place to sit and the one that does not depend on a - // counter this case deliberately leaves alone. + ? // At the cap rather than on the backoff schedule: `failures` + // deliberately never moves, so the schedule would hold this at + // the *initial* delay forever. this.maxBackoffMs : Math.min( requested !== null && Number.isFinite(requested) From 790ef3c8502c7cbbdb272ff1cee0872b725490e8 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Thu, 24 Sep 2026 15:33:57 -0400 Subject: [PATCH 45/55] test(client): stop the hashless-reconcile case racing its own second payload Not this PR's change -- the case is byte-identical to the one on the base branch -- but it went red on this PR's CI run and will keep doing so on a loaded runner. It queued the good payload and the hashless one up front, 20ms apart, then relied on `waitForSkills` before reconciling. `waitForSkills` promises the *first* commit and nothing about the second, so any pause longer than the poll interval between it and the first `writeSkills` -- an `mkdtemp` on a busy runner is enough -- lets the hashless payload commit first. Verification then withholds every object, the reconcile writes nothing, and the assertion fails as `ENOENT` on the read rather than as anything that names the cause. Queueing the hashless payload only after the first reconcile has been asserted removes the race without changing what the case asserts. Reproduced before the fix by standing a 60ms sleep in for the runner (identical ENOENT), and the fixed case survives 300ms in the same spot. Co-Authored-By: Claude Opus 5 --- packages/client/src/__tests__/skills-fdv2.test.ts | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 079203fd..24d63a58 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -2351,10 +2351,6 @@ describe('the missing contentHash', () => { // as a skill key drops out of the keep-set and prune deletes the last // known-good copy. Which is the outcome this transport was written to avoid. endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); - endpoint.queuePoll( - fullPayload([['put-object', putSkill('pdf-extraction', { objectVersion: 4, omitHash: true })]], 'basis-2'), - ); - endpoint.queuePoll([], { status: 304 }); const store = pollStore({ pollIntervalMs: 20 }); store.start(); @@ -2366,6 +2362,15 @@ describe('the missing contentHash', () => { await writeSkills('*', root); expect(readFileSync(written, 'utf8')).toBe(SKILL_BODY); + // Queued only now, and deliberately not up front. `waitForSkills` promises + // the *first* commit and nothing about the second, so a hashless payload + // waiting in the queue commits on the next poll — 20ms later — and a + // reconcile that has not run by then withholds every object and writes + // nothing. That is a real race the assertion above cannot survive, and it + // fails as an ENOENT on the read rather than as anything self-explanatory. + endpoint.queuePoll( + fullPayload([['put-object', putSkill('pdf-extraction', { objectVersion: 4, omitHash: true })]], 'basis-2'), + ); expect(await waitUntil(() => store.diagnostics.hashlessObjects > 0, 5000)).toBe(true); const report = await writeSkills('*', root); From df337a895f9f7408d1bd358f96639f69c3dc7876 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 29 Sep 2026 12:06:08 -0400 Subject: [PATCH 46/55] fix(client): make HTTP 422 a fatal transport failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LaunchDarkly answers 422 when a connection's declared kinds exclude every payload it is assigned, and it picked a non-400 4xx *because* LD SDKs treat those as terminal — the streamer says so in as many words at both places that emit it, each noting the code exists so a misconfigured SDK stops instead of hammering the fleet. Retrying it forever was the precise behavior the status was chosen to prevent. The stated cause was also wrong. The gate is skill delivery enabled for the account *and* a credential that is not view-scoped; with both satisfied the assignment path creates the agent-skill payload row lazily, so an environment holding zero skills is served an empty payload that commits normally. Skill existence is not among the causes, so the message no longer claims it is, and no longer promises that a skill created later arrives without a restart. `classifyStatus` gets its own 422 branch rather than folding into the generic fatal list, because the message is a contract — it is what a customer pastes into a support ticket. It names both causes but weights them: the view-scoped SDK key leads and carries the instruction, because it is the only one of the two a reader can act on. Skill delivery is not enabled per account as a customer-facing step, so a closed gate is a LaunchDarkly-side condition — a kill switch, or a rollout that has not reached them — and telling someone to go enable Agent Skills for their account would send them looking for a setting they do not have. That cause routes to support instead. Routing 422 through the existing give-up path gives the right accounting for free: `failed` and `lastError` are set, `connectionFailures` is untouched (it measures consecutive *recoverable* failures against the retry bound, which a fatal never spends), and `waitForSkills` resolves `false` at once instead of at the caller's timeout. `NoSkillPayloadError` and `diagnostics.payloadUnavailable` are deleted rather than merely unexported. A status handled as neither recoverable nor fatal is a retry loop with no bound and no budget, invisible to `failed` and `connectionFailures` alike, so the shape is forbidden and not just the name: tests assert that classification answers every status with exactly one of the two classes, and assert both absences by source text and on a real diagnostics snapshot. The cost, stated in the docs so it is not rediscovered as a bug: a process whose store gave up needs a restart once the cause is resolved, since `close()` is final and nothing reopens delivery short of a new store. That is deliberately preferred to retrying a permanent rejection for the life of the process. Conforms to ai-sdks-monorepo #29 (TESTING.md §3.25, A.12). Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 4 +- packages/client/agents.md | 6 +- .../client/src/__tests__/skills-fdv2.test.ts | 197 ++++++++++++------ packages/client/src/skills-fdv2.ts | 92 +++----- 4 files changed, 169 insertions(+), 130 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index b401a0b9..91d072d9 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -329,7 +329,7 @@ example above, materializes only what the resolved variation actually asked for. | `FDv2SkillStore(sdkKey, options?)` | The delivery transport: a `SkillStore` fed by LaunchDarkly over the SDK-facing FDv2 channel. `start()`, `waitForSkills(timeoutMs)`, `isInitialized()`, `close()`, `diagnostics`, `failed`, `addListener` / `removeListener`. `close()` is final — `start()` throws afterwards — and `addListener` throws for any kind but `'skill'`. Options (`FDv2SkillStoreOptions`): `mode` (`'stream'` default, or `'poll'`), `baseUri`, `streamUri`, `pollIntervalMs`, `readTimeoutMs`, `initialBackoffMs`, `maxBackoffMs`, `maxConsecutiveFailures`. **Server-side only** — a mobile key or client-side environment ID throws. See [Receiving skills from LaunchDarkly](#receiving-skills-from-launchdarkly). | | `watchSkills(skills, root, options?)` | `writeSkills` plus a re-reconcile on every delivery change. Resolves to `{ report, watcher }` — the initial reconcile's report and a `SkillWatcher`; `await watcher.close()` when done, which also detaches the watcher from the store. Options (`WatchSkillsOptions`): everything `writeSkills` takes, plus `debounceMs` (milliseconds, default `DEFAULT_DEBOUNCE_MS`) and `onReconcile`, called with each delivery-triggered report. Requires a store that implements `addListener`; one watcher per root. Revocation then takes effect within `debounceMs` of arriving rather than at the next restart. | | `SkillWatcher` | What `watchSkills` returns alongside the report: `reconciles` (re-reconciles completed, excluding the initial one), `notify` (the registered change listener), `close()` (idempotent; detaches and awaits any reconcile in flight). | -| `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object` that removed something, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation, and a tombstone for a key never held is reported but not counted), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `payloadUnavailable` (requests LaunchDarkly answered with "no payload of the kind you asked for" — see below), `lastError`. | +| `StoreDiagnostics` | What the transport has seen: `payloadsTransferred`, `skillObjectsReceived`, `objectsIgnored`, `objectsRevoked` (each `delete-object` that removed something, plus each key a full transfer dropped altogether — a version bump is a move, not a revocation, and a tombstone for a key never held is reported but not counted), `payloadsIgnored`, `hashlessObjects`, `connectionFailures`, `lastError`. | | `DEFAULT_BASE_URI` / `DEFAULT_STREAM_URI` | `'https://sdk.launchdarkly.com'` and `'https://stream.launchdarkly.com'` — where `GET /sdk/poll` and `GET /sdk/stream` go by default. Overrides must be `https://` (plain `http://` only to a loopback host), and redirects from either are refused as a fatal failure. | | `DEFAULT_DEBOUNCE_MS` | `500` — the `watchSkills` coalescing window in milliseconds. | | `createSkill(init)` / `createSkillReference(init)` | Build frozen `Skill` / `SkillReference` values. Use `createSkill` to hand `writeSkills` content you already have. | @@ -404,7 +404,7 @@ try { **`addListener` observes skill changes only.** The store notifies changes to `'skill'` objects; flag and segment objects on the same connection are skipped, never dispatched. A listener registered under any other kind therefore throws rather than being accepted and silently never firing. `removeListener` tolerates any kind, so a consumer that detaches on close can do so unconditionally. `watchSkills` is the intended consumer of both. -**A project with no skills yet is a waiting state, not a failure.** Every request declares the payload it wants (`kinds=agent-skill`), and LaunchDarkly answers HTTP 422 when the environment has no such payload — which is the case until somebody creates the first skill in the project. The store says so once, keeps asking at its backoff cap, and picks up that first skill without a restart. `failed` stays `null` throughout, `connectionFailures` stays at zero, and `diagnostics.payloadUnavailable` counts the answers: nonzero and rising beside an empty store means "there is nothing to deliver", not "delivery is broken". +**A 422 means this connection will never be assigned a skill payload, and delivery stops.** Every request declares the payload it wants (`kinds=agent-skill`), and LaunchDarkly answers HTTP 422 when it will not serve one. The cause you can act on is a **view-scoped SDK key**: a key restricted to a view cannot be assigned a skill payload, so check the key's scoping and use one that is not view-scoped. The other cause is that Agent Skills delivery is not enabled for your account, which is not a setting you control — contact LaunchDarkly support if the key is not the problem. Retrying fixes neither, and LaunchDarkly chose the status so that SDKs stop rather than hammer the fleet, so the store gives up: `failed` carries the reason, `lastError` is populated, `connectionFailures` stays at zero (it counts consecutive *recoverable* failures against the retry bound, which a fatal never spends), and `waitForSkills` resolves `false` immediately instead of at your timeout. It is **not** the answer for an environment that merely has no skills yet — with delivery enabled and a non-view-scoped key, an environment holding zero skills is served an empty payload that commits normally. Because delivery has stopped for good, a process that booted while the cause was in effect picks up skills only after a restart. **Nothing above the store changes.** The accessors, integrity verification, and `writeSkills` are transport-agnostic: they see raw objects through the `SkillStore` seam and cannot tell which store produced them. Everything documented above about verification and reconcile semantics applies unchanged. diff --git a/packages/client/agents.md b/packages/client/agents.md index 3b0249d0..11189965 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -59,7 +59,11 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b **No `mv` parameter, deliberately.** It selects the *flag* data model, and delivery overrides whatever a request asks for with the payload's own default for any non-flagging payload, so sending it would state a preference that is ignored. -**HTTP 422 is not a failure.** It is the answer to that declaration when the credential is assigned no agent-skill payload, which is every project in which no skill has ever been created — gonfalon creates that payload row with the first skill and never lazily. So `classifyStatus` maps it to `NoSkillPayloadError`, which the delivery loop treats as its own case: counted under `diagnostics.payloadUnavailable`, logged once per store, retried at `maxBackoffMs` indefinitely, and (through the existing `expected` flag) kept off `connectionFailures`, `lastError` and `failed`. Neither obvious classification is right — as a recoverable failure it spends `maxConsecutiveFailures` and then gives up permanently on an ordinary configuration; as a fatal one, the skill created a minute later never arrives without a process restart. The retry is at the *cap* rather than the initial backoff because `failures` deliberately never moves, so the exponential schedule would sit at the initial delay forever. +**HTTP 422 is fatal, and the platform decided that rather than the SDK inferring it.** Delivery answers 422 when a connection's declared kinds exclude every payload it is assigned, and it picked a non-400 4xx *because* LD SDKs treat those as terminal — the streamer says so in as many words at both places that emit it, each noting the code exists so a misconfigured SDK stops instead of hammering the fleet. So `classifyStatus` returns a `FatalTransportError` and the existing give-up path handles it: `failed` and `lastError` are set, `connectionFailures` is untouched (it measures consecutive *recoverable* failures against the retry bound, and a fatal never retries), and `waitForSkills` resolves `false` at once rather than at the timeout. It gets its own branch in `classifyStatus` rather than joining the generic fatal list because the message is a contract — it is what a customer pastes into a support ticket, so it names the two real causes. + +**The 422 is not about whether any skill exists.** The gate is `payloadvers.SkillDeliveryAllowed`: skill delivery enabled for the account **and** a credential that is not view-scoped. Each way of failing it is permanent from the store's side — the account gate is off (`enable-aic-agent-skills`), the SDK key is view-scoped, or the declared kind is a typo. With the gate open and a non-view-scoped key the assignment path creates the agent-skill payload row **lazily**, so an environment holding zero skills is assigned an empty payload that commits normally through `payload-transferred`. + +**The message leads with the key's scoping, because that is the only cause a reader can act on.** The account gate is not a customer-facing enablement step — it is expected to be flipped on for everyone rather than targeted per account — so a closed gate is a LaunchDarkly-side condition, a kill switch or an unfinished rollout, and instructing someone to go enable Agent Skills for their account would send them looking for a setting they do not have. So the view-scoped key carries the instruction and the gate routes to support. The cost of fatality, stated so it is not rediscovered as a bug: a process whose store gave up needs a restart once the cause is resolved, since `close()` is final and nothing reopens delivery short of a new store. That is deliberately preferred to retrying a permanent rejection for the life of the process. **Polling and streaming have different default hosts.** `GET /sdk/poll` is served from `DEFAULT_BASE_URI` (`sdk.launchdarkly.com`) and `GET /sdk/stream` from `DEFAULT_STREAM_URI` (`stream.launchdarkly.com`), matching the base server-side SDK. Sending the stream request to the polling host fails against a real environment, and the fake endpoint cannot catch it because it serves both from one origin. `baseUri` given alone therefore applies to both, since a relay or private instance usually serves both from one host; only the LaunchDarkly defaults split them, and `streamUri` overrides the stream origin on its own. diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index 24d63a58..cd0971f4 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -52,7 +52,6 @@ import { isSkillEvent, iterSse, MAX_RESPONSE_CHARS, - NoSkillPayloadError, type PollResult, ProtocolReader, RecoverableTransportError, @@ -2172,89 +2171,139 @@ describe('failure handling', () => { expect(classifyStatus(400)).toBeInstanceOf(StaleRequestStateError); expect(classifyStatus(400)).toBeInstanceOf(RecoverableTransportError); expect(classifyStatus(400).message).toContain('base URI'); - // 422 is the third class: recoverable enough to be retried, but its own - // type so the loop can keep it off the failure budget. - expect(classifyStatus(422)).toBeInstanceOf(NoSkillPayloadError); - expect(classifyStatus(422)).toBeInstanceOf(RecoverableTransportError); - expect(classifyStatus(422)).not.toBeInstanceOf(FatalTransportError); - // The message explains the state rather than reciting the status: this is - // the line a user reads when their skills never show up. - expect(classifyStatus(422).message).toContain('no Agent Skills payload'); - }); - - it('never stops delivery on a 422, and never counts it as a failure', async () => { - // A 422 means the credential is assigned no agent-skill payload, which is - // every project in which no skill has ever been created. Counting it as a - // failure would spend the budget and report an ordinary configuration as - // "gave up after N consecutive failures"; so it is counted, said once, and - // retried for as long as the store is open. - endpoint.defaultPollStatus = 422; - const store = pollStore({ maxConsecutiveFailures: 1 }); - store.start(); - // Well past a bound of one, and still asking. - expect(await waitUntil(() => store.diagnostics.payloadUnavailable >= 4)).toBe(true); - expect(store.failed).toBeNull(); - // None of it reads as a failure, because none of it is one. - expect(store.diagnostics.connectionFailures).toBe(0); - expect(store.diagnostics.lastError).toBeNull(); - // Said once, not once per attempt. - expect(logged(warnSpy).match(/Skill delivery is idle/g)).toHaveLength(1); - expect(logged(warnSpy)).toMatch(/no Agent Skills payload/); - expect(logged(warnSpy)).not.toMatch(/Skill delivery failed/); - expect(consoleErrors()).not.toMatch(/gave up/); + // 422 is fatal, and LaunchDarkly chose the status to be terminal rather + // than the SDK inferring it: a connection whose declared kinds exclude + // every payload it is assigned will never be assigned one. + expect(classifyStatus(422)).toBeInstanceOf(FatalTransportError); + expect(classifyStatus(422)).not.toBeInstanceOf(RecoverableTransportError); + }); + + it('answers every status with exactly one of the two classes', () => { + // There is no third class. A status handled as neither recoverable nor + // fatal is a retry loop with no bound and no budget, invisible to both + // `failed` and `connectionFailures` — the shape this suite forbids, not + // just the name it used to go by. + const statuses = [ + 301, 302, 307, 308, 400, 401, 402, 403, 404, 405, 406, 408, 409, 410, 413, 414, 418, 422, 425, 429, 431, 451, 500, + 501, 502, 503, 504, 507, 599, + ]; + for (const status of statuses) { + const err = classifyStatus(status); + const recoverable = err instanceof RecoverableTransportError; + const fatal = err instanceof FatalTransportError; + // Exactly one, for every status: not neither, and not both. + expect([recoverable, fatal].filter(Boolean), `HTTP ${status}`).toHaveLength(1); + } }); - it('waits the backoff cap rather than the initial delay before asking again', async () => { - // The exponential schedule is a function of the failure count, which this - // case deliberately never advances — so reusing it would hold a store - // waiting for its first skill at the *initial* delay forever, reconnecting - // as fast as a fresh connection retries. Asserted by counting requests in a - // window rather than by timing one: at the 5ms initial delay this window - // holds dozens of them, at the cap it holds the one. + it('names the account enablement and the key scoping in the 422 message', () => { + // The message is a contract: it is what a customer pastes into a support + // ticket, so it has to name both real causes without them reading platform + // source. Asserted on substance rather than prose, so the wording can be + // improved without rotting this test. + const message = classifyStatus(422).message; + expect(message).toMatch(/422/); + expect(message).toMatch(/view-scoped/i); + expect(message).toMatch(/not enabled for this account/i); + // Of the two causes only the key's scoping is the reader's to fix, so that + // is the one carrying an instruction. A closed account gate is a + // LaunchDarkly-side condition, so it routes to support rather than sending + // someone to look for a setting they do not have. + expect(message).toMatch(/check whether this key is view-scoped/i); + expect(message).toMatch(/contact LaunchDarkly support/i); + // Not an instruction to go enable it for their own account: skill delivery + // is not enabled per account as a customer-facing step. + expect(message).not.toMatch(/(?:once|after) Agent Skills is enabled for (?:the|your) account/i); + expect(message).not.toMatch(/enable Agent Skills for (?:the|your) account/i); + // And the two things it must not say. Neither is true: the payload row is + // created lazily, so an environment with zero skills commits an empty + // payload normally — and nothing reopens delivery short of a restart. + expect(message).not.toMatch(/no skills|first skill|has no Agent Skills payload/i); + expect(message).not.toMatch(/without a restart|keeps asking|picks (?:one|it) up/i); + }); + + it('stops delivery on a 422 answering the first request', async () => { + // LaunchDarkly picked a non-400 4xx precisely so an SDK asking for a + // payload it will never be assigned stops instead of hammering the fleet. + // So the first response is enough: no retry, and nothing committed. endpoint.defaultPollStatus = 422; - const store = pollStore({ initialBackoffMs: 5, maxBackoffMs: 3000, pollIntervalMs: 5 }); + // A bound well above one, so what stops the loop is provably the + // classification and not an exhausted budget. + const store = pollStore({ maxConsecutiveFailures: 10 }); store.start(); - expect(await waitUntil(() => store.diagnostics.payloadUnavailable >= 1)).toBe(true); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.failed).toMatch(/422/); + expect(store.isInitialized()).toBe(false); + // One request, and it was not retried. Asserted after a window that would + // comfortably hold several retries at this store's 5ms initial backoff. await new Promise((resolve) => setTimeout(resolve, 250)); expect(endpoint.requests).toHaveLength(1); + expect(consoleErrors()).toMatch(/will not retry/); }); - it('picks up a skill payload that appears after a 422', async () => { - // The reason this is not fatal: a skill created after the store started is - // delivered to the store that is already running, with nothing restarted. - endpoint.queuePoll([], { status: 422 }); - endpoint.queuePoll([], { status: 422 }); - endpoint.queuePoll(fullPayload([['put-object', putSkill()]])); - const store = pollStore({ maxConsecutiveFailures: 1 }); + it('accounts a fatal 422 on failed and lastError, and not on connectionFailures', async () => { + // `connectionFailures` measures consecutive *recoverable* failures against + // the retry bound. A fatal never retries, so moving it would put a number + // against a budget nothing will spend and make a store that gave up on its + // first response look like one that exhausted its attempts. This is how the + // give-up path already accounts 401 and 404; 422 is not a special case. + endpoint.defaultPollStatus = 422; + const store = pollStore(); store.start(); - expect(await store.waitForSkills(5000)).toBe(true); - expect(store.getObject(SKILL_OBJECT_KIND, 'pdf-extraction')).not.toBeNull(); - // Both readings, in the order they happened. - expect(store.diagnostics.payloadUnavailable).toBe(2); - expect(store.diagnostics.payloadsTransferred).toBe(1); - expect(store.failed).toBeNull(); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + expect(store.diagnostics.lastError).not.toBeNull(); + expect(store.diagnostics.lastError).toMatch(/422/); + expect(store.diagnostics.connectionFailures).toBe(0); + // And no per-attempt retry warning, because there was no retry. + expect(logged(warnSpy)).not.toMatch(/Skill delivery failed/); + }); + + it('resolves waitForSkills false immediately on a fatal 422, not at the timeout', async () => { + // The observable difference from treating the status as recoverable, and as + // much the point of the classification as the stopped retries are: a boot + // gated on skills behind a long wait would otherwise pay that wait on every + // start against a store that knew the answer on its first response. + endpoint.defaultPollStatus = 422; + const store = pollStore(); + store.start(); + const timeoutMs = 10_000; + const began = performance.now(); + // Waiting this out would take ten seconds; returning early takes the one + // round trip to the loopback endpoint. + expect(await store.waitForSkills(timeoutMs)).toBe(false); + const elapsed = performance.now() - began; + expect(elapsed).toBeLessThan(timeoutMs / 4); + expect(store.failed).not.toBeNull(); }); - it('leaves the store uninitialized on a 422, so a wildcard reconcile prunes nothing', async () => { - // "LaunchDarkly has no skill payload for this environment" and "this - // environment's every skill was revoked" are the two readings of an empty - // answer, and only the second may delete a customer's files. A 422 commits - // no payload, so the readiness probe stays false and the prune is withheld. + it('leaves a store that gave up on a 422 uninitialized, so a wildcard reconcile prunes nothing', async () => { + // Composed with §3.22: "delivery gave up, therefore the store is empty, + // therefore prune" is exactly the inference an implementation assembles + // from two sections, and it deletes a customer's files. The readiness gate + // is what stops it, and this is the path a filesystem-agent deployment + // takes when Agent Skills is not enabled for the account. const root = await scratchRoot(); const stale = path.join(root, 'left-behind'); await mkdir(stale, { recursive: true }); await writeFile(path.join(stale, 'SKILL.md'), 'not ours to delete', 'utf8'); endpoint.defaultPollStatus = 422; - const store = pollStore({ maxConsecutiveFailures: 1 }); + const store = pollStore(); store.start(); - expect(await waitUntil(() => store.diagnostics.payloadUnavailable >= 3)).toBe(true); + expect(await waitUntil(() => store.failed !== null)).toBe(true); + // A store that gave up before any payload committed never initialized, and + // neither wait parks: both answers have already arrived. expect(store.isInitialized()).toBe(false); expect(await store.waitForSkills(100)).toBe(false); _setStore(store); const report = await writeSkills('*', root); - expect(existsSync(path.join(stale, 'SKILL.md'))).toBe(true); + // Retrieval is reported unavailable rather than answered as "no skills". + expect(report.ok).toBe(false); + expect(report.errors.map((action) => action.error).join('\n')).toMatch(/skill retrieval unavailable/); + // Pruning is suppressed for the whole run, and nothing on disk moved. expect(report.actions.some((action) => action.action === 'removed')).toBe(false); + expect(existsSync(path.join(stale, 'SKILL.md'))).toBe(true); + expect(await readFile(path.join(stale, 'SKILL.md'), 'utf8')).toBe('not ours to delete'); }); it('makes backoff exponential and capped', () => { @@ -3696,6 +3745,32 @@ describe('transport contract', () => { expect(FDV2_PAYLOAD_KIND).not.toBe(FDV2_OBJECT_KIND); }); + it('declares no payloadUnavailable field and no NoSkillPayloadError, by source text', () => { + // `StoreDiagnostics` is public API from the moment it ships, and a + // type-level removal is invisible at runtime — so the absence is asserted + // against the source, the way the kind constants above are. A field still + // declared but never incremented would fail the spec's absence assertion, + // and would pass a test that only looked at a snapshot. + expect(source).not.toMatch(/payloadUnavailable/); + // The expected-recoverable class is specified out of existence, not merely + // unexported: a status handled as neither recoverable nor fatal is a retry + // loop with no bound and no budget. + expect(source).not.toMatch(/NoSkillPayloadError/); + // And nothing left of the shape it carried: no idle warning, no retry + // parked at the cap instead of on the backoff schedule. + expect(source).not.toMatch(/delivery is idle/i); + expect(source).not.toMatch(/warnedNoSkillPayload/); + }); + + it('carries no payloadUnavailable key on a real diagnostics snapshot', () => { + // The runtime half of the assertion above: the source check would pass a + // field added dynamically, and this one would pass a field declared and + // never written. Both, because neither alone is the whole fact. + const store = pollStore(); + expect(Object.keys(store.diagnostics)).not.toContain('payloadUnavailable'); + expect('payloadUnavailable' in store.diagnostics).toBe(false); + }); + it('the 401 message names the SDK key', () => { expect(classifyStatus(401).message).toMatch(/SDK key/); }); diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 47338348..72c3aa06 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -321,16 +321,6 @@ export type StoreDiagnostics = { * there delivered nothing either, and does raise it. */ readonly connectionFailures: number; - /** - * Requests answered with "no payload of the kind you asked for" - * ({@link NoSkillPayloadError}). Cumulative, and never reset. - * - * It is deliberately not a `connectionFailures`: nothing is wrong, there is - * nothing to deliver. Nonzero and rising alongside an empty store is the - * difference between "this environment has no skills" and "delivery is - * broken", which is the pair this whole type exists to separate. - */ - readonly payloadUnavailable: number; /** The most recent transport error, if any. Human-readable; do not parse. */ readonly lastError: string | null; }; @@ -798,7 +788,6 @@ function freshDiagnostics(): MutableDiagnostics { payloadsIgnored: 0, hashlessObjects: 0, connectionFailures: 0, - payloadUnavailable: 0, lastError: null, }; } @@ -1142,28 +1131,6 @@ export class RecoverableTransportError extends Error { */ export class StaleRequestStateError extends RecoverableTransportError {} -/** - * An HTTP 422: delivery has no payload of the kind this store declared. - * - * That is the answer for every project in which no skill has ever been created, - * since the agent-skill payload row is created with the first one. Neither of - * the two obvious classifications is right, which is why this is its own class: - * - * - as a failure it would spend `maxConsecutiveFailures` and then give up - * permanently — "gave up after N consecutive failures" — on a configuration - * that is merely waiting for its first skill; - * - as fatal, the skill created a minute later would never arrive, because - * nothing reopens delivery short of a process restart. - * - * `expected` is what keeps it off `connectionFailures`, `lastError` and the - * per-attempt warning; the rest is in the delivery loop. - */ -export class NoSkillPayloadError extends RecoverableTransportError { - constructor(message: string) { - super(message, null, true); - } -} - const REQUEST_ADVICE = 'The request this adapter sent was not understood. It carries only the SDK key and, after the first payload, ' + "a 'basis' selector, so check the base URI and that the endpoint speaks FDv2."; @@ -1238,11 +1205,23 @@ export function classifyStatus(status: number, headers?: Headers | null): Error // be one the server no longer accepts. Recoverable so the selector can be // dropped and a full transfer requested; fatal once that has been tried. if (status === 400) return new StaleRequestStateError(`LaunchDarkly returned HTTP 400. ${REQUEST_ADVICE}`); + // Its own branch rather than the generic fatal list below, because the + // message has to name the two configurations that produce it. LaunchDarkly + // chose 422 *to be* terminal — the streamer says so at both places that emit + // it — so retrying is the one reading the platform ruled out. + // + // The key's scoping leads because it is the only one of the two a reader can + // act on. Skill delivery is not enabled per account as a customer-facing + // step, so a closed gate is a LaunchDarkly-side condition — a kill switch, or + // a rollout that has not reached them — and telling someone to go enable it + // would send them looking for a setting they will not find. if (status === 422) { - return new NoSkillPayloadError( - 'LaunchDarkly has no Agent Skills payload for this environment (HTTP 422). This is what it answers until the ' + - 'first skill is created in this project, so delivery keeps asking and picks one up without a restart. If ' + - 'this environment does have skills, check that this SDK key belongs to it.', + return new FatalTransportError( + 'LaunchDarkly will not deliver Agent Skills on this connection (HTTP 422). The usual cause is a view-scoped ' + + 'SDK key: a key restricted to a view cannot be assigned a skill payload, so check whether this key is ' + + 'view-scoped and use one that is not. Failing that, Agent Skills delivery is not enabled for this account, ' + + 'which is not something you can turn on yourself — contact LaunchDarkly support. Retrying fixes neither, so ' + + 'delivery has stopped; the process must be restarted once the cause is resolved.', ); } if ([405, 406, 414, 501].includes(status)) { @@ -1871,9 +1850,6 @@ export class FDv2SkillStore implements SkillStore { // one that says goodbye having delivered nothing, and only the former escapes // the bound. private reachedServer = false; - // Said once per store rather than once per attempt: the condition holds until - // somebody creates a skill, and delivery keeps asking throughout. - private warnedNoSkillPayload = false; constructor(sdkKey: string, options: FDv2SkillStoreOptions = {}) { const key = requireServerSideCredential(sdkKey); @@ -2171,16 +2147,6 @@ export class FDv2SkillStore implements SkillStore { this.etagBasis = null; repairingState = true; } - if (cause instanceof NoSkillPayloadError) { - this.reader.diagnostics.payloadUnavailable += 1; - if (!this.warnedNoSkillPayload) { - this.warnedNoSkillPayload = true; - warn( - `Skill delivery is idle: ${cause.message} Retrying every ${Math.round(this.maxBackoffMs)}ms; this is ` + - 'the only time it will be said.', - ); - } - } // A connection the server closed while serving it normally ended // without being a failure — see `dispatch` for which ones qualify. It // reconnects like one, but it neither counts against @@ -2203,22 +2169,16 @@ export class FDv2SkillStore implements SkillStore { } } const requested = cause.retryAfterMs; - const delay = - cause instanceof NoSkillPayloadError - ? // At the cap rather than on the backoff schedule: `failures` - // deliberately never moves, so the schedule would hold this at - // the *initial* delay forever. - this.maxBackoffMs - : Math.min( - requested !== null && Number.isFinite(requested) - ? // A server asking for no delay still gets one: honouring - // `Retry-After: 0` literally would reconnect in a loop and burn - // the whole retry bound in milliseconds. - Math.max(requested, this.initialBackoffMs) - : backoffDelayMs(this.failures, this.initialBackoffMs, this.maxBackoffMs), - // `Retry-After` is a request and `maxBackoffMs` is a promise. - this.maxBackoffMs, - ); + const delay = Math.min( + requested !== null && Number.isFinite(requested) + ? // A server asking for no delay still gets one: honouring + // `Retry-After: 0` literally would reconnect in a loop and burn + // the whole retry bound in milliseconds. + Math.max(requested, this.initialBackoffMs) + : backoffDelayMs(this.failures, this.initialBackoffMs, this.maxBackoffMs), + // `Retry-After` is a request and `maxBackoffMs` is a promise. + this.maxBackoffMs, + ); if (!cause.expected) { warn(`Skill delivery failed (${cause.message}); retrying in ${Math.round(delay)}ms`); } From 3eea7e0d541e852ed4002cb1a060470a0e9de57f Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 29 Sep 2026 12:57:54 -0400 Subject: [PATCH 47/55] Update skills-fdv2.ts --- packages/client/src/skills-fdv2.ts | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 72c3aa06..1d03f5dc 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -1205,23 +1205,12 @@ export function classifyStatus(status: number, headers?: Headers | null): Error // be one the server no longer accepts. Recoverable so the selector can be // dropped and a full transfer requested; fatal once that has been tried. if (status === 400) return new StaleRequestStateError(`LaunchDarkly returned HTTP 400. ${REQUEST_ADVICE}`); - // Its own branch rather than the generic fatal list below, because the - // message has to name the two configurations that produce it. LaunchDarkly - // chose 422 *to be* terminal — the streamer says so at both places that emit - // it — so retrying is the one reading the platform ruled out. - // - // The key's scoping leads because it is the only one of the two a reader can - // act on. Skill delivery is not enabled per account as a customer-facing - // step, so a closed gate is a LaunchDarkly-side condition — a kill switch, or - // a rollout that has not reached them — and telling someone to go enable it - // would send them looking for a setting they will not find. + + // View-scoped SDK keys can't carry skills payloads yet, so that is the most likely cause of a 422 if (status === 422) { return new FatalTransportError( 'LaunchDarkly will not deliver Agent Skills on this connection (HTTP 422). The usual cause is a view-scoped ' + - 'SDK key: a key restricted to a view cannot be assigned a skill payload, so check whether this key is ' + - 'view-scoped and use one that is not. Failing that, Agent Skills delivery is not enabled for this account, ' + - 'which is not something you can turn on yourself — contact LaunchDarkly support. Retrying fixes neither, so ' + - 'delivery has stopped; the process must be restarted once the cause is resolved.', + 'SDK key. Check your SDK key or contact LaunchDarkly support.', ); } if ([405, 406, 414, 501].includes(status)) { From ea02aecb00a675e4d54f923821ca7c7c562af50e Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 29 Sep 2026 13:00:54 -0400 Subject: [PATCH 48/55] Update agents.md --- packages/client/agents.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/packages/client/agents.md b/packages/client/agents.md index 11189965..92c64e51 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -59,11 +59,7 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b **No `mv` parameter, deliberately.** It selects the *flag* data model, and delivery overrides whatever a request asks for with the payload's own default for any non-flagging payload, so sending it would state a preference that is ignored. -**HTTP 422 is fatal, and the platform decided that rather than the SDK inferring it.** Delivery answers 422 when a connection's declared kinds exclude every payload it is assigned, and it picked a non-400 4xx *because* LD SDKs treat those as terminal — the streamer says so in as many words at both places that emit it, each noting the code exists so a misconfigured SDK stops instead of hammering the fleet. So `classifyStatus` returns a `FatalTransportError` and the existing give-up path handles it: `failed` and `lastError` are set, `connectionFailures` is untouched (it measures consecutive *recoverable* failures against the retry bound, and a fatal never retries), and `waitForSkills` resolves `false` at once rather than at the timeout. It gets its own branch in `classifyStatus` rather than joining the generic fatal list because the message is a contract — it is what a customer pastes into a support ticket, so it names the two real causes. - -**The 422 is not about whether any skill exists.** The gate is `payloadvers.SkillDeliveryAllowed`: skill delivery enabled for the account **and** a credential that is not view-scoped. Each way of failing it is permanent from the store's side — the account gate is off (`enable-aic-agent-skills`), the SDK key is view-scoped, or the declared kind is a typo. With the gate open and a non-view-scoped key the assignment path creates the agent-skill payload row **lazily**, so an environment holding zero skills is assigned an empty payload that commits normally through `payload-transferred`. - -**The message leads with the key's scoping, because that is the only cause a reader can act on.** The account gate is not a customer-facing enablement step — it is expected to be flipped on for everyone rather than targeted per account — so a closed gate is a LaunchDarkly-side condition, a kill switch or an unfinished rollout, and instructing someone to go enable Agent Skills for their account would send them looking for a setting they do not have. So the view-scoped key carries the instruction and the gate routes to support. The cost of fatality, stated so it is not rediscovered as a bug: a process whose store gave up needs a restart once the cause is resolved, since `close()` is final and nothing reopens delivery short of a new store. That is deliberately preferred to retrying a permanent rejection for the life of the process. +**HTTP 422 is fatal, and the platform decided that rather than the SDK inferring it.** Delivery answers 422 when a connection's declared kinds exclude every payload it is assigned, and it picked a non-400 4xx *because* LD SDKs treat those as terminal. So `classifyStatus` returns a `FatalTransportError` and the existing give-up path handles it: `failed` and `lastError` are set, `connectionFailures` is untouched (it measures consecutive *recoverable* failures against the retry bound, and a fatal never retries), and `waitForSkills` resolves `false` at once rather than at the timeout. **Polling and streaming have different default hosts.** `GET /sdk/poll` is served from `DEFAULT_BASE_URI` (`sdk.launchdarkly.com`) and `GET /sdk/stream` from `DEFAULT_STREAM_URI` (`stream.launchdarkly.com`), matching the base server-side SDK. Sending the stream request to the polling host fails against a real environment, and the fake endpoint cannot catch it because it serves both from one origin. `baseUri` given alone therefore applies to both, since a relay or private instance usually serves both from one host; only the LaunchDarkly defaults split them, and `streamUri` overrides the stream origin on its own. From cc86953baafc0dfd967005dd24faa64b9d31e7f4 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Mon, 28 Sep 2026 13:30:53 -0400 Subject: [PATCH 49/55] docs(client): correct what the kinds declaration made stale Declaring `?kinds=agent-skill` changed what arrives on the connection, and two paragraphs in the client README still described the old shape: - "The connection also carries your flags" told customers a client cannot request only the skill payload and that `objectsIgnored` counts the flag and segment objects that arrive anyway. Both halves are now the opposite of what happens: `FDV2_PAYLOAD_KIND` is on every request, which is what makes the connection carry exactly one payload. - the `addListener` paragraph repeated the same claim in passing, as the reason a non-skill kind is never dispatched. `objectsIgnored` keeps an explanation rather than losing one, because the counter still exists and now means something narrower and worth saying: an object kind this version does not recognise, not the environment's flags. That matches the Python SDK's wording for `objects_ignored` (python-ai-sdk#109, 91c9e20), which rewrote this paragraph rather than deleting it; the two READMEs stay in parity. `payloadsIgnored` needed no change here. The README table lists it without prose, so the "Zero while delivery sends one payload per connection" wording is only on the field's own doc comment in skills-fdv2.ts, and the declaration makes that condition unconditional rather than false. README only. The stale `objectsIgnored` doc comment in skills-fdv2.ts, and the same claim in agents.md, are untouched and still to do. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 91d072d9..3c686f8f 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -402,7 +402,7 @@ try { **`close()` is final.** It stops delivery and does not undo it: `start()` throws on a closed store rather than opening a second connection, since a store that looks live and is not is worse than one that plainly refuses. Retrieval needs no restart — a closed store keeps answering from the content it received — so construct a new store if you need delivery again. -**`addListener` observes skill changes only.** The store notifies changes to `'skill'` objects; flag and segment objects on the same connection are skipped, never dispatched. A listener registered under any other kind therefore throws rather than being accepted and silently never firing. `removeListener` tolerates any kind, so a consumer that detaches on close can do so unconditionally. `watchSkills` is the intended consumer of both. +**`addListener` observes skill changes only.** The store notifies changes to `'skill'` objects; anything else on the connection is skipped, never dispatched. A listener registered under any other kind therefore throws rather than being accepted and silently never firing. `removeListener` tolerates any kind, so a consumer that detaches on close can do so unconditionally. `watchSkills` is the intended consumer of both. **A 422 means this connection will never be assigned a skill payload, and delivery stops.** Every request declares the payload it wants (`kinds=agent-skill`), and LaunchDarkly answers HTTP 422 when it will not serve one. The cause you can act on is a **view-scoped SDK key**: a key restricted to a view cannot be assigned a skill payload, so check the key's scoping and use one that is not view-scoped. The other cause is that Agent Skills delivery is not enabled for your account, which is not a setting you control — contact LaunchDarkly support if the key is not the problem. Retrying fixes neither, and LaunchDarkly chose the status so that SDKs stop rather than hammer the fleet, so the store gives up: `failed` carries the reason, `lastError` is populated, `connectionFailures` stays at zero (it counts consecutive *recoverable* failures against the retry bound, which a fatal never spends), and `waitForSkills` resolves `false` immediately instead of at your timeout. It is **not** the answer for an environment that merely has no skills yet — with delivery enabled and a non-view-scoped key, an environment holding zero skills is served an empty payload that commits normally. Because delivery has stopped for good, a process that booted while the cause was in effect picks up skills only after a restart. @@ -420,7 +420,7 @@ try { **One network timeout, and its default depends on the mode.** `readTimeoutMs` bounds every step of a request, connecting included. In `mode: 'poll'` it bounds the whole request and defaults to 10 seconds; in `mode: 'stream'` it bounds each wait for the next bytes and defaults to 300 seconds, well beyond LaunchDarkly's heartbeat interval. A stream that goes quiet past it reconnects rather than hanging, and a stream that dies mid-body — a reset, a truncated chunk — reconnects the same way. Every delay between retries, including one the server asks for with `Retry-After`, is capped at `maxBackoffMs`. -**The connection also carries your flags.** A client cannot request only the skill payload, so a skills-enabled environment delivers flag and segment objects on the same connection. They are skipped, not evaluated — this store does no evaluation of any kind — and `diagnostics.objectsIgnored` counts them. +**The connection carries only skills.** Every request declares the skill payload, so flag and segment objects no longer arrive on it. Anything that is not a skill is still skipped rather than rejected — this store does no evaluation of any kind — and `diagnostics.objectsIgnored` counts those: a nonzero count means the payload gained an object kind this version does not recognise, not that something failed. > **Beta caveats, worth knowing before you deploy.** Payload signing does not exist on this channel yet, so delivery is TLS-only — the store refuses a cleartext `baseUri`/`streamUri` and refuses to follow a redirect — and the content hash establishes self-consistency, not origin authenticity. The FDv2 protocol is opt-in per account: without it the endpoints return HTTP 403, which the store reports as a fatal error explaining what to do. `ld-relay` does not speak the FDv2 endpoints, so relay-only deployments cannot receive skills. From 78659d8f8c1e58ef34a16a00b273677d314a36fc Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 29 Sep 2026 12:08:24 -0400 Subject: [PATCH 50/55] docs(client): finish correcting what the kinds declaration made stale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README fix left three places still describing the pre-declaration shape. `objectsIgnored`'s own doc comment said what the README paragraph did — "flags, segments, and any future kind" — so the public API docs still told a reader the counter tallies their environment's flags. It now says what it actually counts, matching the README and the Python docstring. `REQUEST_ADVICE` enumerates what the request carries and left `kinds` out. It is the text a user reads on the 400/405/406/414/501 family, which is exactly what an endpoint that does not understand `kinds` would answer, so the omission pointed at the base URI instead of the likely cause. agents.md attributed single-payload delivery to delivery itself ("Delivery provides one payload per credential"), which the paragraph twelve lines above now contradicts: a skill-enabled environment assigns two, and it is the declaration that narrows the connection to one. The reason is now the declaration, which is also what makes the rest of that paragraph — the reader's payload-id check and `payloadsIgnored` — read as the residual it is rather than as a guarantee. Two things Python changed that JS deliberately does not. `payloadUnavailable` documents itself with `{@link NoSkillPayloadError}`, which is fine here because that class is exported; the Python equivalent was private, which is why it was reworded there. And `payloadsIgnored` keeps "Zero while delivery sends one payload per connection": the declaration makes that condition unconditional rather than false, and Python left its wording alone too. Note that Python's agents.md still carries the "one payload per credential" claim this commit fixes, so the two trees diverge on that line until python-ai-sdk catches up. Gate: tsc --noEmit clean, biome check clean, vitest run 1024 passed / 10 skipped in packages/client — the same counts the declaration commit recorded, and the ten are the capability-gated TOCTOU tests. Co-Authored-By: Claude Opus 5 --- packages/client/agents.md | 2 +- packages/client/src/skills-fdv2.ts | 12 +++++++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/packages/client/agents.md b/packages/client/agents.md index 92c64e51..de4a9ae0 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -67,7 +67,7 @@ The `3` after the delimiter is what a `{key, version}` reference pins and what b **`objectsRevoked` counts revocations, not tombstones.** A full transfer revokes by omission, so `payloadTransferred` diffs the committed set against the pending one and pushes the departures into `changes` as tombstones at `(key, version)` granularity — a listener that reads versions needs both halves of a version move. The *counter* is coarser on purpose: only a key the payload dropped altogether counts, because a key surviving under a new version was never revoked, and this is a field operators alert on. `keysFullyRevoked` is where the two granularities part. -**The first payload intent is read, and is assumed to be the skill payload.** Delivery provides one payload per credential and the protocol requires a client to ignore all but the first payload intent, so `payloads[0]` is both what arrives and what the protocol says to read. The cost of that assumption is that an `xfer-full` for somebody *else's* payload would start an empty pending set, and the next `payload-transferred` would publish it — every skill reported revoked, and with pruning on, a customer's files deleted. `ProtocolReader` therefore learns which payload skills arrive on, from the intent's `id` or from the `(p::)` selector, and declines to apply a transfer of any other: once at warning level, counted in `diagnostics.payloadsIgnored`, holding last known good. A transfer that names no payload is applied, since one-payload delivery is the common case. The residual is the first transfer of a connection — before a skill has arrived there is nothing to compare against — which is what the separate warning on a multi-payload intent is for. +**The first payload intent is read, and is assumed to be the skill payload.** The `kinds` declaration narrows the connection to one payload and the protocol requires a client to ignore all but the first payload intent, so `payloads[0]` is both what arrives and what the protocol says to read. The cost of that assumption is that an `xfer-full` for somebody *else's* payload would start an empty pending set, and the next `payload-transferred` would publish it — every skill reported revoked, and with pruning on, a customer's files deleted. `ProtocolReader` therefore learns which payload skills arrive on, from the intent's `id` or from the `(p::)` selector, and declines to apply a transfer of any other: once at warning level, counted in `diagnostics.payloadsIgnored`, holding last known good. A transfer that names no payload is applied, since one-payload delivery is the common case. The residual is the first transfer of a connection — before a skill has arrived there is nothing to compare against — which is what the separate warning on a multi-payload intent is for. **A hashless object is held, not dropped.** Verification withholds it with `missing_content_hash`; the transport's job is to make that loud (an error per object, a summary each time the held store becomes wholly hashless or its withheld set changes, `diagnostics.hashlessObjects`) rather than to work around it. Neither error repeats for a payload re-delivered unchanged, which matters most in polling mode, where the same payload arrives on every interval; each `ProtocolReader` holds its own `HashlessMemory` of what has been said (exposed to tests as `_warnedHashless`), per store rather than module-scoped so two stores in one process do not suppress each other's reports, and capped so a frequently versioned environment cannot accumulate an entry per version. Dropping it at the transport would report `absent` — indistinguishable from "no such skill" — and would let a prune delete the last known-good copy on disk. Never synthesize a hash from the delivered content: that certifies the content against itself and verifies nothing. diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index 1d03f5dc..eb2f376d 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -275,9 +275,10 @@ export type StoreDiagnostics = { /** `put-object` events identified as skills, across all payloads. */ readonly skillObjectsReceived: number; /** - * Objects skipped because they were not skills — flags, segments, and any - * future kind. Skipping is the contract, not a failure; the count exists so a - * mixed payload is visibly mixed. + * Objects skipped because they were not skills. With the skill payload + * declared on every request, this counts an object kind this version does not + * recognise rather than the environment's flags. Skipping is the contract, + * not a failure. */ readonly objectsIgnored: number; /** @@ -1132,8 +1133,9 @@ export class RecoverableTransportError extends Error { export class StaleRequestStateError extends RecoverableTransportError {} const REQUEST_ADVICE = - 'The request this adapter sent was not understood. It carries only the SDK key and, after the first payload, ' + - "a 'basis' selector, so check the base URI and that the endpoint speaks FDv2."; + 'The request this adapter sent was not understood. It carries only the SDK key, a ' + + "'kinds' parameter declaring the skill payload, and, after the first payload, a 'basis' selector, so check " + + 'the base URI and that the endpoint speaks FDv2.'; const FORBIDDEN_ADVICE = 'The FDv2 protocol is opt-in per LaunchDarkly account and is served as HTTP 403 while it is off. Skill ' + From 21c67e09087de2d6b483de2bea72482a7eac4c89 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Tue, 29 Sep 2026 14:20:31 -0400 Subject: [PATCH 51/55] Update skills-fdv2.test.ts --- .../client/src/__tests__/skills-fdv2.test.ts | 23 ++----------------- 1 file changed, 2 insertions(+), 21 deletions(-) diff --git a/packages/client/src/__tests__/skills-fdv2.test.ts b/packages/client/src/__tests__/skills-fdv2.test.ts index cd0971f4..387aab71 100644 --- a/packages/client/src/__tests__/skills-fdv2.test.ts +++ b/packages/client/src/__tests__/skills-fdv2.test.ts @@ -2196,30 +2196,11 @@ describe('failure handling', () => { } }); - it('names the account enablement and the key scoping in the 422 message', () => { - // The message is a contract: it is what a customer pastes into a support - // ticket, so it has to name both real causes without them reading platform - // source. Asserted on substance rather than prose, so the wording can be - // improved without rotting this test. + it('names the key scoping in the 422 message', () => { const message = classifyStatus(422).message; expect(message).toMatch(/422/); expect(message).toMatch(/view-scoped/i); - expect(message).toMatch(/not enabled for this account/i); - // Of the two causes only the key's scoping is the reader's to fix, so that - // is the one carrying an instruction. A closed account gate is a - // LaunchDarkly-side condition, so it routes to support rather than sending - // someone to look for a setting they do not have. - expect(message).toMatch(/check whether this key is view-scoped/i); - expect(message).toMatch(/contact LaunchDarkly support/i); - // Not an instruction to go enable it for their own account: skill delivery - // is not enabled per account as a customer-facing step. - expect(message).not.toMatch(/(?:once|after) Agent Skills is enabled for (?:the|your) account/i); - expect(message).not.toMatch(/enable Agent Skills for (?:the|your) account/i); - // And the two things it must not say. Neither is true: the payload row is - // created lazily, so an environment with zero skills commits an empty - // payload normally — and nothing reopens delivery short of a restart. - expect(message).not.toMatch(/no skills|first skill|has no Agent Skills payload/i); - expect(message).not.toMatch(/without a restart|keeps asking|picks (?:one|it) up/i); + expect(message).toMatch(/Check your SDK key or contact LaunchDarkly support/i); }); it('stops delivery on a 422 answering the first request', async () => { From e707d0db8600dab2e9c38487007bf73c32341b8e Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Mon, 28 Sep 2026 16:01:23 -0400 Subject: [PATCH 52/55] fix(client): record the version mismatch on the log surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `resolveFromStore` withholds a skill when a pin was answered with a different version, and wrote nothing at all: no log record, no signal, and no stated reason for the silence in the code, in `agents.md`, or in TESTING.md. The branch immediately above it — `key_mismatch` — carries fourteen lines explaining which surfaces fire and why. `recordVersionMismatch` fills the gap, following 0f90767's shape: - log record only, `reason_code: version_mismatch`, no product signal. The usual cause of either boundary check firing is a broken custom store adapter, and LaunchDarkly's own counter must not fill with customers' adapter bugs. Pinned in both directions, because an implementation that emitted the signal too would look correct from every other angle; - the record carries `version` (the version **requested**, the meaning it has on every other record) and a record-only `served_version` paralleling `served_key`. No hash fields: verification passed. Keys inserted alphabetically so `JSON.stringify` stays byte-identical to Python's `json.dumps(record, sort_keys=True, separators=(",",":"))`, modulo `language`; - `IntegrityReasonCode` grows from nine tokens to ten. `SkillOutcomeReason` stays a closed five — this changes the detection surface, not the public outcome vocabulary. The code is spelled `version_mismatch` rather than `wrong_version` to keep the `*_mismatch` family consistent and to keep one string out of two closed vocabularies with two meanings. Worth recording even though `wrong_version` is already a public outcome token, and even though neither shipped store can reach the branch: `getSkill` is the documented default and collapses `wrong_version` to `null` exactly as it collapses `integrity_failure`, so without the record an operator on the default accessor has no visibility into a store answering pins with the wrong version. Tests therefore assert it through `getSkill` as well as `getSkillResult`, and use a hand-built store that ignores its version argument — a test built on a shipped store gets `absent` and silently stops covering the case. Also pinned: the key check wins when a store disagrees on both key and version (it decides the caller-visible outcome, not just the code), the two versions are recorded as integers rather than strings, and the listing path reports no mismatch for a store spelling its map keys `key:version`, mirroring 4cfb753's guard. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 20 +- packages/client/agents.md | 18 +- packages/client/src/__tests__/skills.test.ts | 351 +++++++++++++++++-- packages/client/src/skills-core.ts | 134 ++++++- 4 files changed, 475 insertions(+), 48 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index 3c686f8f..d0f67ece 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -449,7 +449,8 @@ The line is a `[LaunchDarkly] ` prefix, the event name, a space, and a single JS | `reason` | yes | Human-readable detail, including byte counts. Wording may change between releases. | | `language` | yes | `typescript`. Distinguishes SDKs in a polyglot fleet; the Python SDK emits the same record with `python`. | | `served_key` | no | Only on `key_mismatch`: the key the store actually answered under, with the same redaction as `skill_key`. Omitted on every other failure mode. | -| `version` | no | The skill version. Omitted when the delivered version was not an integer >= 1, and on `key_mismatch`. | +| `served_version` | no | Only on `version_mismatch`: the version the store actually answered with, as an integer. Omitted on every other failure mode, and never on the same record as `served_key`. | +| `version` | no | The skill version — the version **requested** on `version_mismatch`, the delivered one everywhere else. Always an integer. Omitted when the delivered version was not an integer >= 1, and on `key_mismatch`. | | `expected_hash` | no | The `contentHash` delivered with the content, or `` when it was not 64 lowercase hex characters. Omitted when the failure happened before any hash was read. | | `observed_hash` | no | The sha256 this SDK computed locally. Omitted when the failure happened before hashing. | @@ -465,11 +466,14 @@ Optional fields are **omitted, never null** — the absence of `observed_hash` m | `not_utf8` | The content has no UTF-8 encoding (a lone surrogate), so there are no bytes LaunchDarkly could have hashed. | | `over_size_cap` | The content exceeds the internal `MAX_SKILL_CONTENT_BYTES` cap, so it is inauthentic whatever it hashes to. The reason string names the bound. | | `hash_mismatch` | The content does not hash to the `contentHash` delivered alongside it. | -| `key_mismatch` | The store answered under a different key than the one requested. Carries an extra `served_key` field naming the key it answered under, and — uniquely — records **no** `AgentControl Skill Integrity Failure` signal. | +| `key_mismatch` | The store answered under a different key than the one requested. Carries an extra `served_key` field naming the key it answered under, and records **no** `AgentControl Skill Integrity Failure` signal. | +| `version_mismatch` | You pinned a version and the store answered with a different one. Carries an extra `served_version` field naming the version it answered with, alongside a `version` field carrying the one you asked for, and records **no** `AgentControl Skill Integrity Failure` signal. The caller-facing outcome for this condition is `wrong_version` — see the table below. | -These nine tokens are the whole vocabulary, and the Python SDK emits the same nine for the same conditions — including identical JSON key order — so one parser and one alert rule cover a polyglot fleet. +These ten tokens are the whole vocabulary, and the Python SDK emits the same ten for the same conditions — including identical JSON key order — so one parser and one alert rule cover a polyglot fleet. -`key_mismatch` is the only code that reaches this log record without also reaching the product signal. It is decided after verification has passed, and its usual cause is a bug in a custom `SkillStore` adapter — a stale cache entry, a colliding key, a wrong index lookup — rather than tampering, so it does not inflate LaunchDarkly's own integrity counter. It still reaches this record, because a store substituting one skill for another is worth seeing, and your rule on `ld.skills.integrity_failure` catches it without modification. +**`key_mismatch` and `version_mismatch` are the two codes that reach this log record without also reaching the product signal.** Both are decided after verification has passed, and the usual cause of either is a bug in a custom `SkillStore` adapter — a stale cache entry, a colliding key, a wrong index lookup — rather than tampering, so neither inflates LaunchDarkly's own integrity counter. Both still reach this record, because a store that answers with something other than what was asked for is worth seeing, and your rule on `ld.skills.integrity_failure` catches them without modification. + +**`version_mismatch` is worth alerting on even though `wrong_version` is already a `getSkillResult` outcome.** `getSkill` is the simpler default and collapses `wrong_version` to `null` exactly as it collapses an integrity failure, so this record is the only visibility an operator gets when the code calling `getSkill` treats a `null` as "no skill". Neither store shipped with this SDK can produce it — both answer a pin with exactly that version or with nothing at all, which reads as `absent` — so if you see this code, suspect whatever `SkillStore` implementation is in front of it. #### Fail closed on tampering: `getSkillResult` @@ -493,7 +497,9 @@ switch (outcome.reason) { // Nobody configured this skill, or it was revoked. Ordinary; carry on. return null; case 'wrong_version': - // The pinned version is not what the store holds — a rollout skew, usually. + // The store answered the pin with a *different* version. Not a rollout + // skew — a pin the store cannot satisfy is `absent` — so this one also + // writes an `ld.skills.integrity_failure` record for your SIEM. return null; case 'store_unavailable': // The store could not answer. An outage, not an answer of "no" — retry or @@ -507,7 +513,7 @@ switch (outcome.reason) { | `ok` | the skill | Retrieved and verified. | | `absent` | `null` | The store holds nothing under that key. Not configured, not yet delivered, or revoked. | | `integrity_failure` | `null` | Content was delivered and its identity did not verify, so it was withheld. Two cases reach this token, and both write the `ld.skills.integrity_failure` record above: content that failed hash/size/shape verification, which also records the product signal; and a store that answered under a **different key** than the one requested, which writes the record with `reason_code: key_mismatch` and a `served_key` field but records **no** signal (the check runs after verification has already passed, and a mismatch is usually a store-adapter bug rather than tampering). This is the outcome to fail closed on. | -| `wrong_version` | `null` | A version was pinned and the store answered with a different one. Only a version mismatch — a key mismatch is `integrity_failure`, and there is deliberately no `wrong_key`. | +| `wrong_version` | `null` | A version was pinned and the store answered with a different one. Writes the `ld.skills.integrity_failure` record above with `reason_code: version_mismatch` and a `served_version` field, and — like `key_mismatch` — records **no** product signal. Only a version mismatch reaches this token: a key mismatch is `integrity_failure`, and there is deliberately no `wrong_key`. | | `store_unavailable` | `null` | The store threw. Nothing was retrieved, so nothing is known either way. | `detail` carries the human-readable reason for every non-`ok` outcome and is `null` for `ok`. It is safe to log or surface: it names the key, the requested and held versions, and the failure category, and never skill content or a filesystem path. @@ -516,7 +522,7 @@ switch (outcome.reason) { These five tokens are the whole vocabulary, and the Python SDK publishes the same five for the same conditions. There is deliberately no batch equivalent: `getSkills` and `allSkills` still omit entries they could not return, so call `getSkillResult` per key where the outcome matters. -**`hash_mismatch` deserves a page, not a dashboard.** The other codes are consistent with a malformed store, a bad deployment, or a truncated response. `hash_mismatch` means content and its declared digest disagree, which is the shape of active tampering with skill delivery — in transit, in a cache, or in whatever backs your `SkillStore`. If you serve skill content only from LaunchDarkly, `over_size_cap` and `not_utf8` warrant alerts on the same reasoning: neither should ever occur. `key_mismatch` belongs in the same tier as `hash_mismatch` **if** you use the SDK's own `FDv2SkillStore` and nothing else, since it then also means delivery served the wrong skill; behind a custom store adapter, suspect the adapter first. +**`hash_mismatch` deserves a page, not a dashboard.** The other codes are consistent with a malformed store, a bad deployment, or a truncated response. `hash_mismatch` means content and its declared digest disagree, which is the shape of active tampering with skill delivery — in transit, in a cache, or in whatever backs your `SkillStore`. If you serve skill content only from LaunchDarkly, `over_size_cap` and `not_utf8` warrant alerts on the same reasoning: neither should ever occur. `key_mismatch` and `version_mismatch` belong in the same tier as `hash_mismatch` **if** you use the SDK's own `FDv2SkillStore` and nothing else, since neither is reachable through it and both would then mean delivery served something other than what was asked for; behind a custom store adapter, suspect the adapter first. #### Privilege separation: the agent must not be able to rewrite its own skills diff --git a/packages/client/agents.md b/packages/client/agents.md index de4a9ae0..9534cca7 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -437,9 +437,9 @@ Two decisions here look like unfinished work and are not. Neither should be reve - Do not interpret skill content anywhere — no frontmatter parsing, no YAML dependency, no encoding assumption beyond the one wire-string encode inside integrity verification. See pitfall 3. - Skills telemetry goes through the private emitter seam in `skills-core.ts` and nowhere else. **Never** call `client.track()` for a skills operation: these are LaunchDarkly product-analytics signals, not customer analytics, and `track()` would require an LD context, spend the customer's event volume, and land in their data export. Exactly three signal names exist (`AgentControl Skill Integrity Failure`, `AgentControl Skill Materialized`, `AgentControl Skill Revoked Received`) and they are an allowlist, not a floor — in particular, `AgentControl Skill SDK Reference Returned` and `AgentControl Skill Content Retrieved` must **never** be emitted. Every signal is constructed by a `record*` function in `skills-core.ts`, so the allowlist is enforceable by reading one section of one file. - No filesystem paths and no skill content in telemetry — hashes and byte counts only. Paths belong in the `ReconcileReport`, which is user-facing API. -- The `ld.skills.integrity_failure` log record is a **documented customer-facing contract**, not a debug line. It is constructed in exactly one place — `recordIntegrityFailure` — and written to `console.error` on every failure, unconditionally, because it is the only detection surface a customer with telemetry off has and the only one that can exist where no telemetry destination is reachable. Its keys are inserted in **alphabetical order** so `JSON.stringify` is byte-identical to the Python SDK's `json.dumps(record, sort_keys=True, separators=(",", ":"))` for the same failure, modulo `language`; do not reorder them. Absent optional fields are omitted, never emitted as `null`. Renaming the event, renaming or dropping a field, or emitting a null is a breaking change to a security control — the README documents this record for customers to alert on. -- `reason_code` is a **closed nine-token vocabulary**: `hash_mismatch`, `invalid_key`, `invalid_version`, `key_mismatch`, `missing_content`, `missing_content_hash`, `not_an_object`, `not_utf8`, `over_size_cap`. Eight are one token per call site of `recordIntegrityFailure`; `key_mismatch` is the exception, coming from `recordKeyMismatch` at the retrieval boundary and firing the log record without the signal (see the outcome table below). A tenth token means adding it in the Python SDK, in the README's `reason_code` table, and in the vocabulary test in `src/__tests__/skills.test.ts`, in the same change — a token that exists on one side only silently breaks a customer's cross-language alert rule. `reason_code` deliberately stays **out** of the telemetry properties: the three signals' property keys are the allowlist above, and the vocabulary belongs to the customer-owned detection path rather than to LaunchDarkly's counter. -- `SkillOutcomeReason` is a **closed five-token vocabulary** and the public half of what `resolveFromStore` decides: `absent`, `integrity_failure`, `ok`, `store_unavailable`, `wrong_version`. It is published by `getSkillResult` as API, customers branch on it, and the Python SDK publishes the same five for the same conditions — so a sixth token is a cross-language change, and it means adding it in the Python SDK, in the README's `reason` table, and in the union-exhaustiveness test in `src/__tests__/skills.test.ts`, in the same change. Note this is a **different and coarser** vocabulary from `IntegrityReasonCode`: the nine integrity tokens say *which check failed* and go to the operator through the log record; the five outcome tokens say *what the caller got* and are the actionable programmatic surface. `verifyRawSkill` returns `null` and does not surface which integrity token fired, deliberately — plumbing the finer token into `SkillOutcome` would change that function's return type for a detail the operator already has. +- The `ld.skills.integrity_failure` log record is a **documented customer-facing contract**, not a debug line. It is constructed by the `record*` recorders in `skills-core.ts` and nowhere else — `recordIntegrityFailure` for the eight verification failures, `recordKeyMismatch` and `recordVersionMismatch` for the two boundary ones — and written to `console.error` on every failure, unconditionally, because it is the only detection surface a customer with telemetry off has and the only one that can exist where no telemetry destination is reachable. Its keys are inserted in **alphabetical order** so `JSON.stringify` is byte-identical to the Python SDK's `json.dumps(record, sort_keys=True, separators=(",", ":"))` for the same failure, modulo `language`; do not reorder them. Absent optional fields are omitted, never emitted as `null`. Renaming the event, renaming or dropping a field, or emitting a null is a breaking change to a security control — the README documents this record for customers to alert on. +- `reason_code` is a **closed ten-token vocabulary**: `hash_mismatch`, `invalid_key`, `invalid_version`, `key_mismatch`, `missing_content`, `missing_content_hash`, `not_an_object`, `not_utf8`, `over_size_cap`, `version_mismatch`. Eight are one token per call site of `recordIntegrityFailure`; `key_mismatch` and `version_mismatch` are the exceptions, coming from `recordKeyMismatch` and `recordVersionMismatch` at the retrieval boundary and firing the log record without the signal (see the outcome table below). Both are only reachable once all eight verification checks have passed, so no input produces one of them *and* a verification code; they can compete with **each other**, and the key check runs first, because an answer that is not the requested skill at all makes its version moot. An eleventh token means adding it in the Python SDK, in the README's `reason_code` table, and in the vocabulary test in `src/__tests__/skills.test.ts`, in the same change — a token that exists on one side only silently breaks a customer's cross-language alert rule. `reason_code` deliberately stays **out** of the telemetry properties: the three signals' property keys are the allowlist above, and the vocabulary belongs to the customer-owned detection path rather than to LaunchDarkly's counter. +- `SkillOutcomeReason` is a **closed five-token vocabulary** and the public half of what `resolveFromStore` decides: `absent`, `integrity_failure`, `ok`, `store_unavailable`, `wrong_version`. It is published by `getSkillResult` as API, customers branch on it, and the Python SDK publishes the same five for the same conditions — so a sixth token is a cross-language change, and it means adding it in the Python SDK, in the README's `reason` table, and in the union-exhaustiveness test in `src/__tests__/skills.test.ts`, in the same change. Note this is a **different and coarser** vocabulary from `IntegrityReasonCode`: the ten integrity tokens say *which check failed* and go to the operator through the log record; the five outcome tokens say *what the caller got* and are the actionable programmatic surface. The version mismatch is the one condition with a token in each, and the spellings differ on purpose — `version_mismatch` is the code, `wrong_version` the outcome — so a detection rule matching either string is unambiguous about which surface it was written against. Do not "unify" them: `version_mismatch` keeps the `*_mismatch` family consistent, and one string in two closed vocabularies with two meanings is what the split avoids. `verifyRawSkill` returns `null` and does not surface which integrity token fired, deliberately — plumbing the finer token into `SkillOutcome` would change that function's return type for a detail the operator already has. - `Resolution.reason` is set **explicitly at every construction site**, never derived from `Resolution.error`. Pattern-matching a prose string to decide what a customer's fail-closed branch sees is the fragility the typed outcome exists to remove, and the field is required precisely so a new internal outcome has to pick a public token rather than inherit `'absent'` by omission. The mapping today: | Where the resolution is built | `reason` | @@ -448,18 +448,22 @@ Two decisions here look like unfinished work and are not. Neither should be reve | `resolveFromStore` — `raw` is not an object (null, an array, a scalar) | `absent` | | `resolveFromStore` — `verifyRawSkill` returned `null` | `integrity_failure` | | `resolveFromStore` — `skill.key !== key` (the store answered under another key) | `integrity_failure` | - | `resolveFromStore` — `skill.version !== wantedVersion` | `wrong_version` | + | `resolveFromStore` — `skill.version !== wantedVersion` (the store answered a pin with another version) | `wrong_version` | | `resolveFromStore` — success | `ok` | | `skills-fs.ts` `resolveReference` — deadline exhausted, or no store configured (both set `unavailable`) | `store_unavailable` | The two `integrity_failure` rows are **not** interchangeable, and the difference is the easy thing to get wrong. The `verifyRawSkill` row fires *inside* verification, so it records the `AgentControl Skill Integrity Failure` signal **and** writes the `ld.skills.integrity_failure` log record. The key-mismatch row fires *after* verification has passed, so it goes through `recordKeyMismatch` instead and writes **the log record only** — `reason_code: key_mismatch`, plus a `served_key` field no other record carries, and no product signal. - That asymmetry is the decision, and it is pinned by tests in both directions. The record fires because a substituting store is worth seeing and the record is the customer-owned detection path; reusing the event identity means an existing SIEM rule catches it with the code as the discriminator. The signal does not, because the usual cause is a broken custom store adapter rather than an attacker, and LaunchDarkly's own counter must not fill with customers' adapter bugs — the same false positive the `raw` is-not-an-object row above refuses by reading as `absent`. Do not "fix" either half: emitting the signal here, or dropping the record, each breaks a test that says why. + **The `wrong_version` row writes a record too**, and for the same reasons — `recordVersionMismatch`, `reason_code: version_mismatch`, a record-only `served_version` field beside a `version` field carrying the version *requested*, and no product signal. The two boundary rows are the same decision made at the same place for the same population, so they stay in step: a change that gives one of them a signal has to justify it for both. This row was silent until the record was added; do not read the outcome token as a reason it can stay silent, because `getSkill` collapses `wrong_version` to `null` exactly as it collapses `integrity_failure`, and an operator on the default accessor would otherwise have no visibility into a store answering pins with the wrong version at all. - Keep the check where it is, too. `verifyRawSkill` is unary and decides whether one object is internally self-consistent; a key mismatch is relational. Threading an expected key into it would make the parameter optional at every call site, and a check that silently vanishes when a caller forgets an argument is worse than one at the single site where both values are in hand. + That asymmetry is the decision, and it is pinned by tests in both directions for each row. The records fire because a store answering with something other than what was asked for is worth seeing and the record is the customer-owned detection path; reusing the event identity means an existing SIEM rule catches both with the code as the discriminator. The signals do not, because the usual cause is a broken custom store adapter rather than an attacker, and LaunchDarkly's own counter must not fill with customers' adapter bugs — the same false positive the `raw` is-not-an-object row above refuses by reading as `absent`. Do not "fix" either half of either row: emitting the signal, or dropping the record, each breaks a test that says why. + + Neither is reachable through a shipped store, and that is not an argument for dropping them. `FDv2SkillStore` and `InMemorySkillStore` both answer a pin with exactly that version or with nothing, so `wrong_version` needs a **hand-built** store that ignores its version argument — a test that tries to provoke it from a shipped store gets `absent` and silently stops covering the row. + + Keep both checks where they are, too. `verifyRawSkill` is unary and decides whether one object is internally self-consistent; a key or version mismatch is relational — a disagreement between the object and the request. Threading an expected key or version into it would make the parameter optional at every call site, and a check that silently vanishes when a caller forgets an argument is worse than one at the single site where both values are in hand. Adding an eighth row means answering "which of the five does a caller see?" before writing the code. `unavailable` stays a separate field rather than folding into `reason`: it is narrower, it is what suppresses pruning, and the two bottom rows above never reach an accessor at all. -- `wantedVersion` is passed **into** `SkillStore.getObject(kind, key, version)`, and the post-hoc `skill.version !== wantedVersion` check is kept anyway. The parameter is there because a store may hold several versions of one key and only the store can pick between them; the equality check is a **defense**, not the selection mechanism, because the store is untrusted. Removing either one is wrong: without the parameter a satisfiable pin gets reported as `wrong_version`, and without the check a lying store gets its answer through. Both shipped stores hold several versions of one key and honour the parameter: `getObject` answers a pin with exactly that version, and an omitted version with the newest held. A pin that misses while well-formed versions exist is `absent`; a pin that matches nothing well-formed falls through to the version-less entry, so a malformed object reaches verification and is withheld with a signal rather than reading as a deletion. +- `wantedVersion` is passed **into** `SkillStore.getObject(kind, key, version)`, and the post-hoc `skill.version !== wantedVersion` check is kept anyway. The parameter is there because a store may hold several versions of one key and only the store can pick between them; the equality check is a **defense**, not the selection mechanism, because the store is untrusted. Removing either one is wrong: without the parameter a satisfiable pin gets reported as `wrong_version` — and now writes a `version_mismatch` record accusing a correct store — and without the check a lying store gets its answer through. Both shipped stores hold several versions of one key and honour the parameter: `getObject` answers a pin with exactly that version, and an omitted version with the newest held. A pin that misses while well-formed versions exist is `absent`; a pin that matches nothing well-formed falls through to the version-less entry, so a malformed object reaches verification and is withheld with a signal rather than reading as a deletion. - `getSkill`'s contract — "resolves to `null`, never rejects; throws only when no store is configured" — is **frozen**. It is documented in its JSDoc and in the README, and every existing caller treats that `null` as "no skill", so a reason must be added *alongside* it (as `getSkillResult` was) and never by changing what `getSkill` returns. `getSkillResult` is a projection of the same `Resolution` and shares the single throw; it records no telemetry and writes no log record of its own, because the integrity record already fired inside verification before the resolution returned and reporting it again would double-count one failure. `getSkills` and `allSkills` deliberately have no reporting equivalents yet. - This package has no logger abstraction — every `console.*` call site carries a `biome-ignore` saying so. Introducing a `logger` option is a package-wide API decision affecting unrelated call sites, not a skills change; until one exists, the integrity record has to be self-describing in the string it logs, which is why the event name appears both in the prefix and in the JSON. - Signal names, property keys, wire field names (`contentHash`), the manifest filename and format, and the exported constants are **identical strings** to the Python SDK. A polyglot fleet has to reconcile the same directory identically, so changing one of these is a cross-language breaking change. diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 531978ac..3d8621a6 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -1022,6 +1022,148 @@ describe('getSkill', () => { expect(rest).toHaveLength(1); expect(rest[0]).toEqual(record); }); + + it('records the version mismatch on the log surface but not the product signal', async () => { + // The same split the key mismatch gets, made at the same boundary for the + // same population — a broken custom store adapter — so it is asserted in + // both directions here too. + // + // Asserted through `getSkill` specifically, which is the whole reason the + // record exists. `wrong_version` is a public `SkillOutcomeReason`, so a + // `getSkillResult` caller already learns the condition programmatically; but + // `getSkill` is the documented default and collapses it to `null` exactly as + // it collapses `integrity_failure`, so without this record an operator on + // the default accessor has zero visibility into a store answering pins with + // the wrong version. + // + // Neither shipped store can get here — both answer a pin with exactly that + // version or `null`, which is `absent` — so the store is hand-built. + const lying: SkillStore = { + getObject() { + return rawSkill({ key: 'pdf-extraction', version: 9 }); + }, + allObjects() { + return {}; + }, + }; + _setStore(lying); + + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let errorCalls: unknown[][] = []; + let collapsed: Skill | null; + try { + collapsed = await getSkill('pdf-extraction', { version: 2 }); + } finally { + errorCalls = [...spy.mock.calls]; + spy.mockRestore(); + } + + // The skill is withheld, and the default accessor says only `null`. + expect(collapsed).toBeNull(); + // The signal surface saw nothing at all — not merely no integrity signal. + expect(emitter.records).toEqual([]); + + const lines = errorCalls.filter(([first]) => String(first).includes('ld.skills.integrity_failure')); + expect(lines).toHaveLength(1); + const [first, ...rest] = lines[0]; + const line = String(first); + const record = JSON.parse(line.slice(line.indexOf('{'))) as Record; + + // The event identity is reused deliberately: an existing SIEM rule catches + // this case without being rewritten, and the code is what distinguishes it. + expect(record.event).toBe('ld.skills.integrity_failure'); + expect(record.reason_code).toBe('version_mismatch'); + expect(record.action).toBe('withheld'); + expect(record.language).toBe('typescript'); + expect(typeof record.reason).toBe('string'); + + // Both versions are named. `version` is the one requested — the meaning it + // carries on every other record — and `served_version` the one the store + // answered with, parseable rather than buried in prose. + expect(record.skill_key).toBe('pdf-extraction'); + expect(record.version).toBe(2); + expect(record.served_version).toBe(9); + + expect(rest).toHaveLength(1); + expect(rest[0]).toEqual(record); + }); + + it('records the same version mismatch through getSkillResult, once', async () => { + // The reporting accessor is the same code path, so it produces the same + // record — and only one of it. Recording twice would inflate a customer's + // alert count for one failure. + const lying: SkillStore = { + getObject() { + return rawSkill({ key: 'a', version: 9 }); + }, + allObjects() { + return {}; + }, + }; + _setStore(lying); + + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let errorCalls: unknown[][] = []; + let outcome: Awaited>; + try { + outcome = await getSkillResult('a', { version: 2 }); + } finally { + errorCalls = [...spy.mock.calls]; + spy.mockRestore(); + } + + expect(outcome.reason).toBe('wrong_version'); + expect(outcome.skill).toBeNull(); + + const lines = errorCalls.filter(([first]) => String(first).includes('ld.skills.integrity_failure')); + expect(lines).toHaveLength(1); + const line = String(lines[0][0]); + const record = JSON.parse(line.slice(line.indexOf('{'))) as Record; + expect(record.reason_code).toBe('version_mismatch'); + }); + + it('reports the key mismatch, not the version mismatch, when a store disagrees on both', async () => { + // The two boundary checks can compete, and the order decides both the code + // *and* the caller-visible outcome — so it is fixed rather than left to + // whichever check the implementation happens to reach first. The key check + // runs first: an answer that is not the requested skill at all makes its + // version moot. + const doublyWrong: SkillStore = { + getObject() { + return rawSkill({ key: 'other-key', version: 9 }); + }, + allObjects() { + return {}; + }, + }; + _setStore(doublyWrong); + + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let errorCalls: unknown[][] = []; + let outcome: Awaited>; + try { + outcome = await getSkillResult('asked-for', { version: 2 }); + } finally { + errorCalls = [...spy.mock.calls]; + spy.mockRestore(); + } + + // `integrity_failure`, not `wrong_version` — the outcome a caller is + // expected to fail closed on wins over the one it is invited to tolerate. + expect(outcome.reason).toBe('integrity_failure'); + + const lines = errorCalls.filter(([first]) => String(first).includes('ld.skills.integrity_failure')); + expect(lines).toHaveLength(1); + const line = String(lines[0][0]); + const record = JSON.parse(line.slice(line.indexOf('{'))) as Record; + expect(record.reason_code).toBe('key_mismatch'); + expect(record.served_key).toBe('other-key'); + // One record, not two: the first check to fire returns. + expect('served_version' in record).toBe(false); + }); }); // ─── getSkills ───────────────────────────────────────────────────────── @@ -1145,6 +1287,37 @@ describe('allSkills', () => { expect(errorCalls.filter(([first]) => String(first).includes('ld.skills.integrity_failure'))).toEqual([]); }); + it('files a listed object under its own version, with no version_mismatch for a disagreeing map key', async () => { + // The counterpart to the pinned path's `version_mismatch`, and the same + // asymmetry as the key-mismatch guard above. A listing carries no requested + // version, so there is nothing for the object's version to disagree *with*: + // `resolveFromStore`'s equality check is only reached with a pin. + // + // The map key is the thing that invites a "fix" here, because + // `FDv2SkillStore` spells it `key:version` — so a store whose map key says + // version 3 while the object says 5 looks like a disagreement and is not + // one. Pinned because a check added on the listing path would report a + // mismatch for every store that spells its map keys that way. + const emitter = new RecordingEmitter(); + _setEmitterForTesting(emitter); + _setStore(new DictStore({ 'a:3': rawSkill({ key: 'a', version: 5 }) })); + + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let errorCalls: unknown[][] = []; + let found: Skill[]; + try { + found = await allSkills(); + } finally { + errorCalls = [...spy.mock.calls]; + spy.mockRestore(); + } + + expect(found.map((sk) => [sk.key, sk.version])).toEqual([['a', 5]]); + // Neither surface fires: nothing failed. + expect(emitter.records).toEqual([]); + expect(errorCalls.filter(([first]) => String(first).includes('ld.skills.integrity_failure'))).toEqual([]); + }); + it('keeps an unusable object in the set so verification withholds it with a signal', async () => { // Filtering it here instead would silently shrink the resolved set, which on // the materialization path is indistinguishable from a revocation and would @@ -1691,10 +1864,32 @@ describe('integrity-failure log record', () => { // `allSkills` hands every raw value straight in. const GET = () => getSkill('a'); const ALL = () => allSkills(); + // The tenth code needs a *pinned* retrieval: without a wanted version there is + // nothing for the store's answer to disagree with. + const PINNED = () => getSkill('a', { version: 2 }); /** A one-key store serving exactly what a hostile store might serve. */ const serving = (raw: unknown) => () => new DictStore({ a: raw as RawSkillObject }); + /** + * A store that answers a pin with a well-formed object at another version. + * + * Hand-built on purpose: neither shipped store can produce this outcome. + * `FDv2SkillStore` and `InMemorySkillStore` both answer a pin with exactly + * that version or with `null`, so a pin that misses reads as `absent` — a + * test that tried to provoke this from a shipped store would get `absent` + * instead and silently stop covering the case. Only `getObject` is + * implemented, since the rest of the seam is optional. + */ + const versionLying = (served: number) => (): SkillStore => ({ + getObject(_kind: string, key: string) { + return rawSkill({ key, version: served }); + }, + allObjects() { + return {}; + }, + }); + /** A store serving a wire-shaped object with fields spoiled or dropped. */ const spoiled = (overrides: Partial & { key?: unknown }, drop: Array<'content' | 'contentHash'> = []) => @@ -1721,6 +1916,12 @@ describe('integrity-failure log record', () => { // layer noticed. A store that serves a perfectly valid object under the // wrong key reaches it: everything `verifyRawSkill` checks passes. ['key_mismatch', spoiled({ key: 'served-under-this' }), GET], + // The tenth is the other boundary code, and the same kind of odd one out: + // everything `verifyRawSkill` checks passes, and the disagreement is between + // the object and the *request* rather than inside the object. Reachable only + // with a pin, and only through a store that lies about which version it + // served. + ['version_mismatch', versionLying(9), PINNED], ]; it.each(cases)('logs one record carrying reason_code %s', async (code, makeStore, run) => { @@ -1738,10 +1939,10 @@ describe('integrity-failure log record', () => { }); it('covers the whole reason_code vocabulary and nothing else', () => { - // Eight of the nine are one per call site of `recordIntegrityFailure`; the - // ninth, `key_mismatch`, comes from `recordKeyMismatch` at the retrieval - // boundary. Every language implementation emits the same nine, so a tenth - // in one SDK only is the regression this test exists to catch. + // Eight of the ten are one per call site of `recordIntegrityFailure`; the + // other two, `key_mismatch` and `version_mismatch`, come from the two + // boundary recorders. Every language implementation emits the same ten, so + // an eleventh in one SDK only is the regression this test exists to catch. expect(cases.map(([code]) => code).sort()).toEqual([ 'hash_mismatch', 'invalid_key', @@ -1752,15 +1953,48 @@ describe('integrity-failure log record', () => { 'not_an_object', 'not_utf8', 'over_size_cap', + 'version_mismatch', ]); }); - it('fires the record without the signal for key_mismatch, and never the reverse', async () => { - // The one-directional exception to "the two surfaces agree". Swept across - // the whole vocabulary rather than asserted on the one case, because what - // makes it safe is that *only* this code is signal-free: an implementation - // that dropped the signal for some other code would satisfy a single-case - // assertion. + it('keeps the reason_code and outcome vocabularies apart, spellings included', () => { + // One condition now has a token in each vocabulary, with *different* + // spellings: the code is `version_mismatch`, the outcome `wrong_version`. + // The split is deliberate — the code names which check failed, the outcome + // what the caller got — so a detection rule matching one string is never + // ambiguous about which surface it was written against. Asserting both + // closed sets in one place is what keeps a well-meaning "consistency" fix + // from collapsing them into one spelling. + const codes = cases.map(([code]) => code); + const outcomes: packageIndex.SkillOutcomeReason[] = [ + 'absent', + 'integrity_failure', + 'ok', + 'store_unavailable', + 'wrong_version', + ]; + + expect(codes).toContain('version_mismatch'); + expect(codes).not.toContain('wrong_version'); + expect(outcomes).toContain('wrong_version'); + expect(outcomes).not.toContain('version_mismatch' as packageIndex.SkillOutcomeReason); + // Ten codes, five outcomes. The finer vocabulary grew; the public one did not. + expect(codes).toHaveLength(10); + expect(outcomes).toHaveLength(5); + }); + + it('fires the record without the signal for the two boundary codes, and never the reverse', async () => { + // The one-directional exception to "the two surfaces agree", and there are + // exactly two cases. Swept across the whole vocabulary rather than asserted + // on those two, because what makes it safe is that *only* they are + // signal-free: an implementation that dropped the signal for some other code + // would satisfy a two-case assertion. + // + // Each is asserted separately rather than as a group — an implementation + // that emitted the signal for either one would satisfy every other + // assertion in this block. + const recordOnly = new Set(['key_mismatch', 'version_mismatch']); + for (const [code, makeStore, run] of cases) { _clearState(); _setStore(makeStore()); @@ -1771,7 +2005,9 @@ describe('integrity-failure log record', () => { expect(records, code).toHaveLength(1); const signals = emitter.signals(INTEGRITY_SIGNAL); - if (code === 'key_mismatch') { + if (recordOnly.has(code)) { + // Not merely no integrity signal: the emitter saw nothing at all. + expect(emitter.records, code).toEqual([]); expect(signals, code).toHaveLength(0); } else { expect(signals, code).toHaveLength(1); @@ -1868,6 +2104,56 @@ describe('integrity-failure log record', () => { expect(keys).toEqual(['action', 'event', 'language', 'reason', 'reason_code', 'served_key', 'skill_key']); }); + it('sorts the version_mismatch record too, with served_version in its alphabetical place', async () => { + // The second record-only field, on a third construction path, so the sort + // that makes every other record byte-comparable across SDKs is asserted here + // independently as well. `served_key` and `served_version` sort adjacently, + // so a reader who knows one record's shape can predict the other's — even + // though the two fields never appear on the same record. + _setStore(versionLying(9)()); + + const [{ record }] = await logged(() => getSkill('a', { version: 2 })); + + const keys = Object.keys(record); + expect(keys).toEqual([...keys].sort()); + expect(keys).toEqual([ + 'action', + 'event', + 'language', + 'reason', + 'reason_code', + 'served_version', + 'skill_key', + 'version', + ]); + // No `served_key` here, and no hash fields: verification passed, so the + // hashes are not what disqualified the answer, and the key agreed. + expect('served_key' in record).toBe(false); + expect('expected_hash' in record).toBe(false); + expect('observed_hash' in record).toBe(false); + }); + + it('records both versions as integers, not as strings', async () => { + // The type, not only the value. The sorted-key JSON is compared byte for + // byte across SDKs, and `"served_version":9` and `"served_version":"9"` are + // not the same line — so a language that stringified either number would + // pass a value assertion and still break a cross-language parser. + _setStore(versionLying(9)()); + + const [{ line, record }] = await logged(() => getSkill('a', { version: 2 })); + + for (const field of ['version', 'served_version']) { + expect(typeof record[field], field).toBe('number'); + expect(Number.isInteger(record[field]), field).toBe(true); + } + // `version` is the version **requested**, the meaning it carries on every + // other record; `served_version` is the one the store answered with. + expect(record.version).toBe(2); + expect(record.served_version).toBe(9); + expect(line).toContain('"served_version":9'); + expect(line).toContain('"version":2'); + }); + it('redacts a hostile served_key', async () => { // Unreachable today — verification accepts the served key before this path // runs, so it is well-formed by construction. Asserted anyway, because that @@ -2102,7 +2388,15 @@ describe('getSkillResult', () => { it.each(cases)('reports reason %s', async (reason, makeStore, run) => { _setStore(makeStore()); - const outcome = (await run()) as Awaited>; + // `wrong_version` writes its log record on the way through, like the + // integrity failures do; silenced here so the assertions below read cleanly. + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let outcome: Awaited>; + try { + outcome = (await run()) as Awaited>; + } finally { + spy.mockRestore(); + } expect(outcome.reason).toBe(reason); if (reason === 'ok') { @@ -2236,7 +2530,13 @@ describe('getSkillResult', () => { }, }); - const outcome = await getSkillResult('a', { version: 1 }); + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let outcome: Awaited>; + try { + outcome = await getSkillResult('a', { version: 1 }); + } finally { + spy.mockRestore(); + } expect(outcome.reason).toBe('wrong_version'); expect(outcome.skill).toBeNull(); @@ -2250,13 +2550,18 @@ describe('getSkillResult', () => { // "resolves to `null` — never rejects", and every existing caller treats // that `null` as "no skill". Reporting the reason is additive or it is a // silent breaking change. - for (const [reason, makeStore] of cases) { - if (reason === 'ok') continue; - _clearState(); - _setStore(makeStore()); + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + try { + for (const [reason, makeStore] of cases) { + if (reason === 'ok') continue; + _clearState(); + _setStore(makeStore()); - const wanted = reason === 'wrong_version' ? { version: 2 } : {}; - await expect(getSkill('a', wanted)).resolves.toBeNull(); + const wanted = reason === 'wrong_version' ? { version: 2 } : {}; + await expect(getSkill('a', wanted)).resolves.toBeNull(); + } + } finally { + spy.mockRestore(); } }); @@ -2357,7 +2662,13 @@ describe('getSkillResult', () => { it('detail names the key and both versions on a mismatch, and no path', async () => { _setStore(wrongVersionStore(3)); - const { detail } = await getSkillResult('pdf-extraction', { version: 2 }); + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let detail: string | null; + try { + ({ detail } = await getSkillResult('pdf-extraction', { version: 2 })); + } finally { + spy.mockRestore(); + } expect(detail).toContain('pdf-extraction'); expect(detail).toContain('version 2'); diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 471f4cb0..24e7549e 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -73,20 +73,27 @@ const SIGNAL_REVOKED = 'AgentControl Skill Revoked Received'; const EVENT_INTEGRITY_FAILURE = 'ld.skills.integrity_failure'; /** - * Why a skill was withheld: a closed vocabulary of nine tokens. + * Why a skill was withheld: a closed vocabulary of ten tokens. * * Customers alert on these tokens, and every language implementation emits the - * same nine for the same conditions, so a polyglot fleet writes one detection - * rule rather than two. A tenth token is a cross-SDK change — add it everywhere, - * or not at all. + * same ten for the same conditions, so a polyglot fleet writes one detection + * rule rather than two. An eleventh token is a cross-SDK change — add it + * everywhere, or not at all. * - * Eight of the nine are one per call site of `recordIntegrityFailure`, decided + * Eight of the ten are one per call site of `recordIntegrityFailure`, decided * inside `verifyRawSkill` over a single object, and they fire **both** detection - * surfaces. `key_mismatch` is the exception on both counts: it comes from - * `recordKeyMismatch` at the retrieval boundary, after verification has already - * passed, and it fires the log record only. It shares this vocabulary anyway - * because a customer's detection rule cares that integrity failed, not about - * which layer noticed — see `recordKeyMismatch` for why the signal stays out. + * surfaces. `key_mismatch` and `version_mismatch` are the exceptions on both + * counts: they come from `recordKeyMismatch` and `recordVersionMismatch` at the + * retrieval boundary, after verification has already passed, and they fire the + * log record only. They share this vocabulary anyway because a customer's + * detection rule cares that integrity failed, not about which layer noticed — + * see `recordKeyMismatch` for why the signal stays out of both. + * + * This is a **finer** vocabulary than `SkillOutcomeReason`, which stays five + * tokens, and the version mismatch is the one condition with a token in each: + * `version_mismatch` is the code that says which check failed, `wrong_version` + * the outcome that says what the caller got. The spellings differ deliberately — + * see `recordVersionMismatch`. */ export type IntegrityReasonCode = | 'hash_mismatch' @@ -97,7 +104,8 @@ export type IntegrityReasonCode = | 'missing_content_hash' | 'not_an_object' | 'not_utf8' - | 'over_size_cap'; + | 'over_size_cap' + | 'version_mismatch'; /** What the accessors report when no store is configured. */ export const NO_STORE_MESSAGE = @@ -307,9 +315,10 @@ export function recordIntegrityFailure( /** * Records a store answering under a key other than the one requested. * - * **Log record only — no product signal.** This is the one integrity failure - * that fires one surface rather than both, and the asymmetry is the decision - * rather than an oversight. + * **Log record only — no product signal.** One of the two integrity failures + * that fire one surface rather than both — `recordVersionMismatch` is the + * other, for the same reasons — and the asymmetry is the decision rather than + * an oversight. * * The record fires because a substituting store is a genuine tampering * indicator, and the record is the customer-owned detection path — the only one @@ -366,6 +375,90 @@ export function recordKeyMismatch(requested: unknown, served: unknown): void { console.error(`[LaunchDarkly] ${EVENT_INTEGRITY_FAILURE} ${JSON.stringify(record)}`, record); } +/** + * Records a store answering a version pin with a different version. + * + * **Log record only — no product signal**, the same split `recordKeyMismatch` + * makes and for the same reason: both are decided at the retrieval boundary + * after verification has passed, both name a store that answered *something + * other than what was asked for*, and the overwhelmingly common cause of either + * is a **broken custom store adapter** rather than an attacker. LaunchDarkly's + * own counter must not fill up with customers' adapter bugs. The two stay in + * step: a change that gives one of them a signal has to justify it for both. + * + * The record fires even though neither shipped store can reach this path. + * `FDv2SkillStore` and `InMemorySkillStore` both answer a pin with exactly that + * version or with `null`, so a pin that misses reads as `absent`; getting here + * means the store was asked for version N and answered with a well-formed object + * at version M. The population that does that is the one `key_mismatch` got its + * record for, and the record is the customer-owned detection path — the only one + * that works with telemetry off. + * + * It is **not** made redundant by `wrong_version` being a public + * `SkillOutcomeReason`. That token only reaches a caller who used + * `getSkillResult`; `getSkill` is the documented default and collapses + * `wrong_version` to `null` exactly as it collapses `integrity_failure`, so + * without this record an operator on `getSkill` has *zero* visibility into a + * store answering pins with the wrong version. + * + * The `reason_code` is `version_mismatch`, deliberately not `wrong_version`. + * It keeps the `*_mismatch` family consistent — `hash_mismatch`, `key_mismatch`, + * `version_mismatch` all say "this check found two values that disagree" — and + * it keeps one string out of two closed vocabularies with two meanings, so a + * detection rule matching `wrong_version` cannot be ambiguous about which + * surface it was written against. Widening `SkillOutcomeReason` is not part of + * this: the public outcome vocabulary stays five tokens, and only the detection + * surface changed. + * + * Lives here, beside `recordKeyMismatch`, so the single-emission-site rule still + * holds by reading one module. + * + * `requested` and `served` are both integers by construction rather than by + * redaction — the first is the typed `version` a caller pinned, the second has + * already passed `verifyRawSkill`'s `invalid_version` check — so neither can + * carry a skill body the way a wire *string* can, and both are recorded as + * numbers. The key gets the same shape-check-then-redact as every other key + * that reaches a surface, because a caller can pass anything as a key. + */ +export function recordVersionMismatch(key: unknown, requested: number, served: number): void { + const record: Record = { + action: 'withheld', + event: EVENT_INTEGRITY_FAILURE, + language: LANGUAGE, + // Named apart from the eight, as the key mismatch is, so a reader of the + // line can tell the retrieval boundary from a verification failure. + reason: 'the skill store answered with a different version than the one requested', + reason_code: 'version_mismatch' satisfies IntegrityReasonCode, + // The version the store answered with: record-only, paralleling + // `served_key`, and never added to the signal's allowlist. Burying it in the + // prose `reason` would leave the one datum that makes a broken adapter + // diagnosable unparseable. + served_version: served, + // `skill_key` keeps the meaning it has on every other record — the key the + // *caller asked for*. Here it is not what disagreed, but the record still + // has to name the skill it is about. + skill_key: isValidSkillKey(key) ? key : '', + // `version` likewise keeps its meaning everywhere else: the version + // **requested**, so a rule that groups or filters on it keeps working. That + // this record carries `version` while `recordKeyMismatch`'s does not is + // deliberate — there identity is what disagreed and any pin was incidental; + // here the two versions *are* the disagreement, so a record naming only one + // of them would be undiagnosable. + version: requested, + }; + // No `expected_hash` or `observed_hash`: verification passed, so the hashes + // are not what disqualified the answer and an absent field stays absent + // rather than being emitted as null. + // + // Keys above are in alphabetical order, as in `recordIntegrityFailure`, so the + // emitted line is byte-identical across SDKs for the same input. Note that + // `served_key` and `served_version` sort adjacently, so a reader who knows one + // record's shape can predict the other's. Do not reorder. + // + // biome-ignore lint/suspicious/noConsole: this package has no logger abstraction; integrity failures must be visible + console.error(`[LaunchDarkly] ${EVENT_INTEGRITY_FAILURE} ${JSON.stringify(record)}`, record); +} + /** * Records a materialization. Deliberately carries no `target_path` and no * filesystem path of any kind — the same reasoning that keeps the skill body out @@ -756,6 +849,19 @@ export function resolveFromStore(store: SkillStore, key: string, wantedVersion: }; } if (wantedVersion !== null && skill.version !== wantedVersion) { + // Records the log surface but not the product signal, the same split the key + // mismatch above makes and for the same reason — see + // `recordVersionMismatch`. The asymmetry is pinned by a test in both + // directions, because an implementation that emitted the signal too would + // look correct from every other angle. + // + // Recorded even though neither shipped store can get here: reaching this + // branch means the store answered a pin with a well-formed object at another + // version, which is a broken custom adapter. `wrong_version` being a public + // outcome token is not a substitute — `getSkill` collapses it to `null` + // exactly as it collapses `integrity_failure`, so the record is the only + // visibility an operator on the default accessor has. + recordVersionMismatch(key, wantedVersion, skill.version); return { error: `skill '${key}' version ${wantedVersion} is not available (the store holds version ${skill.version})`, reason: 'wrong_version', From 068143f474025e2fae1c570fd485fb5abb1617c7 Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Mon, 28 Sep 2026 16:02:38 -0400 Subject: [PATCH 53/55] docs(client): stop giving a shared connection as the reason for the skip `1aa1594` corrected `objectsIgnored` and `REQUEST_ADVICE`; two doc comments still give the old shape as the *reason* for a behaviour rather than merely describing it: - `isSkillEvent` says a foreign kind is ignored rather than rejected "because flag and segment objects share the connection". With the declaration in place they do not. The skip is still right and still tested, but the reason is now forward compatibility: erroring on an unrecognised kind would turn a payload that gained one into the reconnect-loop outage this feature must not cause, which is how `agents.md` already puts it; - `addListener`'s doc comment is the twin of the README paragraph adc270f fixed, and repeated the same claim in passing. Co-Authored-By: Claude Opus 5 --- packages/client/src/skills-fdv2.ts | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/packages/client/src/skills-fdv2.ts b/packages/client/src/skills-fdv2.ts index eb2f376d..027f532a 100644 --- a/packages/client/src/skills-fdv2.ts +++ b/packages/client/src/skills-fdv2.ts @@ -440,9 +440,11 @@ export type Tombstone = { readonly key: string; readonly objectVersion: number | /** * Whether one `put-object` / `delete-object` payload is a skill. * - * The kind alone decides it. Every other kind is **ignored, not rejected**, - * because flag and segment objects share the connection and erroring on them - * would turn a normal payload into a reconnect loop. + * The kind alone decides it. Every other kind is **ignored, not rejected**. The + * `kinds` declaration means a flag or segment object should no longer arrive at + * all, but the skip stays: erroring on an unrecognised kind would turn a payload + * that gained one into a reconnect loop, which is the outage this feature must + * not cause. */ export function isSkillEvent(data: unknown): boolean { if (typeof data !== 'object' || data === null || Array.isArray(data)) return false; @@ -2038,10 +2040,10 @@ export class FDv2SkillStore implements SkillStore { * swallowed, because a broken listener must not be able to kill delivery. * * Throws for any `kind` but `'skill'`. This store notifies skill changes and - * nothing else — flag and segment objects on the same connection are skipped, - * never dispatched — so accepting a listener on another kind would hand back a - * watcher that silently never fires, which is indistinguishable from one whose - * objects never changed. + * nothing else — anything else on the connection is skipped, never dispatched + * — so accepting a listener on another kind would hand back a watcher that + * silently never fires, which is indistinguishable from one whose objects + * never changed. */ addListener(kind: string, fn: (raw: RawSkillObject) => unknown): void { if (kind !== SKILL_OBJECT_KIND) { From 91942c5b62245caa2b4f45800ee6a92c8850163c Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 30 Sep 2026 12:27:58 -0400 Subject: [PATCH 54/55] fix(client): shape-check served_version, matching the Python SDK MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Python side of this change landed on `xie/agent-skills` (python-ai-sdk#87), so the parity claim is now checkable by running both recorders rather than by reading one and reasoning about the other. Doing that turned up one divergence on the same input. Python routes `served_version` through `is_valid_skill_version` and records `` when it fails; this side recorded the value verbatim on the strength of its `number` type. For every reachable input the two agree — `verify_raw_skill` / `verifyRawSkill` has already accepted the served version by the time either recorder runs — so the divergence only shows on the path neither SDK can reach today. It is still worth closing, for the two reasons `served_key`'s identical guard already exists on this side: unreachability is a property of the current call order rather than of the recorder, and the check is what keeps the field an integer, which the byte comparison depends on (`3` and `"3"` are not the same line). `requested` stays unchecked in both languages, and that asymmetry is deliberate: it is the caller's own pin rather than a store-controlled value, so it cannot carry skill content, and coercing a mistyped one would hide the caller's mistake from their own log. Verified by executing both implementations over three inputs — the reachable case and each guard — and diffing: byte-identical modulo `language`. The README's `served_version` row now documents the placeholder, as Python's does. Co-Authored-By: Claude Opus 5 --- packages/client/README.md | 2 +- packages/client/src/__tests__/skills.test.ts | 30 ++++++++++++++++++++ packages/client/src/skills-core.ts | 23 +++++++++------ 3 files changed, 45 insertions(+), 10 deletions(-) diff --git a/packages/client/README.md b/packages/client/README.md index d0f67ece..6ef4b807 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -449,7 +449,7 @@ The line is a `[LaunchDarkly] ` prefix, the event name, a space, and a single JS | `reason` | yes | Human-readable detail, including byte counts. Wording may change between releases. | | `language` | yes | `typescript`. Distinguishes SDKs in a polyglot fleet; the Python SDK emits the same record with `python`. | | `served_key` | no | Only on `key_mismatch`: the key the store actually answered under, with the same redaction as `skill_key`. Omitted on every other failure mode. | -| `served_version` | no | Only on `version_mismatch`: the version the store actually answered with, as an integer. Omitted on every other failure mode, and never on the same record as `served_key`. | +| `served_version` | no | Only on `version_mismatch`: the version the store actually answered with, as an integer — or `` when it was not one. Omitted on every other failure mode, and never on the same record as `served_key`. | | `version` | no | The skill version — the version **requested** on `version_mismatch`, the delivered one everywhere else. Always an integer. Omitted when the delivered version was not an integer >= 1, and on `key_mismatch`. | | `expected_hash` | no | The `contentHash` delivered with the content, or `` when it was not 64 lowercase hex characters. Omitted when the failure happened before any hash was read. | | `observed_hash` | no | The sha256 this SDK computed locally. Omitted when the failure happened before hashing. | diff --git a/packages/client/src/__tests__/skills.test.ts b/packages/client/src/__tests__/skills.test.ts index 3d8621a6..fbf0d6b3 100644 --- a/packages/client/src/__tests__/skills.test.ts +++ b/packages/client/src/__tests__/skills.test.ts @@ -2154,6 +2154,36 @@ describe('integrity-failure log record', () => { expect(line).toContain('"version":2'); }); + it('replaces a served_version that is not a valid version', async () => { + // Unreachable today — `verifyRawSkill`'s `invalid_version` check accepts the + // served version before this path runs, so it is an integer by + // construction. Asserted anyway, for the two reasons `served_key`'s guard + // is: that is a property of the current call order rather than of the + // recorder, and the check is what keeps the field an integer the + // cross-language byte comparison can rely on. The Python SDK emits the same + // placeholder for the same input. + const body = 'UNIQUE-SERVED-VERSION-BODY'; + const { recordVersionMismatch } = await import('../skills-core.js'); + + const spy = vi.spyOn(console, 'error').mockImplementation(() => {}); + let calls: unknown[][] = []; + try { + recordVersionMismatch('asked-for', 2, body as unknown as number); + } finally { + calls = [...spy.mock.calls]; + spy.mockRestore(); + } + + const line = String(calls[0][0]); + const record = JSON.parse(line.slice(line.indexOf('{'))) as Record; + expect(record.served_version).toBe(''); + expect(line).not.toContain(body); + // The requested version is the caller's own pin, not a store-controlled + // value, so it is reported as given rather than coerced — hiding a caller's + // mistyped pin from their own log would be the worse failure. + expect(record.version).toBe(2); + }); + it('redacts a hostile served_key', async () => { // Unreachable today — verification accepts the served key before this path // runs, so it is well-formed by construction. Asserted anyway, because that diff --git a/packages/client/src/skills-core.ts b/packages/client/src/skills-core.ts index 24e7549e..2d985e8f 100644 --- a/packages/client/src/skills-core.ts +++ b/packages/client/src/skills-core.ts @@ -413,14 +413,18 @@ export function recordKeyMismatch(requested: unknown, served: unknown): void { * Lives here, beside `recordKeyMismatch`, so the single-emission-site rule still * holds by reading one module. * - * `requested` and `served` are both integers by construction rather than by - * redaction — the first is the typed `version` a caller pinned, the second has - * already passed `verifyRawSkill`'s `invalid_version` check — so neither can - * carry a skill body the way a wire *string* can, and both are recorded as - * numbers. The key gets the same shape-check-then-redact as every other key - * that reaches a surface, because a caller can pass anything as a key. + * The key and `served` are shape-checked and redacted on the same rule as every + * other value that reaches a surface; `requested` is not, and the asymmetry is + * deliberate. `served` cannot actually be hostile on the path that calls this — + * `verifyRawSkill`'s `invalid_version` check accepted it first — but that is a + * property of the current call order rather than of this function, exactly as it + * is for `recordKeyMismatch`'s `served_key`. The check is also what keeps the + * field an **integer**, which the sorted-key JSON needs to stay byte-comparable + * across SDKs: `3` and `"3"` are not the same line. `requested` is the caller's + * own pin rather than a store-controlled value, so it cannot carry skill content, + * and coercing a mistyped one would hide the caller's own mistake from their log. */ -export function recordVersionMismatch(key: unknown, requested: number, served: number): void { +export function recordVersionMismatch(key: unknown, requested: number, served: unknown): void { const record: Record = { action: 'withheld', event: EVENT_INTEGRITY_FAILURE, @@ -432,8 +436,9 @@ export function recordVersionMismatch(key: unknown, requested: number, served: n // The version the store answered with: record-only, paralleling // `served_key`, and never added to the signal's allowlist. Burying it in the // prose `reason` would leave the one datum that makes a broken adapter - // diagnosable unparseable. - served_version: served, + // diagnosable unparseable. Shape-checked for the reason the doc comment + // gives — and the Python SDK emits the same placeholder for the same input. + served_version: isValidSkillVersion(served) ? served : '', // `skill_key` keeps the meaning it has on every other record — the key the // *caller asked for*. Here it is not what disagreed, but the record still // has to name the skill it is about. From 9928c9dee0ed0464ebbe3e19aecd0c730e222f1d Mon Sep 17 00:00:00 2001 From: Christie Williams Date: Wed, 30 Sep 2026 12:51:44 -0400 Subject: [PATCH 55/55] fix(client): reject a non-finite writeSkills timeout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Infinity` passed the guard, because the check was `Number.isNaN(timeout) || timeout < 0` and `Infinity` is neither. It then made `deadline` non-finite, which voided the bound the option documents — "`timeout` bounds the whole call, including content retrieval" — and let the call run unbounded instead. The guard is finiteness now, which covers `Infinity` and `NaN` together and matches the three sibling guards that already read that way: `debounceMs` in skills-watch, and `pollIntervalMs` / `readTimeoutMs` / `waitForSkills` in the FDv2 store. agents.md already claimed `timeout` was "guarded for sign and finiteness"; this makes that true rather than aspirational. `NaN` is worth naming separately even though it was already rejected here, because its failure mode depends on the shape of the comparison rather than on the value: `now > deadline` reads as never expiring and `deadline - now > 0` reads as already expired, so one input can push two implementations in opposite directions with neither detectably wrong from inside its own language. That is why the cross-SDK spec states this as finiteness and not as a `NaN` footnote. The message reports the value with `String` rather than `JSON.stringify`, which renders both `NaN` and `Infinity` as `null` and so named a mistake the caller did not make. Same reasoning, and same wording, as the `debounceMs` guard. Spec: launchdarkly/ai-sdks-monorepo#32 (§3.22, §3.26). Co-Authored-By: Claude Opus 5 --- .../client/src/__tests__/skills-fs.test.ts | 18 ++++++++++++++++++ packages/client/src/skills-fs.ts | 16 +++++++++++++--- 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/packages/client/src/__tests__/skills-fs.test.ts b/packages/client/src/__tests__/skills-fs.test.ts index 3ed7918d..38664919 100644 --- a/packages/client/src/__tests__/skills-fs.test.ts +++ b/packages/client/src/__tests__/skills-fs.test.ts @@ -644,6 +644,24 @@ describe('writeSkills bare-string guard', () => { expect((error as Error).message).toContain('timeout'); }); + it('rejects a non-finite timeout, which would void the bound rather than shorten it', async () => { + // Both values pass the `< 0` half of the guard and make the deadline + // non-finite, so the option's whole promise — "bounds the entire call, + // including content retrieval" — goes silently away and the call runs + // unbounded. `NaN` is the worse one: `now > deadline` reads as never + // expiring while `deadline - now > 0` reads as already expired, so the same + // input can push two implementations in opposite directions. + for (const timeout of [Number.POSITIVE_INFINITY, Number.NaN]) { + const error = await writeSkills([skill('a')], root, { timeout }).catch((e: unknown) => e); + expect(error).toBeInstanceOf(Error); + // The value is named as itself: `JSON.stringify` renders both of these as + // `null`, which would report a mistake the caller did not make. + expect((error as Error).message).toContain(String(timeout)); + } + // Refused before the root was touched, like every other caller value error. + expect(await entryNames(root)).toEqual([]); + }); + it('validates the request shape before creating the root directory', async () => { // A bare string is refused before `resolveRoot` runs, so a mistaken call // does not leave an empty directory behind as a side effect of failing. diff --git a/packages/client/src/skills-fs.ts b/packages/client/src/skills-fs.ts index 52ffc4d8..170eddcf 100644 --- a/packages/client/src/skills-fs.ts +++ b/packages/client/src/skills-fs.ts @@ -169,7 +169,7 @@ const WINDOWS_RESERVED_NAMES: ReadonlySet = new Set([ export type WriteSkillsOptions = { /** Remove formerly-managed skills no longer in the requested set. Default `true`. */ prune?: boolean; - /** Bound on the whole call, in **seconds** (not milliseconds). Default `10`. */ + /** Bound on the whole call, in **seconds** (not milliseconds). Finite. Default `10`. */ timeout?: number; /** How to react to content that could not be retrieved. Default `'keep'`. */ onUnavailable?: OnUnavailable; @@ -220,8 +220,18 @@ export async function writeSkills( if (onUnavailable !== 'keep' && onUnavailable !== 'raise') { throw new Error(`onUnavailable must be "keep" or "raise", got ${JSON.stringify(onUnavailable)}`); } - if (typeof timeout !== 'number' || Number.isNaN(timeout) || timeout < 0) { - throw new Error(`timeout must be a non-negative number of seconds, got ${JSON.stringify(timeout)}`); + // `Infinity` and `NaN` both pass a bare `< 0` check, and either one makes the + // deadline below non-finite — which voids the bound this option promises and + // lets the call run unbounded. `NaN` is the worse of the two, because what it + // voids depends on the *shape* of the comparison: `now > deadline` reads as + // never expiring, and `deadline - now > 0` reads as already expired. So the + // guard is finiteness, the way `debounceMs` and the store's timeouts are. + if (typeof timeout !== 'number' || !Number.isFinite(timeout) || timeout < 0) { + // `String` rather than `JSON.stringify` for the number case: the latter + // serializes both `NaN` and `Infinity` as `null`, which names the wrong + // mistake. + const shown = typeof timeout === 'number' ? String(timeout) : JSON.stringify(timeout); + throw new Error(`timeout must be a non-negative, finite number of seconds, got ${shown}`); } // The request shape is validated before the root is resolved, so a mistaken