From cb5ea3636b61267cc25c84b74711392b65cfbee5 Mon Sep 17 00:00:00 2001 From: nmccready Date: Mon, 6 Jul 2026 19:10:12 -0400 Subject: [PATCH] =?UTF-8?q?docs(spec):=20SPEC-007=20linked=20(editable)=20?= =?UTF-8?q?sources=20=E2=80=94=20npm-link/go-replace=20for=20.agents=20art?= =?UTF-8?q?ifacts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Relative file: scheme (never absolute — sources.lock is git-tracked), reuses Override struct (add link field), symlink into live checkout, managed-clone mode scanned once via SPEC-005, detach = freeze link into vendored copy. Co-Authored-By: Claude Opus 4.8 --- specs/SPEC-007-linked-sources.md | 218 +++++++++++++++++++++++++++++++ 1 file changed, 218 insertions(+) create mode 100644 specs/SPEC-007-linked-sources.md diff --git a/specs/SPEC-007-linked-sources.md b/specs/SPEC-007-linked-sources.md new file mode 100644 index 0000000..0ffb397 --- /dev/null +++ b/specs/SPEC-007-linked-sources.md @@ -0,0 +1,218 @@ +--- +id: SPEC-007 +title: "Linked (editable) sources: npm-link / go-replace for .agents artifacts" +status: Draft +owner: nmccready +created: 2026-07-06 +updated: 2026-07-06 +related: SPEC-003, SPEC-005 +--- + +# [SPEC-007] Feature: Linked (editable) sources + +## Overview + +Add an **editable / linked** source mode to the SPEC-003 source system, +modeled on `npm link` and `go mod replace`. Today a source entry resolves +to an immutable, SHA-pinned tarball snapshot fetched over HTTP — a one-way +copy. A **linked** source instead points a managed artifact in `.agents/` +at a **live local git checkout** via a relative-path symlink, so: + +1. **Edits flow both ways.** The artifact and the upstream working copy are + the same inode; editing either edits both. +2. **The local copy reaps upstream updates** by `git pull` in the checkout + (or a managed clone), with no re-fetch/re-stage step. +3. **Changes can be paid forward upstream** by branching / committing / PRing + from that same checkout. + +Linking is recorded declaratively in `sources.yaml` and `sources.lock` so +the intent is committed, reviewable, and reproducible — replacing today's +undocumented manual `ln -s`. + +## Motivation + +The current model is snapshot-only. When you are *actively developing* a +skill that lives in another repo (or developing the upstream repo itself and +consuming it here), the SPEC-003 flow forces a painful loop: +`edit upstream → commit → push → sync-agents update → test → repeat`. + +Developers work around this with a hand-rolled `ln -s`, which is: + +- **Invisible** — not in `sources.yaml`/`sources.lock`, so `pull`/`update` + and the AGENTS.md index don't know the artifact is a link; a later `pull` + may clobber it. +- **Non-portable** — an absolute symlink or lock path (`/Users/tars/...`) + breaks the instant the repo is cloned on another machine or by another + engineer. `sources.lock` is git-tracked, so this is a correctness bug, not + just an annoyance. +- **One-off** — no lifecycle: no `list` visibility, no clean `detach`, no + re-pin back to a remote snapshot. + +`npm link` and `go mod replace` exist for exactly this. We adopt the +`replace` model: keep the upstream identity entry, redirect resolution to a +**relative** local path. + +## Goals + +- `sync-agents source add --link ` declares a linked source: + writes the upstream `` to `sources.yaml` **and** an override + redirecting it to a relative `file:` path, then wires the symlink. +- Bare `sync-agents source add --link ` (no path) **manages a clone**: + clones the upstream repo into `.agents/.sources/-/`, then links + the artifact out of it. `update` runs `git pull` in that clone. +- `sync-agents source add --link ` (path, no entry) derives + `owner/repo` (and type/in-repo path) from the checkout's git remote + + layout. +- `pull` / `update` **skip the HTTP fetcher** for linked entries (nothing to + download); `update` optionally `git pull`s a managed clone. +- `index` **follows the symlink** so AGENTS.md still enumerates the artifact. +- `source list` marks linked entries (`link → ../foo-skill`). +- `detach` converts a link into a real vendored copy (materialize the files, + drop the override) — the inverse of linking. +- All persisted paths are **relative** (`file:` scheme); the on-disk symlink + is created **relative** too. Absolute paths are rejected at parse time. + +## Non-goals + +- **Watching / copy-on-change.** We symlink; we do not run a file watcher. + Symlink *is* the bidirectional link (same as `npm link`). +- **Cross-VCS.** Managed clones are git-only in v1. +- **Automatic pay-forward.** Committing/pushing/PRing upstream stays a manual + git action in the checkout — we do not wrap `git push`/`gh pr`. +- **Re-running SPEC-005 quarantine per sync.** See Security. + +## Path grammar & the `file:` scheme + +Linked paths use npm's `file:` convention, resolved **relative to the +`.agents/` directory** (the `sources.yaml` location): + +``` +file:../foo-skill # sibling of the repo root (common: checkout in ~/code) +file:./vendor/foo-skill # inside the project +``` + +- **Absolute paths are a parse error** (`file:/Users/...`, `file:/abs`) — they + do not round-trip through git. Error names the fix (use a `file:../` relative + path). +- The literal after `file:` is `path.Clean`ed and must stay a relative path + (may traverse up with `..`). +- Portability across machines requires the same sibling layout on each — this + is inherent to relative links and shared with `go mod replace`. Documented, + not worked around. + +## Manifest & lock schema + +Linking reuses the existing `Override` mechanism (`match` glob), adding one +field — a link is conceptually "pin to a local path instead of a SHA": + +`sources.yaml` +```yaml +version: 1 +sources: + - skill:nmccready/foo-skill # upstream identity — unchanged +overrides: + - match: skill:nmccready/foo-skill* + link: file:../foo-skill # relative to .agents/; mutually exclusive with pin_sha +``` + +`sources.lock` +```yaml +version: 1 +entries: + - entry: skill:nmccready/foo-skill + link: file:../foo-skill # relative, echoing the override + resolved_sha: 9f3a… # HEAD of the checkout at link time — INFORMATIONAL only + content_hash: "" # empty: a live checkout has no stable content hash + fetched_at: 2026-07-06T23:00:00Z + managed_clone: false # true when sync-agents owns the clone under .agents/.sources/ +``` + +Schema changes: + +- `Override.Link string` (yaml `link,omitempty`). Mutually exclusive with + `PinSHA` — validated on load (a link cannot also be SHA-pinned). +- `LockEntry.Link string`, `LockEntry.ManagedClone bool`. `ResolvedSHA` for a + linked entry is informational (records provenance at link time; not used for + drift/verification). `ContentHash` is empty for links. + +## Behavior + +### `source add --link` + +| invocation | behavior | +|---|---| +| `--link ` | write `` to `sources`; write override `link: file:` (path made relative to `.agents/`); create relative symlink `.agents// → /`; write lock. | +| `--link ` (no path) | clone `/` → `.agents/.sources/-/`; link artifact out; `managed_clone: true`. | +| `--link ` (no entry) | read `/.git` remote to derive `owner/repo`; infer type + in-repo path from layout; proceed as row 1. | + +`--link` refuses if an absolute path is supplied. If the target symlink +already exists as a real dir/file, refuse unless `--force`. + +### `pull` + +For each entry: if it has a `link` override (or a `link` lock record), +**skip the fetcher entirely**. Verify the symlink exists and resolves; if the +target is missing (checkout moved/deleted), emit a clear warning and leave the +dangling link (do not silently re-fetch a remote snapshot — that would mask a +broken dev setup). Non-linked entries: unchanged SPEC-003 path. + +### `update` + +Linked, `managed_clone: true`: `git -C .agents/.sources/- pull +--ff-only`; refresh `resolved_sha` in lock (informational). Linked, +`managed_clone: false` (user checkout): **no-op** — the user drives their own +git; we only refresh the informational `resolved_sha` from current HEAD. +Non-linked: unchanged. + +### `index` + +`DiscoverArtifacts` must `filepath.EvalSymlinks` / stat-through the link so a +linked skill's `SKILL.md` is read and enumerated in AGENTS.md exactly like a +vendored one. (Today it walks real files; confirm symlinked dirs are +followed.) + +### `source list` + +Linked entries render with a `link → file:../foo-skill` annotation and, for +managed clones, the clone's short SHA. Snapshot entries unchanged. + +### `detach` + +Extend existing `detach` to handle links: replace the symlink with a real +copy of the current target contents, remove the `link` override, and record a +normal snapshot lock entry (content-hash the materialized files). This is the +"freeze my linked dev copy into a vendored artifact" operation. + +## Security (SPEC-005 interaction) + +A linked source is a **local checkout the operator is actively developing** — +running the quarantine scanner and hardened-extraction gate on every `pull` +is noise, and there is no tarball to extract. Therefore: + +- **Linking a local `--link ` is trusted by default** — no quarantine, + no scan on link or on subsequent `pull`. +- **Managed-clone linking (`--link `) fetches remote code**, so the + **initial clone is scanned once** through the SPEC-005 scanner (same + heuristics as tarball pull). Findings gate the link unless `--trust`. + Subsequent `git pull`s during `update` are **not** re-scanned (the operator + now owns the checkout) — but `sync-agents check`/scan can be run on demand. +- Rationale: matches the [ClawHub supply-chain] posture — untrusted *remote + first contact* is scanned; an operator's own working tree is not policed. + +## Test plan (bats + Go) + +- Parse: `file:../x` ok; `file:/abs` rejected; `file:./x` ok; empty rejected. +- `Override.Link` + `PinSHA` together → load error. +- `add --link `: symlink created relative; manifest+lock written + relative; re-`pull` is a no-op and does not clobber the link. +- `add --link `: managed clone created + scanned; `managed_clone: true`. +- Dangling target → `pull` warns, does not re-fetch. +- `index`: linked skill appears in AGENTS.md. +- `detach`: link → real copy, override removed, content_hash populated. +- Portability: lock/symlink contain no absolute paths (grep gate in test). + +## Rollout + +Additive, backwards-compatible: no `link` override ⇒ identical to today. +Ships in 1.x (per the standing no-2.0 decision). New `.agents/.sources/` +dot-dir is invisible to sync/index (same convention as `.quarantine`).