Skip to content

Replaced external.dir with declared packs referenced by name (0.10.0) #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dzykovic wants to merge 2 commits into from
Closed

Replaced external.dir with declared packs referenced by name (0.10.0) #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8686f93
Replaced external.dir with declared packs referenced by name and rele…
dzykovic Jul 28, 2026
d5ddb90
Made the packs migration and the mirror guard idempotent under re-run…
dzykovic Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 126 additions & 12 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,12 @@ jobs:
[ "$fail" -eq 0 ] && echo "OK: all examples stamped at $V"
exit $fail

external-packs:
name: external packs (materialize + track)
packs:
name: packs (declare + mirror + track)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Remote pack lands in the tracked dir, and a bump is a readable diff
- name: A declared pack mirrors into the repo, and a bump is a readable diff
shell: bash
run: |
set -euo pipefail
Expand All @@ -98,16 +98,19 @@ jobs:
project:
name: ext-test
sync_version: "$V"
external:
dir: "intelligence/external"
packs:
shared-intel:
url: file://$PACK
ref: v1.0.0
mirror: "intelligence/external/shared-intel"
sources:
rules:
- "intelligence/rules"
- "git+file://$PACK@v1.0.0#rules"
- "@shared-intel/rules"
agents:
- "intelligence/agents"
skills:
- "git+file://$PACK@v1.0.0#skills"
- "@shared-intel/skills"
targets:
agents: { enabled: true, output: "AGENTS.md" }
claude: { enabled: true, output: ".claude" }
Expand Down Expand Up @@ -145,26 +148,66 @@ jobs:
git -C "$PACK" add -A
git -C "$PACK" -c user.email=ci@example.com -c user.name=ci commit -qm v11
git -C "$PACK" tag v1.1.0
sed -i 's/@v1.0.0/@v1.1.0/g' intelligence/config.yaml
# The pin lives in ONE place — the pack declaration, not each source.
sed -i 's/ref: v1.0.0/ref: v1.1.0/' intelligence/config.yaml
bash intelligence/sync/scripts/sync.sh >/dev/null
git diff --stat -- intelligence/external
git diff -- intelligence/external | grep -q '^+Be careful AND explicit\.$'
git diff -- intelligence/external | grep -q '^+ref=v1.1.0$'
test ! -e intelligence/external/shared-intel/skills/shared-do-thing
test -f intelligence/external/shared-intel/skills/shared-ship-it/SKILL.md

# A MOVED upstream (same pack, new url) refreshes the mirror. The
# mirror is declared per pack, so a stamped directory stays this
# pack's even after its url changes — freezing it at the old content
# while the generated output follows the new repo would break exactly
# the reviewability the mirror exists for, and do it silently.
git add -A && git -c user.email=ci@example.com -c user.name=ci commit -qm bumped
MOVED=$RUNNER_TEMP/moved-intel
git clone -q "$PACK" "$MOVED"
printf -- '---\nname: shared-baseline\n---\n\nFrom the moved repo.\n' > "$MOVED/rules/shared-baseline.md"
git -C "$MOVED" add -A
git -C "$MOVED" -c user.email=ci@example.com -c user.name=ci commit -qm moved
# Same pin, new home — the clone carries the tags, so move v1.1.0 onto
# the moved commit rather than adding a ref the config does not use.
git -C "$MOVED" tag -f v1.1.0 >/dev/null
sed -i "s|url: file://$PACK\$|url: file://$MOVED|" intelligence/config.yaml
out=$(bash intelligence/sync/scripts/sync.sh 2>&1)
case "$out" in *"not a pack's"*) echo "$out"; echo "a moved upstream must refresh, not freeze"; exit 1 ;; esac
grep -q "^url=file://$MOVED\$" intelligence/external/shared-intel/.pack
grep -q '^From the moved repo\.$' intelligence/external/shared-intel/rules/shared-baseline.md
git checkout -- intelligence/config.yaml

# A directory that is not ours is never cleared.
rm -rf intelligence/external
mkdir -p intelligence/external/shared-intel
echo precious > intelligence/external/shared-intel/PRECIOUS.md
bash intelligence/sync/scripts/sync.sh >/dev/null
test -f intelligence/external/shared-intel/PRECIOUS.md

# external.dir inside a configured source is refused, not obeyed.
sed -i 's|dir: "intelligence/external"|dir: "intelligence/rules/packs"|' intelligence/config.yaml
# A mirror inside a configured source is refused, not obeyed.
git checkout -- intelligence/config.yaml
sed -i 's|mirror: "intelligence/external/shared-intel"|mirror: "intelligence/rules/packs"|' intelligence/config.yaml
rc=0; bash intelligence/sync/scripts/sync.sh >/dev/null 2>&1 || rc=$?
[ "$rc" -eq 1 ] || { echo "expected exit 1 for external.dir inside a source, got $rc"; exit 1; }
[ "$rc" -eq 1 ] || { echo "expected exit 1 for a mirror inside a source, got $rc"; exit 1; }

# An undeclared pack reference FAILS the run — it must never degrade
# into a silently skipped source the way a missing local path does.
git checkout -- intelligence/config.yaml
sed -i 's|"@shared-intel/rules"|"@typo-pack/rules"|' intelligence/config.yaml
rc=0; err=$(bash intelligence/sync/scripts/sync.sh 2>&1 >/dev/null) || rc=$?
[ "$rc" -eq 1 ] || { echo "expected exit 1 for an undeclared pack, got $rc"; exit 1; }
case "$err" in *"typo-pack"*) ;; *) echo "$err"; echo "error must name the pack"; exit 1 ;; esac

# A pack with no mirror keeps working and leaves NOTHING in the repo.
git checkout -- intelligence/config.yaml
sed -i '/mirror: "intelligence\/external\/shared-intel"/d' intelligence/config.yaml
rm -rf intelligence/external
bash intelligence/sync/scripts/sync.sh >/dev/null
test ! -e intelligence/external
test -f .claude/rules/shared-baseline.md

echo "OK: external packs materialize, track, refresh and refuse unsafe dirs"
echo "OK: packs declare once, mirror, refresh, and fail closed on a bad reference"

smoke:
name: smoke (sync against examples)
Expand Down Expand Up @@ -539,6 +582,77 @@ jobs:
s=$(bash intelligence/sync/scripts/sync.sh); echo "$s" | grep -q 'IS_STATUS=ok'
test -f AGENTS.md

- name: 0.10.0 converts inline git+ sources into declared packs
shell: bash
run: |
set -euo pipefail
PROJ=$RUNNER_TEMP/packmig
mkdir -p "$PROJ/intelligence"/{rules,agents,skills}
cp -r intelligence/sync "$PROJ/intelligence/sync"
printf '# Always-on context\n' > "$PROJ/intelligence/rules/context.md"
# A 0.9.0-shaped config: one global external dir, the pack identity
# duplicated per section, and one UNPINNED spec — the case where ref
# is absent but a subpath is present.
cat > "$PROJ/intelligence/config.yaml" <<'EOF'
project:
name: packmig
sync_version: "0.9.0"
external:
dir: "intelligence/external"
sources:
rules:
- "intelligence/rules"
- "git+https://example.invalid/org/shared.git@v1.2.0#rules"
agents:
- "intelligence/agents"
skills:
- "intelligence/skills"
- 'git+https://example.invalid/org/shared.git@v1.2.0#skills'
- "git+https://example.invalid/org/other.git#skills"
targets:
agents:
enabled: true
output: "AGENTS.md"
header: |
# Packmig
EOF
git -C "$PROJ" init -q
git -C "$PROJ" add -A
git -C "$PROJ" -c user.email=ci@example.com -c user.name=ci commit -q -m init

cd "$PROJ"
out=$(REPO_URL="file://$GITHUB_WORKSPACE" bash intelligence/sync/scripts/update.sh --yes)
echo "$out"
case "$out" in *"IS_STATUS=migrated"*) ;; *) echo "expected IS_STATUS=migrated"; exit 1 ;; esac

# Post-conditions of the breaking change.
! grep -q '^external:' intelligence/config.yaml
grep -q '^packs:' intelligence/config.yaml
! grep -qE '^[[:space:]]*-[[:space:]]*["'\'']?git\+' intelligence/config.yaml
# One declaration per distinct repo, each carrying its own pin.
test "$(grep -c '^ url: ' intelligence/config.yaml)" -eq 2
test "$(grep -c '^ ref: v1.2.0$' intelligence/config.yaml)" -eq 1
# external.dir became a per-pack mirror, so vendored content keeps
# landing where it already is.
grep -q '^ mirror: "intelligence/external/shared"$' intelligence/config.yaml
# An unpinned spec keeps its subpath and gains NO ref — the field is
# optional, and a subpath read as a ref would clone a missing branch.
grep -q '"@other/skills"' intelligence/config.yaml
test "$(grep -c '^ ref: ' intelligence/config.yaml)" -eq 1
# Both sections reference the same pack by name — the pin has one home.
test "$(grep -c '"@shared/' intelligence/config.yaml)" -eq 2

# Idempotent: the whole chain re-runs without touching the config.
cp intelligence/config.yaml "$RUNNER_TEMP/packmig.yaml"
out2=$(REPO_URL="file://$GITHUB_WORKSPACE" bash intelligence/sync/scripts/update.sh --yes)
case "$out2" in *"IS_STATUS=ok"*) ;; *) echo "expected IS_STATUS=ok on re-run"; exit 1 ;; esac
diff -u "$RUNNER_TEMP/packmig.yaml" intelligence/config.yaml

# An unreachable pack degrades to a warning, exactly as an unreachable
# inline spec did — the local sources still sync.
s=$(bash intelligence/sync/scripts/sync.sh); echo "$s" | grep -q 'IS_STATUS=ok'
test -f AGENTS.md

- name: Multi-version chain walks the whole registry forward, idempotent
shell: bash
run: |
Expand Down
18 changes: 18 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,24 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
Update intelligence-sync: fetch the latest engine from https://github.com/ainova-systems/intelligence-sync and run its update flow to migrate this project to the newest version. Leave my rules, agents, and project skills untouched. If it fails, read the CHANGELOG "### Breaking" entries between my version and the latest, base your fix plan on them, make sure you are running the latest scripts, and retry; ask me only if it still fails.
```

## [0.10.0] — 2026-07-28

A remote source is now declared once as a **pack** and referenced by name, so its url and its pin live in exactly one place.

### Breaking

- **`external: { dir: … }` is replaced by `packs:`; `migrate_to_0_10_0` rewrites the config.** 0.9.0 put the mirror location in one global block while the identity of each pack — url and ref — stayed duplicated inside every `sources.*` entry that used it. A pack spanning rules, agents and skills therefore carried its `url@ref` three times, and nothing detected the drift when only two of them were bumped: rules pinned at one commit and skills at another is a config that looks fine and reads wrong. `packs:` inverts it. Each pack is declared once (`url`, optional `ref`, optional `mirror`) and referenced as `@<pack>[/<subpath>]`, so the pin has a single home and the mirror is a per-pack path rather than one global directory. The migration is automatic and comment-preserving: every inline `git+` spec becomes a declared pack plus an `@name` reference, `external.dir` becomes each pack's `mirror:` so vendored content keeps landing where it already is, and the `external:` block is dropped. **Post-condition: no `external:` key in `config.yaml`, and a `packs:` block declaring every remote previously reached inline.** Idempotent — a config with no `external:` key and no `git+` token is left untouched.

### Added

- **`packs:` — declared remote sources, referenced by name.** `packs.<name>.{url,ref,mirror}` sits at three levels, exactly like `targets:`, so it is read by the existing `get_nested_yaml_value` and adds no parser. `mirror:` is both the location and the switch: present, the pack is materialized there and committed; absent, it stays transient in the run cache, which is 0.9.0's default behaviour and needs no flag to express. Because the mirror is *declared*, the whole derive-a-name machinery 0.9.0 needed is gone — no basename extraction, no charset sanitizing, no `pack-<key>` fallback, no collision suffix — and a pack name is now purely a reference handle that never becomes a path component. **An undeclared `@pack` reference fails the run** (exit 1, naming the pack and listing the declared ones), deliberately unlike a missing local path, which only warns: the config claims to know that name, so a typo must not silently drop a whole rule set. That check runs up front, in `validate_pack_refs`, because `resolve_source_dir` is always called inside `$( )` — an error raised there would exit the substitution subshell, not the sync. Inline `git+<url>[@<ref>][#<subpath>]` specs keep working as *anonymous* packs: no name, no mirror, always transient. CI job `packs` covers declaration, mirroring, idempotency, the refresh diff, the never-clear guard, the unsafe-mirror refusal, the undeclared-pack failure and the transient path, against a `file://` pack repo.

### Fixed

- **`get_nested_yaml_value` cut values at the last colon, not the first.** The value strip was a greedy `.*:[[:space:]]*`, which on `url: https://host/repo.git` matched through `https:` and yielded `//host/repo.git`. It is now anchored with `[^:]*:`, and an unquoted value additionally drops a trailing ` # comment` per YAML while a `#` inside quotes stays content. This was latent in 0.9.0 — nothing read a URL through this helper — but it also silently truncated any `models.<ide>.<tier>` value containing a colon.
- **A pack could lose every subpath but the last when the run cache was unset.** `materialize_pack` recorded its "already cleared this run" claim only when `IS_REMOTE_CACHE` was exported, yet cleared the directory unconditionally. Since `resolve_source_dir` runs in a command substitution — a fresh subshell each call — a caller that had not exported the cache re-cleared the mirror on every entry, so a pack referenced for rules, agents and skills kept only the last. The claim now uses the same cache-root fallback the clone does.
- **Empty-array expansion under `set -u`.** The new migration iterates with `${#arr[@]}` bounds rather than `"${!arr[@]}"`, which bash 3.2 — the macOS default, and a supported target — treats as unbound when the array is empty.

## [0.9.0] — 2026-07-28

Remote packs can now be materialized into a tracked directory, so an upstream bump is reviewable instead of invisible.
Expand Down
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Zero dependencies. Just bash + awk. Linux, macOS, Windows (Git Bash / WSL).

## Ready-made packs

[intelligence-dev-packs](https://github.com/ainova-systems/intelligence-dev-packs) is a set of shared AI-first engineering packs - git/PR/review discipline plus an optional spec-driven development lifecycle - that you consume straight through this engine as remote `git+` sources, instead of authoring every rule and skill from scratch. Extracted from production AI-coded systems.
[intelligence-dev-packs](https://github.com/ainova-systems/intelligence-dev-packs) is a set of shared AI-first engineering packs - git/PR/review discipline plus an optional spec-driven development lifecycle - that you consume straight through this engine as declared `packs:`, instead of authoring every rule and skill from scratch. Extracted from production AI-coded systems.

## How It Works

Expand Down Expand Up @@ -152,7 +152,7 @@ Each `intelligence/<module>/` (e.g. `sync/`, future `domain/`) is self-contained
- [go-api-with-opencode](examples/go-api-with-opencode/) -- Go API with opencode (subagents in `.opencode/agents/`, skills via `.agents/skills/`)
- [dotnet-api-with-react-frontend](examples/dotnet-api-with-react-frontend/) -- .NET backend + React frontend
- [platform-with-submodules](examples/platform-with-submodules/) -- Multi-component platform with git submodules
- [with-remote-skills](examples/with-remote-skills/) -- Pulls shared rules/skills from a remote git repo (`git+<url>` sources), materialized into a tracked `external:` dir
- [with-remote-skills](examples/with-remote-skills/) -- Pulls shared rules/skills from a remote git repo declared under `packs:`, mirrored into a tracked directory

## Documentation

Expand Down
11 changes: 8 additions & 3 deletions docs/ADAPTERS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,10 @@ Source `lib/common.sh` for these utilities:
| `map_access_to_claude_tools(access)` | Tool string for access level |
| `map_access_to_claude_disallowed(access)` | Disallowed tools string |
| `read_yaml_list(config, section)` | Read list from `config.yaml` |
| `resolve_source_dir(repo_root, src)` | Map a source entry to a local dir — `"$repo_root/$src"`, or a shallow clone for a remote `git+<url>` spec |
| `source_is_remote(src)` | True (0) if a source entry is a remote `git+` spec |
| `resolve_source_dir(repo_root, src)` | Map a source entry to a local dir — `"$repo_root/$src"` for a path, or a shallow clone for a pack reference (`@<name>[/<subpath>]`) or an inline `git+<url>` spec |
| `source_is_local_path(src)` | True (0) if a source entry is a plain repo-relative path — i.e. neither of the two below. Use this, not a negated `source_is_remote`, whenever a token is about to be pattern-matched against a real directory |
| `source_is_pack(src)` | True (0) if a source entry references a pack declared under `packs:` (`@<name>`) |
| `source_is_remote(src)` | True (0) if a source entry is an inline remote `git+` spec |
| `get_target_field(config, target, field)` | Read a field from a target's config block |

### Transformation Patterns
Expand Down Expand Up @@ -140,7 +142,10 @@ sync_to_myide() {
while IFS= read -r src; do
[ -z "$src" ] && continue
# resolve_source_dir maps a source entry to a local dir: "$repo_root/$src"
# for a local path, or a shallow clone for a remote `git+<url>` spec.
# for a local path, or a shallow clone for a pack reference (`@<name>`)
# or an inline `git+<url>` spec. A pack's url/ref/mirror are read from
# config.yaml, which it takes from $IS_CONFIG_FILE (exported by sync.sh)
# unless you pass the config as a third argument.
local dir
dir="$(resolve_source_dir "$repo_root" "$src")"
[ -d "$dir" ] || continue
Expand Down
Loading