An optimisation loop that cannot fool itself #87

Workflow file for this run

	name: verify

	on:
	push:
	branches: [main]
	pull_request:

	jobs:
	verify:
	runs-on: ubuntu-latest
	steps:
	# fetch-depth 0 because check 24 compares the installable payload against the tag whose
	# version the manifests declare, and the default shallow checkout carries no tags at all.
	# Without this the check cannot measure and skips, which is honest but proves nothing on
	# the one machine that gates every push.
	- uses: actions/checkout@v4
	with:
	fetch-depth: 0

	# The gate needs jq for the JSON checks and the install dry run.
	- name: Install jq
	run: sudo apt-get update -qq && sudo apt-get install -y -qq jq

	# Check 19 runs `claude plugin validate --strict`; without a working CLI it skips, and the
	# falsifiability suite skips its case 19 with it. Install it so CI proves the real thing.
	#
	# `npm install -g` alone was not enough and produced a false green for as long as the step
	# has existed. The npm package is a shim that fetches a platform-native binary in its
	# postinstall; that did not land here, so every `claude plugin validate` exited non-zero
	# with "claude native binary not installed" — and check 19's old output parsing read that
	# error as no findings and printed ok. CI has never once validated these manifests.
	#
	# So install it, then prove the binary answers before trusting the step. `claude
	# --version` failing here is a setup failure and should stop the run, not be discovered
	# later as a check that silently measures nothing.
	- name: Install Claude Code CLI
	run: \|
	set -u
	# The npm package is a shim; the real binary arrives as a platform-specific optional
	# dependency. Installing the shim alone produced a false green for as long as this
	# step existed: the optional dep never landed, every `claude plugin validate` exited
	# with "native binary not installed", and check 19's old output parsing read that
	# error as "no findings" and printed ok. CI validated these manifests exactly zero
	# times before this was found.
	#
	# The reason the optional dep never landed is worth naming, because it will recur:
	# the shim pins its platform deps to its own version, and the platform packages are
	# published on a lag. At the time of writing the shim was 2.1.237 and linux-x64 was
	# 2.1.236, so npm resolved nothing and said nothing. Ask for the platform package by
	# name at whatever version is actually published, and use the binary it ships — it is
	# self-contained and needs no shim.
	npm install -g @anthropic-ai/claude-code \|\| true
	case "$(uname -s)-$(uname -m)" in
	Linux-x86_64) PKG=linux-x64 ;;
	Linux-aarch64) PKG=linux-arm64 ;;
	Darwin-arm64) PKG=darwin-arm64 ;;
	Darwin-x86_64) PKG=darwin-x64 ;;
	*) PKG="" ;;
	esac
	if [ -n "$PKG" ]; then
	PREFIX="$RUNNER_TEMP/ccbin"
	npm install --prefix "$PREFIX" "@anthropic-ai/claude-code-$PKG" \|\| true
	BIN="$PREFIX/node_modules/@anthropic-ai/claude-code-$PKG/claude"
	if [ -x "$BIN" ] && "$BIN" --version >/dev/null 2>&1; then
	sudo ln -sf "$BIN" /usr/local/bin/claude
	fi
	fi
	# Report rather than fail. Check 19 runs its own two-way control and skips with a
	# reason when the validator is unusable, so an unavailable CLI degrades to a visible
	# skip in the gate output instead of a red run over something this repo does not own.
	claude --version \|\| echo "::warning::claude CLI unusable on this runner; check 19 will skip"

	- name: The manifests were actually validated, not skipped
	run: \|
	# Check 19 is allowed to skip, so a green gate does not prove it ran. On this runner
	# it must run: that is the whole point of installing the CLI above, and a silent
	# regression back to skipping is exactly how it went unnoticed before.
	./.claude/verify.sh > "$RUNNER_TEMP/gate.txt" 2>&1 \|\| true
	if grep -q '^skip plugin manifests valid' "$RUNNER_TEMP/gate.txt"; then
	grep '^skip plugin manifests valid' "$RUNNER_TEMP/gate.txt"
	echo "check 19 skipped on a runner where the CLI was installed on purpose"
	exit 1
	fi
	grep '^ok plugin manifests valid' "$RUNNER_TEMP/gate.txt"

	- name: Run the verification gate
	run: ./.claude/verify.sh

	# A gate that cannot fail proves nothing. This used to break one skill description and
	# call it done, which left the other checks unproven — and two of them were not enforcing
	# anything at the time. Every declared check now gets its own mutation, and check 16 of
	# the gate fails if a check is added without one.
	- name: Prove every check is falsifiable
	run: ./tests/gate-falsifiability.sh

	# The comparison the README cites. It asserts the decision each mechanism produces, so a
	# regression turns it red rather than quietly changing a number in a table nobody re-runs.
	- name: The mechanisms still decide what the README says they do
	run: ./tests/compare-baseline.sh

	- name: Confirm the tree is restored
	run: git diff --exit-code

	# Everything above runs the gate. This installs for real, on Linux, into a disposable HOME.
	# It exists because the worst bug this repo has shipped was Linux-only and invisible on the
	# machine it was written on: hooks called /usr/bin/jq by absolute path, which is a macOS
	# path, so off macOS the verify gate silently stopped blocking and the session hook injected
	# nothing at all. A dry run cannot catch that. Only installing and firing the hooks can.
	install-linux:
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v4

	- name: Install jq
	run: sudo apt-get update -qq && sudo apt-get install -y -qq jq

	# Runs install.sh into throwaway HOMEs and asserts the resulting tree each time: a plain
	# install, CLAUDE_CONFIG_DIR pointed elsewhere, a home path with a space, jq removed from
	# PATH, a bash user, a second run for idempotency, both uninstall paths, and an install
	# over a home that already holds somebody else's skills, settings and MCP servers. Two
	# further cases exercise the curl bootstrap and the plugin marketplace against the
	# published repo. This runner is the environment the matrix exists for — bash by default,
	# no Homebrew, the shape of a cloud VM or a container.
	- name: Install Claude Code CLI
	run: \|
	set -u
	# The npm package is a shim; the real binary arrives as a platform-specific optional
	# dependency. Installing the shim alone produced a false green for as long as this
	# step existed: the optional dep never landed, every `claude plugin validate` exited
	# with "native binary not installed", and check 19's old output parsing read that
	# error as "no findings" and printed ok. CI validated these manifests exactly zero
	# times before this was found.
	#
	# The reason the optional dep never landed is worth naming, because it will recur:
	# the shim pins its platform deps to its own version, and the platform packages are
	# published on a lag. At the time of writing the shim was 2.1.237 and linux-x64 was
	# 2.1.236, so npm resolved nothing and said nothing. Ask for the platform package by
	# name at whatever version is actually published, and use the binary it ships — it is
	# self-contained and needs no shim.
	npm install -g @anthropic-ai/claude-code \|\| true
	case "$(uname -s)-$(uname -m)" in
	Linux-x86_64) PKG=linux-x64 ;;
	Linux-aarch64) PKG=linux-arm64 ;;
	Darwin-arm64) PKG=darwin-arm64 ;;
	Darwin-x86_64) PKG=darwin-x64 ;;
	*) PKG="" ;;
	esac
	if [ -n "$PKG" ]; then
	PREFIX="$RUNNER_TEMP/ccbin"
	npm install --prefix "$PREFIX" "@anthropic-ai/claude-code-$PKG" \|\| true
	BIN="$PREFIX/node_modules/@anthropic-ai/claude-code-$PKG/claude"
	if [ -x "$BIN" ] && "$BIN" --version >/dev/null 2>&1; then
	sudo ln -sf "$BIN" /usr/local/bin/claude
	fi
	fi
	# Report rather than fail. Check 19 runs its own two-way control and skips with a
	# reason when the validator is unusable, so an unavailable CLI degrades to a visible
	# skip in the gate output instead of a red run over something this repo does not own.
	claude --version \|\| echo "::warning::claude CLI unusable on this runner; check 19 will skip"

	- name: Install matrix (all environments, plus the network lanes)
	run: ./tests/install-matrix.sh

	# The marketplace case is allowed to skip when there is no CLI, which meant it skipped in
	# every job: the CLI was installed only in the verify job, and the matrix ran only in the
	# others. One of the three advertised install lanes was therefore unproven while every
	# required check stayed green. The CLI is installed above on purpose, so a skip here is a
	# regression, not an environment fact.
	- name: The marketplace lane actually ran
	run: \|
	./tests/install-matrix.sh marketplace > "$RUNNER_TEMP/mkt.txt" 2>&1 \|\| true
	cat "$RUNNER_TEMP/mkt.txt"
	if grep -q '^skip plugin marketplace lane' "$RUNNER_TEMP/mkt.txt"; then
	echo "marketplace case skipped on a runner where the CLI was installed on purpose"
	exit 1
	fi
	grep -q '^ok plugin marketplace lane' "$RUNNER_TEMP/mkt.txt"

	- name: install.sh, for real
	run: ./install.sh

	- name: The installed tree matches the repo
	run: ./bin/doctor --drift

	# Expected counts are derived from the repo at run time, not hardcoded — a literal here
	# went stale the first time a skill was added and failed a green tree.
	- name: Everything landed
	run: \|
	set -e
	count() { ls -d $1 2>/dev/null \| wc -l \| tr -d ' '; }
	for spec in "$HOME/.claude/skills//:claude/skills//" \
	"$HOME/.claude/agents/.md:claude/agents/.md" \
	"$HOME/.claude/commands/.md:claude/commands/.md" \
	"$HOME/.claude/hooks/.sh:claude/hooks/.sh"; do
	pat="${spec%:}"; src="${spec##:}"
	got=$(count "$pat"); want=$(count "$src")
	[ "$got" = "$want" ] && [ "$want" != 0 ] \|\| { echo "$pat: got $got, repo has $want"; exit 1; }
	echo "ok $pat -> $got (repo: $want)"
	done
	test -x "$HOME/.claude/hooks/verify-gate.sh"

	- name: The Stop gate blocks on Linux, with and without jq on PATH
	run: \|
	set -e
	mkdir -p /tmp/g/repo/.claude /tmp/g/home/.config/agents /tmp/g/tmp /tmp/g/bin
	printf '#!/usr/bin/env bash\necho "seeded failure"\nexit 1\n' > /tmp/g/repo/.claude/verify.sh
	chmod +x /tmp/g/repo/.claude/verify.sh
	printf '%s %s\n' "$(sha256sum /tmp/g/repo/.claude/verify.sh \| cut -d' ' -f1)" \
	/tmp/g/repo/.claude/verify.sh > /tmp/g/home/.config/agents/verify-trust
	for t in bash sh cat cut grep sed awk tr rm mkdir env dirname basename sha256sum; do
	ln -sf "$(command -v $t)" /tmp/g/bin/$t
	done
	out=$(printf '{"session_id":"ci"}' \| env HOME=/tmp/g/home TMPDIR=/tmp/g/tmp \
	CLAUDE_PROJECT_DIR=/tmp/g/repo bash claude/hooks/verify-gate.sh)
	echo "$out" \| jq -e '.decision=="block"' >/dev/null \|\| { echo "did not block with jq"; exit 1; }
	echo "ok blocks with jq"
	sed 's#/usr/bin/jq#/nonexistent/jq#' claude/hooks/verify-gate.sh > /tmp/g/nojq.sh
	out=$(printf '{"session_id":"ci2"}' \| env PATH=/tmp/g/bin HOME=/tmp/g/home \
	TMPDIR=/tmp/g/tmp CLAUDE_PROJECT_DIR=/tmp/g/repo bash /tmp/g/nojq.sh)
	echo "$out" \| jq -e '.decision=="block"' >/dev/null \|\| { echo "did not block without jq"; exit 1; }
	echo "ok blocks with no jq reachable"

	- name: The session hook injects the routing block on Linux
	run: \|
	set -e
	n=$(printf '{"hook_event_name":"SessionStart"}' \| bash claude/hooks/inject-session-context.sh \| wc -c)
	[ "$n" -gt 2000 ] \|\| { echo "session hook injected only $n bytes"; exit 1; }
	echo "ok session hook injected $n bytes"

	- name: uninstall.sh --dry-run
	run: ./uninstall.sh --dry-run

	# macOS is the platform this repo calls primary, and until now CI never touched it. Every
	# macOS-only assumption — BSD sed and find, `shasum` instead of `sha256sum`, `stat -f` — was
	# only ever exercised on the author's laptop, which is the definition of works-on-my-machine.
	install-macos:
	runs-on: macos-latest
	steps:
	- uses: actions/checkout@v4

	- name: Install matrix on macOS
	run: ./tests/install-matrix.sh

	- name: Run the verification gate
	run: ./.claude/verify.sh

	# Windows via Git Bash, which is what `shell: bash` selects on a windows runner and what a
	# Windows user running these scripts actually has. Native PowerShell is not a target and the
	# README says so; this proves the Git Bash and WSL-shaped path, which is the one people use.
	#
	# It gates like every other platform. It started as continue-on-error because Windows had
	# never been supported and the job existed to report the truth rather than block merges; it
	# went green in three commits, so the crutch is gone.
	install-windows:
	runs-on: windows-latest
	defaults:
	run:
	shell: bash
	steps:
	- uses: actions/checkout@v4

	- name: What this runner actually has
	run: \|
	bash --version \| head -1
	uname -s
	for t in jq git sed find cp chmod mktemp shasum sha256sum; do
	printf '%-12s %s\n' "$t" "$(command -v $t \|\| echo MISSING)"
	done

	- name: Install matrix on Windows (Git Bash)
	run: ./tests/install-matrix.sh

	- name: Run the verification gate
	run: ./.claude/verify.sh

	# Alpine, because BusyBox is not GNU. Its sed, grep, find and stat take different flags, and
	# there is no bash until you install it — which is exactly the shape of a slim container
	# image. Nothing else in this workflow would catch a GNU-ism.
	install-alpine:
	runs-on: ubuntu-latest
	container: alpine:latest
	steps:
	- name: Tools a slim image does not ship
	run: apk add --no-cache bash git jq curl findutils coreutils grep sed nodejs npm

	- uses: actions/checkout@v4

	# The container runs as root against a checkout owned by another uid, so git refuses to
	# operate on it at all. That is a property of running in a container, not of this repo,
	# but it fails loudly and confusingly: overlay.sh dies mid-run and check 17 reports the
	# overlay as broken. Declare the checkout safe so the gate measures the repo rather than
	# the runner's file ownership.
	- name: Let git operate on a checkout root does not own
	run: git config --global --add safe.directory "$GITHUB_WORKSPACE"

	- name: What this container actually has
	run: \|
	bash --version \| head -1
	for t in jq git sed find cp chmod mktemp sha256sum shasum; do
	printf '%-12s %s\n' "$t" "$(command -v $t \|\| echo MISSING)"
	done

	- name: Install matrix on Alpine
	run: bash ./tests/install-matrix.sh

	- name: Run the verification gate
	run: bash ./.claude/verify.sh

Navigation Menu

An optimisation loop that cannot fool itself #87

Workflow file

An optimisation loop that cannot fool itself #87

Uh oh!

Workflow file for this run