CI has never validated the plugin manifests

The real cause of the last three red runs, and of every green one before them.
`npm install -g @anthropic-ai/claude-code` installs a shim that fetches a
platform-native binary in its postinstall. On CI that binary never landed, so
every `claude plugin validate` exited non-zero with "claude native binary not
installed" — and check 19's old output parsing read that error as "no findings"
and printed ok. The step has been in the workflow for as long as the check has,
and it has never once validated anything.

CI now runs the postinstall and asserts `claude --version` before the gate, so a
broken install stops the run where it happens instead of resurfacing later as a
check that measures nothing.

The control runs both directions now, because each catches a different lie. A
validator that rejects everything is indistinguishable from a broken repo, which
is exactly the state CI was in. A validator that accepts everything is
indistinguishable from a healthy one. It has to accept a known-good manifest and
reject a known-bad one before this check will repeat what it says.

The good control carries an author field: without it --strict warns about
missing attribution and the control fails itself, which reads precisely like the
broken validator it exists to detect.

Proven against two stubs, one erroring and one exiting 0 on everything: both
produce a skip naming the reason, and the accounting still balances.

verify.sh 22/22, gate-falsifiability.sh 23/23.

Author: itsvedantkumar

.claude/verify.sh

@@ -600,17 +600,30 @@ fi
 #
 # So hand the validator a manifest that must be rejected, and if it accepts it, say the check
 # cannot measure here rather than reporting a pass on its behalf.
+# The control runs both ways, because each direction catches a different lie. A validator that
+# rejects everything looks identical to a broken repo: on CI the `claude` shim was on PATH while
+# its native binary was missing, so every invocation exited non-zero with an installation error
+# and nothing here could tell that apart from a genuinely bad manifest. A validator that accepts
+# everything looks identical to a healthy one. Require it to accept a good manifest AND reject a
+# bad one before believing a word it says about this repo's.
+ctl_state=usable
 if command -v claude >/dev/null 2>&1; then
-  ctl=$(mktemp -d); mkdir -p "$ctl/.claude-plugin"
-  printf '{"name":42,"version":"nope"}\n' > "$ctl/.claude-plugin/plugin.json"
-  ctl_ok=1
-  claude plugin validate --strict "$ctl" >/dev/null 2>&1 && ctl_ok=0
+  ctl=$(mktemp -d)
+  mkdir -p "$ctl/good/.claude-plugin" "$ctl/bad/.claude-plugin"
+  # The good manifest carries author too: without it --strict warns about missing attribution
+  # and the control fails itself, which reads exactly like a broken validator.
+  printf '{"name":"probe","version":"0.0.1","description":"control","author":{"name":"control"}}\n' \
+    > "$ctl/good/.claude-plugin/plugin.json"
+  printf '{"name":42,"version":"nope"}\n'                               > "$ctl/bad/.claude-plugin/plugin.json"
+  if ! ctl_out=$(claude plugin validate --strict "$ctl/good" 2>&1); then
+    ctl_state="cannot run: $(printf '%s' "$ctl_out" | grep -v '^$' | head -1)"
+  elif claude plugin validate --strict "$ctl/bad" >/dev/null 2>&1; then
+    ctl_state="accepts anything: it passed a manifest with name:42"
+  fi
   rm -rf "$ctl"
-else
-  ctl_ok=1
 fi
-if command -v claude >/dev/null 2>&1 && [ "$ctl_ok" = 0 ]; then
-  skip "plugin manifests valid" "validator accepted a manifest with name:42 — it is not validating here, so this check would only be reporting its own silence"
+if command -v claude >/dev/null 2>&1 && [ "$ctl_state" != usable ]; then
+  skip "plugin manifests valid" "validator $ctl_state — reporting its answer would be reporting its own silence"
 elif command -v claude >/dev/null 2>&1; then
   errs=""
   # Exactly one warning is expected: claude/CLAUDE.md is the source for the global and overlay

.github/workflows/verify.yml

@@ -16,10 +16,23 @@ jobs:
       - name: Install jq
         run: sudo apt-get update -qq && sudo apt-get install -y -qq jq
 
-      # Check 19 runs `claude plugin validate --strict`; without the CLI it skips, and the
+      # Check 19 runs `claude plugin validate --strict`; without a working CLI it skips, and the
       # falsifiability suite skips its case 19 with it. Install it so CI proves the real thing.
+      #
+      # `npm install -g` alone was not enough and produced a false green for as long as the step
+      # has existed. The npm package is a shim that fetches a platform-native binary in its
+      # postinstall; that did not land here, so every `claude plugin validate` exited non-zero
+      # with "claude native binary not installed" — and check 19's old output parsing read that
+      # error as no findings and printed ok. CI has never once validated these manifests.
+      #
+      # So install it, then prove the binary answers before trusting the step. `claude
+      # --version` failing here is a setup failure and should stop the run, not be discovered
+      # later as a check that silently measures nothing.
       - name: Install Claude Code CLI
-        run: npm install -g @anthropic-ai/claude-code
+        run: |
+          npm install -g @anthropic-ai/claude-code
+          node "$(npm root -g)/@anthropic-ai/claude-code/install.cjs" || true
+          claude --version
 
       - name: Run the verification gate
         run: ./.claude/verify.sh