Benchmark the review, across harnesses, and report what it says

Everything measured so far proves mechanisms fire. None of it shows the skills
produce a better review, which is the claim that actually matters and the one
this repo had no evidence for.

tests/evals/run.sh sends an identical prompt under three arms — Claude Code's
built-ins alone, plus vstack's skills, plus gstack's — with every arm loading
skills the same way, through project scope, so the only variable is which skills
are present. Fixtures carry planted defects and decoys that look suspicious and
are correct, findings come back as JSON with line numbers, and scoring is
arithmetic. clean.py has nothing wrong with it at all, so every finding there is
a false positive; that is the only unambiguous precision measurement in the set.

The first pilot at one sample says none 3/3, vstack 2/3, gstack 2/3, zero false
positives each. No arm beats the baseline. That is recorded here before running a
larger sample, so the number cannot be chosen after seeing it.

Two versions of this benchmark produced flattering nonsense before being caught,
and both are documented in the file because they are the failure mode of anyone
benchmarking their own project. Pointing CLAUDE_CONFIG_DIR at an empty directory
removes authentication along with the configuration, so every baseline run
returned "Not logged in", scored zero, and handed vstack a 5/6-to-0/6 victory
over a CLI that was never logged in. And omitting `< /dev/null` makes the CLI
wait for stdin and truncate. Each arm now reports which skills the session
actually loaded, from the CLI's own init event, and an arm that did not load its
harness is marked INVALID rather than scored.

Also from the external review: the deploy tier is opt-in behind --with-deploy
rather than installed by default, because someone who wants better Claude
behaviour does not need this author's deployment stack. And bin/cloudflare-mcp
now explains itself and exits cleanly when its server is not built, instead of
dying with a raw "Cannot find module" that reads like vstack is broken.

Author: itsvedantkumar

bin/cloudflare-mcp

@@ -1,9 +1,32 @@
 #!/bin/sh
+# cloudflare-mcp — thin wrapper around a locally-built Cloudflare MCP server.
+#
+# This one is the author's, and it is not vendored: the server itself lives in a separate
+# checkout. Nothing in this repo builds it, so on anyone else's machine the path below does not
+# exist. It used to exec straight into it and die with a raw "Cannot find module", which reads
+# like vstack is broken rather than like an optional integration is simply not set up.
+#
+# Point CLOUDFLARE_MCP_DIST at your own build if you have one. Otherwise this is inert, and the
+# only cost is the line in .claude.json.
 set -eu
+
 if [ -f "$HOME/.config/agents/secrets.env" ]; then
   . "$HOME/.config/agents/secrets.env"
-else
-  echo "cloudflare-mcp: missing $HOME/.config/agents/secrets.env" >&2
-  exit 1
 fi
-exec node "${CLOUDFLARE_MCP_DIST:-$HOME/Projects/cloudflare-mcp/dist/index.js}"
+
+DIST="${CLOUDFLARE_MCP_DIST:-$HOME/Projects/cloudflare-mcp/dist/index.js}"
+
+if [ ! -f "$DIST" ]; then
+  echo "cloudflare-mcp: not set up on this machine, so this server is doing nothing." >&2
+  echo "  expected a built server at: $DIST" >&2
+  echo "  set CLOUDFLARE_MCP_DIST to your own build, or remove the cloudflare-mcp entry" >&2
+  echo "  from ~/.claude.json if you do not use Cloudflare. vstack does not ship this server." >&2
+  exit 0
+fi
+
+if [ -z "${CLOUDFLARE_API_TOKEN:-}" ]; then
+  echo "cloudflare-mcp: CLOUDFLARE_API_TOKEN is not set in ~/.config/agents/secrets.env" >&2
+  exit 0
+fi
+
+exec node "$DIST"

setup-machine.sh

@@ -3,7 +3,8 @@
 # nothing. Idempotent: every tool is checked before it is installed, so a second run is a
 # fast no-op rather than a reinstall.
 #
-#   ./setup-machine.sh                 core + claude + deploy tiers
+#   ./setup-machine.sh                 core + claude
+#   ./setup-machine.sh --with-deploy   also vercel and wrangler
 #   ./setup-machine.sh --with-security also trivy, gitleaks, nmap, nuclei
 #   ./setup-machine.sh --check         report what is present, install nothing
 #   ./setup-machine.sh --dry-run       print what would be installed
@@ -14,16 +15,22 @@
 #                                                          with node or the Xcode tools
 #   claude    the Claude Code CLI itself
 #   conductor the Conductor Mac app, several agents in parallel (macOS only)
-#   deploy    vercel, wrangler                          — the autonomous deploy chain
+#   deploy    vercel, wrangler                          — opt-in. Someone who wants better
+#                                                          Claude behaviour does not need this
+#                                                          author's deployment stack, and
+#                                                          installing it by default made a
+#                                                          personal toolchain look like a
+#                                                          requirement of the product.
 #   security  trivy, gitleaks, nmap, nuclei             — the /security command
 #
 # This script installs software. It never removes any, and it never touches your dotfiles.
 set -uo pipefail
 
-WITH_SECURITY=0; CHECK=0; DRY=0; APT_UPDATED=0
+WITH_SECURITY=0; WITH_DEPLOY=0; CHECK=0; DRY=0; APT_UPDATED=0
 for a in "$@"; do
   case "$a" in
     --with-security) WITH_SECURITY=1 ;;
+    --with-deploy)   WITH_DEPLOY=1 ;;
     --check)         CHECK=1 ;;
     --dry-run)       DRY=1 ;;
     -h|--help)       sed -n '2,20p' "$0"; exit 0 ;;
@@ -307,9 +314,13 @@ else
 fi
 
 note ""
+if [ "$WITH_DEPLOY" = 1 ]; then
 note "== deploy"
 ensure_npm vercel   vercel
 ensure_npm wrangler wrangler
+else
+note "== deploy (skipped — pass --with-deploy for vercel and wrangler)"
+fi
 
 if [ "$WITH_SECURITY" = 1 ]; then
   note ""

tests/evals/fixtures/__pycache__/clean.cpython-314.pyc

@@ -0,0 +1,20 @@
+from typing import Iterable
+
+def normalise(names: Iterable[str]) -> list[str]:
+    """Lowercase and strip each name, dropping blanks."""
+    out = []
+    for n in names:
+        n = n.strip().lower()
+        if n:
+            out.append(n)
+    return out
+
+def dedupe(values: list[str]) -> list[str]:
+    """Preserve first-seen order while removing duplicates."""
+    seen = set()
+    out = []
+    for v in values:
+        if v not in seen:
+            seen.add(v)
+            out.append(v)
+    return out

tests/evals/fixtures/__pycache__/leak.cpython-314.pyc

@@ -0,0 +1,13 @@
+def read_config(path):
+    f = open(path)
+    # PLANTED: on a parse error the handle is never closed; the happy path is fine
+    data = f.read()
+    if not data.strip():
+        raise ValueError("empty config")
+    f.close()
+    return data
+
+def read_safe(path):
+    # DECOY: with-statement is correct; no leak here
+    with open(path) as f:
+        return f.read()

tests/evals/fixtures/__pycache__/off-by-one.cpython-314.pyc

@@ -0,0 +1,10 @@
+def chunk(items, size):
+    out = []
+    # PLANTED: range stops one chunk early when len(items) is not a multiple of size
+    for i in range(0, len(items) - size, size):
+        out.append(items[i:i + size])
+    return out
+
+def last_n(items, n):
+    # DECOY: negative slicing is correct here even though it looks like it could underflow
+    return items[-n:] if n > 0 else []

tests/evals/fixtures/__pycache__/sql-injection.cpython-314.pyc

@@ -0,0 +1,15 @@
+import sqlite3
+
+def find_user(conn, username):
+    # PLANTED: user input concatenated straight into SQL
+    q = "SELECT id, email FROM users WHERE name = '" + username + "'"
+    return conn.execute(q).fetchall()
+
+def find_by_role(conn, role_id):
+    # DECOY: looks like string building, but the value is bound, not interpolated
+    q = "SELECT id, email FROM users WHERE role = ?" + ""
+    return conn.execute(q, (role_id,)).fetchall()
+
+def audit_label(username):
+    # DECOY: concatenation into a log string is not injection
+    return "lookup:" + username