Upgrade exemplar to multi-resource stencil with full test coverage

.claude-plugin/marketplace.json

@@ -7,8 +7,8 @@
   "plugins": [
     {
       "name": "clify",
-      "description": "Generate A+ Node.js CLIs from API documentation. Copy a hand-crafted exemplar, mechanically substitute API-specific content, then verify with a deterministic validation gate.",
-      "version": "0.2.0",
+      "description": "Generate A+ Node.js CLIs from API documentation. Copies a hand-crafted exemplar (structurally inspired by google/agents-cli), mechanically substitutes API-specific content, and verifies via a deterministic validation gate.",
+      "version": "0.3.0",
       "source": "./"
     }
   ]

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "clify",
-  "version": "0.2.0",
-  "description": "Generate A+ Node.js CLIs from API documentation. Copy a hand-crafted exemplar, mechanically substitute API-specific content, then verify with a deterministic validation gate.",
+  "version": "0.3.0",
+  "description": "Generate A+ Node.js CLIs from API documentation. Copies a hand-crafted exemplar (structurally inspired by google/agents-cli), mechanically substitutes API-specific content, and verifies via a deterministic validation gate.",
   "author": {
     "name": "codeyogi911",
     "url": "https://github.com/codeyogi911"
@@ -11,9 +11,7 @@
   "license": "MIT",
   "keywords": ["cli-generator", "api", "openapi", "codegen", "agent"],
   "skills": [
-    { "name": "clify-scaffold", "source": "skills/clify-scaffold/SKILL.md" },
-    { "name": "clify-validate", "source": "skills/clify-validate/SKILL.md" },
-    { "name": "clify-sync",     "source": "skills/clify-sync/SKILL.md" }
+    { "name": "clify", "source": "skills/clify/SKILL.md" }
   ],
   "capabilities": ["network", "codegen", "file-write"]
 }

.github/workflows/test.yml

@@ -16,7 +16,12 @@ jobs:
       - name: clify unit tests
         run: npm test
       - name: validate exemplar
-        run: node bin/clify.mjs validate examples/jsonplaceholder-cli
+        run: node bin/clify.mjs validate examples/exemplar-cli
       - name: exemplar tests
-        working-directory: examples/jsonplaceholder-cli
+        working-directory: examples/exemplar-cli
         run: npm test
+      - name: scaffold-init smoke (full round-trip)
+        run: |
+          tmp=$(mktemp -d)
+          node bin/clify.mjs scaffold-init demo-api --target "$tmp"
+          node bin/clify.mjs validate "$tmp/demo-api-cli"

README.md

@@ -5,7 +5,7 @@
 </p>
 
 <p align="center">
-  A Claude Code plugin that generates Node.js CLIs from API documentation by copying a hand-crafted exemplar, mechanically substituting API-specific content, and verifying with a deterministic validation gate. Inspired by <a href="https://github.com/google/agents-cli">google/agents-cli</a>.
+  A Claude Code plugin that generates Node.js CLIs from API documentation by copying a hand-crafted exemplar, mechanically substituting API-specific content, and verifying with a deterministic validation gate. Structurally inspired by <a href="https://github.com/google/agents-cli">google/agents-cli</a>.
 </p>
 
 <p align="center">
@@ -20,21 +20,21 @@
 
 MCPs are heavy; CLIs are the natural fit for agents. clify produces CLIs that are:
 
-- **Tested by default.** Every generated repo ships with smoke + integration tests and CI on Node 20 & 22.
+- **Tested by default.** Every generated repo ships with smoke + integration + auth tests and CI on Node 20 & 22.
 - **Verifiable.** A deterministic validation gate (`clify validate`) checks 8 categories — manifest consistency, coverage bookkeeping, structural reachability, declared nuances, secrets, CI, and tests.
-- **Reproducible.** The deterministic phases (copy + rename + validate) are pure JS — same pass/fail for any agent.
+- **Reproducible.** The deterministic phases (copy + rename + validate + grade) are pure JS — same pass/fail for any agent.
 - **Honest.** Endpoints that get dropped require an explicit reason in `coverage.json`. No silent omissions.
 
 ## Architecture
 
-clify is **a binary + skills**, on the same model as `google/agents-cli`:
+clify is **a binary + one skill**:
 
 | Layer | What it does | What runs it |
 |---|---|---|
-| **Skills** | Fetch docs, parse free-form HTML/Markdown, consult the user on tradeoffs, rewrite per-API content. | LLM (Claude/Codex) |
-| **`bin/clify.mjs`** | `clify validate <dir>` — full validation gate. `clify scaffold-init <api-name>` — copy exemplar + rename. `clify sync-check <dir>` — re-fetch docs, hash diff. | Pure JS, no LLM |
+| **Skill (`clify`)** | Fetch docs, parse free-form HTML/Markdown, consult the user on tradeoffs, rewrite per-API content. | LLM (Claude) |
+| **`bin/clify.mjs`** | `validate` — run the gate. `scaffold-init` — copy exemplar + rename. `sync-check` — re-fetch docs, hash diff. `eval` — grade against an `evals/cases/` file. | Pure JS, no LLM |
 
-Anything verifiable by code goes in the binary. Anything requiring judgment stays in skills. The binary is what makes verification reproducible across agent vendors.
+Anything verifiable by code goes in the binary. Anything requiring judgment stays in the skill. The binary is what makes verification reproducible across agent vendors. The earlier `clify-validate` and `clify-sync` skills are gone — they were thin wrappers around binary verbs an agent can call directly without a SKILL.md to read first.
 
 ## Install
 
@@ -44,31 +44,39 @@ In a Claude Code session:
 /plugin install codeyogi911/clify
 ```
 
-Or clone and link:
+Then verify the install — `/skills` should list `clify` as available, and the binary should run:
+
+```
+clify --help
+clify --version            # 0.3.0
+```
+
+Or clone and link for local development:
 
 ```
 git clone https://github.com/codeyogi911/clify
 cd clify
 npm install
-npm link  # makes `clify` available as a CLI
+npm link                   # makes `clify` available globally
 ```
 
 ## Use
 
 Inside Claude Code:
 
 ```
-/clify-scaffold https://docs.example.com/api
+/clify https://docs.example.com/api
 ```
 
-The skill walks the 13-step pipeline (fetch → parse → consult → **ask where to put it** → init → substitute → validate → simplify → report). The generated CLI is **its own project**, in its own directory, with its own `git init` — by default a sibling of your current directory, but the skill asks before creating files. Output is `<chosen-parent>/<api-name>-cli/`, never nested inside the calling repo unless you explicitly ask for that.
+The skill walks six phases (fetch & detect → parse & group → consult → init → substitute → validate, simplify, report). The generated CLI is **its own project**, in its own directory, with its own `git init` — by default a sibling of your current directory, but the skill asks before creating files. Output is `<chosen-parent>/<api-name>-cli/`, never nested inside the calling repo unless you explicitly ask for that.
 
 You can also call the binary verbs directly:
 
 ```
 clify validate ./my-api-cli              # check a generated repo
 clify scaffold-init demo-api --target .  # copy exemplar + rename only
 clify sync-check ./my-api-cli            # detect upstream doc drift
+clify eval zoho-inventory --repo ./zoho-inventory-cli
 clify --help
 ```
 
@@ -84,40 +92,55 @@ Every check in [`references/validation-gate.md`](references/validation-gate.md)
 - CI runs `npm test` on Node 20 + 22
 - `npm test` exits 0
 
-The exemplar at [`examples/jsonplaceholder-cli/`](examples/jsonplaceholder-cli/) is the canonical A+ implementation. Run the gate against it:
+The exemplar at [`examples/exemplar-cli/`](examples/exemplar-cli/) is the canonical A+ implementation — a fictional API designed to exercise every nuance the scaffolder must handle. Run the gate against it:
 
 ```
-node bin/clify.mjs validate examples/jsonplaceholder-cli
+node bin/clify.mjs validate examples/exemplar-cli
 ```
 
+## Evals
+
+`evals/` holds graded test cases. Each case file describes a real API and the attributes the scaffolder must produce:
+
+```
+clify eval zoho-inventory --repo ./zoho-inventory-cli
+```
+
+The harness runs the validation gate plus deterministic graders (auth scheme, resource count, declared nuances, knowledge files). The LLM-as-judge structural similarity score is a placeholder — wired in a follow-up. See [`evals/README.md`](evals/README.md).
+
 ## Testing
 
 ```
-npm test                  # clify unit tests + deliberate-break tests
+npm test                  # clify unit + deliberate-break tests
+npm run test:exemplar     # the exemplar's own smoke + integration + auth tests
 npm run validate-exemplar # run the gate against the bundled exemplar
 ```
 
-The clify CI workflow (`.github/workflows/test.yml`) runs both, plus the exemplar's own test suite, on Node 20 and 22.
+The clify CI workflow (`.github/workflows/test.yml`) runs both on Node 20 and 22.
 
 ## Repo layout
 
 ```
 clify/
-├── bin/clify.mjs                 top-level binary (verbs)
+├── bin/clify.mjs                       top-level binary (verbs)
 ├── lib/
-│   ├── validate.mjs              validation gate impl
-│   ├── scaffold-init.mjs         file-copy + rename
-│   └── sync-check.mjs            hash-diff
-├── skills/
-│   ├── clify-scaffold/SKILL.md   the 13-step generation pipeline
-│   ├── clify-validate/SKILL.md   wraps `clify validate`
-│   └── clify-sync/SKILL.md       wraps `clify sync-check` + regeneration
-├── examples/jsonplaceholder-cli/ canonical A+ exemplar
+│   ├── validate.mjs                    validation gate impl
+│   ├── scaffold-init.mjs               file-copy + rename
+│   └── sync-check.mjs                  hash-diff
+├── skills/clify/SKILL.md               the 6-phase generation pipeline (lean)
+├── examples/
+│   ├── exemplar-cli/                   canonical A+ stencil
+│   └── legacy/jsonplaceholder-cli/     reference: simple-API single-skill shape
+├── evals/
+│   ├── cases/                          one .json per real API
+│   ├── grade.mjs                       deterministic graders
+│   └── run.mjs                         eval harness
 ├── references/
-│   ├── conventions.md            contracts every generated CLI honors
-│   └── validation-gate.md        every check the gate enforces
-├── test/clify.test.mjs           unit + deliberate-break tests
-└── .github/workflows/test.yml    CI (Node 20, 22)
+│   ├── conventions.md                  contracts every generated CLI honors
+│   ├── validation-gate.md              every check the gate enforces
+│   └── scaffold-pipeline.md            per-phase detail (loaded on demand)
+├── test/clify.test.mjs                 unit + deliberate-break tests
+└── .github/workflows/test.yml          CI (Node 20, 22)
 ```
 
 ## License

bin/clify.mjs

@@ -1,33 +1,35 @@
 #!/usr/bin/env node
 // clify — top-level CLI for the codegen plugin.
-// Verbs that do deterministic work: validate, scaffold-init, sync-check.
-// Verb that delegates to LLM: scaffold (informational, points to /clify-scaffold skill).
+// Verbs that do deterministic work: validate, scaffold-init, sync-check, eval.
+// Verb that delegates to LLM: scaffold (informational, points to the /clify skill).
 import { readFileSync } from "node:fs";
 import { dirname, join, resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 import { validate } from "../lib/validate.mjs";
 import { scaffoldInit } from "../lib/scaffold-init.mjs";
 import { syncCheck } from "../lib/sync-check.mjs";
+import { runEval, formatReport as formatEvalReport } from "../evals/run.mjs";
 
 const __dirname = dirname(fileURLToPath(import.meta.url));
 const REPO_ROOT = resolve(__dirname, "..");
 const VERSION = JSON.parse(readFileSync(join(REPO_ROOT, "package.json"), "utf8")).version;
 
 const HELP = `clify ${VERSION}
-Generate, validate, and sync agent-friendly CLIs from API documentation.
+Generate, validate, sync, and grade agent-friendly CLIs from API documentation.
 
 Usage:
   clify validate <dir>                 Run the validation gate (deterministic, no LLM)
   clify scaffold-init <api-name>       Copy the exemplar to <api-name>-cli/ and rename
                                        (use --target <dir> to choose parent directory)
   clify sync-check <dir>               Re-fetch docs, hash, print diff summary
-  clify scaffold <url>                 (LLM step) Use /clify-scaffold <url> in Claude Code
+  clify eval <case> [--repo <dir>]     Grade a candidate repo against an evals/cases/ file
+  clify scaffold <url>                 (LLM step) Use /clify <url> in Claude Code
   clify --version
   clify --help
 
-The validate, scaffold-init, and sync-check verbs are pure JS and reproducible across
-agent vendors. The scaffold verb is informational — full scaffold needs LLM judgment
-and runs as the /clify-scaffold skill in Claude Code.
+The validate, scaffold-init, sync-check, and eval verbs are pure JS and reproducible
+across agent vendors. The scaffold verb is informational — full scaffold needs LLM
+judgment and runs as the /clify skill in Claude Code.
 `;
 
 async function main() {
@@ -48,6 +50,7 @@ async function main() {
     case "validate": return await runValidate(rest);
     case "scaffold-init": return runScaffoldInit(rest);
     case "sync-check": return await runSyncCheck(rest);
+    case "eval":      return await runEvalVerb(rest);
     case "scaffold":   return runScaffoldInfo(rest);
     default:
       process.stderr.write(`error: unknown verb '${verb}'\n\n${HELP}`);
@@ -107,7 +110,7 @@ function runScaffoldInit(args) {
   try {
     const result = scaffoldInit({ apiName, target });
     if (json) process.stdout.write(JSON.stringify(result, null, 2) + "\n");
-    else process.stdout.write(`scaffolded ${result.apiName} → ${result.dir}\n  (next: cd into the dir, git init if you want history, then run /clify-scaffold's substitution phase)\n`);
+    else process.stdout.write(`scaffolded ${result.apiName} → ${result.dir}\n  (next: cd into the dir, git init if you want history, then run the /clify skill's substitution phase)\n`);
   } catch (err) {
     process.stderr.write(`error: ${err.message}\n`);
     process.exit(1);
@@ -137,7 +140,20 @@ async function runSyncCheck(args) {
 
 function runScaffoldInfo(args) {
   const url = args.find((a) => !a.startsWith("-")) || "<url>";
-  process.stdout.write(`Full scaffolding needs LLM judgment (parsing free-form docs, deciding endpoint→action mapping).\nRun in Claude Code:  /clify-scaffold ${url}\n\nThe scaffold skill calls 'clify scaffold-init' and 'clify validate' as deterministic\nsub-steps, but the parsing and substitution phases are LLM-driven.\n`);
+  process.stdout.write(`Full scaffolding needs LLM judgment (parsing free-form docs, deciding endpoint→action mapping).\nRun in Claude Code:  /clify ${url}\n\nThe clify skill calls 'clify scaffold-init' and 'clify validate' as deterministic\nsub-steps, but the parsing and substitution phases are LLM-driven.\n`);
+}
+
+async function runEvalVerb(args) {
+  const caseName = args.find((a) => !a.startsWith("-"));
+  if (!caseName) { process.stderr.write("Usage: clify eval <case> [--repo <dir>] [--skip-tests] [--json]\n"); process.exit(2); }
+  let repoDir;
+  for (let i = 0; i < args.length; i++) if (args[i] === "--repo" && args[i + 1]) repoDir = args[++i];
+  const skipTests = args.includes("--skip-tests");
+  const json = args.includes("--json");
+  const report = await runEval({ caseName, repoDir, skipTests });
+  if (json) process.stdout.write(JSON.stringify(report, null, 2) + "\n");
+  else process.stdout.write(formatEvalReport(report));
+  process.exit(report.ok ? 0 : 1);
 }
 
 main().catch((err) => {

evals/README.md

@@ -0,0 +1,74 @@
+# clify evals
+
+Eval harness for the clify scaffolder. Each case file describes a real API and a set of expected attributes; the harness grades a candidate generated repo against those expectations.
+
+## Layout
+
+```
+evals/
+├── cases/                 one .json per real API
+│   └── zoho-inventory.json
+├── grade.mjs              deterministic graders (file presence, .clify.json fields)
+├── run.mjs                harness entry — loads case, runs validate + graders
+└── README.md              this file
+```
+
+## Run a case
+
+The harness assumes the candidate repo already exists (you've run the scaffolder). Pass its path via `--repo`, or let it default to `./<case-name>-cli/` relative to your cwd.
+
+```
+clify eval zoho-inventory                          # default candidate path
+clify eval zoho-inventory --repo ../zoho-cli       # explicit candidate
+clify eval zoho-inventory --json                   # machine-readable
+clify eval zoho-inventory --skip-tests             # skip running candidate's npm test
+```
+
+Or directly:
+
+```
+node evals/run.mjs zoho-inventory --repo /path/to/zoho-inventory-cli
+```
+
+## What the harness checks
+
+1. **Validation gate** — runs `clify validate <candidate>`. Every category must pass. The case file's `expected.must_pass_validation_gate: true` makes this a hard requirement.
+2. **Deterministic graders** (`grade.mjs`):
+   - `auth_scheme` matches `expected.auth_scheme` (or one of `auth_scheme_fallbacks`).
+   - Resource count ≥ `expected.resources_min`.
+   - Every `expected.must_include_nuances` is declared in `.clify.json.nuances`.
+   - Every `expected.must_have_knowledge_files` exists under `knowledge/`.
+3. **Structural similarity** to the exemplar — *placeholder*. Today this returns `null` (skipped). The plan is to score files-present, helper-signatures-match, and modular-skills-layout against the exemplar via an LLM-as-judge sibling session. Until that's wired, the deterministic graders carry the load.
+
+## Adding cases
+
+Create `evals/cases/<name>.json`:
+
+```json
+{
+  "name": "<name>",
+  "url": "<docs URL>",
+  "description": "Why this API is interesting as an eval target.",
+  "expected": {
+    "auth_scheme": "bearer",
+    "resources_min": 5,
+    "must_include_nuances": ["pagination"],
+    "must_have_knowledge_files": [],
+    "must_pass_validation_gate": true,
+    "structural_similarity_to_exemplar": 0.85
+  }
+}
+```
+
+## Why Zoho Inventory is the inaugural case
+
+It exercises every nuance the scaffolder is supposed to handle:
+
+- **OAuth refresh** — multiple scopes, refresh_token grant against `accounts.zoho.<region>`. Forces the scaffolder to either model OAuth as `bearer` (with refresh in `login.mjs`) or extend the auth schemes.
+- **Multi-region routing** — `.com`, `.eu`, `.in`, `.com.au`, `.ca` each have their own accounts host *and* API host. Forces the scaffolder to handle a `<API>_REGION` env var feeding `BASE_URL` resolution.
+- **Offset pagination** — `page_context` with `has_more_page`. Different from the cursor pattern the exemplar demonstrates; tests whether the scaffolder generalizes pagination.
+- **Multipart uploads** — image attachments on items.
+- **Composite resources** — composite items (assemblies that bundle other items).
+- **Plan-tier rate limits** — different free/paid limits.
+
+If the scaffolder produces a passing CLI for Zoho Inventory from its docs URL alone, it has demonstrated the agents-cli quality bar.