From 325fa9b805485f9c49b568eb5014be84edc9645e Mon Sep 17 00:00:00 2001 From: gurvinder-dhillon Date: Mon, 6 Apr 2026 18:50:33 -0700 Subject: [PATCH 1/5] feat(project-kickoff): research-first approach with suggested answers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add Phase 0: silent research & inference before any questions - Extracts facts from user's message - WebSearches the concept to understand the domain and infer stack - Assigns HIGH/MEDIUM/UNKNOWN confidence tier to every answer - Rewrite Phase 1: pre-filled confirmation block instead of blank form - Every item shows [Pre-filled], [Suggested: X — reason], or [Needed] - Uses AskUserQuestion tool for binary/choice questions - User confirms or corrects, not fills from scratch - Phase 2 reuses Phase 0 research — no duplicate WebSearch - Add Gotchas section: personal/pro mismatch, org repos, SSH vs HTTPS, Windows paths, WebSearch unavailable, minimal-context fallback Co-Authored-By: Claude Sonnet 4.6 --- skills/project-kickoff/SKILL.md | 317 +++++++++++++++++++------------- 1 file changed, 187 insertions(+), 130 deletions(-) diff --git a/skills/project-kickoff/SKILL.md b/skills/project-kickoff/SKILL.md index 9bbef82..7fcb808 100644 --- a/skills/project-kickoff/SKILL.md +++ b/skills/project-kickoff/SKILL.md @@ -4,122 +4,185 @@ description: > Your CEO/PM assistant for starting, migrating, reviving, or bootstrapping any project — coding or non-coding. Use this skill whenever the user wants to: kick off a new project, set up a workspace or repo, migrate an existing project to a new repo, revive a defunct project, use an existing repo as inspiration, or bootstrap anything from scratch. This includes coding projects (Python, Node, Go, etc.), research projects, documentation, knowledge bases, note-taking systems, brainstorming workspaces, social media content workflows, video production setups, and learning projects. Trigger especially when the user says things like "help me set up", "start a new project", "I want to build", "kick off", "bootstrap", "migrate my project", "set up a workspace", "I found this repo and want to build on it", or describes wanting to begin something new even without saying the word "project". - This skill handles the full setup: discovery interview → tool checks → repo setup → reference fetching → CLAUDE.md + skills + MCP config → testing setup → README → verification → first task creation. + This skill handles the full setup: research & inference → discovery confirmation → tool checks → repo setup → reference fetching → CLAUDE.md + skills + MCP config → testing setup → README → verification → first task creation. --- # Project Kickoff -You are the user's project setup partner — part CEO, part PM, part senior engineer. Your job is to get any project from zero to "ready to work" in one structured conversation. You handle the full stack: workspace, GitHub, references, CLAUDE.md, skills, MCP servers, testing, README, and the first task. +You are the user's project setup partner — part CEO, part PM, part senior engineer. Your job is to get any project from zero to "ready to work" in one structured conversation. -Move through the phases in order. Never skip verification (Phase 9) or the first task (Phase 10). Always explain what you're about to do before running any command. +**Core principle:** Research first, ask second. Before presenting any questions, spend ~30 seconds researching and inferring answers. Then present a pre-filled confirmation block — not a blank form. The user should be confirming your work, not doing it themselves. --- -## Phase 1: Discovery Interview +## Phase 0: Research & Inference -Ask ALL of the following in a single, well-formatted message grouped by category. Do not ask them one at a time — that creates a tedious 30-message back-and-forth. Present them as a friendly intake form. +Run this phase silently before presenting any questions. Do not announce it — just do the work. -If the user has already provided some answers in their initial message, pre-fill those and only ask for what's missing. +### Step 1 — Extract from user's message -``` -Group A — Project Identity - 1. Project concept / rough idea? (we'll workshop the final name and description together next) - 2. Personal or professional? - 3. Category: coding / documentation / research / knowledge-base / brainstorming / - social-media / video / note-taking / learning / other (describe)? - 4. Tech stack (if coding — language, frameworks, tools you have in mind)? - 5. Any existing project, repo, or URL you're building on or drawing inspiration from? - 6. Any branding preferences? (colors, tone — e.g. minimal, playful, technical, bold) - -Group B — Workspace & Source Control - 7. Directory path where this should live? (e.g. ~/Projects/my-thing) - 8. GitHub setup: - a) No GitHub needed - b) New public repo - c) New private repo - d) Already have a GitHub repo — share the URL - e) Clone an existing internet repo and push to my own GitHub account - 9. GitHub username? (needed for b, c, e) - -Group C — References - 10. Any reference materials to pull in? (npm/pip packages, GitHub repos, URLs, - local files, papers, documents) — these will be fetched and stored locally, - gitignored so they never get committed. - -Group D — Infrastructure - 11. Any API keys, secrets, or cloud services this project will use? - (e.g. OpenAI, AWS, Stripe — we'll set up .env.example for these) - 12. Any MCP servers you know you'll want? - (e.g. web search, GitHub, database, browser automation — or say "suggest") - 13. Any specific Claude Code skills to add? - (or say "suggest" — I'll recommend based on your stack) - -Group E — Project Management & Documentation - 14. How do you want to track tasks? - (local TASKS.md / GitHub Issues / Linear / Notion / other) - 15. Where should project docs/wiki live? - (local docs/ folder / GitHub Wiki / Notion / other) - 16. What's the very first thing you'll work on after setup? - -Group F — Testing & Verification - 17. Coding projects: preferred testing framework? - (or say "suggest" — I'll pick based on your language) - 18. Non-coding projects: how will you verify/review your work? - (checklists, peer review, publish criteria, etc.) - 19. Do you need CI/CD? (GitHub Actions, etc.) -``` +Parse the user's initial message and extract every fact explicitly stated or strongly implied: +- Project concept / domain +- Tech stack hints (language names, framework names, tool names) +- Personal vs professional signals ("side project", "my team", "startup", "work") +- GitHub preferences (public/private mentioned, username/org mentioned, repo URL mentioned) +- Directory path (if mentioned) +- Any API services mentioned (OpenAI, Stripe, AWS, etc.) +- Any repos or URLs mentioned as references or inspiration + +### Step 2 — WebSearch the concept + +Run 2–3 targeted searches to understand the domain and infer sensible defaults: +1. `"[concept] github open source"` — find existing tools, understand naming patterns, assess landscape +2. `"[concept] [inferred language] best practices"` — confirm tech stack defaults +3. (optional) `"[concept] site:github.com stars:>100"` — find popular reference repos + +Use the results to: +- Understand what already exists (avoids naming conflicts in Phase 2) +- Confirm or refine the inferred tech stack +- Identify common patterns, testing frameworks, and tooling conventions for this type of project + +### Step 3 — Build draft answers + +For every question in Phase 1, assign a confidence tier: -Wait for the user's responses before proceeding. +| Tier | Confidence | Action | +|------|-----------|--------| +| HIGH | 90%+ | Mark as `[Pre-filled]` — present in confirmation block, user just confirms | +| MEDIUM | 60–90% | Mark as `[Suggested]` — show with brief reasoning, easy to override | +| UNKNOWN | <60% | Mark as `[Needed]` — ask directly with 2–3 concrete examples | + +Typical inference rules: +- "side project" / "personal" / "fun" → Personal +- "my team" / "company" / "startup" / "work" / "we" / API keys → Professional +- Language/framework explicitly named → HIGH confidence stack +- Common patterns from WebSearch → MEDIUM confidence stack +- No stack mentioned, ambiguous domain → UNKNOWN, ask with examples +- GitHub username/org mentioned → HIGH +- No path mentioned → suggest `~/Projects/` --- -## Phase 2: Project Name, Description & Branding +## Phase 1: Discovery Confirmation + +Present a single pre-filled confirmation block. Do NOT send a blank form. Every question must have either a pre-filled value, a suggestion, or a clear "Needed" marker. + +**Opening line:** "Here's what I found — confirm or correct anything:" + +**Format for each item:** +- `[Pre-filled]` — high confidence, shown without question mark +- `[Suggested: X — reason]` — medium confidence, shown as a proposal +- `[Needed]` — unknown, shown with examples to reduce typing + +**Use `AskUserQuestion` tool** for any binary or small-choice questions (personal vs professional, public vs private, task tracker choice) where the user can click rather than type. -This phase runs before touching any tooling or files. A strong project identity shapes the repo name, README, and everything else downstream. +**Example confirmation block:** -### Step 1 — Research similar projects +``` +Here's what I found — confirm or correct anything: + +── Project Identity ────────────────────────────── +1. Concept: CLI tool that converts Markdown files to PDF + [Pre-filled — from your message] + +2. Type: Personal side project + [Suggested — you said "side project"] + +3. Category: Coding — Python CLI tool + [Suggested — Markdown/PDF processing is typically Python] + +4. Stack: Python 3.11+, Click (CLI), weasyprint (PDF), pytest + [Suggested — standard stack for this type of tool; alternatives: pypandoc, reportlab] + +5. Inspiration: None mentioned + [Pre-filled — let me know if you have a reference repo] + +6. Branding: Minimal / technical + [Suggested — CLI tools default to this tone] -Use WebSearch to look for existing projects with similar names or solving the same problem. Check: -- GitHub (search the concept) -- Product Hunt / npm / PyPI / crates.io as appropriate -- General web search for "[concept] tool", "[concept] app", "[concept] project" +── Workspace & Source Control ──────────────────── +7. Directory: ~/Projects/md-to-pdf + [Suggested — based on your project concept] -Goals: avoid naming conflicts, avoid trademark/copyright collisions, understand the competitive landscape well enough to suggest differentiated names. +8. GitHub: New private repo + [Pre-filled — you said "private"] -### Step 2 — Suggest name options +9. Username: [Needed — please provide your GitHub username or org] -Propose **3–5 name options** that are: -- **Catchy and memorable** — easy to say, spell, and remember -- **SEO-friendly** — descriptive enough that people searching for this type of tool will find it -- **Available** — not already a major open-source project or product -- **Modern** — avoid generic or dated naming patterns +── References ──────────────────────────────────── +10. References: None mentioned + [Pre-filled — mention any repos/URLs/docs to pull in as context] -For each option provide: -- The name -- Why it works (one line) -- Suggested GitHub repo slug +── Infrastructure ──────────────────────────────── +11. API keys: None + [Suggested — no external services implied] -Format as a quick table for easy scanning. +12. MCP servers: None + [Suggested — not needed for a local CLI tool] -### Step 3 — Suggest description and tags +13. Skills: None beyond project-kickoff + [Suggested] -After the user picks a name (or proposes their own), draft: +── Project Management ──────────────────────────── +14. Tasks: GitHub Issues + [Suggested — standard for public/private repos] -1. **One-liner** (for README headline, GitHub description field): ~10–15 words, describes what it does and who it's for. Should contain the primary keyword naturally. -2. **Short description** (for README intro, npm/PyPI/GitHub "About"): 2–3 sentences. Lead with the problem it solves, then what it does, then why it's different. -3. **Tags / topics**: 5–8 relevant GitHub topics or keywords (lowercase, hyphenated). These improve discoverability on GitHub and search engines. +15. Docs: Local docs/ folder + [Suggested] + +16. First task: "Implement core markdown→PDF conversion" + [Suggested — most natural starting point] + +── Testing ─────────────────────────────────────── +17. Framework: pytest + [Pre-filled — standard for Python] + +18. Verification: Run pytest, all tests pass + [Pre-filled] + +19. CI/CD: GitHub Actions — lightweight test runner + [Suggested] + +──────────────────────────────────────────────── +Reply with corrections to any items above (e.g. "9: gurvinder, 7: ~/Work/md-to-pdf"). +Items with [Pre-filled] are confirmed unless you say otherwise. +``` -### Step 4 — Confirm with user +### Handling the response -Present all of the above and get explicit sign-off on: final name, one-liner, and tags. Only proceed to Phase 3 once confirmed. +- If the user replies "looks good" or similar → proceed with all pre-filled/suggested values +- If the user corrects specific items → update those items, proceed +- If the user asks "what about X" → answer and update +- The ONLY item that always requires explicit input: GitHub username/org (item 9), unless it was extracted from the initial message --- -## Phase 3: Tool Availability Check +## Phase 2: Project Name, Description & Branding + +**Use the research already done in Phase 0** — do not repeat the same WebSearch. Pull from the landscape findings. + +### Suggest name options + +Propose **3–5 name options** based on the concept and competitive landscape already researched: +- **Catchy and memorable** — easy to say, spell, recall +- **SEO-friendly** — contains the primary keyword naturally +- **Available** — not already a major project (check Phase 0 findings) +- **Modern** — no generic or dated patterns + +Format as a table: name | why it works | GitHub slug + +### Suggest one-liner and tags -Before touching the filesystem, detect the OS and check which tools are available. +After name confirmation: +1. **One-liner** (~12 words): what it does + who it's for, keyword-rich +2. **Short description** (2–3 sentences): problem → solution → differentiator +3. **Tags** (5–8): lowercase, hyphenated GitHub topics -### OS Detection +Get explicit sign-off before proceeding to Phase 3. + +--- + +## Phase 3: Tool Availability Check ```bash case "$(uname -s 2>/dev/null)" in @@ -128,12 +191,7 @@ case "$(uname -s 2>/dev/null)" in MINGW*|MSYS*|CYGWIN*) OS_TYPE="Windows" ;; *) OS_TYPE="Windows" ;; esac -echo "Detected OS: $OS_TYPE" -``` - -### Tool Check -```bash git --version 2>/dev/null && echo "git: OK" || echo "git: MISSING" gh --version 2>/dev/null && echo "gh: OK" || echo "gh: MISSING" gh auth status 2>/dev/null && echo "gh-auth: OK" || echo "gh-auth: NOT AUTHENTICATED" @@ -142,11 +200,9 @@ npm --version 2>/dev/null && echo "npm: OK" || echo "npm: MISSING" uv --version 2>/dev/null && echo "uv: OK" || echo "uv: MISSING" ``` -**Required for all projects:** `git`, `gh` (if GitHub is involved) - -If `gh-auth` is NOT AUTHENTICATED, stop and ask the user to run `gh auth login` before proceeding. +Stop if `gh-auth` is NOT AUTHENTICATED — ask the user to run `gh auth login`. -**Install commands for missing tools:** +Install commands for missing tools: | Tool | macOS | Linux | |------|-------|-------| @@ -155,13 +211,13 @@ If `gh-auth` is NOT AUTHENTICATED, stop and ask the user to run `gh auth login` | `node/npm` | `brew install node` | `sudo apt install nodejs npm` | | `uv` | `curl -LsSf https://astral.sh/uv/install.sh \| sh` | same | -Never install tools without explicit user approval. +Never install without explicit user approval. --- ## Phase 4: Repository Setup -**A — New repo (no existing code):** +**A — New repo:** ```bash mkdir -p && cd git init @@ -169,7 +225,7 @@ git init gh repo create --public/--private --source=. --remote=origin --push ``` -**B — Clone from internet, push to personal GitHub:** +**B — Clone from internet, push to own GitHub:** ```bash git clone && cd git remote rename origin upstream @@ -180,12 +236,11 @@ git push -u origin main **C — Existing local directory:** ```bash -cd -git init +cd && git init gh repo create --public/--private --source=. --remote=origin --push ``` -Create `.gitignore` early — must include: +Minimum `.gitignore`: ``` opensrc/ .env @@ -195,38 +250,32 @@ node_modules/ .DS_Store ``` -After creating the repo, apply topics: +After creating the repo: ```bash +# Set topics gh repo edit / --add-topic --add-topic -``` -Enable branch protection: -```bash -gh api repos///branches/main/protection \ - --method PUT \ - -f 'required_status_checks=null' \ - -F enforce_admins=false \ - -F 'required_pull_request_reviews[required_approving_review_count]=0' \ - -F allow_force_pushes=false \ - -F allow_deletions=false \ - -f restrictions=null +# Branch protection +gh api repos///branches/main/protection --method PUT --input - <<'EOF' +{"required_status_checks":null,"enforce_admins":false,"required_pull_request_reviews":null,"restrictions":null,"allow_force_pushes":false,"allow_deletions":false} +EOF ``` --- ## Phase 5: Reference Fetching -For each reference the user provided: +For each reference confirmed in Phase 1: -**GitHub repo → opensrc (preferred) or git clone:** ```bash -opensrc owner/repo # if opensrc is available +# GitHub repo (preferred) +opensrc owner/repo # fallback: git clone opensrc/ --depth=1 -``` -**URL → WebFetch:** -Fetch and save to `references/.md`. +# URL → save locally +# Use WebFetch, save to references/.md +``` Always ensure `opensrc/` is in `.gitignore`. @@ -239,13 +288,13 @@ Always ensure `opensrc/` is in `.gitignore`. ```markdown # - + ## Rules - Never install packages globally - [Python] Use uv: `uv venv && source .venv/bin/activate` -- Never commit `.env` files — use `.env.example` +- Never commit `.env` — use `.env.example` - Always run tests before committing ## Testing @@ -273,7 +322,7 @@ npx skills add ### MCP Servers (.mcp.json) -Only create if at least one MCP is approved. Use `${VAR_NAME}` for secrets: +Only create if at least one MCP was confirmed. Use `${VAR_NAME}` for all secrets: ```json { @@ -303,7 +352,7 @@ echo 'def test_placeholder(): assert True' > tests/test_basic.py npm install --save-dev vitest ``` -**CI (`github/workflows/ci.yml`):** +**CI (`.github/workflows/ci.yml`):** ```yaml name: CI on: [push, pull_request] @@ -312,11 +361,11 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - run: + - run: - run: ``` -**Non-coding projects:** Create `CHECKLIST.md` with review criteria instead. +**Non-coding:** Create `CHECKLIST.md` with review criteria. --- @@ -327,7 +376,7 @@ jobs: > -<2–3 sentence description: problem → solution → differentiator> +<2–3 sentences: problem → solution → differentiator> ## Installation @@ -356,8 +405,6 @@ MIT ## Phase 9: Verify the Setup -Run through this checklist and report results: - ``` [ ] Project directory exists [ ] git repo initialized @@ -370,17 +417,16 @@ Run through this checklist and report results: [ ] First commit created ``` -Fix any failures before proceeding. +Fix any failures before Phase 10. --- ## Phase 10: Create First Task -**GitHub Issues:** ```bash gh issue create \ - --title "" \ - --body "## Goal\n\n\n## Notes\n" + --title "" \ + --body "## Goal\n\n\n## Notes\n" ``` --- @@ -396,5 +442,16 @@ Project setup complete. Tasks: First task: -Next: cd && +Next: cd ``` + +--- + +## Gotchas + +- **Personal vs professional signal mismatch:** User says "personal" but mentions API keys, a team, or "we" → treat as professional. Professional affects `.env.example` depth and CI rigor. +- **Org repos:** If the repo is under a GitHub org (not personal account), `gh repo create /` requires the user to be an org member with repo creation rights. Confirm org membership if unsure. +- **SSH vs HTTPS:** If `git push` fails with "Permission denied (publickey)", the user's git protocol is SSH but no key is configured. Switch remote to HTTPS: `git remote set-url origin https://github.com//.git` +- **Windows paths:** `mkdir -p` does not exist — use `mkdir` or `New-Item -ItemType Directory -Force`. Always branch on `$OS_TYPE` from Phase 3. +- **Phase 0 WebSearch failure:** If WebSearch is unavailable, skip it and lower all confidence tiers by one level (HIGH → MEDIUM, MEDIUM → UNKNOWN). Still present a confirmation block, but with fewer pre-filled items. +- **User provides very little context:** If the initial message is 5 words or fewer (e.g. "new python project"), set everything to UNKNOWN and ask — but still use AskUserQuestion tool for binary choices. From c172f63c778e30a5414bb9bbf613b28097c4c127 Mon Sep 17 00:00:00 2001 From: gurvinder-dhillon Date: Mon, 6 Apr 2026 18:55:57 -0700 Subject: [PATCH 2/5] feat(project-kickoff): proactive opensrc reference fetching MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0: extract npm packages, GitHub repos, and URLs from user's message into an explicit fetch queue. Phase 1: item 10 now shows pre-filled list of what will be fetched with opensrc, rather than asking the user to remember. Phase 5: complete rewrite — always runs (new and existing projects), uses `npx opensrc` for packages/repos, WebFetch for URLs, verifies opensrc/ in .gitignore before fetching, documents refs in CLAUDE.md. .gitignore: add comment linking to vercel-labs/opensrc. Co-Authored-By: Claude Sonnet 4.6 --- .gitignore | 5 ++- skills/project-kickoff/SKILL.md | 75 ++++++++++++++++++++++++++++----- 2 files changed, 68 insertions(+), 12 deletions(-) diff --git a/.gitignore b/.gitignore index a3c2c9d..844789c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,8 @@ -# References — local only, never commit +# opensrc — local reference sources fetched via `npx opensrc` +# See https://github.com/vercel-labs/opensrc opensrc/ + +# Other local references — never commit references/ # Environment / secrets diff --git a/skills/project-kickoff/SKILL.md b/skills/project-kickoff/SKILL.md index 7fcb808..2d13248 100644 --- a/skills/project-kickoff/SKILL.md +++ b/skills/project-kickoff/SKILL.md @@ -28,7 +28,11 @@ Parse the user's initial message and extract every fact explicitly stated or str - GitHub preferences (public/private mentioned, username/org mentioned, repo URL mentioned) - Directory path (if mentioned) - Any API services mentioned (OpenAI, Stripe, AWS, etc.) -- Any repos or URLs mentioned as references or inspiration +- **References to fetch** — build an explicit queue for Phase 5: + - npm packages mentioned (e.g. "using zod and express") → `npx opensrc ` + - GitHub repos mentioned as inspiration or reference → `npx opensrc owner/repo` + - URLs mentioned (docs, blog posts, papers) → WebFetch → `references/.md` + - If the user says "I want to build something like X" where X is a known repo → add X to the fetch queue ### Step 2 — WebSearch the concept @@ -110,8 +114,11 @@ Here's what I found — confirm or correct anything: 9. Username: [Needed — please provide your GitHub username or org] ── References ──────────────────────────────────── -10. References: None mentioned - [Pre-filled — mention any repos/URLs/docs to pull in as context] +10. References: Will fetch with opensrc in Phase 5: + - → npx opensrc + - → npx opensrc + [Pre-filled — from your message; add more repos, packages, or URLs] + (or: None mentioned — let me know if you want any pulled in as context) ── Infrastructure ──────────────────────────────── 11. API keys: None @@ -265,19 +272,65 @@ EOF ## Phase 5: Reference Fetching -For each reference confirmed in Phase 1: +**Always run this phase** — for new projects and existing projects alike. References give Claude Code deep implementation context (actual source, not just docs). Never skip it when the user has mentioned any package, repo, or URL. +### Step 1 — Verify .gitignore has opensrc/ + +Before fetching anything: +```bash +grep -q "opensrc/" .gitignore || echo "opensrc/" >> .gitignore +``` + +### Step 2 — Fetch packages and repos with opensrc + +Use `npx opensrc` for every npm package and GitHub repo in the Phase 0 fetch queue. No global install needed — `npx` works everywhere. + +```bash +# npm package — auto-detects version from lockfile if present +npx opensrc +npx opensrc @ # specific version + +# GitHub repo +npx opensrc owner/repo +npx opensrc owner/repo@v1.2.3 # specific tag + +# Multiple at once +npx opensrc react react-dom next + +# Full GitHub URL also works +npx opensrc https://github.com/owner/repo +``` + +On first run, opensrc may offer to update `.gitignore` and create `AGENTS.md` — accept both. + +opensrc stores sources under `opensrc/` with a `sources.json` index for agent discovery. + +**Fallback if npx/npm is unavailable:** ```bash -# GitHub repo (preferred) -opensrc owner/repo -# fallback: -git clone opensrc/ --depth=1 +git clone https://github.com// opensrc/-- --depth=1 +``` + +### Step 3 — Fetch URLs with WebFetch + +For each URL, doc page, or paper in the fetch queue: +1. Use the WebFetch tool to retrieve the content +2. Save to `references/.md` +3. Add a 2–3 line summary of what it contains at the top of the file -# URL → save locally -# Use WebFetch, save to references/.md +### Step 4 — Document in CLAUDE.md + +Add a **Key References** section to CLAUDE.md listing what was fetched: + +```markdown +## Key References + +- `opensrc/owner--repo/` — +- `references/.md` — ``` -Always ensure `opensrc/` is in `.gitignore`. +### For existing projects + +When invoked on a project that already exists (Phase 4 scenario C or D), still run this phase fully if the user mentioned any packages, repos, or URLs. Do not skip because the project is already set up — references are always additive context. --- From 55eb0ac146a7743c7eed06ea4b1af1ab139e3564 Mon Sep 17 00:00:00 2001 From: gurvinder-dhillon Date: Mon, 6 Apr 2026 18:57:42 -0700 Subject: [PATCH 3/5] chore: consolidate all references under opensrc/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove references/ from .gitignore — opensrc/ is the single home for all local context: npm packages, repos, and fetched docs/URLs. Update skill Phase 5 and CLAUDE.md templates accordingly. Co-Authored-By: Claude Sonnet 4.6 --- .../skill-creator/references/schemas.md | 430 ++++++++++++++++++ .gitignore | 7 +- CLAUDE.md | 2 +- skills/project-kickoff/SKILL.md | 7 +- 4 files changed, 438 insertions(+), 8 deletions(-) create mode 100644 .agents/skills/skill-creator/references/schemas.md diff --git a/.agents/skills/skill-creator/references/schemas.md b/.agents/skills/skill-creator/references/schemas.md new file mode 100644 index 0000000..b6eeaa2 --- /dev/null +++ b/.agents/skills/skill-creator/references/schemas.md @@ -0,0 +1,430 @@ +# JSON Schemas + +This document defines the JSON schemas used by skill-creator. + +--- + +## evals.json + +Defines the evals for a skill. Located at `evals/evals.json` within the skill directory. + +```json +{ + "skill_name": "example-skill", + "evals": [ + { + "id": 1, + "prompt": "User's example prompt", + "expected_output": "Description of expected result", + "files": ["evals/files/sample1.pdf"], + "expectations": [ + "The output includes X", + "The skill used script Y" + ] + } + ] +} +``` + +**Fields:** +- `skill_name`: Name matching the skill's frontmatter +- `evals[].id`: Unique integer identifier +- `evals[].prompt`: The task to execute +- `evals[].expected_output`: Human-readable description of success +- `evals[].files`: Optional list of input file paths (relative to skill root) +- `evals[].expectations`: List of verifiable statements + +--- + +## history.json + +Tracks version progression in Improve mode. Located at workspace root. + +```json +{ + "started_at": "2026-01-15T10:30:00Z", + "skill_name": "pdf", + "current_best": "v2", + "iterations": [ + { + "version": "v0", + "parent": null, + "expectation_pass_rate": 0.65, + "grading_result": "baseline", + "is_current_best": false + }, + { + "version": "v1", + "parent": "v0", + "expectation_pass_rate": 0.75, + "grading_result": "won", + "is_current_best": false + }, + { + "version": "v2", + "parent": "v1", + "expectation_pass_rate": 0.85, + "grading_result": "won", + "is_current_best": true + } + ] +} +``` + +**Fields:** +- `started_at`: ISO timestamp of when improvement started +- `skill_name`: Name of the skill being improved +- `current_best`: Version identifier of the best performer +- `iterations[].version`: Version identifier (v0, v1, ...) +- `iterations[].parent`: Parent version this was derived from +- `iterations[].expectation_pass_rate`: Pass rate from grading +- `iterations[].grading_result`: "baseline", "won", "lost", or "tie" +- `iterations[].is_current_best`: Whether this is the current best version + +--- + +## grading.json + +Output from the grader agent. Located at `/grading.json`. + +```json +{ + "expectations": [ + { + "text": "The output includes the name 'John Smith'", + "passed": true, + "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'" + }, + { + "text": "The spreadsheet has a SUM formula in cell B10", + "passed": false, + "evidence": "No spreadsheet was created. The output was a text file." + } + ], + "summary": { + "passed": 2, + "failed": 1, + "total": 3, + "pass_rate": 0.67 + }, + "execution_metrics": { + "tool_calls": { + "Read": 5, + "Write": 2, + "Bash": 8 + }, + "total_tool_calls": 15, + "total_steps": 6, + "errors_encountered": 0, + "output_chars": 12450, + "transcript_chars": 3200 + }, + "timing": { + "executor_duration_seconds": 165.0, + "grader_duration_seconds": 26.0, + "total_duration_seconds": 191.0 + }, + "claims": [ + { + "claim": "The form has 12 fillable fields", + "type": "factual", + "verified": true, + "evidence": "Counted 12 fields in field_info.json" + } + ], + "user_notes_summary": { + "uncertainties": ["Used 2023 data, may be stale"], + "needs_review": [], + "workarounds": ["Fell back to text overlay for non-fillable fields"] + }, + "eval_feedback": { + "suggestions": [ + { + "assertion": "The output includes the name 'John Smith'", + "reason": "A hallucinated document that mentions the name would also pass" + } + ], + "overall": "Assertions check presence but not correctness." + } +} +``` + +**Fields:** +- `expectations[]`: Graded expectations with evidence +- `summary`: Aggregate pass/fail counts +- `execution_metrics`: Tool usage and output size (from executor's metrics.json) +- `timing`: Wall clock timing (from timing.json) +- `claims`: Extracted and verified claims from the output +- `user_notes_summary`: Issues flagged by the executor +- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising + +--- + +## metrics.json + +Output from the executor agent. Located at `/outputs/metrics.json`. + +```json +{ + "tool_calls": { + "Read": 5, + "Write": 2, + "Bash": 8, + "Edit": 1, + "Glob": 2, + "Grep": 0 + }, + "total_tool_calls": 18, + "total_steps": 6, + "files_created": ["filled_form.pdf", "field_values.json"], + "errors_encountered": 0, + "output_chars": 12450, + "transcript_chars": 3200 +} +``` + +**Fields:** +- `tool_calls`: Count per tool type +- `total_tool_calls`: Sum of all tool calls +- `total_steps`: Number of major execution steps +- `files_created`: List of output files created +- `errors_encountered`: Number of errors during execution +- `output_chars`: Total character count of output files +- `transcript_chars`: Character count of transcript + +--- + +## timing.json + +Wall clock timing for a run. Located at `/timing.json`. + +**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact. + +```json +{ + "total_tokens": 84852, + "duration_ms": 23332, + "total_duration_seconds": 23.3, + "executor_start": "2026-01-15T10:30:00Z", + "executor_end": "2026-01-15T10:32:45Z", + "executor_duration_seconds": 165.0, + "grader_start": "2026-01-15T10:32:46Z", + "grader_end": "2026-01-15T10:33:12Z", + "grader_duration_seconds": 26.0 +} +``` + +--- + +## benchmark.json + +Output from Benchmark mode. Located at `benchmarks//benchmark.json`. + +```json +{ + "metadata": { + "skill_name": "pdf", + "skill_path": "/path/to/pdf", + "executor_model": "claude-sonnet-4-20250514", + "analyzer_model": "most-capable-model", + "timestamp": "2026-01-15T10:30:00Z", + "evals_run": [1, 2, 3], + "runs_per_configuration": 3 + }, + + "runs": [ + { + "eval_id": 1, + "eval_name": "Ocean", + "configuration": "with_skill", + "run_number": 1, + "result": { + "pass_rate": 0.85, + "passed": 6, + "failed": 1, + "total": 7, + "time_seconds": 42.5, + "tokens": 3800, + "tool_calls": 18, + "errors": 0 + }, + "expectations": [ + {"text": "...", "passed": true, "evidence": "..."} + ], + "notes": [ + "Used 2023 data, may be stale", + "Fell back to text overlay for non-fillable fields" + ] + } + ], + + "run_summary": { + "with_skill": { + "pass_rate": {"mean": 0.85, "stddev": 0.05, "min": 0.80, "max": 0.90}, + "time_seconds": {"mean": 45.0, "stddev": 12.0, "min": 32.0, "max": 58.0}, + "tokens": {"mean": 3800, "stddev": 400, "min": 3200, "max": 4100} + }, + "without_skill": { + "pass_rate": {"mean": 0.35, "stddev": 0.08, "min": 0.28, "max": 0.45}, + "time_seconds": {"mean": 32.0, "stddev": 8.0, "min": 24.0, "max": 42.0}, + "tokens": {"mean": 2100, "stddev": 300, "min": 1800, "max": 2500} + }, + "delta": { + "pass_rate": "+0.50", + "time_seconds": "+13.0", + "tokens": "+1700" + } + }, + + "notes": [ + "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value", + "Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent", + "Without-skill runs consistently fail on table extraction expectations", + "Skill adds 13s average execution time but improves pass rate by 50%" + ] +} +``` + +**Fields:** +- `metadata`: Information about the benchmark run + - `skill_name`: Name of the skill + - `timestamp`: When the benchmark was run + - `evals_run`: List of eval names or IDs + - `runs_per_configuration`: Number of runs per config (e.g. 3) +- `runs[]`: Individual run results + - `eval_id`: Numeric eval identifier + - `eval_name`: Human-readable eval name (used as section header in the viewer) + - `configuration`: Must be `"with_skill"` or `"without_skill"` (the viewer uses this exact string for grouping and color coding) + - `run_number`: Integer run number (1, 2, 3...) + - `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors` +- `run_summary`: Statistical aggregates per configuration + - `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields + - `delta`: Difference strings like `"+0.50"`, `"+13.0"`, `"+1700"` +- `notes`: Freeform observations from the analyzer + +**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually. + +--- + +## comparison.json + +Output from blind comparator. Located at `/comparison-N.json`. + +```json +{ + "winner": "A", + "reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.", + "rubric": { + "A": { + "content": { + "correctness": 5, + "completeness": 5, + "accuracy": 4 + }, + "structure": { + "organization": 4, + "formatting": 5, + "usability": 4 + }, + "content_score": 4.7, + "structure_score": 4.3, + "overall_score": 9.0 + }, + "B": { + "content": { + "correctness": 3, + "completeness": 2, + "accuracy": 3 + }, + "structure": { + "organization": 3, + "formatting": 2, + "usability": 3 + }, + "content_score": 2.7, + "structure_score": 2.7, + "overall_score": 5.4 + } + }, + "output_quality": { + "A": { + "score": 9, + "strengths": ["Complete solution", "Well-formatted", "All fields present"], + "weaknesses": ["Minor style inconsistency in header"] + }, + "B": { + "score": 5, + "strengths": ["Readable output", "Correct basic structure"], + "weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"] + } + }, + "expectation_results": { + "A": { + "passed": 4, + "total": 5, + "pass_rate": 0.80, + "details": [ + {"text": "Output includes name", "passed": true} + ] + }, + "B": { + "passed": 3, + "total": 5, + "pass_rate": 0.60, + "details": [ + {"text": "Output includes name", "passed": true} + ] + } + } +} +``` + +--- + +## analysis.json + +Output from post-hoc analyzer. Located at `/analysis.json`. + +```json +{ + "comparison_summary": { + "winner": "A", + "winner_skill": "path/to/winner/skill", + "loser_skill": "path/to/loser/skill", + "comparator_reasoning": "Brief summary of why comparator chose winner" + }, + "winner_strengths": [ + "Clear step-by-step instructions for handling multi-page documents", + "Included validation script that caught formatting errors" + ], + "loser_weaknesses": [ + "Vague instruction 'process the document appropriately' led to inconsistent behavior", + "No script for validation, agent had to improvise" + ], + "instruction_following": { + "winner": { + "score": 9, + "issues": ["Minor: skipped optional logging step"] + }, + "loser": { + "score": 6, + "issues": [ + "Did not use the skill's formatting template", + "Invented own approach instead of following step 3" + ] + } + }, + "improvement_suggestions": [ + { + "priority": "high", + "category": "instructions", + "suggestion": "Replace 'process the document appropriately' with explicit steps", + "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior" + } + ], + "transcript_insights": { + "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script", + "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods" + } +} +``` diff --git a/.gitignore b/.gitignore index 844789c..7463360 100644 --- a/.gitignore +++ b/.gitignore @@ -1,10 +1,7 @@ -# opensrc — local reference sources fetched via `npx opensrc` -# See https://github.com/vercel-labs/opensrc +# opensrc — all local reference sources (packages, repos, fetched docs) +# fetched via `npx opensrc` — see https://github.com/vercel-labs/opensrc opensrc/ -# Other local references — never commit -references/ - # Environment / secrets .env .env.* diff --git a/CLAUDE.md b/CLAUDE.md index 3f2714c..227d761 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ A curated collection of Claude Code skills for running AI agents at the executiv - Skills live in `skills//SKILL.md` — one skill per directory - Every SKILL.md must have `name:` and `description:` frontmatter fields -- Never edit files under `opensrc/` — read-only reference context +- Never edit or commit files under `opensrc/` — all reference material lives there, gitignored - Run the validator before committing: `bash .github/scripts/validate.sh` - All task tracking happens in GitHub Issues at https://github.com/openqa-labs/zenith/issues diff --git a/skills/project-kickoff/SKILL.md b/skills/project-kickoff/SKILL.md index 2d13248..84c7e7e 100644 --- a/skills/project-kickoff/SKILL.md +++ b/skills/project-kickoff/SKILL.md @@ -314,9 +314,11 @@ git clone https://github.com// opensrc/-- --depth=1 For each URL, doc page, or paper in the fetch queue: 1. Use the WebFetch tool to retrieve the content -2. Save to `references/.md` +2. Save to `opensrc/.md` 3. Add a 2–3 line summary of what it contains at the top of the file +Everything goes under `opensrc/` — packages, repos, and fetched docs alike. One gitignored directory, no exceptions. + ### Step 4 — Document in CLAUDE.md Add a **Key References** section to CLAUDE.md listing what was fetched: @@ -325,7 +327,7 @@ Add a **Key References** section to CLAUDE.md listing what was fetched: ## Key References - `opensrc/owner--repo/` — -- `references/.md` — +- `opensrc/.md` — ``` ### For existing projects @@ -348,6 +350,7 @@ When invoked on a project that already exists (Phase 4 scenario C or D), still r - Never install packages globally - [Python] Use uv: `uv venv && source .venv/bin/activate` - Never commit `.env` — use `.env.example` +- Never commit `opensrc/` — all reference material lives there, gitignored - Always run tests before committing ## Testing From 098d0ebc9ada1387ba0e335b408dcf577e0fc434 Mon Sep 17 00:00:00 2001 From: gurvinder-dhillon Date: Mon, 6 Apr 2026 19:00:12 -0700 Subject: [PATCH 4/5] feat(project-kickoff): include opensrc instructions in generated CLAUDE.md The CLAUDE.md template now tells the agent about opensrc so it can fetch additional context during regular work, not just at kickoff. Co-Authored-By: Claude Sonnet 4.6 --- skills/project-kickoff/SKILL.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/skills/project-kickoff/SKILL.md b/skills/project-kickoff/SKILL.md index 84c7e7e..b237d26 100644 --- a/skills/project-kickoff/SKILL.md +++ b/skills/project-kickoff/SKILL.md @@ -350,7 +350,10 @@ When invoked on a project that already exists (Phase 4 scenario C or D), still r - Never install packages globally - [Python] Use uv: `uv venv && source .venv/bin/activate` - Never commit `.env` — use `.env.example` -- Never commit `opensrc/` — all reference material lives there, gitignored +- `opensrc/` is gitignored — all local reference material lives there (packages, repos, fetched docs) + - Add more context anytime: `npx opensrc ` + - See `opensrc/sources.json` for what's already been fetched + - Never edit files inside `opensrc/` — they are read-only context - Always run tests before committing ## Testing From 38892078d955ac689866159d5da1f72267dc13fc Mon Sep 17 00:00:00 2001 From: gurvinder-dhillon Date: Mon, 6 Apr 2026 19:03:26 -0700 Subject: [PATCH 5/5] docs: explicit opensrc usage instructions in CLAUDE.md (skill + root) Tell the agent exactly when to use opensrc: before working with any package/repo, and for saving reference docs/specs. Both the skill's generated CLAUDE.md template and the project root CLAUDE.md now say the same thing. Co-Authored-By: Claude Sonnet 4.6 --- CLAUDE.md | 6 +++++- skills/project-kickoff/SKILL.md | 9 +++++---- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 227d761..54284e7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,11 @@ A curated collection of Claude Code skills for running AI agents at the executiv - Skills live in `skills//SKILL.md` — one skill per directory - Every SKILL.md must have `name:` and `description:` frontmatter fields -- Never edit or commit files under `opensrc/` — all reference material lives there, gitignored +- `opensrc/` is gitignored — use it for all local reference material: + - **Before working with a package or external repo**, fetch its source for context: `npx opensrc ` + - **To save reference docs, specs, or web pages**, fetch and save them as `opensrc/.md` + - Check `opensrc/sources.json` to see what's already been fetched before fetching again + - Never edit files inside `opensrc/` — read-only context only - Run the validator before committing: `bash .github/scripts/validate.sh` - All task tracking happens in GitHub Issues at https://github.com/openqa-labs/zenith/issues diff --git a/skills/project-kickoff/SKILL.md b/skills/project-kickoff/SKILL.md index b237d26..57e3dfd 100644 --- a/skills/project-kickoff/SKILL.md +++ b/skills/project-kickoff/SKILL.md @@ -350,10 +350,11 @@ When invoked on a project that already exists (Phase 4 scenario C or D), still r - Never install packages globally - [Python] Use uv: `uv venv && source .venv/bin/activate` - Never commit `.env` — use `.env.example` -- `opensrc/` is gitignored — all local reference material lives there (packages, repos, fetched docs) - - Add more context anytime: `npx opensrc ` - - See `opensrc/sources.json` for what's already been fetched - - Never edit files inside `opensrc/` — they are read-only context +- `opensrc/` is gitignored — use it for all local reference material: + - **Before working with a package or external repo**, fetch its source for context: `npx opensrc ` + - **To save reference docs, specs, or web pages**, fetch and save them as `opensrc/.md` + - Check `opensrc/sources.json` to see what's already been fetched before fetching again + - Never edit files inside `opensrc/` — read-only context only - Always run tests before committing ## Testing