English | 简体中文
A meta-skill that optimizes other skills. You supply real usage examples — it runs the loop and ships a better, traceable skill.
Skill quality isn't about writing more — it's about verifying against real usage. Teams rarely write a high-quality skill on the first try, and no maintainer should have to master prompt engineering. What matters is whether a skill stably produces the expected output under real input.
| Traditional | Skill Optimizer |
|---|---|
| Hand-written by intuition | Agent optimizes automatically |
| Trial-and-error by hand | Agent blind-tests the skill |
| Unclear why it fails | Agent scores the gap |
| Changes untraceable | Versioned & reported |
| Depends on individuals | Process-driven & reusable |
Humans stop writing optimization logic. They only supply real usage examples — the agent does the rest.
Requires Python 3.11+. Scripts are standard-library only — no
pip install, runs offline. This skill is itself a standard Agent Skill: drop it where your host discovers project skills.
git clone https://github.com/nxl801/skill-optimizer.git
# The skill then lives at:
# .agents/skills/optimizing-skills-from-human-usage/| Host | Project location |
|---|---|
| GitHub Copilot | .agents/skills/ (also .github/skills/, .claude/skills/) |
| Claude Code | .claude/skills/ |
| OpenAI Codex | .agents/skills/ |
You don't write optimization logic — you supply real usage examples, run, and review.
① Prepare a task — three inputs feed one optimization task:
skill-optimization-task/
├── target-skill/
│ └── SKILL.md # the skill to optimize
├── usage-set/
│ ├── usage-001/
│ │ ├── input/ # a real call: prompt + context + attachments
│ │ └── expected/ # human reference output + acceptance criteria
│ └── usage-002/ ...
└── optimization-config.yaml # rounds / thresholds / stop conditions
Isolation red line — Runner sees only
input/, neverexpected/. That is the foundation of blind testing.
② Run optimization — point your host agent at the task directory. First, verify the deterministic preflight locally:
make demo # end-to-end deterministic demo (8 steps, prints each)
make preflight # full preflight on the bundled example
make ci # = tests + lint + preflight (reproduces CI locally)Then ask the host agent to load optimizing-skills-from-human-usage and run the loop on your task. (The Quick Start itself runs only the deterministic preflight — it validates and freezes task data; the semantic loop is orchestrated by the host agent.)
③ Review the result — read reports/final-report.md, ship only optimized-skill/.
For every usage pair, the loop runs blind and isolated — four roles in a closed loop, walled off from the answers:
- Runner blind-runs the skill as a real user — never sees the expected answer.
- Evaluator scores it across 3 layers — never edits the skill.
- Diagnosis finds the root cause — never edits directly.
- Optimizer applies one fix per round — never touches the usage set.
Separation of duties is the institutional guarantee of quality: no role is both player and referee.
A Human Reference Usage Pair is not a traditional unit test — it's a complete real-world usage scenario:
Usage Pair = Input + Reference output + Acceptance + References
- Input — the prompt, context, and attachments a real user supplies (a full call, not a fragment)
- Reference output — a high-quality result a human expert would produce (a baseline, not the only answer)
- Acceptance — Hard Assertions + Rubric (machine-checkable criteria)
- References — specs, docs, sample code (so the skill starts on the same footing as a human)
The agent need not replicate human prose — it must meet or exceed human expectations.
Skill Optimizer is a Meta-Skill (a skill that optimizes other skills). A Parent Agent (orchestrator) coordinates 4 isolated subagents:
| Subagent | Role | Does ✓ | Does NOT ✗ |
|---|---|---|---|
| Runner | Blind-runs the skill as a real user | Executes the task | Never sees the expected answer |
| Evaluator | 3-layer scoring, judges pass/fail | Only scores | Never edits the skill |
| Diagnosis | Root-cause attribution | Only analyzes | Never edits the skill directly |
| Optimizer | One main fix per round | Only edits the candidate | Never touches the usage set |
The optimization loop is hardened into a fail-closed, audited, crash-recoverable pipeline. These are not add-ons — they are part of how every run works:
- Independent multi-judge review. Every candidate's semantic review runs ≥2 genuinely independent certified reviewers (distinct session + fingerprint + seed). The variant is clean only when all of them pass with unanimous conclusions. Any disagreement, timeout, invalid schema, or reused session ⇒ reject. (Separate from the Final Panel.)
- Dry-run reliability gate. A pure detector flags any Runner output that is empty / placeholder / template-only / missing-artifact. A round whose dry-run ratio exceeds a frozen threshold routes to a legal non-promotion terminal — and never consumes the Hidden Holdout.
- Real cross-process crash recovery. If the process crashes between rounds, a fresh process reopens the same store, verifies the checkpoint hash + audit chain, and resumes — without rerunning the baseline, recreating the previous candidate, or duplicating bridge steps.
- Tamper-evident audit + content-addressed evidence. Every transition, execution, and artifact is hash-chained; promotion re-verifies the full evidence set before a Champion can change.
Full system-layer design, the judge-pool and dry-run-gate diagrams, and acceptance evidence (
847 passed, 5 skipped) live indocs/skill-evolution-system/.
- 🔁 Closed loop — test → evaluate → diagnose → patch → retest, until pass or stop condition
- 🧪 Blind & isolated — Runner can't see
expected/; Evaluator sees only anonymous A/B - 🎯 3-layer evaluation — Hard Assertions (veto) → Quality Rubric (scored) → Pairwise Blind (head-to-head)
- 🩺 Diagnose before patching — 13 failure types (10 optimizable skill defects + 3 audit defects); only skill defects enter the fix loop
- 🛡️ Anti-overfitting — no hardcoded answers; 70/30 Optimization / Validation split
- 📊 Baseline comparison — every improvement backed by comparable scores
- 📦 Auditable output — optimized skill + full version history + evaluation report, not a black box
- 🔐 Fail-closed reliability — multi-judge review, dry-run gate, cross-process crash recovery, tamper-evident audit
- 🌐 Cross-agent — open
SKILL.mdformat + standard-library Python; binds to no SDK / MCP
| Layer | Name | Purpose |
|---|---|---|
| 1️⃣ | Hard Assertions | Factual assertions that must pass — one-vote veto |
| 2️⃣ | Quality Rubric | Multi-dimensional scored (correctness / completeness / usability) |
| 3️⃣ | Pairwise Blind Comparison | Candidate vs human reference; Judge blind-picks the better one |
"Surpass human" requires all 6 conditions (AND, none may be missing): all hard gates pass · no factual errors · no forbidden behavior · Rubric beats the human reference · pairwise majority prefers the candidate · no critical regression on validation. Surpassing has strict criteria — not a subjective feeling.
| 🚫 Must NOT | ✅ Must |
|---|---|
| Copy the Reference Output | Changes must generalize into reusable rules |
| Write case-specific answers | Validation Set does not participate in daily edits |
| Modify the Usage Set / lower the Acceptance bar |
The usage set is split 70% Optimization / 30% Validation. One general rule covers thousands of cases; one hardcoded answer covers exactly one. Grinding the score on the optimization set is meaningless — generalization is the real quality.
🚫 Red line — never hardcode a human reference answer into the skill. This is the first gate against overfitting.
Diagnosis classifies failures into 13 types: 10 skill-side defects (optimizable — Trigger / Rule / Context / Navigation / Workflow / Tool / Output / Safety / Over-instruction / Environment) and 3 benchmark-audit defects (not optimizable — fix the truth, not the prompt).
The Optimizer fixes one main problem per round and must emit 6 accountable artifacts: Observation · Hypothesis · Planned Change · Expected Effect · Regression Risk · Generalization Check. Full taxonomy in references/diagnosis-taxonomy.md.
The deliverable isn't just a new skill — it's traceable optimization evidence.
skill-optimization-result/
├── optimized-skill/ # final version
├── versions/candidate-v{0..N}/ # versioned history
├── runs/{baseline,iterations,validation,final}/
├── evaluations/ # evaluation evidence
└── reports/
├── final-report.md # baseline / per-round changes / results / regression / grade / open issues
└── final-report.json
The core is an open skill — not locked to any SDK.
| Design Principle | Meaning |
|---|---|
| Core flow lives in the Skill | No host-specific logic |
| Subagents provided by the host | Adapts to Copilot / Claude / Codex / ... |
| Scripts are standard Python | No third-party deps, runs offline |
| No MCP / SDK dependency | Portable & auditable |
From templates/optimization-config.yaml (overridable within the schema):
| Setting | Default |
|---|---|
| Optimization / Validation split | 70 / 30 |
| Holdout minimum | 4 pairs (fewer → no holdout, higher generalization risk) |
| Maximum iterations | 6 |
| Maximum no-improvement rounds | 2 |
| Minimum meaningful score improvement | 2 points |
| Candidate regression threshold | fixed at 2.0 points (not configurable) |
| Passing score | 85/100 with all hard gates |
| Exceed-human score delta | 3 points + pairwise majority |
| Final stability | 3 independent Runners per pair |
| Final panel | 3 independent Evaluators per Runner output (9 judgments per pair) |
| Primary changes per iteration | exactly 1 |
Start with one minimal closed loop. One existing skill + 3–5 usage pairs is enough; scale after proving value.
- Pick an existing skill
- Prepare 3–5 usage pairs
- Run Skill Optimizer
- Review the report
- Review the optimized skill yourself
- Add more usage pairs & skills as you go
Recommended first scenarios — the easier it is to define input and expected output, the better a first project:
| Scenario | Why |
|---|---|
| PR Review | Clear input, clear expected diff, easy to judge |
| Linting / Static Analysis | Enumerable rules, binary pass/fail |
| Test Case Design | Coverage is quantifiable, gaps detectable |
| Doc / Architecture Review | Spec/principles are checkable, deviations detectable |
A Makefile provides one-command access (see CONTRIBUTING.md):
make test # 121-test offline suite
make lint # ruff check
make preflight # full deterministic preflight on the bundled example
make demo # end-to-end demo (8 steps, prints each)
make ci # = test + lint + preflight (reproduces CI locally)Repository layout
skill-optimizer/
├── README.md # this file (English)
├── README.zh-CN.md # 简体中文版
├── Makefile / pyproject.toml / LICENSE
├── docs/
│ ├── images/en/ # rendered slide images (22)
│ └── skill-evolution-system/ # system-layer design & acceptance evidence
└── .agents/skills/optimizing-skills-from-human-usage/
├── SKILL.md # ← the skill entry point
├── references/ # 7 authoritative references
├── schemas/ # 24 JSON Schemas / artifact contracts
├── scripts/ # 24 deterministic tools (stdlib)
├── templates/ # config / prompts / report templates
├── examples/simple-document-review/ # runnable example
└── tests/test_optimizer_scripts.py # 121 regression tests
Key documentation
| Doc | What's inside |
|---|---|
SKILL.md |
Entry point — orchestrator role, isolation gate, loop, stops, grades |
references/workflow.md |
The 14-step authoritative state machine |
references/subagent-roles.md |
Per-role receive / forbidden / write boundaries |
references/cross-agent-compatibility.md |
Mandatory isolation preflight before any real pair |
docs/skill-evolution-system/ |
System-layer reliability design & acceptance evidence |
examples/simple-document-review/ |
Runnable end-to-end example |
- Host isolation is external — this package specifies and validates the protocol; file / tool / network barriers are enforced by the host. If a host cannot enforce role boundaries, the correct outcome is
SAFETY_BLOCKED, not a lower-integrity single-agent run. - Benchmark audit first — independent Benchmark Auditors validate each frozen input/expected pair; a non-
VALIDaudit blocks optimization. - Anti-overfitting is fail-closed — the hardcoding detector scans only frozen Optimization IDs (never Validation); release requires
CLEANwith no override. - Semantic scoring is still model judgment — blinding, evidence-bearing rubrics, deterministic hard checks, and multi-judge aggregation reduce bias but don't eliminate it.
- Output is text-first — the current contract evaluates one UTF-8 text deliverable per execution; multi-file/binary artifacts need a deterministic textual comparison artifact.
| Doc | Purpose |
|---|---|
CONTRIBUTING.md |
How to contribute safely — dev setup, red lines, fork→PR CI |
SECURITY.md |
Security policy — threat model & vulnerability reporting |
CODE_OF_CONDUCT.md |
Contributor Covenant |
CHANGELOG.md |
Release history |
| Issues | Bugs & features |
Contributions welcome — please read CONTRIBUTING.md first. This project has a few inviolate rules: stdlib-only, scripts must never read expected/, one schema per artifact type.
License: Code & docs are MIT — see LICENSE. Deck content (SVG source, speaker notes) is MIT; projects/icons/tabler-outline/ is Tabler Icons (MIT).
The core of skill optimization is no longer "who writes better prompts" — it's "who can supply high-quality human reference usage sets."
Human Usage Set → Agent Loop → Optimized Skill → Reusable Skill Asset
Built as a meta-skill · Drive Agent-Skill auto-optimization with a real Human Reference Usage Set