Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/evals/docs-drop-pointer-stamps-batch5-a37f0c58.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Changed

- Pointer records no longer carry an as-of date or recheck trigger; stored values keep theirs.
7 changes: 7 additions & 0 deletions .changes/planning/docs-drop-pointer-stamps-batch5-1e6d9b43.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Changed

- Pointer records no longer carry an as-of date or recheck trigger; stored values keep theirs.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Changed

- Pointer records no longer carry an as-of date or recheck trigger; stored values keep theirs.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Changed

- Pointer records no longer carry an as-of date or recheck trigger; stored values keep theirs.
17 changes: 7 additions & 10 deletions plugins/evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,10 @@ cookbook (`anthropics/claude-cookbooks` `misc/building_evals.ipynb`), fetched 20

`claude plugin eval` is what runs and scores a plugin's eval cases. For what one run does and what
it reports, read
[How an eval run works](https://code.claude.com/docs/en/plugin-evals#how-an-eval-run-works) live
(as of 2026-11-03; recheck when that section changes how a run is graded or reported). It needs a
recent Claude Code; for the minimum version, see
[Requirements](https://code.claude.com/docs/en/plugin-evals#requirements) (as of 2026-10-11;
recheck when that section's version floor changes). Every run and every judge grader is a metered
model call on your own account.
[How an eval run works](https://code.claude.com/docs/en/plugin-evals#how-an-eval-run-works) live.
It needs a recent Claude Code; for the minimum version, see
[Requirements](https://code.claude.com/docs/en/plugin-evals#requirements). Every run and every
judge grader is a metered model call on your own account.

This plugin is the practice around that command rather than a second runner. `/evals:plugin-eval`
preflights the machine and the target, validates the case files, prices the suite before it spends,
Expand All @@ -45,8 +43,8 @@ work.

Here `/evals:design` writes a skill's cases and `claude plugin eval init` a whole plugin's, both in
the suite layout the CLI reads, and `/evals:validate` checks them. For that layout, read
[Write a case manually](https://code.claude.com/docs/en/plugin-evals#write-a-case-manually) live
(as of 2026-11-03; recheck when that section changes the case layout). This repository converts
[Write a case manually](https://code.claude.com/docs/en/plugin-evals#write-a-case-manually) live.
This repository converts
`skill-creator` `evals.json` cases into the suite rather than keeping both:
`skills/plugin-eval/scripts/convert-evals-json.py <plugin-dir>` converts one plugin's `evals.json`
cases into its `evals/` suite and removes the converted files.
Expand Down Expand Up @@ -112,8 +110,7 @@ Eight `userConfig` keys. Two set the run's cost ceiling, read by `/evals:plugin-
- **`max_cost_usd`** (number, default `5`): the ceiling passed to the CLI as `--max-cost-usd`.
Raise it for a suite whose estimate is higher. For what the ceiling bounds and what a run that
reaches it returns, read the `--max-cost-usd` row of
[Command options](https://code.claude.com/docs/en/plugin-evals#command-options) live (as of
2026-11-03; recheck when that row changes).
[Command options](https://code.claude.com/docs/en/plugin-evals#command-options) live.
- **`unlimited_cost`** (boolean, default `false`): removes the ceiling and nothing else. The
estimate still prints before the run, so an expensive suite is still visible before it starts.

Expand Down
9 changes: 3 additions & 6 deletions plugins/evals/skills/design/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,9 +195,6 @@ convert it; the overwrite rule below applies.
- **Pointer**: for the eval directory, the case layout and the `case.yaml` required fields, see
[Eval suite reference](https://code.claude.com/docs/en/plugin-evals#eval-suite-reference) and
[case.yaml fields](https://code.claude.com/docs/en/plugin-evals#case-yaml-fields).
- **As of**: 2026-10-31
- **Recheck trigger**: either section changes the default eval directory, the files a case
directory holds, or the fields `case.yaml` requires.

`claude plugin eval` runs each case with the plugin and again without it. The with arm answers
"does the output meet these expectations"; the with/without delta answers a comparative claim (the
Expand All @@ -206,9 +203,9 @@ agents and hooks together). `/evals:plugin-eval` reads that delta. Whether the s
delta question: the no-plugin arm has no skill to invoke, so read it from the with-arm
`tool_used: Skill` indicator, which is unscored.

Verified 2026-09-11 against <https://code.claude.com/docs/en/plugin-evals.md> and
`claude plugin eval --help` (recorded in `docs/native-surfaces.md`, the `plugin eval` entry).
Recheck when a Claude Code release changes `--ablation`'s default or which graders it excludes from scoring.
For how the arms run and which graders are scored, read
<https://code.claude.com/docs/en/plugin-evals.md> and `claude plugin eval --help` (recorded in
`docs/native-surfaces.md`, the `plugin eval` entry) live.

## Phase 3: grading hygiene gate

Expand Down
5 changes: 0 additions & 5 deletions plugins/evals/skills/methodology/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,6 @@ The target-number and rewrite rules are this repository's reading of the Achieva

- **Pointer**: for what grounds a target, see
<https://platform.claude.com/docs/en/test-and-evaluate/develop-tests#define-your-success-criteria>.
- **As of**: 2026-10-02
- **Recheck trigger**: that section changes what it names as grounds for a target.

## Agent evals: grader hygiene

Expand All @@ -100,9 +98,6 @@ When the thing under test is an agent that acts over many turns and changes an e
- **Pointer**: Anthropic's "Demystifying evals for AI agents" (2026-01-09)
<https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents>. Read through a fetch
summary with quoted sentences, not a raw-text check.
- **As of**: 2026-10-06
- **Recheck trigger**: the post is revised, or the develop-tests page above starts covering agent
evals; then the pointer moves there.

## Route by repository kind

Expand Down
6 changes: 2 additions & 4 deletions plugins/evals/skills/methodology/reference/eval-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,7 @@ prompt and leave `max_tokens` headroom rather than fencing the answer with a tig

- **Pointer**: when sizing `max_tokens` for a model that thinks, fetch
[Budget rules and tuning](https://platform.claude.com/docs/en/build-with-claude/extended-thinking#budget-rules-and-tuning)
live. **As of**: 2026-11-03. **Recheck trigger**: that section changes how thinking tokens count
against `max_tokens`.
live.

## Effort as an eval axis

Expand All @@ -65,8 +64,7 @@ suite, a flat cost-performance curve across levels means the task is not bound b
compute, and higher effort buys cost without score; a steep curve locates the cheapest level
that holds the target. Sweep model and effort together, since a stronger model at low effort can
beat a weaker one at high effort on both axes (basis:
`platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#tune-effort`,
verified 2026-09-09; recheck on that section changing).
`platform.claude.com/docs/en/about-claude/models/optimizing-for-cost-and-intelligence#tune-effort`).

When the bundled `claude-api` skill resolves in this session, its `hillclimb` subcommand
automates this search over a suite; this repository's step map for it is
Expand Down
3 changes: 0 additions & 3 deletions plugins/evals/skills/methodology/reference/hillclimb.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,6 @@ approval to change the target.

- **Pointer**: for how a subcommand workflow is started, see
<https://code.claude.com/docs/en/skills#work-on-claude-api-projects>.
- **As of**: 2026-10-01
- **Recheck trigger**: a docs page states whether the model may start a `/claude-api` subcommand
workflow.

## Whether it can drive a `claude plugin eval` suite

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Default: a judge rubric is a list of checkable pass/fail claims. The 1-to-5 reci
[eval-audit.md, When the grader is an LLM judge](https://github.com/anthropics/skills/blob/8a1541c4a3ffa5a20a5a91de0dcf3f0bab1d1ef4/skills/claude-api/shared/evals/eval-audit.md#when-the-grader-is-an-llm-judge)
disagree on rubric form.
- Correlate with <https://claude.dev/blog/automating-eval-design-and-hillclimbing#validating-the-grader>,
which departs from the platform page on rubric form as of 2026-10-01.
which departs from the platform page on rubric form.
- **As of**: 2026-10-01
- **Recheck trigger**: the platform page changes its Example evals section, or a commit to
`anthropics/skills` changes the LLM-judge section of `eval-audit.md`.
Expand Down
Loading
Loading