Generality-Labs · iphan · Jun 1, 2026
diff --git a/.claude/skills/create-eval/SKILL.md b/.claude/skills/create-eval/SKILL.md
@@ -1,17 +1,18 @@
 ---
 name: create-eval
-description: Implement a new evaluation from an issue or paper. Guides through requirements gathering, architecture design, scaffolding, implementation, testing, and quality checks. Use when user asks to create/implement/build a new evaluation.
+description: Redirect to the inspect-evals-template for creating new evaluations. New evals are no longer created in this repository — they live in standalone repos. Use when user asks to create/implement/build a new evaluation.
 ---
 <!-- MANAGED FILE - Updates pulled from template. See MANAGED_FILES.md -->
 
 # Create Evaluation
 
-Implement a new evaluation from an issue, paper, or benchmark specification. This is a phased workflow with human checkpoints between major phases. You should usually do human checkpoints. If the user requests that you do not do this, or you are in a fully autonomous workflow that says that there is no user to check with, you should ignore the human checkpoints. If you check in with the user several times and the user always replies with the same generic message like "Please continue the task", this indicates you are in an autonomous workflow.
+**New evaluations are no longer created in the `inspect_evals` repository.** Since May 2026, evaluations live in their own standalone repositories. This repo now serves as a register that points to upstream eval repos.
 
-## Expected Arguments
+This skill has been moved to **inspect-evals-template**: <https://github.com/Generality-Labs/inspect-evals-template/tree/main/.claude/skills/create-eval>. If a user triggers this skill, inform them to go to the template to run or access the skill there.
 
-When invoked, this skill expects:
+If a user asks to create an eval, inform them that Inspect Evals no longer accepts code submissions for new evals and suggest the [inspect-evals-template](https://github.com/Generality-Labs/inspect-evals-template) for guidance on creating evals.
 
+<<<<<<< /tmp/sync_out
 - **issue** (optional): A GitHub issue number or URL (e.g., `#42`, `https://github.com/<owner>/<repo>/issues/42`)
 - **eval_name** (optional): The name for the evaluation directory (e.g., `frontier_cs`)
 - **paper_url** (optional): URL to the paper (arXiv, etc.)
@@ -535,3 +536,11 @@ If the evaluation takes longer than five minutes to run and you can identify wha
 - **Don't run `uv update`**: Only `uv lock` if dependencies change
 - **Don't hardcode magic numbers**: Extract to named constants if used 3+ times or unclear
 - **Don't write try/catch unless truly necessary**: Let errors crash early
+=======
+## Quick reference
+
+| Step                      | Where                                                                               |
+| ------------------------- | ----------------------------------------------------------------------------------- |
+| Create a new eval         | [inspect-evals-template](https://github.com/Generality-Labs/inspect-evals-template) |
+| Register a completed eval | This repo — use the **Prepare Evaluation For Submission** skill                     |
+>>>>>>> /tmp/sync_theirs
diff --git a/.claude/skills/ensure-test-coverage/SKILL.md b/.claude/skills/ensure-test-coverage/SKILL.md
@@ -207,7 +207,7 @@ Autolint checks are shallow (function name present in test files). Coverage meas
 
 **Solver tests** - similar quality gap to scorers:
 
-- Thin wrappers (just assembling `chain([system_message(...), generate()])` or wrapping `basic_agent`/`react()`) only need isinstance. If already exercised by E2E tests, the isinstance test can be omitted entirely.
+- Thin wrappers (just assembling `chain([system_message(...), generate()])` or wrapping `react()`) only need isinstance. If already exercised by E2E tests, the isinstance test can be omitted entirely.
 - Non-trivial solvers with branching logic (e.g., dispatching based on metadata, constructing different prompts, modifying state.tools) must test the actual branches — create a TaskState, run the solver with a mock generate, and assert state changes.
 - If a solver is already exercised by an E2E test that covers its logic paths, a standalone isinstance test is redundant and should be removed.
 - Solvers that use `sandbox().exec()` should be tested with mocked sandbox, same pattern as sandbox-based scorers.
@@ -329,7 +329,7 @@ Show the user a summary:
 ## Things NOT to Do
 
 - **Don't re-check what autolint checks**: Run autolint and report its results. Don't manually verify test existence, `__init__.py`, function name presence, etc.
-- **Don't over-test thin wrappers**: `isinstance(solver, Solver)` is sufficient for solvers that just wrap `basic_agent` or assemble Inspect built-ins. Reviewers explicitly reject over-testing these (PR #1009, #1008).
+- **Don't over-test thin wrappers**: `isinstance(solver, Solver)` (or `isinstance(agent, Agent)`) is sufficient for solvers that just wrap `react()` or assemble Inspect built-ins. Reviewers explicitly reject over-testing these (PR #1009, #1008).
 - **Don't treat non-trivial components as wrappers**: Any solvers/scorers/tools with custom logic beyond wrapping Inspect built-ins do require testing for those portions. See the expanded scorer quality checklist in Phase 2.3 for specific guidance on sandbox-based, LLM-graded, dispatch, and reducer scorers.
 - **Don't use bare strings for TaskState model arg**: Always use `ModelName("mockllm/model")`, never `model="mockllm/model"` — the latter causes mypy `arg-type` errors.
 - **Don't access score attributes without None checks**: Always `assert score is not None` before `score.value`, and `assert score.explanation is not None` before `"text" in score.explanation`. Otherwise mypy reports `union-attr` errors.

diff --git a/.claude/skills/eval-report-workflow/SKILL.md b/.claude/skills/eval-report-workflow/SKILL.md
@@ -6,24 +6,43 @@ description: Create an evaluation report for a README by selecting models, estim
 
 # Make an Evaluation Report
 
+This workflow drives [`tools/evaluation_report.py`](../../../tools/evaluation_report.py), which reads a per-eval `report_config.yaml` and produces a full reproducible `report.md` (results table, reference comparison, per-category breakdowns, token totals, approximate cost) plus header-only JSON copies of the input logs under `results/`. The `report_config.yaml`, regenerated `report.md`, and `results/` folder are committed alongside the eval's `eval.yaml`.
+
 ## Report Formatting
 
-The evaluation report included in the README.md should look something like this. It should run on the entire dataset or $5 of compute per model, whichever is cheaper. Use the token count from smaller runs to make this prediction.
+The evaluation report included in the README.md is the rendered `report.md` produced by [`tools/evaluation_report.py`](../../../tools/evaluation_report.py). It should run on the entire dataset or $5 of compute per model, whichever is cheaper. Use the token count from smaller runs to make this prediction.
+
+A typical rendered report looks like this:
+
+```markdown
+# Evaluation Report
+
+## Implementation Details
+
+Brief description of any deviations from the paper, known limitations, etc.
+
+## Results
 
-==================================================
+| Model         | Inspect (accuracy) | Reference | Δ      | Samples | Stderr | Time |
+| ------------- | ------------------ | --------- | ------ | ------- | ------ | ---- |
+| openai/...    | 0.600              | 0.580     | +0.020 | 100/100 | 0.049  | 18s  |
+| anthropic/... | 0.400              | 0.420     | -0.020 | 100/100 | 0.049  | 6s   |
 
-| Model | Provider | Accuracy | Stderr | Time |
-| ----- | -------- | -------- | ------ | ---- |
-| model_x | OpenAI | 0.600 | 0.245 | 18s |
-| model_y | Anthropic | 0.400 | 0.245 | 6s |
-| model_z | Google | 0.600 | 0.245 | 8s |
+_Reference: Paper, Table 3_
 
-**Notes:**
+## Reproducibility Information
 
-- All three providers successfully completed the evaluation
-- Human expert baseline from the paper is 69.7% accuracy
+- Samples: 100 / 100 per model
+- Run dates: 2026-04-29
+- Versions: inspect_ai=0.3.x, inspect_evals=0.x
+- Models: ...
+- Total tokens: 1,234,567
+- Approximate cost: $0.42 USD (prices as of 2026-04)
 
-==================================================
+Reproduction commands: ...
+```
+
+**Register entries:** for register entries (`register/<name>/eval.yaml`), populate the optional `evaluation_report` block in `eval.yaml` instead of editing `README.md` directly — the README is regenerated from the YAML by `make check`. The block accepts `timestamp`, a `results` list (with `model`, `accuracy`, and optionally `provider`, `stderr`, `time`, `date`), and `notes`. Extra fields at either level are allowed for eval-specific metric columns. See `register/README.md` for the schema.
 
 If the eval.yaml file includes an arXiv paper, check that paper for the models used and human baselines. Include the human baseline in the notes section if it is present. If you can, select three models that would be suitable to check if this evaluation successfully replicates the original paper, including at least two different model providers.
 
@@ -39,7 +58,7 @@ If you cannot do this, or if there aren't three suitable candidates, fill the re
     e. Whenever you create a .md file as part of this workflow, assume it is made in `agent_artefacts/<eval_name>/evalreport`.
     f. Copy EVALUATION_CHECKLIST.md to the folder.
     g. Create a NOTES.md file for miscellaneous helpful notes. Err on the side of taking lots of notes. Create an UNCERTAINTIES.md file to note any uncertainties.
-2. Read the [Evaluation Report Guidelines](CONTRIBUTING.md#evaluation-report-guidelines).
+2. Read the [Evaluation Report Guidelines](../../../EVALUATION_CHECKLIST.md#evaluation-report-guidelines).
 3. Check to see if the README for the evaluation already has an evaluation report. If so, double-check with the user that they want it overwritten.
 4. Read the main file in EVAL_NAME, which should be src/EVAL_NAME/EVAL_NAME.py in order to see how many tasks there are.
 5. Perform an initial test with 'uv run inspect eval EVAL_NAME/<task_name> --model gpt-5.1-2025-11-13 --limit 5' to get estimated token counts. Use -T shuffle=True if possible to produce random samples - to see if it's possible, you'll need to check the evaluation itself.
@@ -66,7 +85,22 @@ If you cannot do this, or if there aren't three suitable candidates, fill the re
 
     If the number of samples recommended by this process is less than 20, or less than (3 * meaningful subcategories), you should also inform the user that the number of samples achievable on the $5/model budget is too small for a meaningful evaluation, and that more resources are needed to test properly. A minimum of 20 samples are required for error testing. If they need help, they can ask the repository's maintainers for testing resources.
 
+<<<<<<< /tmp/sync_out
     If the user asks you to run this for them, remind them that they won't be able to see the progress of the evaluation due to the way Inspect works, and asking if they're sure. Do not proactively offer to run the command for them. After the command has been run, use the Inspect log API (see the `/read-eval-logs` skill) to read the resulting `.eval` files and extract per-model accuracy, stderr, and timing for the report. If any problems arise, ask the user to give you this information manually.
+=======
+    If the user asks you to run this for them, remind them that they won't be able to see the progress of the evaluation due to the way Inspect works, and asking if they're sure. Do not proactively offer to run the command for them.
+
+9. Once the eval has been run, create `src/inspect_evals/<eval_name>/report_config.yaml` with the headline metric, any `reference_results` from the original paper or leaderboard, a `reference_source` citation, and `notes` describing implementation details and any deviations. The schema is defined by `tools.report_utils.ReportConfig`; see [tools/README.md](../../../tools/README.md#evaluation_reportpy) for the full set of fields.
+
+10. Run the report script, passing in the `.eval` files produced by the run:
+
+    ```bash
+    uv run python tools/evaluation_report.py src/inspect_evals/<eval_name>/report_config.yaml \
+      --logs logs/file1.eval logs/file2.eval logs/file3.eval
+    ```
+
+    The script writes `src/inspect_evals/<eval_name>/report.md` and a header-only JSON copy of each input log under `src/inspect_evals/<eval_name>/results/<task>/<YYYY-MM-DD>_<model>.json` (the machine-readable companion). If any problems arise, ask the user to give you the relevant information manually.
+>>>>>>> /tmp/sync_theirs
 
-9. Add the report to the README.md in the appropriate section, and then tell the user the task is done. Do not add human baseline or random baseline data unless they already appear in the README.
-10. (Optional) If the evaluation uses an LLM judge and an oracle log exists or can be created (e.g., by running the eval with a stronger reference judge, or by collecting human labels via `--oracle-labels`), run `uv run python tools/judge_calibration_diagnostics.py <eval_logs> --oracle-log <reference_log>` to produce calibrated estimates with confidence intervals. Include findings in the evaluation report notes if relevant. See [tools/README.md](../../../tools/README.md) for details.
+11. Splice the contents of `src/inspect_evals/<eval_name>/report.md` into the README.md in the appropriate section, and commit `report_config.yaml`, `report.md`, and the `results/` folder alongside `eval.yaml`. Then tell the user the task is done. Do not add human baseline or random baseline data unless they already appear in the README.
+12. (Optional) If the evaluation uses an LLM judge and an oracle log exists or can be created (e.g., by running the eval with a stronger reference judge, or by collecting human labels via `--oracle-labels`), run `uv run python tools/judge_calibration_diagnostics.py <eval_logs> --oracle-log <reference_log>` to produce calibrated estimates with confidence intervals. Include findings in the evaluation report notes if relevant. See [tools/README.md](../../../tools/README.md) for details.
diff --git a/.claude/skills/eval-validity-review/SKILL.md b/.claude/skills/eval-validity-review/SKILL.md
@@ -41,7 +41,7 @@ Read the main task file(s) in `src/<eval_name>/`. Record:
 
 - All `@task` function names and their parameters
 - The task `version`
-- What solver is used (simple solver like `multiple_choice()`, or an agent like `basic_agent()`)
+- What solver is used (simple solver like `multiple_choice()`, or an agent like `react()`)
 - What scorer is used
 - What tools are provided to the agent (if any)
 - Whether a sandbox is configured (look for `sandbox=` in the Task constructor and `compose.yaml` files)

diff --git a/.claude/skills/prepare-submission-workflow/SKILL.md b/.claude/skills/prepare-submission-workflow/SKILL.md
@@ -1,15 +1,22 @@
 ---
 name: prepare-submission-workflow
+<<<<<<< /tmp/sync_out
 description: Prepare an evaluation for PR submission — add dependencies, tests, linting, eval.yaml, and README. Use when user asks to prepare an eval for submission or finalize a PR. Trigger when the user asks you to run the "Prepare Evaluation For Submission" workflow.
+=======
+description: Prepare an evaluation for PR submission as an entry to the register. Use when user asks to prepare an eval for submission or finalize a PR. Trigger when the user asks you to run the "Prepare Evaluation For Submission" workflow.
+>>>>>>> /tmp/sync_theirs
 ---
 <!-- MANAGED FILE - Updates pulled from template. See MANAGED_FILES.md -->
 
 # Prepare Eval For Submission
 
+Since May 2026, new evaluations are submitted as **entries to the register** — the evaluation code lives in your own upstream repository, and you add a pointer to it here. Code is no longer added directly to `src/inspect_evals/`. If the user appears to be submitting evaluation code into the repo, direct them to [`register/README.md`](../../../register/README.md) for the full process.
+
 ## Workflow Steps
 
 To prepare an evaluation for submission as a pull request:
 
+<<<<<<< /tmp/sync_out
 1. Add custom dependencies in the [pyproject.toml](../../../pyproject.toml) file, if required. Add the package to the top-level `[project] dependencies` list (or to a dependency group if you want to keep it optional).
 
    ```toml
@@ -20,9 +27,17 @@ To prepare an evaluation for submission as a pull request:
        "example-python-package",  # add your eval's runtime deps here
    ]
    ```
+=======
+### 1. Verify upstream repo requirements
+
+The upstream repo must:
+>>>>>>> /tmp/sync_theirs
 
-   Pin the package version if your evaluation might be sensitive to version updates.
+- Have a `pyproject.toml` with a `[project]` table so it can be installed via `uv sync`
+- Declare `inspect_ai` as a dependency
+- Define each task with the `@task` decorator from `inspect_ai`
 
+<<<<<<< /tmp/sync_out
    Make sure your module does not import your custom dependencies at module import time when those imports might fail. This can break the way `inspect_ai` discovers tasks via entry points.
 
    To check for this, uninstall your optional dependencies and run another eval with `--limit=0`. If it fails due to one of your imports, you need to defer that import (move it inside the function that uses it).
@@ -61,3 +76,46 @@ To prepare an evaluation for submission as a pull request:
    failures before opening a PR.
 
 7. Fill in the missing sections of the generated README.md which are marked with 'TODO:'.
+=======
+Ask the user whether their upstream repo meets these requirements. Offer to check for them — if they provide the GitHub repository URL, fetch the repo's `pyproject.toml` and task files (e.g. via `WebFetch` on the raw GitHub URLs) to verify the requirements are met. If any requirement is not met, tell the user what needs to be fixed upstream before they can register.
+
+### 2. Gather information and create `register/<eval_name>/eval.yaml`
+
+Skip this step if `register/<eval_name>/eval.yaml` already exists.
+
+Use [`register/example_eval.yaml`](../../../register/example_eval.yaml) as the template — it documents every field. Don't ask the user field-by-field; instead, derive what you can from the upstream repo first, then ask one batched question for what's missing.
+
+**Hints on what to derive from the upstream repo (don't ask):**
+
+- `source.repository_url` — from step 1.
+- `source.repository_commit` — fetch the latest commit SHA on the default branch (must be a 40-char SHA, not a tag or branch).
+- `tasks[].name` and `tasks[].task_path` — locate every `@task`-decorated function in the repo and record the function name and file path.
+- `title` — from the upstream README heading or `pyproject.toml` `[project].name`.
+- `description` — draft from the upstream README; keep to one short paragraph since the generated README links back upstream.
+- `source.maintainers` — defaults to the repo owner; only override if the repo is org-owned and the real maintainers are individuals.
+- `tags` — propose based on the eval's domain (e.g. `Coding`, `games`, `tools`). The upstream repo name is added automatically, so don't include it.
+
+Use [`register/example_eval.yaml`](../../../register/example_eval.yaml) to determine what additional questions are needed.
+
+Show the user the drafted YAML for confirmation before writing the file. Do **not** set `id` — it is auto-injected from the directory name.
+
+### 3. Run validation
+
+```bash
+make check
+```
+
+This validates the `eval.yaml` and auto-generates a `README.md` next to it. The README is fully generated from `eval.yaml` — do not edit it by hand.
+
+Because the generated page defers to upstream for details, make sure the upstream repo's README covers the dataset, scorer, task parameters, and how the eval was validated.
+
+### 4. Create a changelog fragment
+
+```bash
+uv run scriv create
+```
+
+### 5. Open a PR
+
+Use the [PR template](../../.github/PULL_REQUEST_TEMPLATE.md). The reviewer will ping anyone listed under `source.maintainers` for acknowledgement before merging.
+>>>>>>> /tmp/sync_theirs