From 28604a75b399ad160924b0a2d90f1f6defaea67c Mon Sep 17 00:00:00 2001 From: Isabelle Phan Date: Fri, 5 Jun 2026 11:56:35 -0700 Subject: [PATCH 1/3] add validation results and clean up readme --- README.md | 342 ++++++-------------------------------- src/lab_bench_2/README.md | 71 +++++++- 2 files changed, 124 insertions(+), 289 deletions(-) diff --git a/README.md b/README.md index 953e06a..5fb36d0 100644 --- a/README.md +++ b/README.md @@ -1,194 +1,80 @@ -# Inspect Eval Template +# LABBench Evaluations -A template repository for building [Inspect AI](https://inspect.aisi.org.uk/) -evaluations, part of the [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) -registry. Supports multiple evaluations in a single repository. +Inspect AI implementations of the **LABBench** family of biology-research +benchmarks, built for the [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) +registry. -## Important Features +This repository is the home for LABBench evaluations: it currently ships +[`lab_bench_2`](src/lab_bench_2/README.md) (LAB-Bench 2), and is structured to +host future versions and related LABBench evals alongside it, each as its own +self-contained evaluation under `src/`. -This template contains: +## Evaluations -- [Skills](#skills) — Claude Code skills to help produce evaluations, improve - their quality, and speed up velocity. -- [Documentation](#documentation) — guides on best practices and recommended - standards. -- [Examples](#examples) — example evaluations that show how to produce - evaluations in Inspect. +| Eval | Description | Docs | +| ---- | ----------- | ---- | +| `lab_bench_2` | LAB-Bench 2 ([arXiv:2604.09554](https://arxiv.org/abs/2604.09554)) — language models and research agents on life-science tasks spanning literature reasoning, database access, figures, tables, protocols, source quality, sequence analysis, cloning, patents, and clinical trials. | [src/lab_bench_2/README.md](src/lab_bench_2/README.md) | -## Getting started - -1. **Fork this repository** — click "Use this template" (or fork) on GitHub - to create your own copy, then clone it: - - ```bash - git clone https://github.com//.git - cd - ``` - -2. **Install dependencies and run**: - - ```bash - # Install dependencies - uv sync +## Quickstart - # Run an evaluation - uv run inspect eval / --model openai/gpt-4o +This project uses [uv](https://docs.astral.sh/uv/) for dependency management. - # Run tests - uv run pytest +```bash +# Install dependencies +uv sync --dev --all-extras - # Run linters - uv run ruff check && uv run ruff format --check && uv run mypy . - ``` +# Run an evaluation (see the eval's README for tags, modes, and solvers) +uv run inspect eval lab_bench_2 --model openai/gpt-5-nano -## Quickstart +# View logs +uv run inspect view -1. Fork and clone (see above), then `uv sync` -2. Copy an example: `cp -r src/examples/simple_qa src/my_eval` -3. Register it in `pyproject.toml`: +# Run tests +uv run pytest +``` - ```toml - [project.entry-points.inspect_ai] - my_eval = "my_eval" - ``` +If you don't want to pass `--model` each time, set it in a `.env` file: -4. Run it: `uv run inspect eval my_eval/my_task --model openai/gpt-5-nano` +```bash +INSPECT_EVAL_MODEL=anthropic/claude-opus-4-1-20250805 +ANTHROPIC_API_KEY= +``` -See `src/examples/` for complete working examples, including a real -benchmark adaptation (GPQA). +See each evaluation's README for its parameters, dataset details, and scoring — +for `lab_bench_2`, [src/lab_bench_2/README.md](src/lab_bench_2/README.md). -## Structure +## Repository structure Each evaluation lives in its own directory under `src/` and is registered via -entry points in `pyproject.toml`. Tests go in `tests//`. +entry points in `pyproject.toml`; its tests live in `tests//`. ```text src/ - / # Your evaluation - __init__.py # Exports task function(s) for Inspect discovery - .py # Task implementation - eval.yaml # Evaluation metadata - examples/ # Example evaluations (not registered) + lab_bench_2/ # LAB-Bench 2 evaluation (task, dataset, scorers, solvers) + utils/ # Shared helpers tests/ - / # Tests for your evaluation - examples/ # Tests for examples + lab_bench_2/ # Tests for lab_bench_2 ``` -## Adding evaluations - -1. Create a new directory under `src/` (e.g., `src/my_eval/`) — copying one of - `src/examples/` is the easiest start. It needs an `__init__.py` that exports - your `@task` functions and an `eval.yaml` (see `src/examples/gpqa/eval.yaml` - for a fully populated example). -2. Add an entry point in `pyproject.toml`: - - ```toml - [project.entry-points.inspect_ai] - my_eval = "my_eval" - ``` - -3. Add the module name to `[tool.setuptools.packages.find]` include list -4. Add a mypy override for the new module - -## Examples - -The `src/examples/` directory contains three working evaluations demonstrating -common patterns. These are not registered as evaluations — they exist purely -as reference implementations you can copy from. Keep them in place: they're -managed files (kept up-to-date by `sync-template.yml`) and don't ship as -part of your eval's wheel. - -### Simple Q&A (`src/examples/simple_qa/`) - -A straightforward question-answering evaluation using `match()` scoring. -Demonstrates dataset conversion, prompt templates, and few-shot examples. -Similar to evaluations like GPQA. - -### LLM-as-Judge (`src/examples/llm_judge/`) - -An evaluation that uses a language model to grade open-ended responses via -`model_graded_qa()`. Demonstrates custom grading templates and judge model -configuration. Similar to evaluations like Healthbench. - -### GPQA (`src/examples/gpqa/`) - -A real-world benchmark adaptation demonstrating external dataset loading, -multiple-choice scoring, and domain filtering. Adapted from the -[inspect_evals implementation](https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/gpqa). -Includes a fully-populated `eval.yaml` showing all available metadata fields. - -### Agentic (`src/examples/agentic/`) - -An agent-based evaluation where the model uses `bash()` and `python()` tools -in a Docker sandbox via `basic_agent()`. Demonstrates sandbox configuration -and tool-use scoring. Similar to evaluations like GAIA. - -For more examples of production evaluations, see the -[inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) registry, -which contains reference implementations of benchmarks like GPQA, GAIA, -HumanEval, and many others. - -## Skills - -The `.claude/skills/` directory ships Claude Code skills that activate when -you ask Claude to perform the matching task — e.g. "create a new evaluation" -or "review this PR against the template standards". They are managed files, -kept up to date by `sync-template.yml`. The reliable way to invoke one is -*"Please run the /SKILL_NAME skill on EVAL_NAME."* - -### Authoring - -- **create-eval** — implement a new evaluation from an issue, paper, or - benchmark spec, with checkpoints between phases. -- **investigate-dataset** — explore HuggingFace, CSV, or JSON datasets to - understand their structure and data quality. -- **ensure-test-coverage** — review existing tests or create missing ones for - a single evaluation against repo conventions. - -### Reviewing - -- **eval-quality-workflow** — fix or review a single evaluation against the - standards in [EVALUATION_CHECKLIST.md](EVALUATION_CHECKLIST.md). -- **eval-validity-review** — assess whether an evaluation accurately measures - what it claims to measure and has good methodological standards. -- **review-pr-workflow** — review a PR against the agent-checkable standards - in `EVALUATION_CHECKLIST.md` and `BEST_PRACTICES.md`; designed to run in CI. - -### Running and analysing - -- **eval-report-workflow** — produce a README results table by selecting - models, estimating costs, and running evaluations. -- **read-eval-logs** — view and analyse Inspect `.eval` log files via the - Python API and CLI. -- **check-trajectories-workflow** — use Inspect Scout to scan agent - trajectories for failures, formatting issues, reward hacking, and refusals. - -### Submission - -- **prepare-submission-workflow** — finalize an evaluation for PR submission - (dependencies, tests, lint, `eval.yaml`, README). +## Development -## Features +Always run tooling through `uv` (not bare `python`/`pytest`/`pip`): -- **Multiple evaluation support** — add as many evaluations as needed, each - with its own directory, tests, and metadata -- **Automated quality checks** — pre-commit hooks and CI for ruff, mypy, - and pytest -- **Managed file sync** — template updates are synced automatically via - GitHub Actions with three-way merging to preserve your customizations - (see [MANAGED_FILES.md](MANAGED_FILES.md)) +```bash +uv sync # install/sync dependencies +uv run pytest # run tests +uv run ruff check && uv run ruff format --check +uv run mypy src tests # type-check source and tests +make check # full check suite (ruff, mypy, build, autolint, ...) +``` ## CI workflows -The template includes several GitHub Actions workflows that run -automatically. - ### Standard checks (always active) - **Checks** (`checks.yml`) — runs ruff, mypy, POSIX code check, unlisted-eval check, package build, autolint, generated-docs check, - and large-file scan on every push and PR. Each check is individually - enforceable — see [Checks and enforcement](#checks-and-enforcement). + and large-file scan on every push and PR. - **Markdown Lint** (`markdown-lint.yml`) — lints markdown files on PRs - **PR Template Check** (`pr-template-check.yml`) — verifies PR body contains the required checklist @@ -197,135 +83,15 @@ automatically. - **Sync Upstream** (`sync-upstream.yml`) — weekly sync of managed files from [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) -### Repository settings for sync workflows - -The sync workflows create branches and open PRs automatically. Two -things need to be set up in your repository for this to work: - -#### 1. Workflow permissions and PR creation - -1. Go to **Settings > Actions > General > Workflow permissions** -2. Select **Read and write permissions** -3. Check **Allow GitHub Actions to create and approve pull requests** - -#### 2. `SYNC_PAT` secret - -Sync runs often include changes to files under `.github/workflows/` -(template-managed workflow files). GitHub blocks the default -`GITHUB_TOKEN` from creating or updating workflow files, so the sync -needs a separate token with the `Workflows` permission. Without it, -the sync run fails on push with `refusing to allow a GitHub App to -create or update workflow … without 'workflows' permission`. - -To set it up: - -1. At github.com → Settings → Developer settings → Personal access - tokens → Fine-grained tokens, generate a new token. Set - **Resource owner** to your org, **Repository access** to this - repo, and **Repository permissions** to: Contents = Read and - write, Pull requests = Read and write, Workflows = Read and write - (Metadata = Read is added automatically). -2. If your org requires admin approval for fine-grained PATs, - approve it on the org's PAT page. -3. Add the token as a repository secret named `SYNC_PAT` (Settings > - Secrets and variables > Actions > New repository secret). - -See [MANAGED_FILES.md](MANAGED_FILES.md) for details on which files are -synced and how the three-way merge preserves your customizations. - -### Claude-powered workflows - -These workflows use Claude to automate code review and issue resolution. -To enable them, add one of these secrets to your repository settings -(Settings > Secrets and variables > Actions > Secrets): - -- **`ANTHROPIC_API_KEY`** — an Anthropic API key (recommended for most - users). Create one at . -- **`ANTHROPIC_ROLE_ARN`** — an AWS IAM role ARN for Bedrock access via - OIDC (for organisations using AWS Bedrock). - -Without either secret, the workflow is skipped. - -At time of writing, each Claude review costs roughly **$0.50–$2** using -Claude Opus 4.6 ($5/$25 per million tokens input/output). - -- **Claude Code Review** (`claude-review.yaml`) — reviews PRs against - the evaluation standards in EVALUATION_CHECKLIST.md and - BEST_PRACTICES.md. - - Three modes, controlled by repository variables (Settings > Variables - > Actions): - - 1. **Disabled** (default if no secret is set) — workflow skips - entirely. - 2. **On-demand only** (`AUTO_REVIEW_ALL_COMMITS` unset or `false`) — - review fires only when someone comments `/review` on a PR or - dispatches the workflow manually. Recommended starting point. - 3. **Every commit** (`AUTO_REVIEW_ALL_COMMITS=true`) — review fires - on every push to a non-draft PR. Highest cost; use when the team - wants a review on each iteration. - - Optional `CLAUDE_MODEL` repository variable overrides the model - (defaults to `claude-opus-4-6-20250725` for API key users). - -## Checks and enforcement - -This template is calibrated against the -[inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) registry's -quality standards. Those standards are **recommended, not required** in the -template — meeting them is what we suggest if you want a smooth path to -registry submission, but the template doesn't block your work if you don't. - -Each check has an `ENFORCE_` setting in -[`tools/enforcement.config`](tools/enforcement.config). When -`ENFORCE_=true`, the check blocks merge on failure. When `=false`, it -reports as advisory in the PR (visible in the run logs but doesn't prevent -merging). To change enforcement for your repo, edit that file and commit — -the change is git-tracked and reviewable. - -The same file is honoured locally by `make check` (which runs -`tools/run_checks.sh`): advisory failures are reported with a `⚠` and a -"not enforced" note; only enforced failures cause `make check` to exit -non-zero. Environment variables override the file for one-off local runs: - -```bash -ENFORCE_AUTOLINT=true bash tools/run_checks.sh -``` - -All toggles live in [`tools/enforcement.config`](tools/enforcement.config) — edit and commit to change behaviour for your fork. The defaults committed there are: - -**Default-enforced** (blocking unless `ENFORCE_=false`): - -- `ENFORCE_RUFF` — Ruff format + lint -- `ENFORCE_MYPY` — Mypy static types -- `ENFORCE_POSIX_CHECK` — POSIX-only Python idioms (`tools/check_posix_code.py`) -- `ENFORCE_UV_LOCK` — `uv.lock` in sync with `pyproject.toml` -- `ENFORCE_UNLISTED_EVALS` — eval directories must be registered in - `pyproject.toml` entry-points and have an `eval.yaml` -- `ENFORCE_PACKAGE` — package builds and inspects cleanly - -**Default-advisory** (reported but non-blocking unless `ENFORCE_=true`): - -- `ENFORCE_AUTOLINT` — inspect_evals structural standards (eval.yaml schema, - README sections, test patterns, etc.) via `tools/run_autolint.py` -- `ENFORCE_GENERATED_DOCS` — auto-generated README sections committed -- `ENFORCE_MARKDOWN_LINT` — markdown style (`.markdownlint.yaml`) -- `ENFORCE_LARGE_FILES` — no files >10MB -- `ENFORCE_PR_TEMPLATE` — PR body contains the template checklist - -### Quick recipes - -- **Strict mode (registry-ready)**: set every `ENFORCE_*` to `true`. Your - PRs now fail on the same standards inspect_evals applies. -- **Loose mode (default)**: leave everything unset. Correctness blocks - (Ruff/Mypy/POSIX/lock/registration/build); style and structure are - advisory. -- **Per-check**: set just the toggles you want — e.g. - `ENFORCE_AUTOLINT=true` if you want eval-structure rules enforced but - don't care about markdown style. +## Template & managed files -To opt out of a default-enforced check, set its variable to `false`. To opt -into a default-advisory check, set its variable to `true`. +This repository is built on the +[inspect-eval-template](https://github.com/ArcadiaImpact/inspect-eval-template). +Some files (CI workflows, quality tooling, Claude Code skills, shared docs) are +**template-managed** and kept up to date via an automated sync; your evaluation +code, tests, and READMEs are yours. See [MANAGED_FILES.md](MANAGED_FILES.md) for +the split and [AGENTS.md](AGENTS.md) for agent-assisted workflows and the +available skills. ## Documentation diff --git a/src/lab_bench_2/README.md b/src/lab_bench_2/README.md index 95387a8..51ae7b4 100644 --- a/src/lab_bench_2/README.md +++ b/src/lab_bench_2/README.md @@ -276,6 +276,75 @@ than vendoring (copying) the code into this repository. ## Evaluation Report -TODO: A brief summary of results for your evaluation implementation compared against a standard set of existing results. +Model under test: **GPT-5.2** (`gpt-5.2`). The **Paper acc (GPT-5.2)** column is +the reference accuracy from the original LAB-Bench 2 reports +([arXiv:2604.09554](https://arxiv.org/abs/2604.09554)), matched per case to the +nearest paper run configuration. The **Accuracy**, **Stderr**, and **Time** +columns are from our runs. + +| tag | mode | solver | Paper config matched | N | Paper acc | Accuracy | Stderr | Time | +| ----------- | -------- | ------- | -------------------- |------:|-----------:|---------:|-------:|---------:| +| litqa3 | N/A | tools | `@tools,high` | 168† | 0.815 | 0.799 | 0.032 | 9m59s | +| dbqa2 | N/A | bare | `bare` | 86 | 0.070 | 0.058 | 0.025 | 1m51s | +| cloning | inject | tools | `@tools,high` | 14 | 0.286 | 0.286 | 0.125 | 1h41m39s | +| figqa2-img | file | bare | `bare` | 101 | 0.564 | 0.525 | 0.050 | 1m10s | +| seqqa2 | retrieve | bare | `bare` | 200 | 0.095 | 0.115 | 0.023 | 2m31s | +| protocolqa2 | file | agentic | `@tools,high`\* | 125 | 0.416 | 0.390 | 0.044 | 13m57 | + +\* The paper has no client-side-sandbox (`agentic`) config; `@tools,high` +(server-side tools) is the closest augmented baseline. + +† `litqa3`: 9 of the 168 grader calls were blocked by the content filter (see +the refusals note below) and excluded as unscored; the reported accuracy and +stderr are over the 159 scored samples. + +### Notes + +- Every GPT-5.2 run tracks its matched paper reference within stderr (≤ ~1 + stderr). +- Grader refusals are excluded from the results. The LLM judge + (`claude-sonnet-4-5`) is occasionally blocked by Anthropic's content filter on + biosecurity-adjacent questions, which + returns an empty response with `stop_reason=content_filter`. The sample + is marked **unscored** (`verdict_source="refusal"`) rather than counted wrong. +- LLM-judge tags (`litqa3`, `dbqa2`, `figqa2-img`, `protocolqa2`) are graded by + `claude-sonnet-4-5`; deterministic tags (`seqqa2`, `cloning`) use no grader + and are unaffected by refusals. + +### Reproducibility + +- **Model under test:** `openai/gpt-5.2`. **Grader (LLM-judge tags only):** + `anthropic/claude-sonnet-4-5` — the eval default; deterministic tags + (`seqqa2`, `cloning`) use no grader. +- **Eval version:** `1-A`. **Dataset:** `EdisonScientific/labbench2`, split + `train`, pinned to revision `27d12d72af24e3f70db8a99df63e567366cbdb80`. +- **Samples:** each row runs its full set of mode-compatible samples (the `N` + column); `seqqa2` `retrieve` is the retrieve-compatible subset (~200 of 400). +- **Date:** June 2026. +- **Reasoning effort:** `bare` rows use the model default; the `@tools,high` + rows (to fill) use `--reasoning-effort high` to match the paper's augmented + configuration. + +Commands (one per row): + +```bash +# Bare runs +uv run inspect eval lab_bench_2 -T tags=dbqa2 -T solver=bare --model openai/gpt-5.2 +uv run inspect eval lab_bench_2 -T tags=figqa2-img -T mode=file -T solver=bare --model openai/gpt-5.2 +uv run inspect eval lab_bench_2 -T tags=seqqa2 -T mode=retrieve -T solver=bare --model openai/gpt-5.2 + +# @tools,high runs +uv run inspect eval lab_bench_2 -T tags=litqa3 -T solver=tools \ + --model openai/gpt-5.2 --reasoning-effort high +uv run inspect eval lab_bench_2 -T tags=cloning -T mode=inject -T solver=tools \ + --model openai/gpt-5.2 --reasoning-effort high +uv run inspect eval lab_bench_2 -T tags=protocolqa2 -T mode=file -T solver=agentic \ + --model openai/gpt-5.2 --reasoning-effort high +``` ## Changelog + +### [1-A] - 2026-06-05 + +- Add the initial implementation of LABBench2, which supports 15 tags, 3 file processing +modes, 5 scorers and 3 solvers From 6b3eadbea3a144177ad33a4a4d37cceed5df28e8 Mon Sep 17 00:00:00 2001 From: Isabelle Phan Date: Fri, 5 Jun 2026 17:41:52 -0700 Subject: [PATCH 2/3] Update eval.yaml --- src/lab_bench_2/README.md | 2 +- src/lab_bench_2/eval.yaml | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lab_bench_2/README.md b/src/lab_bench_2/README.md index 51ae7b4..3faf7bf 100644 --- a/src/lab_bench_2/README.md +++ b/src/lab_bench_2/README.md @@ -3,7 +3,7 @@ LABBench2 evaluates language models and research agents on life-science tasks spanning literature reasoning, database access, figures, tables, protocols, source quality, sequence analysis, cloning, patents, and clinical trials. -Contributed by [@iphan](https://github.com/iphan) +Contributed by [@iphan](https://github.com/iphan), [@ItsTania](https://github.com/ItsTania), [@lewtun](https://github.com/lewtun), [@Sunishchal](https://github.com/Sunishchal) diff --git a/src/lab_bench_2/eval.yaml b/src/lab_bench_2/eval.yaml index 9d528bb..e5406a5 100644 --- a/src/lab_bench_2/eval.yaml +++ b/src/lab_bench_2/eval.yaml @@ -9,9 +9,12 @@ arxiv: https://arxiv.org/abs/2604.09554 group: Reasoning contributors: - iphan + - ItsTania + - lewtun + - Sunishchal version: "1-A" tasks: - name: lab_bench_2 - dataset_samples: 168 + dataset_samples: 1,912 metadata: requires_internet: true From 2f11b03316ea408929a7059928cf905f3dca6c98 Mon Sep 17 00:00:00 2001 From: Isabelle Phan Date: Mon, 8 Jun 2026 11:17:03 -0700 Subject: [PATCH 3/3] refine readme --- src/lab_bench_2/README.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/src/lab_bench_2/README.md b/src/lab_bench_2/README.md index 3faf7bf..d02126d 100644 --- a/src/lab_bench_2/README.md +++ b/src/lab_bench_2/README.md @@ -286,10 +286,10 @@ columns are from our runs. | ----------- | -------- | ------- | -------------------- |------:|-----------:|---------:|-------:|---------:| | litqa3 | N/A | tools | `@tools,high` | 168† | 0.815 | 0.799 | 0.032 | 9m59s | | dbqa2 | N/A | bare | `bare` | 86 | 0.070 | 0.058 | 0.025 | 1m51s | -| cloning | inject | tools | `@tools,high` | 14 | 0.286 | 0.286 | 0.125 | 1h41m39s | +| cloning | inject | tools | `@tools,high` | 14§ | 0.286 | 0.286 | 0.125 | 1h41m39s | | figqa2-img | file | bare | `bare` | 101 | 0.564 | 0.525 | 0.050 | 1m10s | | seqqa2 | retrieve | bare | `bare` | 200 | 0.095 | 0.115 | 0.023 | 2m31s | -| protocolqa2 | file | agentic | `@tools,high`\* | 125 | 0.416 | 0.390 | 0.044 | 13m57 | +| protocolqa2 | file | agentic | `@tools,high`\* | 125‡ | 0.416 | 0.390 | 0.044 | 13m57 | \* The paper has no client-side-sandbox (`agentic`) config; `@tools,high` (server-side tools) is the closest augmented baseline. @@ -298,18 +298,28 @@ columns are from our runs. the refusals note below) and excluded as unscored; the reported accuracy and stderr are over the 159 scored samples. +‡ `protocolqa2`: 2 of the 125 grader calls were blocked by the content filter +(see the refusals note below) and excluded as unscored; the reported +accuracy and stderr are over the 123 scored samples. + +§ `cloning`: its low score is partly a refusal artifact, not pure capability. +The model under test declined ~40% of this small 14-question set on biosecurity +grounds (refusing to give actionable cloning protocols), plus 1 OpenAI API-level +content-filter block. + ### Notes - Every GPT-5.2 run tracks its matched paper reference within stderr (≤ ~1 stderr). - Grader refusals are excluded from the results. The LLM judge (`claude-sonnet-4-5`) is occasionally blocked by Anthropic's content filter on - biosecurity-adjacent questions, which - returns an empty response with `stop_reason=content_filter`. The sample - is marked **unscored** (`verdict_source="refusal"`) rather than counted wrong. + biosecurity-adjacent questions, which returns an empty response with + `stop_reason=content_filter`. The sample is retried up to 3× and, if still + blocked, marked **unscored** (`verdict_source="refusal"`) rather than counted + wrong. - LLM-judge tags (`litqa3`, `dbqa2`, `figqa2-img`, `protocolqa2`) are graded by - `claude-sonnet-4-5`; deterministic tags (`seqqa2`, `cloning`) use no grader - and are unaffected by refusals. + `claude-sonnet-4-5`; deterministic tags (`seqqa2`, `cloning`) use no grader, + so they are unaffected by *grader* refusals. ### Reproducibility