From 92177940020da0f8e7ff55538cce950c801627e1 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Thu, 16 Jul 2026 14:17:01 +1000 Subject: [PATCH 1/9] Recommend model-roles in custom-scorers docs --- docs/custom-scorers.qmd | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/custom-scorers.qmd b/docs/custom-scorers.qmd index f9abc8c13c..604ad0050f 100644 --- a/docs/custom-scorers.qmd +++ b/docs/custom-scorers.qmd @@ -149,7 +149,13 @@ Next, we'll take a look at the source code for a couple of the built in scorers ## Models in Scorers -You'll often want to use models in the implementation of scorers. Use the `get_model()` function to get either the currently evaluated model or another model interface. For example: +You'll often want to use models in the implementation of scorers. The best way of doing this is to specify the grader model using [model roles](models.qmd#model-roles), allowing them to be chosen at runtime. By default, model-graded scorers look for the `"grader"` model role: + +```python +grader_model = get_model(role="grader") +``` + +You can also use the `get_model()` function to get either the currently evaluated model or another model interface. For example: ``` python # use the model being evaluated for grading From 7ac98d1862ffa2a0497a23ab116bb63d5b36f432 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Thu, 16 Jul 2026 16:07:45 +1000 Subject: [PATCH 2/9] Scoring policy docs v1 --- docs/_quarto.yml | 2 + docs/scoring-policy.qmd | 166 ++++++++++++++++++++++++++++++++++++++++ docs/scoring.qmd | 1 + 3 files changed, 169 insertions(+) create mode 100644 docs/scoring-policy.qmd diff --git a/docs/_quarto.yml b/docs/_quarto.yml index 9f4b889b7b..b5e98fa35f 100644 --- a/docs/_quarto.yml +++ b/docs/_quarto.yml @@ -81,6 +81,8 @@ inspect-docs: contents: - href: standard-scorers.qmd - href: custom-scorers.qmd + - text: Scoring Policy + href: scoring-policy.qmd - text: Model Grading href: model-graded.qmd - text: Scoring Metrics diff --git a/docs/scoring-policy.qmd b/docs/scoring-policy.qmd new file mode 100644 index 0000000000..a63469e44b --- /dev/null +++ b/docs/scoring-policy.qmd @@ -0,0 +1,166 @@ +--- +title: Scoring Policy +llms-description: How to route a sample-ending failure in a scorer to the right outcome (return a Score, raise, or Score.unscored()) so that metrics reflect the model under test rather than the run machinery. +--- + +## Overview + +The point of a scorer is to assess the performance of a model under test (MUT) against the given task. +When the MUT's response is assessed as correct, the outcome is straightforward: return `Score(value=CORRECT)`. +However there are multiple reasons that the response passed to a scorer can fail to be correct, and distinguishing between the different cases has significant effects downstream on the MUT's metrics. + +The three main outcomes are: + +- return a `Score` + This is a verdict on the model, which counts in the denominator of the resultant metrics. +- raise an error + This indicates a machinery malfunction, and will be retried if [`retry_on_error`](handling-errors.qmd#sample-retries) is set. It is counted against the `fail_on_error` [failure threshold](handling-errors.qmd#failure-threshold) and will only be scored if [`score_on_error`](handling-errors.qmd#scoring-errored-samples) is set. +- return `Score.unscored()` + This indicates that the scorer was unable to render a verdict, typically because a grader model's response was a refusal or not in the correct format. The sample will be excluded from the resultant metrics. + +The organising question is "whose behaviour does the outcome reflect: the model under test, or the run machinery?" Note that a grader model is part of the *measurement instrument*, not the subject of measurement, so a grader failure reflects the machinery, not the MUT. + +Choosing the wrong outcome distorts the headline metric. Scoring an infrastructure failure `0.0` deflates accuracy and opts the sample out of retries; raising on a genuine model failure inflates accuracy by shrinking the denominator. The rest of this page matches each kind of failure to the right outcome. + +## The three outcomes + +Using `accuracy` as an example, an eval run with one correct answer, one incorrect answer, one sample that errored, and one sample that triggered a content filter on the grader model will report an accuracy of **0.5**: + +| Sample | Scorer outcome | Value | Counts in denominator? | +|---|---|---|---| +| 1 | `Score(CORRECT)` | `1.0` | yes | +| 2 | `Score(INCORRECT)` | `0.0` | yes | +| 3 | `raise` | n/a | no (errored) | +| 4 | `Score.unscored()` | `NaN` | no (unscored) | + +: {tbl-colwidths=[15,35,15,35]} + +Only samples 1 and 2 are scored, so `accuracy = 1 / 2 = 0.5`, with `scored_samples = 2` and `unscored_samples = 1`; the errored sample is recorded separately via `sample.error`. Contrast the two ways of getting this wrong: + +- Had samples 3 and 4 been scored `0.0`, accuracy would read `0.25`, penalising the model for an infrastructure error and a grader failure it had no control over. +- Had they simply been dropped with no accounting, the distinction would be invisible, and a run that errored on nine of ten samples could report `100%` over the one that happened to succeed. + +The [`score_on_error`](handling-errors.qmd#scoring-errored-samples) run option changes this picture for errors raised during *solving*: rather than dropping the sample, it runs the scorer on whatever partial state was reached and counts the result. If sample 3 had errored mid-solve and its partial state scored `INCORRECT`, it would enter the denominator and accuracy would read `1 / 3`. It does **not**, however, rescue a `raise` from the scorer itself (the scorer is the thing that failed), so it is not a substitute for returning a `Score` when you want a sample counted. + +A quick way to decide which outcome applies is to ask **what the correct remedial action is**: + +| Correct remedy | Outcome | +|---|---| +| Retry the sample and it will probably work | `raise` (transient) | +| Fix your code, environment, or dataset and re-run | `raise` (defect; must be loud even though a retry won't fix it) | +| Nothing, because due diligence is done and the instrument can't say | `Score.unscored()` | +| The model did this to itself | return a `Score` | + +: {tbl-colwidths=[60,40]} + +### Verdicts on the model + +A failure that is a genuine statement about the model's behaviour is a score, and it stays in the denominator. This includes output that doesn't parse: failing to follow the requested output format is an instruction-following failure, which is a legitimate capability miss. + +``` python +return Score( + value=INCORRECT, + explanation=f"Expected JSON, could not parse: {state.output.completion}", + metadata={ + "reason": "invalid_response_format", + }, +) +``` + +Inspect provides two labels for model-fault outcomes. Both map to `0.0` by default and both stay in the denominator, so the choice is about what the label *means* and what downstream tooling can partition on, not about the number: + +- **`INCORRECT`**: the model produced an answer that is wrong or unusable (including a format or instruction-following failure). +- **`NOANSWER`**: the model produced *nothing to grade*, such as an empty completion or a detected refusal. + +::: {.callout-note appearance="simple"} +If you intend to split on `NOANSWER` in a custom metric (for example to report an answer rate), be aware that the default `mean` epoch reducer collapses `NOANSWER` to `0.0` *before* metrics run, erasing the label. Use a value-preserving epoch reducer (see [Reducing Epochs](metrics.qmd)) if you need the distinction to survive. +::: + +Never let a model-fault outcome raise; raising removes the sample from the denominator and inflates accuracy. + +### Malfunctions + +A failure that tells you nothing about the model, because the run machinery went wrong, should `raise`. There are two flavours, and raising is correct for both: + +- **Transient**: an API error, a network blip, a container that died. A retry may succeed, and `retry_on_error` will attempt it. +- **A defect**: a bug in your solver or scorer, a broken environment, a missing or unrunnable ground-truth artifact. A retry won't help, but raising surfaces the problem loudly against `fail_on_error` instead of silently biasing the result. + +Raising routes the sample to Inspect's error-handling machinery rather than baking a run-time policy decision into the scorer at authoring time. See [Handling Errors](handling-errors.qmd) for `retry_on_error`, `fail_on_error`, and `score_on_error`. + +Give the raise a message that lets a human reading the errored sample attribute it after the fact: what failed, why it isn't a verdict on the model, and a bounded tail of the relevant output or traceback. + +::: {.callout-note appearance="simple"} +Deterministic machinery should not get the unscored escape hatch. Sandbox execution, file I/O, your own runner, JSON your code wrote, and the grader *API call* itself either work or are broken; hence a failure is always transient or a defect, so it should `raise`. `Score.unscored()` is reserved for instruments that are inherently *stochastic* (see below). Unscoring a deterministic failure hides a bug. +::: + +### Unscoreable samples + +Some failures tell you nothing about the model, yet nothing malfunctioned either: the run worked, but the measurement instrument could not render a verdict on this sample after reasonable due diligence. The canonical case is a probabilistic instrument such as a grader model, which has an inherent, irreducible per-sample failure rate. Sometimes it emits a parseable verdict; sometimes it refuses or produces an unparseable one, and no retry or code fix will change that for this sample. + +For these, return `Score.unscored()`. This produces a `Score` whose `value` is `NaN`, which Inspect excludes from metric computation and counts as `unscored_samples` on the `EvalScore`: + +``` python +return Score.unscored( + answer=state.output.completion, + explanation="Grader did not return a parseable verdict after 3 retries.", + metadata={"reason": "grader_parse_failure"}, +) +``` + +Always report the unscored rate. A non-trivial rate can invalidate the run regardless of how the individual samples are handled, because it is a fact about your measurement instrument, not about the model. + +### Attribution: whose output failed? + +The three outcomes above turn on a single question: *whose output* failed to parse or comply. + +- **The model's output** failing to parse is a verdict on the model → `INCORRECT` (detect this before invoking a grader where possible). +- **The instrument's output** failing (a grader model's verdict, or a harness the scorer invokes) is never a verdict on the model → apply the malfunction / unscoreable decision above. + +Beware misidentifying the author of a payload. A JSON file written by *your own runner* is instrument output, even though the pipeline started from model output; a JSON blob the *model itself* emitted is model output. The author determines the bucket, not where the data happens to sit. + +Where a reliable signal exists, classify from it rather than collapsing every failure into one bucket. For example, bracketing test output with sentinel markers lets a scorer tell "the tests ran and failed" (a verdict → score) apart from "the test process never started" (infrastructure → raise). + +## Model-graded scorers + +When a model grades the model under test, the grader is the *measurement instrument*, and a failure to parse the **grader's** verdict is an instrument failure, not a verdict on the model. Charging it to the model as `0.0` measures the grader's competence and biases results toward whichever grader you chose. Handle grader-verdict failures in this order: + +1. **Constrain the grader's output structurally** rather than regex-parsing free text; prefer a tool call with a schema-validated payload. This removes most format-failure ambiguity at the source. +2. **Retry the grader only**, bounded, with the model's output held fixed. Note that `retry_on_error` re-runs the *whole sample* including model generation, which re-rolls the answer and can inflate scores, so grader-only retry is a separate loop inside the scorer. +3. **Use a capable, pinned grader.** A grader too weak to follow its own output format is an eval-validity problem to flag, not something to charge to the model. +4. **Only then** return `Score.unscored()` for the residual, and report the grader-failure rate as a first-class number. + +The `raise` vs `unscored()` line still applies within grading: the grader *API call* raising or timing out is a malfunction → `raise` (the transport failed, not the verdict), whereas a well-formed-but-unparseable *verdict* after bounded retry is an instrument miss → `unscored()`. + +## Recording the reason + +Whatever the outcome, set `explanation` to a human-readable message, and record a machine-readable reason in `Score.metadata` so failures are filterable in the log viewer and in analysis dataframes: + +``` python +return Score( + value=INCORRECT, + explanation="Model response was not valid JSON.", + metadata={"reason": "invalid_response_format"}, +) +``` + +A consistent vocabulary makes failures groupable across evals. For example, use `invalid_response_format` and `refusal` for model-fault outcomes, and `grader_parse_failure`, `grader_refusal`, or `grader_no_tool_call` for instrument failures. Keeping the actor in the value (a `grader_` prefix) lets a single query separate instrument failures from model failures. + +## Recommended run settings + +The scorer decides *what a sample's outcome means*; the run configuration decides *how much failure a run tolerates*. Keep the two separate: return the right `Score` / `raise` / `unscored()` from the scorer, and set the tolerance at the run level. + +- Set `fail_on_error` to a proportion that reflects how many machinery failures would invalidate your run. Tolerating a small fraction such as `0.1` is a reasonable default for real runs; the default (`True`) aborts the run on the first error, which is useful during development but brittle for long runs. Setting `fail_on_error` to `False` can produce *highly misleading results* if a significant number of samples fail. +- Enable `retry_on_error` to recover from transient malfunctions. + +``` bash +inspect eval task.py --fail-on-error=0.1 --retry-on-error=3 +``` + +Because errored and unscored samples both leave the denominator, a run can report a clean, high accuracy computed over only a fraction of its samples. When reporting or reviewing results, check the coverage (how many samples were actually scored) alongside the headline metric. The counts are recorded on each `EvalScore` as `scored_samples` and `unscored_samples`; errored samples are visible per-sample via `sample.error`. + +## See also + +- [Custom Scorers](custom-scorers.qmd): writing scorers and the `Score`, `Value`, and `Target` types. +- [Model Grading](model-graded.qmd): built-in model-graded scorers and how to customise them. +- [Scoring Metrics](metrics.qmd): metrics, epoch reducers, and custom metrics. +- [Handling Errors](handling-errors.qmd): `retry_on_error`, `fail_on_error`, and `score_on_error` at the run level. diff --git a/docs/scoring.qmd b/docs/scoring.qmd index dbff57da68..37eb28be52 100644 --- a/docs/scoring.qmd +++ b/docs/scoring.qmd @@ -9,6 +9,7 @@ Scoring turns the raw `output` a model produces for each sample into a `Score`, |---|---| | [Standard Scorers](standard-scorers.qmd) | The built-in scorers (text matching, multiple choice, math, model grading, perplexity) and how to choose among them. | | [Custom Scorers](custom-scorers.qmd) | Write your own scorers using the `Score`, `Value`, and `Target` types, including scorers that call models or inspect a sandbox. | +| [Scoring Policy](scoring-policy.qmd) | Route a sample-ending failure to the right outcome (score it, `raise`, or `Score.unscored()`) so metrics reflect the model, not the run machinery. | | [Model Grading](model-graded.qmd) | Use another model to grade open-ended answers; customise templates, instructions, grader models, and chat history. | | [Scoring Metrics](metrics.qmd) | Built-in metrics, grouping, clustered standard errors, custom metrics, and reducing epochs. | | [Multiple Scorers](multiple-scorers.qmd) | Use several scorers together, emit multiple scores from one scorer, and reduce multiple scores into one. | From ddd77ca928efde0ee4c705d21607ec2cedf92254 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Thu, 16 Jul 2026 17:10:10 +1000 Subject: [PATCH 3/9] Document pattern vs match outcomes for non-matching output match/includes/exact score a non-match INCORRECT; pattern/answer score NOANSWER when the pattern does not match at all. Note both map to 0.0 and cross-reference Scoring Policy for the wrong-answer vs no-answer distinction. Co-Authored-By: Claude Opus 4.8 --- docs/_builtin-scorers.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/_builtin-scorers.md b/docs/_builtin-scorers.md index 2d06b0b35f..cebb6f1972 100644 --- a/docs/_builtin-scorers.md +++ b/docs/_builtin-scorers.md @@ -40,6 +40,18 @@ Inspect includes both text matching scorers as well as model graded scorers. Bel ::: +## When the output doesn't match + +These scorers use the `CORRECT` and `INCORRECT` constants, which the default metrics convert to `1.0` and `0.0` (see [Custom Scorers](custom-scorers.qmd#score) for the `Value` types and `value_to_float()`). They differ in how they treat output that does not match the target: + +`includes()`, `match()`, and `exact()` +: Score a non-matching output `INCORRECT`, so the sample stays in the denominator as a `0.0`. + +`pattern()` (and `answer()`, which builds on it) +: Score `INCORRECT` when the pattern matches but the captured value is wrong, and `NOANSWER` when the pattern does not match at all. `NOANSWER` also converts to `0.0` by default, so it does not change accuracy; it is a distinct label for "no answer was found in the expected form" rather than "the model answered incorrectly". + +Whether a failure to produce output in the expected format should count as a wrong answer (`INCORRECT`) or as no answer (`NOANSWER`) is a judgement call, since failing to follow the requested format is itself an instruction-following failure. See [Scoring Policy](scoring-policy.qmd#verdicts-on-the-model) for the distinction and how it interacts with metrics. + ## Metrics Each scorer provides one or more built-in metrics. Most report `accuracy` and `stderr`; `exact()` and `f1()` report `mean` and `stderr`; and the perplexity scorers report `perplexity_per_token` and `perplexity_per_seq`. You can override these by passing your own `metrics` to the `Task`: From c4218b872befcf623c51f5f3e404bb53ab5ee4e8 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Thu, 16 Jul 2026 17:10:10 +1000 Subject: [PATCH 4/9] Score grader parse failures as unscored in custom-scorers example The model_graded_qa example returned INCORRECT when the grader's verdict could not be parsed, charging an instrument failure to the model. Return Score.unscored() instead, and link Scoring Policy for when to score, raise, or unscore. Co-Authored-By: Claude Opus 4.8 --- docs/custom-scorers.qmd | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/custom-scorers.qmd b/docs/custom-scorers.qmd index 604ad0050f..ff76cd44e5 100644 --- a/docs/custom-scorers.qmd +++ b/docs/custom-scorers.qmd @@ -131,6 +131,8 @@ return Score.unscored( Unscored samples are skipped by aggregate metrics and epoch reducers and are counted toward `EvalScore.unscored_samples` rather than included as zeros. This works for scalar, dict-valued, and list-valued scorers. +For guidance on *when* to return `Score.unscored()` versus a score or an error, see [Scoring Policy](scoring-policy.qmd). + ## Score Value `Value` is union over the main scalar types as well as a `list` or `dict` of the same types: @@ -252,15 +254,21 @@ def model_graded_qa( explanation=result.completion, ) else: - return Score( - value=INCORRECT, + # The grader failed to produce a parseable verdict. That is a + # failure of the measurement instrument, not a wrong answer from the + # model, so record it as unscored (excluded from metrics) rather than + # charging the model INCORRECT. See Scoring Policy for the rationale. + return Score.unscored( explanation="Grade not found in model output: " + f"{result.completion}", + metadata={"reason": "grader_parse_failure"}, ) return score ``` +The grade-parse-failure branch returns `Score.unscored()` rather than `INCORRECT`: a grader that cannot render a verdict is an instrument failure, not a verdict on the model. See [Scoring Policy](scoring-policy.qmd#model-graded-scorers) for when to score, raise, or return `Score.unscored()`. + Note that the call to `model_grader.generate()` is done with `await`. This is critical to ensure that the scorer participates correctly in the scheduling of generation work. Note also we use the `input_text` property of the `TaskState` to access a string version of the original user input to substitute it into the grading template. Using the `input_text` has two benefits: (1) It is guaranteed to cover the original input from the dataset (rather than a transformed prompt in `messages`); and (2) It normalises the input to a string (as it could have been a message list). From b7284770baeb346c9725884d884947160bc7e909 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Thu, 16 Jul 2026 17:10:10 +1000 Subject: [PATCH 5/9] Link scoring policy for unparseable grader verdicts Co-Authored-By: Claude Opus 4.8 --- docs/model-graded.qmd | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/model-graded.qmd b/docs/model-graded.qmd index 9c81a2c10a..e385b65833 100644 --- a/docs/model-graded.qmd +++ b/docs/model-graded.qmd @@ -169,6 +169,8 @@ Grade extraction binds to the last grade. The default `grade_pattern` (`(?is).*( Structural delimiters are neutralized. The default templates wrap content in `[BEGIN DATA]` / `[END DATA]` markers. Before formatting the prompt, Inspect rewrites any `[BEGIN DATA]` / `[END DATA]` markers that appear in the model-controlled `answer`, the `question`, the `criterion`, and any `metadata` values (for example to `[END-DATA]`) so a model cannot inject a fake delimiter and smuggle grading instructions into the prompt. The `instructions` you provide are author-controlled and left untouched. +When the grader's verdict still cannot be parsed after these precautions, that is a failure of the measurement instrument rather than a verdict on the model. See [Scoring Policy](scoring-policy.qmd#model-graded-scorers) for how to handle it (bounded grader-only retry, then `Score.unscored()`), and why it should not be scored `INCORRECT`. + ## Reproducible Grading By default the grader model runs with the provider's default generation settings. In practice this often means a non-zero temperature (e.g. 1.0) and no fixed seed, so grades for borderline answers can change from run to run and across epochs. From 3304da9d7976a3bca636fe484b0863fef39f8f4b Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Thu, 16 Jul 2026 17:10:10 +1000 Subject: [PATCH 6/9] Link scoring policy from handling-errors Pairs the run-level error options with the scorer authoring decision. Co-Authored-By: Claude Opus 4.8 --- docs/handling-errors.qmd | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/handling-errors.qmd b/docs/handling-errors.qmd index 94eb61ca89..202cd0f61c 100644 --- a/docs/handling-errors.qmd +++ b/docs/handling-errors.qmd @@ -92,6 +92,8 @@ Consequently, when enabling `retry_on_error` you should do some post-hoc analysi Some evaluations are designed so that an error during the agent run is itself a meaningful (often failing) outcome — for example, a tool-using agent that crashes after producing partial state, or a benchmark where "the model errored" should count as a scoreable result rather than as missing data. +This page covers the run-level options for tolerating errors. For the complementary authoring decision, namely what a scorer should return for a given failure (a `Score`, a `raise`, or `Score.unscored()`), see [Scoring Policy](scoring-policy.qmd). + The `score_on_error` option causes errored samples to be scored anyway (using whatever `TaskState` was reached before the error), and prevents `fail_on_error` from crashing the eval mid-run: ``` bash From deff1721befc074952d1fe072a520c625fb13ad4 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Thu, 16 Jul 2026 17:10:10 +1000 Subject: [PATCH 7/9] Restructure scoring policy to lead with concept, and cross-reference siblings Lead with what scorers do and why before the prescriptive "if you're writing a scorer" guidance, and cross-reference custom-scorers, model-graded, and metrics rather than restating them. Co-Authored-By: Claude Opus 4.8 --- docs/scoring-policy.qmd | 47 ++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 24 deletions(-) diff --git a/docs/scoring-policy.qmd b/docs/scoring-policy.qmd index a63469e44b..fa8b99484b 100644 --- a/docs/scoring-policy.qmd +++ b/docs/scoring-policy.qmd @@ -5,33 +5,28 @@ llms-description: How to route a sample-ending failure in a scorer to the right ## Overview -The point of a scorer is to assess the performance of a model under test (MUT) against the given task. -When the MUT's response is assessed as correct, the outcome is straightforward: return `Score(value=CORRECT)`. -However there are multiple reasons that the response passed to a scorer can fail to be correct, and distinguishing between the different cases has significant effects downstream on the MUT's metrics. +A scorer's job is to turn what a model did on a sample into a verdict about the model's capability on the task, which Inspect then aggregates into the metrics that summarise the evaluation. In the clean case this is simple: the model under test (MUT) produced an answer, the scorer judges it correct or incorrect, and that verdict counts. -The three main outcomes are: +The complication is that not everything which ends a sample is a statement about the model. The model can genuinely fail, with a wrong answer or output that doesn't follow the requested format. But the *run machinery* can also break (an API error, a dead container, a bug in your own code), and where a grader model or other measurement instrument is involved, the *instrument itself* can fail to produce a reading. These are different in kind, and only the first is a verdict on the model. -- return a `Score` - This is a verdict on the model, which counts in the denominator of the resultant metrics. -- raise an error - This indicates a machinery malfunction, and will be retried if [`retry_on_error`](handling-errors.qmd#sample-retries) is set. It is counted against the `fail_on_error` [failure threshold](handling-errors.qmd#failure-threshold) and will only be scored if [`score_on_error`](handling-errors.qmd#scoring-errored-samples) is set. -- return `Score.unscored()` - This indicates that the scorer was unable to render a verdict, typically because a grader model's response was a refusal or not in the correct format. The sample will be excluded from the resultant metrics. +Distinguishing them matters because each affects the metric denominator differently, so conflating them distorts the headline number (see the [worked example](#the-three-outcomes) below). The organising question throughout is: **whose behaviour does the outcome reflect, the model under test or the run machinery?** A grader model counts as machinery here, because it is part of the measurement instrument, not the subject of measurement. -The organising question is "whose behaviour does the outcome reflect: the model under test, or the run machinery?" Note that a grader model is part of the *measurement instrument*, not the subject of measurement, so a grader failure reflects the machinery, not the MUT. +## The three outcomes -Choosing the wrong outcome distorts the headline metric. Scoring an infrastructure failure `0.0` deflates accuracy and opts the sample out of retries; raising on a genuine model failure inflates accuracy by shrinking the denominator. The rest of this page matches each kind of failure to the right outcome. +A scorer has three ways to record what happened on a sample, and each routes to different downstream handling: -## The three outcomes +- **Return a `Score`** is a verdict on the model. It counts in the denominator of the resultant metrics. +- **Raise an error** signals a machinery malfunction. The sample is marked errored: excluded from metrics, retried if [`retry_on_error`](handling-errors.qmd#sample-retries) is set, counted against the [`fail_on_error`](handling-errors.qmd#failure-threshold) threshold, and scored only if [`score_on_error`](handling-errors.qmd#scoring-errored-samples) is set. +- **Return `Score.unscored()`** records that the scorer could not render a verdict even though nothing malfunctioned. The sample is excluded from the metrics but counted separately. -Using `accuracy` as an example, an eval run with one correct answer, one incorrect answer, one sample that errored, and one sample that triggered a content filter on the grader model will report an accuracy of **0.5**: +The choice has real consequences for the reported numbers. Using `accuracy` as an example, an eval run with one correct answer, one incorrect answer, one sample that errored, and one sample that triggered a content filter on the grader model will report an accuracy of **0.5**: -| Sample | Scorer outcome | Value | Counts in denominator? | -|---|---|---|---| -| 1 | `Score(CORRECT)` | `1.0` | yes | -| 2 | `Score(INCORRECT)` | `0.0` | yes | -| 3 | `raise` | n/a | no (errored) | -| 4 | `Score.unscored()` | `NaN` | no (unscored) | +| Sample | Scorer outcome | Value | Counts in denominator? | +|--------|--------------------|-------|------------------------| +| 1 | `Score(CORRECT)` | `1.0` | yes | +| 2 | `Score(INCORRECT)` | `0.0` | yes | +| 3 | `raise` | n/a | no (errored) | +| 4 | `Score.unscored()` | `NaN` | no (unscored) | : {tbl-colwidths=[15,35,15,35]} @@ -42,7 +37,9 @@ Only samples 1 and 2 are scored, so `accuracy = 1 / 2 = 0.5`, with `scored_sampl The [`score_on_error`](handling-errors.qmd#scoring-errored-samples) run option changes this picture for errors raised during *solving*: rather than dropping the sample, it runs the scorer on whatever partial state was reached and counts the result. If sample 3 had errored mid-solve and its partial state scored `INCORRECT`, it would enter the denominator and accuracy would read `1 / 3`. It does **not**, however, rescue a `raise` from the scorer itself (the scorer is the thing that failed), so it is not a substitute for returning a `Score` when you want a sample counted. -A quick way to decide which outcome applies is to ask **what the correct remedial action is**: +## Choosing an outcome + +If you are writing a scorer, the practical question is which of the three outcomes to return for a given failure. A quick test is to ask **what the correct remedial action would be**: | Correct remedy | Outcome | |---|---| @@ -53,6 +50,8 @@ A quick way to decide which outcome applies is to ask **what the correct remedia : {tbl-colwidths=[60,40]} +The sections below work through each outcome, and how to tell which applies when the cause is ambiguous. + ### Verdicts on the model A failure that is a genuine statement about the model's behaviour is a score, and it stays in the denominator. This includes output that doesn't parse: failing to follow the requested output format is an instruction-following failure, which is a legitimate capability miss. @@ -73,7 +72,7 @@ Inspect provides two labels for model-fault outcomes. Both map to `0.0` by defau - **`NOANSWER`**: the model produced *nothing to grade*, such as an empty completion or a detected refusal. ::: {.callout-note appearance="simple"} -If you intend to split on `NOANSWER` in a custom metric (for example to report an answer rate), be aware that the default `mean` epoch reducer collapses `NOANSWER` to `0.0` *before* metrics run, erasing the label. Use a value-preserving epoch reducer (see [Reducing Epochs](metrics.qmd)) if you need the distinction to survive. +If you intend to split on `NOANSWER` in a custom metric (for example to report an answer rate), be aware that the default `mean` epoch reducer collapses `NOANSWER` to `0.0` *before* metrics run, erasing the label. Use a value-preserving epoch reducer (see [Reducing Epochs](metrics.qmd#reducing-epochs)) if you need the distinction to survive. ::: Never let a model-fault outcome raise; raising removes the sample from the denominator and inflates accuracy. @@ -97,7 +96,7 @@ Deterministic machinery should not get the unscored escape hatch. Sandbox execut Some failures tell you nothing about the model, yet nothing malfunctioned either: the run worked, but the measurement instrument could not render a verdict on this sample after reasonable due diligence. The canonical case is a probabilistic instrument such as a grader model, which has an inherent, irreducible per-sample failure rate. Sometimes it emits a parseable verdict; sometimes it refuses or produces an unparseable one, and no retry or code fix will change that for this sample. -For these, return `Score.unscored()`. This produces a `Score` whose `value` is `NaN`, which Inspect excludes from metric computation and counts as `unscored_samples` on the `EvalScore`: +For these, return `Score.unscored()`. This produces a `Score` whose `value` is `NaN`, which Inspect excludes from metric computation and counts as `unscored_samples` on the `EvalScore` (see [Unscored Samples](custom-scorers.qmd#unscored-samples) for the API details): ``` python return Score.unscored( @@ -126,7 +125,7 @@ When a model grades the model under test, the grader is the *measurement instrum 1. **Constrain the grader's output structurally** rather than regex-parsing free text; prefer a tool call with a schema-validated payload. This removes most format-failure ambiguity at the source. 2. **Retry the grader only**, bounded, with the model's output held fixed. Note that `retry_on_error` re-runs the *whole sample* including model generation, which re-rolls the answer and can inflate scores, so grader-only retry is a separate loop inside the scorer. -3. **Use a capable, pinned grader.** A grader too weak to follow its own output format is an eval-validity problem to flag, not something to charge to the model. +3. **Use a capable, pinned grader.** A grader too weak to follow its own output format is an eval-validity problem to flag, not something to charge to the model. Pin its generation settings so grades are reproducible (see [Reproducible Grading](model-graded.qmd#reproducible-grading)). 4. **Only then** return `Score.unscored()` for the residual, and report the grader-failure rate as a first-class number. The `raise` vs `unscored()` line still applies within grading: the grader *API call* raising or timing out is a malfunction → `raise` (the transport failed, not the verdict), whereas a well-formed-but-unparseable *verdict* after bounded retry is an instrument miss → `unscored()`. From b3b8b702c20bfd0752892de5c7b3f468f9405893 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Thu, 16 Jul 2026 17:43:50 +1000 Subject: [PATCH 8/9] Include answer in examples --- docs/scoring-policy.qmd | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/scoring-policy.qmd b/docs/scoring-policy.qmd index fa8b99484b..869b089e6e 100644 --- a/docs/scoring-policy.qmd +++ b/docs/scoring-policy.qmd @@ -102,7 +102,10 @@ For these, return `Score.unscored()`. This produces a `Score` whose `value` is ` return Score.unscored( answer=state.output.completion, explanation="Grader did not return a parseable verdict after 3 retries.", - metadata={"reason": "grader_parse_failure"}, + metadata={ + "reason": "grader_parse_failure", + "grader_payload": grader_output.message, + }, ) ``` @@ -137,6 +140,7 @@ Whatever the outcome, set `explanation` to a human-readable message, and record ``` python return Score( value=INCORRECT, + answer=state.output.completion, explanation="Model response was not valid JSON.", metadata={"reason": "invalid_response_format"}, ) From 8ff2ed124e1f1196fe29fc739fb1de2194b66f5a Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 17 Jul 2026 17:38:15 +1000 Subject: [PATCH 9/9] Fix typos --- docs/scoring-policy.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/scoring-policy.qmd b/docs/scoring-policy.qmd index 869b089e6e..cfbf7cb2a0 100644 --- a/docs/scoring-policy.qmd +++ b/docs/scoring-policy.qmd @@ -94,7 +94,7 @@ Deterministic machinery should not get the unscored escape hatch. Sandbox execut ### Unscoreable samples -Some failures tell you nothing about the model, yet nothing malfunctioned either: the run worked, but the measurement instrument could not render a verdict on this sample after reasonable due diligence. The canonical case is a probabilistic instrument such as a grader model, which has an inherent, irreducible per-sample failure rate. Sometimes it emits a parseable verdict; sometimes it refuses or produces an unparseable one, and no retry or code fix will change that for this sample. +Some failures tell you nothing about the model, yet nothing malfunctioned either: the run worked, but the measurement instrument could not render a verdict on this sample after reasonable due diligence. The canonical case is a probabilistic instrument such as a grader model, which has an inherent, irreducible per-sample failure rate. Sometimes it emits a parseable verdict; sometimes it refuses or produces an unparsable one, and no retry or code fix will change that for this sample. For these, return `Score.unscored()`. This produces a `Score` whose `value` is `NaN`, which Inspect excludes from metric computation and counts as `unscored_samples` on the `EvalScore` (see [Unscored Samples](custom-scorers.qmd#unscored-samples) for the API details): @@ -131,7 +131,7 @@ When a model grades the model under test, the grader is the *measurement instrum 3. **Use a capable, pinned grader.** A grader too weak to follow its own output format is an eval-validity problem to flag, not something to charge to the model. Pin its generation settings so grades are reproducible (see [Reproducible Grading](model-graded.qmd#reproducible-grading)). 4. **Only then** return `Score.unscored()` for the residual, and report the grader-failure rate as a first-class number. -The `raise` vs `unscored()` line still applies within grading: the grader *API call* raising or timing out is a malfunction → `raise` (the transport failed, not the verdict), whereas a well-formed-but-unparseable *verdict* after bounded retry is an instrument miss → `unscored()`. +The `raise` vs `unscored()` line still applies within grading: the grader *API call* raising or timing out is a malfunction → `raise` (the transport failed, not the verdict), whereas a well-formed-but-unparsable *verdict* after bounded retry is an instrument miss → `unscored()`. ## Recording the reason