docs: add Scoring Policy page (score vs raise vs unscored)

docs/_builtin-scorers.md

@@ -40,6 +40,18 @@ Inspect includes both text matching scorers as well as model graded scorers. Bel
 
 :::
 
+## When the output doesn't match
+
+These scorers use the `CORRECT` and `INCORRECT` constants, which the default metrics convert to `1.0` and `0.0` (see [Custom Scorers](custom-scorers.qmd#score) for the `Value` types and `value_to_float()`). They differ in how they treat output that does not match the target:
+
+`includes()`, `match()`, and `exact()`
+:   Score a non-matching output `INCORRECT`, so the sample stays in the denominator as a `0.0`.
+
+`pattern()` (and `answer()`, which builds on it)
+:   Score `INCORRECT` when the pattern matches but the captured value is wrong, and `NOANSWER` when the pattern does not match at all. `NOANSWER` also converts to `0.0` by default, so it does not change accuracy; it is a distinct label for "no answer was found in the expected form" rather than "the model answered incorrectly".
+
+Whether a failure to produce output in the expected format should count as a wrong answer (`INCORRECT`) or as no answer (`NOANSWER`) is a judgement call, since failing to follow the requested format is itself an instruction-following failure. See [Scoring Policy](scoring-policy.qmd#verdicts-on-the-model) for the distinction and how it interacts with metrics.
+
 ## Metrics
 
 Each scorer provides one or more built-in metrics. Most report `accuracy` and `stderr`; `exact()` and `f1()` report `mean` and `stderr`; and the perplexity scorers report `perplexity_per_token` and `perplexity_per_seq`. You can override these by passing your own `metrics` to the `Task`:

docs/_quarto.yml

@@ -81,6 +81,8 @@ inspect-docs:
       contents:
         - href: standard-scorers.qmd
         - href: custom-scorers.qmd
+        - text: Scoring Policy
+          href: scoring-policy.qmd
         - text: Model Grading
           href: model-graded.qmd
         - text: Scoring Metrics

docs/custom-scorers.qmd

@@ -131,6 +131,8 @@ return Score.unscored(
 
 Unscored samples are skipped by aggregate metrics and epoch reducers and are counted toward `EvalScore.unscored_samples` rather than included as zeros. This works for scalar, dict-valued, and list-valued scorers.
 
+For guidance on *when* to return `Score.unscored()` versus a score or an error, see [Scoring Policy](scoring-policy.qmd).
+
 ## Score Value
 
 `Value` is union over the main scalar types as well as a `list` or `dict` of the same types:
@@ -149,7 +151,13 @@ Next, we'll take a look at the source code for a couple of the built in scorers
 
 ## Models in Scorers
 
-You'll often want to use models in the implementation of scorers. Use the `get_model()` function to get either the currently evaluated model or another model interface. For example:
+You'll often want to use models in the implementation of scorers. The best way of doing this is to specify the grader model using [model roles](models.qmd#model-roles), allowing them to be chosen at runtime. By default, model-graded scorers look for the `"grader"` model role:
+
+```python
+grader_model = get_model(role="grader")
+```
+
+You can also use the `get_model()` function to get either the currently evaluated model or another model interface. For example:
 
 ``` python
 # use the model being evaluated for grading
@@ -246,15 +254,21 @@ def model_graded_qa(
                 explanation=result.completion,
             )
         else:
-            return Score(
-                value=INCORRECT,
+            # The grader failed to produce a parseable verdict. That is a
+            # failure of the measurement instrument, not a wrong answer from the
+            # model, so record it as unscored (excluded from metrics) rather than
+            # charging the model INCORRECT. See Scoring Policy for the rationale.
+            return Score.unscored(
                 explanation="Grade not found in model output: "
                 + f"{result.completion}",
+                metadata={"reason": "grader_parse_failure"},
             )
 
     return score
 ```
 
+The grade-parse-failure branch returns `Score.unscored()` rather than `INCORRECT`: a grader that cannot render a verdict is an instrument failure, not a verdict on the model. See [Scoring Policy](scoring-policy.qmd#model-graded-scorers) for when to score, raise, or return `Score.unscored()`.
+
 Note that the call to `model_grader.generate()` is done with `await`. This is critical to ensure that the scorer participates correctly in the scheduling of generation work.
 
 Note also we use the `input_text` property of the `TaskState` to access a string version of the original user input to substitute it into the grading template. Using the `input_text` has two benefits: (1) It is guaranteed to cover the original input from the dataset (rather than a transformed prompt in `messages`); and (2) It normalises the input to a string (as it could have been a message list).

docs/handling-errors.qmd

@@ -92,6 +92,8 @@ Consequently, when enabling `retry_on_error` you should do some post-hoc analysi
 
 Some evaluations are designed so that an error during the agent run is itself a meaningful (often failing) outcome — for example, a tool-using agent that crashes after producing partial state, or a benchmark where "the model errored" should count as a scoreable result rather than as missing data.
 
+This page covers the run-level options for tolerating errors. For the complementary authoring decision, namely what a scorer should return for a given failure (a `Score`, a `raise`, or `Score.unscored()`), see [Scoring Policy](scoring-policy.qmd).
+
 The `score_on_error` option causes errored samples to be scored anyway (using whatever `TaskState` was reached before the error), and prevents `fail_on_error` from crashing the eval mid-run:
 
 ``` bash

docs/model-graded.qmd

@@ -169,6 +169,8 @@ Grade extraction binds to the last grade. The default `grade_pattern` (`(?is).*(
 
 Structural delimiters are neutralized. The default templates wrap content in `[BEGIN DATA]` / `[END DATA]` markers. Before formatting the prompt, Inspect rewrites any `[BEGIN DATA]` / `[END DATA]` markers that appear in the model-controlled `answer`, the `question`, the `criterion`, and any `metadata` values (for example to `[END-DATA]`) so a model cannot inject a fake delimiter and smuggle grading instructions into the prompt. The `instructions` you provide are author-controlled and left untouched.
 
+When the grader's verdict still cannot be parsed after these precautions, that is a failure of the measurement instrument rather than a verdict on the model. See [Scoring Policy](scoring-policy.qmd#model-graded-scorers) for how to handle it (bounded grader-only retry, then `Score.unscored()`), and why it should not be scored `INCORRECT`.
+
 ## Reproducible Grading
 
 By default the grader model runs with the provider's default generation settings. In practice this often means a non-zero temperature (e.g. 1.0) and no fixed seed, so grades for borderline answers can change from run to run and across epochs.