feat(scorer): add claim_support scorer (#4143)

[avalyset]

Draft for #4143. Opening as draft per the issue author's note that maintainers own the in-tree decision.

Adds a claim_support scorer that assesses whether a claimed answer is substantiated by the conversation transcript, rather than whether it is correct in absolute terms — the faithfulness/support distinction raised in the issue.

Design (incorporating the issue thread):

• Model-graded style, additive — new _claim_support.py + export, no changes to existing scorers.
• Public rubric SUPPORTED / PARTIAL / UNSUPPORTED mapped onto existing Inspect score values: SUPPORTED→CORRECT, PARTIAL→PARTIAL, UNSUPPORTED→INCORRECT, grader parse failure→NOANSWER (matching the direction in model_graded_qa / model_graded_fact silently default to INCORRECT when the judge's verdict doesn't match the grade regex #4026/fix(scorer): mark model_graded grade-parse failure as unscored (not INCORRECT) #4048). No parallel score vocabulary.
• Key boundary: absence of evidence is not support. A negative claim ("I made no network calls") is only SUPPORTED if the transcript can actually represent that class of event; otherwise PARTIAL/UNSUPPORTED. This surfaces overclaims instead of laundering them through a plausible rationale.
• v1 is transcript-only and explicitly not a runtime-truth scorer: it asserts "supported by the Inspect transcript," not "actually happened in the environment." An alternate evidence renderer / external evidence summary is a deliberate future extension point, not in this PR.
• Parameterised template and model; explanation preserves the grader response and metadata preserves both the grader prompt and the parsed grade.
• Tests cover supported, partial, unsupported, malformed-grader-output (→NOANSWER), literal-brace robustness, and the absence-of-evidence boundary reaching the grader prompt.

Happy to adjust rubric wording, add a transcript renderer hook consistent with include_history, or move out-of-tree as a package if maintainers prefer.

[Rul1an]

Reading the diff, the thing I cared most about survived intact: the rubric won't let absence of evidence pass as support, so a claim like "I made no network calls" only lands as SUPPORTED when the transcript can actually represent that class of event. That's the whole reason this scorer earns its place, and having it spelled out in the template instead of left to the grader's discretion is reassuring. Mapping SUPPORTED/PARTIAL/UNSUPPORTED onto CORRECT/PARTIAL/INCORRECT, with NOANSWER on a parse failure, keeps it consistent with the other model-graded scorers, and the additive shape makes it easy to reason about.

Where I'd push a little is that the boundary lives in the prompt today, so it holds only as long as the grader honours it. A test that drives the negative-claim case directly would make it stick: a "no network calls" claim against a transcript with no network-event surface, asserted to come back not-CORRECT. That locks the absence-isn't-support rule the way the malformed-grader case is already locked, and it's the regression I'd least want to discover after the fact.

The merge call is the maintainers' to make, not mine. From where I sit as the person who raised the issue, this is the shape I was hoping for, and it's good to see it come through the flow.

[avalyset]

Thanks — that case is already covered. test_claim_support_absence_boundary_reaches_grader drives exactly the "no network calls" claim against a transcript with no network-event surface and asserts INCORRECT (not CORRECT), and also checks the absence-of-evidence rule reaches the grader prompt. test_claim_support_absence_partial_maps_to_partial pins the PARTIAL branch of the same claim to PARTIAL / not-CORRECT. Together they lock both rubric-permitted grades for an absence claim to non-CORRECT.

One honest limit: these lock the grade→score mapping, not grader fidelity — i.e. they prove PARTIAL/UNSUPPORTED map to non-CORRECT, not that a real grader never returns SUPPORTED for an absence claim. That last property isn't deterministically unit-testable with a mock grader, so it's left to the rubric wording rather than a test.

[dragonstyle]

Thanks for this PR. After considering it, I think this better fits an external package. While I think this represents a real need, I don't think there is an established approach to addressing that we can canonicalize within Inspect. An external package will allow this to evolve and grow more easily, and if/when it is stable and widely used we can roll it from extension into Inspect core.

[avalyset]

Packaged it as an extension per your suggestion — inspect-claim-support is on PyPI now (pip install inspect-claim-support), exposing the scorer as inspect_claim_support/claim_support. Same rubric and tests from this PR, with chat_history reimplemented against the public API so it has no internal dependencies. @Rul1an the negative-claim boundary test came across intact. Happy to keep it maintained out here, and it's straightforward to pull into core later if it gets traction.