Skip to content

docs(extensions): add Inspect Claim Support to the extensions gallery#4364

Merged
jjallaire merged 1 commit into
UKGovernmentBEIS:mainfrom
avalyset:docs/gallery-inspect-claim-support
Jun 26, 2026
Merged

docs(extensions): add Inspect Claim Support to the extensions gallery#4364
jjallaire merged 1 commit into
UKGovernmentBEIS:mainfrom
avalyset:docs/gallery-inspect-claim-support

Conversation

@avalyset

Copy link
Copy Markdown
Contributor

This scorer started as #4166 (addressing #4143). The conclusion there was that
it fits better as an external package than in Inspect core, so I've packaged it
as a standalone extension (PyPI: inspect-claim-support, GitHub:
avalyset/inspect-claim-support, MIT). This PR lists it in the extensions
gallery, where external packages live — the natural follow-through from that
decision.

What it covers: a faithfulness / claim-support scorer. It assesses whether a
claimed answer is actually substantiated by the conversation transcript, not
whether it is correct in absolute terms — a grading axis Inspect doesn't have
built-in coverage for. The rubric's core property is that absence of evidence is
not treated as support: a negative claim ("I made no network calls") only scores
SUPPORTED if the transcript can actually represent that class of event;
otherwise PARTIAL or UNSUPPORTED, never SUPPORTED. A grader parse failure returns
Score.unscored() — instrument failure, kept out of the accuracy denominator —
rather than recording it as a no-answer from the model under test. The
implementation uses only Inspect's public API.

Added under the Analysis category, next to CJE — the closest neighbour, also
model-graded scorer work.

Candour: this is a new package, added here for discoverability. Happy to hold
it until it's more established if you'd rather see some traction first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants