Skip to content

Add claim-support benchmark for transcript-grounded scoring#4478

Closed
dpaul0501 wants to merge 1 commit into
UKGovernmentBEIS:mainfrom
dpaul0501:claim-support-benchmark
Closed

Add claim-support benchmark for transcript-grounded scoring#4478
dpaul0501 wants to merge 1 commit into
UKGovernmentBEIS:mainfrom
dpaul0501:claim-support-benchmark

Conversation

@dpaul0501

Copy link
Copy Markdown

This PR contains:

  • New features
  • Changes to dev-tools e.g. CI config / github tooling
  • Docs
  • Bug fixes
  • Code refactor

What is the current behavior? (You can also link to an open issue here)

claim_support() is available as a transcript-aware model-graded scorer, but it only evaluates the final claim in a single pass.

What is the new behavior?

This PR adds an optional atomic-claim verification flow for claim_support() via decompose_claims=True.

  • Splits compound claims into atomic subclaims
  • Verifies each subclaim against the transcript/tool trace
  • Aggregates subclaim verdicts into a final score
  • Keeps the scorer model-agnostic, so any supported LLM backend can act as the judge

It also adds:

  • a small benchmark-style regression test comparing baseline vs atomic verification
  • a security-trace example task
  • documentation updates so claim_support() appears in the built-in scorer list

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No breaking change.

claim_support() keeps its existing behavior by default. Existing users do not need to change anything unless they want the new atomic-claim mode, in which case they can opt into decompose_claims=True.

Other information:

This is meant to make claim verification more robust for agent and security-trace evaluations, especially when answers contain multiple facts bundled into one response.

@dragonstyle

Copy link
Copy Markdown
Collaborator

Duplicate of #4166

@dragonstyle dragonstyle marked this as a duplicate of #4166 Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants