Add all scorers for LABBench2 [3/n] by iphan · Pull Request #3 · Generality-Labs/lab-bench

iphan · 2026-05-29T18:25:16Z

This PR contains

Description

Add all scorers and enable additional tags (cloning, dbqa2, figqa2*, seqqa2, suppqa2, tableqa2*).
The reference implementation is at EdisonScientific/labbench2 evals/evaluators.py,

Judge based scorers:
a. Add recall and exact match scorers. They have the same logic as the existing semantic scorer, but just use a different prompt.
b. Request structured output from the grader, so the verdict is read from a typed field. The existing regex-scraping logic is kept as fallback in case a provider doesn't honor the schema or returns non-compliant output.
Deterministic scorers for cloning and seqqa2 tags
a. The scorer.py file from this PR was used as a starting point, with proper code attribution in the commit history.
b. Instead of vendoring in code from the reference implementation, the cloning_reward() is used directly.

Validation

Automated checks

make check
make test

Manual tests

# Verify LLM semantic judge
uv run inspect eval lab_bench_2 --model ollama/llama3.2 --limit 3 --model-role grader=ollama/llama3.2
dataset: EdisonScientific/labbench2

total time:                                                       0:00:13
ollama/llama3.2                                                   1,435 tokens [I: 1,053, O: 382]
grader (role)                                                  1,095 tokens [I: 908, O: 187]

semantic_judge_scorer
accuracy               0.000
stderr                 0.000

# Verify LLM recall judge
uv run inspect eval lab_bench_2 --model ollama/llama3.2 --limit 10 --model-role grader=ollama/llama3.2 -T tag=dbqa2

tag: dbqa2, dataset: EdisonScientific/labbench2

total time:                                                  0:01:13
ollama/llama3.2                                              20,636 tokens [I: 18,453, O: 2,183]
grader (role)                                             18,807 tokens [I: 17,890, O: 917]

recall_judge_scorer
accuracy             0.200
stderr               0.133

# Verify LLM exact match judge
uv run inspect eval lab_bench_2 --model openai/gpt-5-nano --limit 5 --model-role grader=ollama/llama3.2 -T tag=figqa2-img -T mode=file
tag: figqa2-img, mode: file, dataset: EdisonScientific/labbench2
total time:                                              0:00:41
openai/gpt-5-nano                                        30,627 tokens [I: 24,853, O: 5,774, R: 5,376]
ollama/llama3.2                                          2,247 tokens [I: 1,735, O: 512]
grader (role)                                               2,247 tokens [I: 1,735, O: 512]

exact_match_judge_scorer
accuracy                  0.400
stderr                    0.245

# Verify cloning judge
uv run inspect eval lab_bench_2 --model openai/gpt-5-nano --limit 5 --model-role grader=ollama/llama3.2 -T tag=cloning -T mode=inject
tag: cloning, mode: inject, dataset: EdisonScientific/labbench2

total time:                                           0:01:02
openai/gpt-5-nano                                     129,707 tokens [I: 90,288, O: 39,419, R: 38,720]

cloning_scorer
accuracy        0.000
stderr          0.000


# Verify seqqa2 judge
uv run inspect eval lab_bench_2 --model openai/gpt-5-nano --limit 10 --model-role grader=ollama/llama3.2 -T tag=seqqa2 -T mode=file --sample-shuffle
sample_shuffle: True, tag: seqqa2, mode: file, dataset: EdisonScientific/labbench2

total time:                                          0:01:17
openai/gpt-5-nano                                    193,336 tokens [I: 148,395, O: 44,941, R: 44,288]

seqqa2_scorer
accuracy       0.400
stderr         0.163

Reuse the original's recall prompt, but since it has no parseable verdict instruction, so append VERDICT_FORMAT_SUFFIX to emit a `result:` line for parse_judge_verdict.

Route figqa2*/tableqa2*/suppqa2 to an exact-match LLM judge using STRUCTURED_EVALUATION_PROMPT_EXACT_MATCH Enable the seven configs in SUPPORTED_TAGS. The base figqa2/tableqa2/suppqa2 configs are text-only (mode is a no-op); the -img/-pdf variants are file-bearing and support only `file` mode.

Code lifted from UKGovernmentBEIS/inspect_evals#1307 Add cloning_scorer, scoring CloningQA protocols via labbench2's cloning_reward pipeline. Call upstream cloning_reward directly (no local fork). Enable cloning in SUPPORTED_TAGS (14 samples, file-bearing). Extend the mypy override to cover the labbench2 package. Co-Authored-By: Lewis Tunstall <lewtun@users.noreply.github.com>

Map the cloning reward to CORRECT/INCORRECT (was a raw float), record the numeric reward under metadata["cloning_score"], and drop the vestigial "route" metadata. Pass validator_params as a plain dict defaulting to {}, matching the loader's parsed shape.

parse_judge_verdict matched `result = "correct"` inside a grader's code-dump response scoring it CORRECT. Exclude `=` from the separator so assignments no longer match; every supported verdict format (colon, arrow, markdown) still parses.

Code lifted from UKGovernmentBEIS/inspect_evals#1307 Add seqqa2_scorer, dispatching to labbench2's per-type VALIDATORS. Co-Authored-By: Lewis Tunstall <lewtun@users.noreply.github.com>

Request structured output from the grader, reading the verdict from a typed field instead of regex-scraping the completion — mirroring upstream's LLMJudgeEvaluator. Fall back to parse_judge_verdict when a provider doesn't honor the schema or returns non-compliant output.

ItsTania

Hi! Thought I'd share some of my comments from my static review - the main question I have is whether we should raise sample-level errors or mark samples as INCORRECT when the issue is with the LLM judge scorer rather than the solver.

Tomorrow I'll go through the tests and also run the eval with different configurations too :)) Is there anything in particular you think I should try or look at?

ItsTania · 2026-06-01T12:55:47Z

+| --------------- | ------- | ------------ | ---------------------- |----------------------------------------|
+| `cloning`       | 14      | Yes          | any (all modes)        | Cloning protocols; reward-scored.      |
+| `dbqa2`         | 86      | No           | any (mode is a no-op)  | Database access; recall judge.         |
+| `figqa2`        | 101     | No           | any (mode is a no-op)  | Figure QA; exact-match judge.          |


Question - is figqa2-img and figqa2-pdf subsets of figqa2?

Reading forward ^ doesn't seem to be right. It could be worth explaining what the relationship between these tags are - are they made up of the same questions with just different data files? are they subsets?

This is explained in section 2.1 of the paper. How much of the paper's info do you typically add to this README?

FigQA2 and TableQA2.
The original FigQA and TableQA benchmarks presented questions that required nuanced interpretation of information presented in scientific figures and tables in a multiple-choice format. Importantly, in both cases the figure or table in question was presented to the model as an image file and thus measured only the ability to read the image and choose the best option. For the new FigQA2 and TableQA2, all new questions have been written, and we’ve created three different task variants for each figure or table:

(i) Image: Similar to the original Fig/TableQA tasks, but are open-response. Require only reading and understanding the figure or table as an image, which is provided to the model along with the question.
(ii) Paper: Rather than the figure image, the entire PDF of the source paper is provided to the model. The correct figure or table containing the information needed to answer must be parsed and interpreted from the distracting context of the full paper.
(iii) Retrieval: Here the task is similar to LitQA3, where no context is provided, and the model/agent must retrieve the correct source paper and properly locate the information in the correct figure or table.

Thanks for commenting the section! The amount of info I add into the README is usually what is needed to run and interpret the eval results - I didn't read the paper in depth so it was a point of confusion on my end was what combination of tags to run.

I also get that a bloated README is also unhelpful, so I think you with what you think.

ItsTania · 2026-06-01T14:13:41Z

+or <https://go.dev/dl/>. Without Go, protocol execution fails gracefully: PCR-based
+samples score 0.0 with an explanatory reason rather than crashing the run.


Design question - Would it be better to raise a sample level error? It won't crash the run, but it will exclude the same from the aggregate score.

(e.g say 1 sample did not get scored because go didn't run, 8/10 would rather be 8/9 (8 correct of the 9 samples scored))

ItsTania · 2026-06-01T14:55:22Z

+            verdict = parse_judge_verdict(result.completion)
+            explanation = result.completion
+            verdict_source = "fallback"
+


Eval design question I'm asking myself - if the judge model failed to produce a correct output, should the sample be marked incorrect (0/1) or errored (0/0) as the error was not introduced by the solver?

What does the OG implementation do? (I can try check this tomorrow!)

edit: same with lines 149 to 153 (bad sample) and line 157 (missing ground truth)

Here's another example of a discussion that might be relevant to discussing what we should do here :)) (UKGovernmentBEIS/inspect_evals#1679 (comment))

I think this is not as black and white, because not all models support structured outputs. That is why I added the fallback rather than mark it as an error.

ItsTania · 2026-06-01T15:00:25Z

+        metadata = state.metadata or {}
+        question_type = cast(str | None, metadata.get("type"))
+        if not question_type:
+            return Score(
+                value=INCORRECT,
+                explanation="SeqQA2 evaluation requires question type metadata.",
+            )
+
+        validator = VALIDATORS.get(question_type)
+        if validator is None:
+            return Score(
+                value=INCORRECT,
+                explanation=f"No validator found for type: {question_type}",
+            )


Same as above!

(Also thinking about this further - is this something we check for/ validate during the record to sample mapping?)

The behavior here is matching the OG implementation:
https://github.com/EdisonScientific/labbench2/blob/c028ecdcf144b55ffcd92b68be45081df5628c20/src/labbench2/cloning/rewards.py#L172-L176
https://github.com/EdisonScientific/labbench2/blob/c028ecdcf144b55ffcd92b68be45081df5628c20/evals/evaluators.py#L131-L145

ItsTania · 2026-06-01T15:01:16Z

+        if extracted is None:
+            return Score(
+                value=INCORRECT,
+                explanation="Failed to extract answer from model output.",


Is it worth storing the model output in metadata for troubleshooting?

Co-authored-by: Tania <120768997+ItsTania@users.noreply.github.com>

iphan

Thanks for the review!
I updated the README per your suggestions and changed the default mode to file.

the main question I have is whether we should raise sample-level errors or mark samples as INCORRECT when the issue is with the LLM judge scorer rather than the solver.

Your suggestion would indeed improve the accuracy of the eval. However, the current behavior here is maintaining the same logic as the OG implementation.
This is the same question that was raised a couple times about this benchmark regarding faithfulness vs. improvement in the port.
In the hopes of getting evals results closer to the paper's results, I chose to lean towards faithfulness. But let me know if you typically handle this tradeoff differently.

iphan · 2026-06-01T22:25:40Z

+| --------------- | ------- | ------------ | ---------------------- |----------------------------------------|
+| `cloning`       | 14      | Yes          | any (all modes)        | Cloning protocols; reward-scored.      |
+| `dbqa2`         | 86      | No           | any (mode is a no-op)  | Database access; recall judge.         |
+| `figqa2`        | 101     | No           | any (mode is a no-op)  | Figure QA; exact-match judge.          |


This is explained in section 2.1 of the paper. How much of the paper's info do you typically add to this README?

FigQA2 and TableQA2.
The original FigQA and TableQA benchmarks presented questions that required nuanced interpretation of information presented in scientific figures and tables in a multiple-choice format. Importantly, in both cases the figure or table in question was presented to the model as an image file and thus measured only the ability to read the image and choose the best option. For the new FigQA2 and TableQA2, all new questions have been written, and we’ve created three different task variants for each figure or table:

(i) Image: Similar to the original Fig/TableQA tasks, but are open-response. Require only reading and understanding the figure or table as an image, which is provided to the model along with the question.
(ii) Paper: Rather than the figure image, the entire PDF of the source paper is provided to the model. The correct figure or table containing the information needed to answer must be parsed and interpreted from the distracting context of the full paper.
(iii) Retrieval: Here the task is similar to LitQA3, where no context is provided, and the model/agent must retrieve the correct source paper and properly locate the information in the correct figure or table.

iphan · 2026-06-01T22:29:59Z

+# Excluding ``=`` from the separator stops the pattern from matching code-like
+# grader output such as ``result = "correct"`` (an assignment, not a verdict).
 _GRADE_PATTERN = re.compile(
-    r"\bresult\b[^A-Za-z]*(correct|incorrect|unsure)\b",
+    r"\bresult\b[^A-Za-z=]*(correct|incorrect|unsure)\b",


The original is using the structured output, this fallback regex parsing is added for the models which don't support structured output

iphan · 2026-06-01T22:45:32Z

+        metadata = state.metadata or {}
+        question_type = cast(str | None, metadata.get("type"))
+        if not question_type:
+            return Score(
+                value=INCORRECT,
+                explanation="SeqQA2 evaluation requires question type metadata.",
+            )
+
+        validator = VALIDATORS.get(question_type)
+        if validator is None:
+            return Score(
+                value=INCORRECT,
+                explanation=f"No validator found for type: {question_type}",
+            )


The behavior here is matching the OG implementation:
https://github.com/EdisonScientific/labbench2/blob/c028ecdcf144b55ffcd92b68be45081df5628c20/src/labbench2/cloning/rewards.py#L172-L176
https://github.com/EdisonScientific/labbench2/blob/c028ecdcf144b55ffcd92b68be45081df5628c20/evals/evaluators.py#L131-L145

iphan · 2026-06-01T22:46:32Z

+            verdict = parse_judge_verdict(result.completion)
+            explanation = result.completion
+            verdict_source = "fallback"
+


I think this is not as black and white, because not all models support structured outputs. That is why I added the fallback rather than mark it as an error.

ItsTania

I saw your other PR and am looking at it now!

With all of the 'should be be scored as 0 or errored' questions - its a broader discussion that we're having at the moment. I've noted all of the instances this question comes up in Lab bench and will use them as useful case studies (I'll share the doc with you on slack too :))

I think its fine to merge this as is, and I'll get back to you/ raise PRs in the future depending on the inspect eval approach!

iphan and others added 7 commits May 29, 2026 11:22

Add recall judge and enable dbqa2

d57f3d2

Reuse the original's recall prompt, but since it has no parseable verdict instruction, so append VERDICT_FORMAT_SUFFIX to emit a `result:` line for parse_judge_verdict.

Port seqqa2 scorer and enable the seqqa2 tag

efe825a

Code lifted from UKGovernmentBEIS/inspect_evals#1307 Add seqqa2_scorer, dispatching to labbench2's per-type VALIDATORS. Co-Authored-By: Lewis Tunstall <lewtun@users.noreply.github.com>

Tidy seqqa2 scorer, extract answer helpers, add tests

a5e4e06

iphan changed the title ~~Add all scorers for LABBench2~~ Add all scorers for LABBench2 [3/n] May 29, 2026

iphan force-pushed the lb2_scorers branch from 936b05a to a3b9811 Compare May 29, 2026 18:59

iphan force-pushed the lb2_scorers branch from a3b9811 to cd7f600 Compare May 29, 2026 19:25

iphan marked this pull request as ready for review May 29, 2026 21:32

ItsTania reviewed Jun 1, 2026

View reviewed changes

Pass answer in the cloning scorer output

a1694d9

iphan force-pushed the lb2_scorers branch from 987e0c9 to a1694d9 Compare June 1, 2026 22:03

Apply README suggestions from code review

224c4fa

Co-authored-by: Tania <120768997+ItsTania@users.noreply.github.com>

iphan commented Jun 1, 2026

View reviewed changes

Update readme and add more debugging to scorer

bf2d357

iphan requested a review from ItsTania June 2, 2026 17:31

ItsTania approved these changes Jun 3, 2026

View reviewed changes

iphan merged commit 5f66102 into main Jun 3, 2026
15 checks passed

iphan deleted the lb2_scorers branch June 3, 2026 17:47

		or <https://go.dev/dl/>. Without Go, protocol execution fails gracefully: PCR-based
		samples score 0.0 with an explanatory reason rather than crashing the run.

Uh oh!

Conversation

iphan commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR contains

Description

Validation

Automated checks

Manual tests

Uh oh!

ItsTania left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iphan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ItsTania left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

iphan commented May 29, 2026 •

edited

Loading