Add simplest lab_bench_2 case [1/n]

[iphan]

This PR contains

More details in the design doc

Description

Initial port of LAB-Bench 2 (EdisonScientific/labbench2) into this inspect_evals template as the lab_bench_2 evaluation. This is Phase 1 of a phased port (see implementation plan details):

• One tag only: litqa3 (literature-QA subset of 168 samples).
• Bare solver only — a single-turn generate() matching the benchmark's "bare" configuration. The solver task arg accepts any Inspect solver for overrides.
• One scorer — a semantic LLM judge (grader model role, default anthropic/claude-sonnet-4-5, temperature 0) returning correct / incorrect / unsure; only correct scores 1.0. Reports accuracy and stderr.
• Dataset sourced from the public EdisonScientific/labbench2 HF dataset, pinned to a specific commit for reproducibility.
• mode task arg (file / inject / retrieve) wired through but a no-op for litqa3 (no per-question files); included now to keep the surface stable for later tags.

Validation

Automated Checks

make check
make test

Manual

# Run the eval and check results
uv run inspect eval lab_bench_2 --model ollama/llama3.2 --limit 10 --model-role grader=ollama/llama3.2

[celiawaggoner]

nit: use a different model here as an example of overriding the default

[celiawaggoner]

Is this going to evolve into a "run config file" like Matt implemented support for in UKGovernmentBEIS/inspect_ai#3928? If we go with the run config file approach, we could achieve some of the configuration functionality that Justin was talking about in Slack and in these GH comments: Comment 1 and Comment 2. E.g. some users might want to run the implementation most faithful to the paper (i.e. same grader models, same temperature) and other users might want more control.

At the moment, this file has the content that would be submitted to inspect_evals when hooking this eval into the register.

Do you think the "run config file" would add value here as a way to document the paper defaults in code and make them easy to run? And, would using the "run config file" approach add more complexity to the implementation?

[iphan]

ah yes, we can definitely add config files for the paper settings. It would be very useful for users who want to reproduce the paper results. I didn't add it yet, because I don't have all the options working yet.
It shouldn't add much complexity. I think I just need to add config files with the settings from the paper. This eval.yaml file will remain as is, because we need for the inspect_evals submission. I will name the config files differently: paper_config_bare.yaml and paper_config_tools_high.yaml

[celiawaggoner]

Cool, sounds good!

This eval.yaml file will remain as is, because we need for the inspect_evals submission.

This actually doesn't need to live in the eval repo, it will be part of the PR that's raised to the inspect_evals repo and then live in the register/ dir of inspect_evals, e.g. here: https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/register/hangman-bench/eval.yaml

[iphan]

Ah thanks for pointing this out. I will remove this file then and submit it with the inspect_evals' PR.

I had included it here, because the examples all have this yaml file. Maybe adding a comment at the top that this file is meant for the registry would be helpful for users?

[iphan]

oh actually we do need the eval.yaml file: after removing it several CI checks failed:

• https://github.com/Generality-Labs/lab-bench/actions/runs/26593496388/job/78358221311?pr=1
• https://github.com/Generality-Labs/lab-bench/actions/runs/26593496388/job/78358221516?pr=1

[celiawaggoner]

Whoops, sorry @iphan! 🤦‍♀️ Let's add it back for now.. I'll add to our backlog that we can probably remove this from the template and update the CI (but will catch up on context about why it's included in the template first)

[celiawaggoner]

why remove this? should it be trust_remote_code=True with a comment explaining why it's needed?

[iphan]

Since datasets version 4.0.0, this input param was removed from load_dataset_builder(), see code here

Then this should be updated in the template:

• either specify the datasets version to use
• or remove this option altogether

[celiawaggoner]

ohh i see, thank you! we'll update the template

[ItsTania]

updated in Generality-Labs#18 !

[ItsTania]

Looks good! (No comments from my end - the design doc is great + useful thanks)