Add agentic solvers [4/n]

[iphan]

This PR contains

Description

Add 2 new agentic solvers:

• a tools() solver giving the model provider-native, server-side WebSearch and CodeExecution
• a agentic() solver giving the model python/bash (and optionally web_search) tools in a Docker sandbox

Validation

Automated Checks

make check
make test

Manual Tests

uv run inspect eval lab_bench_2 --model openai/gpt-5-nano --limit 3 --model-role grader=ollama/llama3.2 -T solver=tools
solver: tools, dataset: EdisonScientific/labbench2
total time:                                0:03:52
openai/gpt-5-nano                          128,522 tokens [I: 112,224, CW: 0, CR: 4,864, O: 11,434, R: 10,112]
ollama/llama3.2                            1,244 tokens [I: 970, O: 274]
grader (role)                                                  1,244 tokens [I: 970, O: 274]

semantic_judge_scorer
accuracy               1.000
stderr                 0.000

uv run inspect eval lab_bench_2 --model openai/gpt-5-nano --limit 3 --model-role grader=ollama/llama3.2 -T solver=tools -T tag=cloning -T mode=retrieve
tag: cloning, mode: retrieve, solver: tools, dataset: EdisonScientific/labbench2

total time:                                 0:01:08
openai/gpt-5-nano                           33,599 tokens [I: 7,636, CW: 0, CR: 9,728, O: 16,235, R: 14,976]

cloning_scorer
accuracy        0.000
stderr          0.000


# Artificially reduce the turn limit, and see the custom prompt asking the model to submit answer now
uv run inspect eval lab_bench_2 --model openai/gpt-5-nano --limit 3 --model-role grader=ollama/llama3.2 -T solver=agentic -T tag=seqqa2 -T mode=retrieve
tag: seqqa2, mode: retrieve, solver: agentic, dataset: EdisonScientific/labbench2

total time:                                 0:01:10
openai/gpt-5-nano                           53,454 tokens [I: 27,232, CW: 0, CR: 7,552, O: 18,670, R: 11,840]

seqqa2_scorer
accuracy       0.667
stderr         0.333

[iphan]

This agent with final warning mechanism was provided by Sunishchal from RAND. I preserved his contribution logic here. Let me know if you'd like to adjust/modify the agent's behavior

[ItsTania]

Super interesting agent design! Might have to implement this for evals where a common failure mode is not submitting

[ItsTania]

No notes - looks good to me! The doco makes the three options provided easy to understand.

I ran uv run inspect eval lab_bench_2/lab_bench_2 -T tag=litqa3 --limit 1, with -T solver=tools and -T solver=agentic and works on my end.

Things to note:

• A warning to let us know whether web_search tool is provided in the agentic case could be useful (but isn't blocking). (I didn't have a key set, so the LLM went around this my using python to do its websearch expect it didn't parse the data right)
• Not related to this PR, but there might be some scoring issues we need to test for in a separate PR? One of my test cases happened to be marked as incorrect when I think it should be marked as correct (I'll upload image in a following comment)

[ItsTania]

Thanks for the comment - this is useful.

Is the version of Inspect AI pinned (in the .toml right)? is it possible to pin to the corresponding docker image version and update when the Inspect AI version updates?

This is just a suggestion though - I don't image the aisiuk/inspect-tool-support image will change drastically

[iphan]

I dug into this and decided to keep the base image unpinned, because version-tracking isn't actually possible for the image: it is only published as latest on Docker Hub, there are no per-version tags to pin to.
inspect_ai is locked to an exact version in uv.lock, and Inspect version-checks the sandbox's tool-support at startup, so a protocol mismatch would surface clearly

[ItsTania]

Super interesting agent design! Might have to implement this for evals where a common failure mode is not submitting

[ItsTania]

I think this should be marked as correct?

[ItsTania]

I think this should be marked as correct?

Extra context - it seemed to fail at the scoring stage

verdict:

verdict_source:

fallback

edit: this might be a local set up issue? not sure - I should have the right API keys set... Just letting you know to check for something similar before you run your big runs!!

[iphan]

I think this should be marked as correct?

Extra context - it seemed to fail at the scoring stage

verdict:
verdict_source:
fallback

edit: this might be a local set up issue? not sure - I should have the right API keys set... Just letting you know to check for something similar before you run your big runs!!

Interesting, which model did you use for the scorer?
Since it is using the fallback, it looks like a model which doesn't support structured output.

In the fallback case, the explanation field is the output of the judge. Since it is empty, it looks like the judge returned nothing, which was parsed to incorrect.
I'll merge for now, and can do a small follow up PR to adjust this behavior. It shouldn't be a blocker for my validation runs, because they use Sonnet as the judge, which does support structured output.

Thanks again for the thorough review!