Add agentic solvers [4/n]

src/lab_bench_2/README.md

@@ -73,9 +73,50 @@ See `uv run inspect eval --help` for all available options.
 
 - `tag` (str): Which LAB-Bench 2 subset to run. Supported tags: ``cloning``, ``dbqa2``, ``figqa2`` (and ``figqa2-img`` / ``figqa2-pdf``), ``litqa3``, ``patentqa``, ``protocolqa2``, ``seqqa2``, ``sourcequality``, ``suppqa2``, ``tableqa2`` (and ``tableqa2-img`` / ``tableqa2-pdf``), ``trialqa``. (default: `'litqa3'`)
 - `mode` (Mode): How a question's data files are delivered to the model. A no-op for tags without files (such as litqa3). Options: ``file``: Files uploaded via API. PDFs/images attached as context; other files as document attachments., ``inject``: Text file contents concatenated into the prompt as text., ``retrieve``: Only file names/stems are given; prompt instructs the agent to retrieve the necessary sequences or data from a source of its choosing. File contents are withheld. (default: `'file'`)
-- `solver` (Solver | None): The solver to run. Defaults to ``bare()`` (the benchmark's "bare" configuration: a plain single-turn ``generate()``) when not provided. Pass any Inspect solver to override, e.g. ``-T solver=bare`` on the CLI. (default: `None`)
+- `solver` (SolverType): The solver to run. Options: ``bare``: a plain single-turn `generate()`., ``tools``: the server-side agentic configuration. The model is given provider-native, **server-side** tools — WebSearch and CodeExecution — and runs Inspect's tool-use loop., ``agentic``: the client-side agentic configuration. The model is given ``python``/``bash`` (and, with an external provider key, ``web_search``) tools in a Docker sandbox. (default: `'bare'`)
 <!-- /Parameters: Automatically Generated -->
 
+## Solvers
+
+The benchmark can run each model in three configurations, selected via the `solver`
+parameter:
+
+- **`bare`** (default): a plain single-turn `generate()` — no tools.
+- **`tools`**: the server-side agentic configuration. The model is given provider-native,
+  **server-side** tools — WebSearch and CodeExecution — and runs Inspect's
+  tool-use loop, which drives each provider's server-side tool round-trips. The
+  internal provider is auto-selected for the active model, so no external search
+  keys or local sandbox are required.
+- **`agentic`**: the client-side agentic configuration. The model is given **sandboxed**
+  `python` / `bash` tools (plus `web_search` when an external provider key is set —
+  `TAVILY_API_KEY`, `EXA_API_KEY`, or `GOOGLE_CSE_API_KEY`) inside a **Docker sandbox**,
+  and must call `submit()` to answer. A question's data files are copied into the sandbox
+  working directory. The initial docker image build should take ~ 2 minutes on a standard personal computer.
+
+Reasoning effort is set with Inspect's built-in `--reasoning-effort` flag (it
+applies to the model under test only, not the grader). The paper's "tools,high"
+case is:
+
+```bash
+uv run inspect eval lab_bench_2/lab_bench_2 -T tag=litqa3 -T solver=tools --reasoning-effort high
+```
+
+### WebFetch is omitted
+
+The reference benchmark's tool set also includes a **WebFetch** tool (Anthropic
+`web_fetch`, Google `url_context`) for retrieving the full content of a specific
+URL. It is **omitted here because Inspect has no native `web_fetch` wrapper** in
+any release, and no mechanism to pass a raw provider-native tool through. The
+capability exists at the providers — it simply is not surfaced through Inspect.
+Revisit if Inspect adds a web-fetch tool.
+
+One consequence: under `solver=tools`, `mode="retrieve"` (where the model is given
+only file names and must obtain the data itself) is degraded, since the model
+cannot reliably fetch a specific record's full content — prefer `inject` or `file`.
+This limitation does not apply to `solver=agentic`: the question's files are copied
+into the sandbox, so `mode="retrieve"` (filenames in the prompt, contents on disk) is
+the recommended pairing there.
+
 ## Dataset
 
 This eval uses the public `EdisonScientific/labbench2` dataset on Hugging Face, pinned to a specific commit for reproducibility.
@@ -111,16 +152,16 @@ Not every sample is compatible with each mode of data uploading; if incompatible
 Each sample in the dataset contains flags for compatible modes - this may change and sample counts
 can be verified by running with the configuration you intend before drawing conclusions from sample counts.
 
-For most tags, those that uses files requires the `file` mode. For example;
+For most tags, those that use files requires the `file` mode. For example;
 
 `uv run inspect eval lab_bench_2 -T tag=sourcequality -T mode=retrieve`
 
- Will result in no samples being loaded in. This is also true for tags protocolqa2`,`sourcequality`,`figqa2-img`,`figqa2-pdf`,`tableqa2-img`,`tableqa2-pdf`.
+ Will result in no samples being loaded in. This is also true for tags `protocolqa2`,`sourcequality`,`figqa2-img`,`figqa2-pdf`,`tableqa2-img`,`tableqa2-pdf`.
 
 Note that the base `figqa2`, `tableqa2`, and `suppqa2` tags have no files (mode is a no-op). Their image/PDF variants do have files and are impacted by the above.
 
 `seqqa2` is the exception: all of its samples are compatible with `file` and `inject`, while only a
-subset of this tag can be used with`retrieve` (so `mode="retrieve"` loads fewer samples).
+subset of this tag can be used with `retrieve` (so `mode="retrieve"` loads fewer samples).
 
 ## Scoring
 

src/lab_bench_2/__init__.py

@@ -17,7 +17,15 @@
     semantic_judge_scorer,
     seqqa2_scorer,
 )
-from lab_bench_2.solvers import bare
+from lab_bench_2.solvers import (
+    SolverType,
+    agentic,
+    bare,
+    native_tools,
+    sandbox_tools,
+    solver_for_type,
+    tools,
+)
 
 __all__ = [
     "DEFAULT_GRADER_MODEL",
@@ -26,15 +34,21 @@
     "LAB_BENCH_2_DATASET_SPLIT",
     "SUPPORTED_TAGS",
     "Mode",
+    "SolverType",
+    "agentic",
     "bare",
     "cloning_scorer",
     "exact_match_judge_scorer",
     "lab_bench_2",
     "load_lab_bench_2_dataset",
+    "native_tools",
     "parse_judge_verdict",
     "recall_judge_scorer",
     "record_to_sample",
+    "sandbox_tools",
     "scorer_for_tag",
     "semantic_judge_scorer",
     "seqqa2_scorer",
+    "solver_for_type",
+    "tools",
 ]

src/lab_bench_2/lab_bench_2.py

@@ -8,12 +8,11 @@
 from __future__ import annotations
 
 from inspect_ai import Task, task
-from inspect_ai.solver import Solver
 
 from lab_bench_2.dataset import load_lab_bench_2_dataset
 from lab_bench_2.prompt_composer import Mode
 from lab_bench_2.scorers import scorer_for_tag
-from lab_bench_2.solvers import bare
+from lab_bench_2.solvers import SolverType, sandbox_for_solver, solver_for_type
 from utils.metadata import load_version_from_yaml
 
 SUPPORTED_TAGS = (
@@ -41,7 +40,7 @@
 def lab_bench_2(
     tag: str = "litqa3",
     mode: Mode = "file",
-    solver: Solver | None = None,
+    solver: SolverType = "bare",
 ) -> Task:
     """LAB-Bench 2 evaluation task.
 
@@ -61,17 +60,24 @@ def lab_bench_2(
             - ``retrieve``: Only file names/stems are given; prompt instructs
               the agent to retrieve the necessary sequences or data from a
               source of its choosing. File contents are withheld.
-        solver: The solver to run. Defaults to ``bare()`` (the benchmark's "bare"
-            configuration: a plain single-turn ``generate()``) when not provided.
-            Pass any Inspect solver to override, e.g. ``-T solver=bare`` on the CLI.
+        solver: The solver to run. Options:
+
+            - ``bare``: a plain single-turn `generate()`.
+            - ``tools``: the server-side agentic configuration. The model
+            is given provider-native, **server-side** tools — WebSearch and
+            CodeExecution — and runs Inspect's tool-use loop.
+            - ``agentic``: the client-side agentic configuration. The model is
+            given ``python``/``bash`` (and, with an external provider
+            key, ``web_search``) tools in a Docker sandbox.
     """
     if tag not in SUPPORTED_TAGS:
         raise NotImplementedError(
             f"tag={tag!r} is not implemented yet; supported tags: {list(SUPPORTED_TAGS)}."
         )
     return Task(
         dataset=load_lab_bench_2_dataset(tag=tag, mode=mode),
-        solver=solver or bare(),
+        solver=solver_for_type(solver),
         scorer=scorer_for_tag(tag),
+        sandbox=sandbox_for_solver(solver),
         version=EVAL_VERSION,
     )

src/lab_bench_2/solvers/Dockerfile

@@ -0,0 +1,14 @@
+# Scientific Python stack for the agentic solver's sandboxed python/bash tools.
+# Versions are pinned for reproducible builds. pydna constrains biopython,
+# numpy, pandas, and scipy, so bump them as a set (re-resolve, don't pin
+# individually). The base image is intentionally left unpinned: it must track
+# the inspect tool-support protocol of the installed inspect_ai.
+FROM aisiuk/inspect-tool-support
+
+RUN pip install --no-cache-dir \
+    biopython==1.87 \
+    pydna==5.5.12 \
+    primer3-py==2.3.0 \
+    pandas==3.0.3 \
+    numpy==2.4.6 \
+    scipy==1.17.1

src/lab_bench_2/solvers/__init__.py

@@ -0,0 +1,28 @@
+"""Solvers for the LAB-Bench 2 evaluation.
+
+The reference benchmark runs each model in two configurations: `bare` (no tools,
+single-turn) and an agentic configuration with provider-native tools. This package
+groups the solver factories and the registry that maps a ``SolverType`` to them.
+"""
+
+from lab_bench_2.solvers.agent import agentic
+from lab_bench_2.solvers.registry import (
+    SOLVERS_BY_TYPE,
+    SolverType,
+    sandbox_for_solver,
+    solver_for_type,
+)
+from lab_bench_2.solvers.sandbox_tools import sandbox_tools
+from lab_bench_2.solvers.solvers import bare, native_tools, tools
+
+__all__ = [
+    "SOLVERS_BY_TYPE",
+    "SolverType",
+    "agentic",
+    "bare",
+    "native_tools",
+    "sandbox_for_solver",
+    "sandbox_tools",
+    "solver_for_type",
+    "tools",
+]