feat: swe bench scorer

ATTRIBUTION

@@ -25,4 +25,13 @@ This section lists third-party assets and their attributions.
   (instance IDs such as `django__django-16899`). Cite: Jimenez et al.,
   "SWE-bench: Can Language Models Resolve Real-world Github Issues?", ICLR 2024.
 
+### mini-swe-agent
+
+- Source: https://github.com/SWE-agent/mini-swe-agent/tree/v2.3.0
+- License: MIT
+- Copyright: Copyright (c) 2025 Kilian A. Lieret and Carlos E. Jimenez
+- Description: The SWE-bench service templates and optional Qwen tool schemas,
+  parser, and model extension are derived from mini-swe-agent v2.3.0. The full
+  upstream notice is retained in the swebench_service package.
+
 For a complete list of dependencies and their licenses, see the requirements files in the `requirements/` directory.

examples/10_Agentic_Inference/README.md

@@ -194,3 +194,65 @@ Update the first `datasets` entry (`name` and `path`), `model_params.name`, and
 uv run inference-endpoint benchmark from-config \
   --config examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml
 ```
+
+## SWE-bench Accuracy
+
+Both `qwen_agentic_benchmark.yaml` and `kimi_agentic_benchmark.yaml` include the
+SWE-bench accuracy dataset. The benchmark framework skips its built-in endpoint
+phase for the SWE-bench dataset. Instead,
+`SWEBenchScorer` submits the run to a native SWE-bench service. The service host
+owns Docker, `mini-swe-agent`, and the `swebench` evaluation harness, and it
+drives requests to the configured endpoint.
+
+Keep `accuracy_config.num_repeats: 1`: the scorer performs one external
+evaluation run per benchmark. Optional `accuracy_config.extras.subset` and
+`split` are used consistently for dataset loading, preflight, and scoring.
+
+`accuracy_config.extras.swebench_service_url` points the benchmark client to
+the service. Service mode follows the LiveCodeBench-style external-service
+convention for heavyweight evaluation work and supports exactly one endpoint URL
+in `endpoint_config.endpoints`; that URL must be reachable from the service
+host. Treat the service host as trusted infrastructure: it receives the endpoint
+URL and optional endpoint API key needed to run mini-swe-agent.
+For non-loopback service deployments, bind it on a private network or start it
+with `--auth-token` and set
+`accuracy_config.extras.swebench_service_auth_token`.
+
+`accuracy_config.extras.workers` sets the agent run's parallelism (`--workers`).
+If unset, it defaults to the load pattern's `target_concurrency` (for
+`concurrency`/`agentic_inference` patterns), else 10. `max_eval_workers`
+(default 10, `--max_workers`) sets the eval harness's parallelism.
+
+Qwen tool-call runs should set
+`accuracy_config.extras.swebench_template: qwen_tools`. The selected packaged
+template also activates the service's `QwenToolsModel` through mini-swe-agent's
+`model_class` hook.
+
+Start the service on the host that has Docker:
+
+```bash
+uv run --project src/inference_endpoint/evaluation/swebench_service \
+  python -m swebench_service --host 0.0.0.0 --port 18080
+```
+
+Then select the matching model config and run it from the repo root:
+
+```bash
+CONFIG=examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml
+# For Kimi, use examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml.
+
+# PERF (default): agentic performance and inline scoring; skips SWE-bench.
+uv run inference-endpoint benchmark from-config --config "$CONFIG"
+
+# BOTH: agentic performance followed by SWE-bench.
+uv run inference-endpoint benchmark from-config --config "$CONFIG" --mode both
+
+# ACC: SWE-bench only; skips the agentic performance dataset.
+uv run inference-endpoint benchmark from-config --config "$CONFIG" --mode acc
+```
+
+The default `PERF` mode does not load, preflight, or submit external evaluation
+scorers. Use `--mode both` or `--mode acc` whenever SWE-bench should run.
+
+See `accuracy/RUNBOOK.md` for preconditions, sanity checks, and common failure
+modes.

examples/10_Agentic_Inference/accuracy/RUNBOOK.md

@@ -0,0 +1,74 @@
+# SWE-bench Accuracy Smoke-Test Runbook
+
+End-to-end validation for the SWE-bench accuracy pipeline. Unit tests mock all
+subprocesses, so running the real pipeline is the only way to catch Docker,
+HuggingFace access, or mini-swe-agent wiring issues.
+
+## 0. Preconditions
+
+- Docker daemon running on the SWE-bench service host.
+- Docker Hub auth or a pre-seeded image cache on the service host.
+- Network egress to PyPI and HuggingFace Hub from the service host.
+- Endpoint URL reachable from the service host.
+- `uv` binary on PATH (`curl -LsSf https://astral.sh/uv/install.sh | sh`).
+- Parent endpoints env already synced (`uv sync --extra dev` from repo root).
+
+## 1. Start the SWE-bench service
+
+From the repo root:
+
+```bash
+uv run --project src/inference_endpoint/evaluation/swebench_service \
+  python -m swebench_service --host 0.0.0.0 --port 18080
+```
+
+Sanity check:
+
+```bash
+curl http://localhost:18080/health
+```
+
+## 2. End-to-end test (requires live endpoint)
+
+Select the config for the model under test:
+
+```bash
+CONFIG=examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml
+# For Kimi, use examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml.
+
+# PERF (default): agentic performance only; skips SWE-bench.
+uv run inference-endpoint benchmark from-config --config "$CONFIG"
+
+# BOTH: agentic performance followed by SWE-bench.
+uv run inference-endpoint benchmark from-config --config "$CONFIG" --mode both
+
+# ACC: SWE-bench only.
+uv run inference-endpoint benchmark from-config --config "$CONFIG" --mode acc
+```
+
+Both configs include a performance dataset and the SWE-bench accuracy dataset.
+The default `PERF` mode skips external evaluation, so it does not require a
+running SWE-bench service. Client preflight and submission require `--mode acc`
+or `--mode both`; `ACC` also skips the performance dataset.
+
+Scorer preflight calls the service `/health` endpoint. It does not check Docker
+or pre-pull images on the benchmark client.
+
+The service is trusted infrastructure. It receives one endpoint URL and optional
+endpoint credentials, runs Docker-backed evaluations, and serves artifacts. For
+non-loopback deployments, bind it on a private network or start it with
+`--auth-token TOKEN` and set
+`accuracy_config.extras.swebench_service_auth_token: TOKEN`.
+
+Qwen SWE-bench configs opt in with
+`accuracy_config.extras.swebench_template: qwen_tools`. The service loads its
+packaged Qwen template and activates `QwenToolsModel` through mini-swe-agent's
+`model_class` hook. Omit this setting for Kimi and other non-Qwen runs.
+
+## Common failure modes
+
+| Symptom                              | Likely cause                              | Fix                                                |
+| ------------------------------------ | ----------------------------------------- | -------------------------------------------------- |
+| `swebench_service_url is required`   | Client config missing service URL         | Set `accuracy_config.extras.swebench_service_url`  |
+| Service health check fails           | Service not running or unreachable        | Start the service or fix client-to-service routing |
+| Docker error during `run_evaluation` | Docker daemon not running on service host | Start Docker on the service host and retry         |

examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml

@@ -23,6 +23,15 @@ datasets:
       num_trajectories_to_issue: 990 # Should be integer multiple of 990.
       # Required benchmark default; set to true only for faster optimization/debug runs.
       stop_issuing_on_first_user_complete: false
+  - name: swe_bench # Default PERF skips external evaluation; use ACC or BOTH.
+    type: "accuracy"
+    accuracy_config:
+      eval_method: "swe_bench_scorer"
+      num_repeats: 1
+      extras:
+        swebench_service_url: "http://localhost:18080"
+        num_instances: 200
+        # workers: 8 # parallel agent workers; defaults to target_concurrency if unset.
 
 settings:
   runtime:

examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml

@@ -0,0 +1,52 @@
+name: "qwen-agentic-benchmark"
+version: "1.0"
+type: "online"
+
+model_params:
+  name: "Qwen/Qwen3.6-35B-A3B"
+  temperature: 1.0
+  top_k: 20
+  top_p: 0.95
+  repetition_penalty: 1.0
+  presence_penalty: 1.5
+  max_new_tokens: 8192
+  chat_template_kwargs:
+    preserve_thinking: true
+
+datasets:
+  - name: agentic_coding
+    type: performance
+    path: ./agentic_combined.jsonl
+    accuracy_config:
+      eval_method: agentic_inference_inline # required benchmark default.
+    agentic_inference:
+      turn_timeout_s: 14400.0
+      enable_salt: true # do not change.
+      inject_tool_delay: true # do not change.
+  - name: swe_bench # Default PERF skips external evaluation; use ACC or BOTH.
+    type: "accuracy"
+    accuracy_config:
+      eval_method: "swe_bench_scorer"
+      num_repeats: 1
+      extras:
+        swebench_service_url: "http://localhost:18080"
+        num_instances: 200
+        workers: 10
+        max_eval_workers: 10
+        swebench_template: "qwen_tools"
+
+settings:
+  runtime:
+    min_duration_ms: 0
+    max_duration_ms: 36000000
+
+  load_pattern:
+    type: agentic_inference
+    target_concurrency: 8 # Submission-specific concurrency.
+
+endpoint_config:
+  endpoints:
+    - "http://localhost:30000"
+  api_type: openai
+
+report_dir: logs/qwen_agentic

pyproject.toml

@@ -33,6 +33,8 @@ source-exclude = [
     # numpy<2 / prm800k that conflict with the parent env). Invoked only via
     # `uv run --project`, never imported - keep it out of the parent wheel.
     "inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/**",
+    # Isolated native uv service that owns SWE-bench Docker execution.
+    "inference_endpoint/evaluation/swebench_service/**",
 ]
 
 [project]
@@ -94,6 +96,7 @@ dependencies = [
     # Fix pytz-2024 import warning
     "pytz==2026.1.post1",
     "urllib3==2.7.0",
+    "pyyaml==6.0.3",
 ]
 
 [project.optional-dependencies]