feat: swe bench scorer#342
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for SWE-bench accuracy evaluation by adding a new accuracy-only SWEBench dataset, a SWEBenchScorer that runs evaluations using mini-swe-agent in an isolated environment, and associated configuration templates, tests, and runbooks. Feedback on the changes focuses on improving the robustness of the SWEBenchScorer implementation, specifically by safely handling missing or null values when parsing configuration templates, benchmark configurations, and evaluation results, as well as gracefully handling cases where the Docker binary is missing from the system's PATH during preflight checks.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
aa7f559 to
bb9b307
Compare
There was a problem hiding this comment.
Pull request overview
Adds first-class SWE-bench accuracy support to the benchmarking system by introducing an external (subprocess-driven) scorer and a predefined SWE-bench dataset, plus wiring in config/schema handling and examples to run the workflow.
Changes:
- Introduces
SWEBenchScorer(external evaluation viauv run --project ... mini-extra+swebenchharness) and integrates it into the scoring/benchmark lifecycle (preflight, phase skipping, reporting). - Adds predefined
swe_benchdataset with caching + anACCURACY_ONLYguard to prevent using it as a performance dataset. - Updates schema/templates/examples and adds unit tests for the scorer + dataset + benchmark wiring.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Locks pyyaml as a direct dependency. |
| pyproject.toml | Adds pyyaml==6.0.3 to runtime dependencies. |
| src/inference_endpoint/evaluation/scoring.py | Adds scorer preflight hooks, shared subprocess logging helper, and new SWEBenchScorer implementation. |
| src/inference_endpoint/dataset_manager/predefined/swe_bench/init.py | Adds SWE-bench predefined dataset with HF download + parquet cache and ACCURACY_ONLY=True. |
| src/inference_endpoint/dataset_manager/dataset.py | Introduces Dataset.ACCURACY_ONLY class flag. |
| src/inference_endpoint/dataset_manager/init.py | Exposes/imports SWEBench dataset. |
| src/inference_endpoint/config/schema.py | Adds swe_bench_scorer to ScorerMethod and injects concurrency → SWE-bench workers default. |
| src/inference_endpoint/commands/benchmark/execute.py | Runs scorer preflight(), prevents accuracy-only datasets as perf, skips endpoint phases for external scorers, and improves sample counting/reporting. |
| src/inference_endpoint/config/templates/online_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/offline_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/concurrency_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| examples/10_Agentic_Inference/swebench_template.yaml | Adds a SWE-bench/mini-swe-agent config template used by the external scorer. |
| examples/10_Agentic_Inference/swe_bench_accuracy.yaml | Adds a runnable example config for SWE-bench external accuracy evaluation. |
| examples/10_Agentic_Inference/README.md | Documents how to sync/run the SWE-bench accuracy subproject. |
| examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/accuracy/RUNBOOK.md | Adds an ops runbook for validating the SWE-bench accuracy pipeline. |
| examples/10_Agentic_Inference/accuracy/pyproject.toml | Adds isolated uv subproject deps for SWE-bench evaluation tooling. |
| tests/unit/evaluation/test_swe_bench_scorer.py | Adds unit coverage for scorer behavior, config patching, and preflight. |
| tests/unit/dataset_manager/test_swe_bench_dataset.py | Adds unit coverage for dataset registration, caching, and subset mapping. |
| tests/unit/commands/test_benchmark.py | Adds tests for concurrency→workers injection, ACCURACY_ONLY enforcement, preflight propagation, and phase skipping behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
8d66f43 to
ae04494
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #342 +/- ##
=======================================
Coverage ? 81.31%
=======================================
Files ? 143
Lines ? 18981
Branches ? 0
=======================================
Hits ? 15434
Misses ? 3547
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
da45a99 to
fe442f8
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 39 out of 41 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
src/inference_endpoint/evaluation/swebench_service/swebench_service/qwen_tools.py:193
isinstance(view_range, list | tuple)will raiseTypeErrorat runtime becauseisinstancedoesn’t accept PEP604 union types. This breaks thestr_replace_editorview-range path (and should also fail the added unit tests). Use a tuple of types instead.
if command == "view":
view_range = args.get("view_range")
if view_range is not None:
if not isinstance(view_range, list | tuple) or len(view_range) != 2:
errors.append(
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 39 out of 41 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (3)
examples/10_Agentic_Inference/README.md:253
- The note about
TestMode.PERF“skipping accuracy datasets” is incorrect per the current benchmark behavior (e.g._load_datasets(..., TestMode.PERF)still loads accuracy datasets and runs scorer preflight intests/unit/commands/test_benchmark.py:734+). This sentence should instead explain that--mode accis used to skip the performance phase and run only the SWE-bench accuracy workflow.
`--mode acc` is required because `type: online` configs default to
`TestMode.PERF`, which skips accuracy datasets.
src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/swebench_template.yaml:20
- The
<IMPORTANT>tag is closed as</important>(case mismatch). If downstream tooling/prompt parsing depends on these tags, this can break parsing or reduce instruction reliability.
src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/swebench_qwen_tools_template.yaml:20 - Same
<IMPORTANT>/</important>case mismatch as in the default template; worth fixing for consistency and to avoid any tag-sensitive prompt processing issues.
<IMPORTANT>This is an interactive process where you will think and issue AT LEAST ONE command, see the result, then think and issue your next command(s).</important>
| for acc_cfg in accuracy_cfgs: | ||
| scorer_cls, extractor_cls = _resolve_accuracy_components( | ||
| acc_cfg.name, acc_cfg.accuracy_config | ||
| ) | ||
| assert acc_cfg.accuracy_config is not None |
| if ( | ||
| parsed.scheme not in {"http", "https"} | ||
| or not parsed.netloc | ||
| or parsed.hostname is None | ||
| or parsed.path not in {"", "/"} | ||
| or parsed.params | ||
| or parsed.query | ||
| or parsed.fragment | ||
| ): | ||
| raise SetupError(message) | ||
| return parsed._replace(path="/", params="", query="", fragment="").geturl() |
|
@arekay-nv I've addressed the comments. Appreciate if you could take another look |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 40 out of 42 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
src/inference_endpoint/evaluation/swebench_service/swebench_service/qwen_tools.py:193
isinstance(view_range, list | tuple)will raiseTypeErrorat runtime (isinstancedoesn't accept PEP 604 union types). This breaks thestr_replace_editorview_range validation path and will surface as an unhandled exception instead of aFormatError.
Use a normal tuple of types for the check.
if view_range is not None:
if not isinstance(view_range, list | tuple) or len(view_range) != 2:
errors.append(
|
Also reverted a previous change that made perf (default) runs execute accuracy eval when accuracy config is present. Default mode should probably be --mode both and runs whatever config is specified in .yaml file, but that's out of this PR's scope. |
| settings: | ||
| runtime: | ||
| min_duration_ms: 0 | ||
| max_duration_ms: 36000000 | ||
|
|
||
| load_pattern: | ||
| type: agentic_inference | ||
| target_concurrency: 8 # Submission-specific concurrency. |
There was a problem hiding this comment.
client:
warmup_connections: 0
max_idle_time: 0.5
I saw that Kimi's yaml has this warmup connection settings but Qwen does not, from the code, it looks like this will create a HTTP connection that "warms up" the connection from Endpoint client to the LLM server but does not seem to do much in this case. Kimi effectively does not send any request for this. I believe this should be the same for Qwen as well.
There was a problem hiding this comment.
I think this is to bypass some port/connection limit issue that Harshil faced during experiments. I haven't run into this issue from my end, hence not specifying them for Qwen to keep the example yaml minimal.
# Conflicts: # AGENTS.md
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 40 out of 42 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
src/inference_endpoint/commands/benchmark/execute.py:1579
- This get_raw_outputs() call can be triggered for external scorers (SKIP_ENDPOINT_PHASE=True, e.g. SWEBenchScorer) even though they don’t score endpoint outputs. That can cause an unnecessary scan of events.jsonl (often with an empty UUID bound) and noisy warnings in ACC-only external runs where events.jsonl may not exist. Consider skipping the whole response_counts/OSL block for external scorers.
out_df = scorer_instance.get_raw_outputs(accuracy_uuids)
What does this PR do?
Adds SWE-bench accuracy evaluation to the agentic inference workflow (per #310). A new
SWEBenchdataset loadsSWE-bench_Verified/_Lite, andSWEBenchScorerdrives the eval by shelling out tomini-swe-agentand theswebenchharness in an isolated uv subproject, bypassing the built-in accuracy phase.Agent-run parallelism (
extras.workers) defaults to the load pattern'starget_concurrencywhen unset. Includes example configs, an optional Qwen tool-call patch, and docs.Type of change
Related issues
Closes #310
Testing
Checklist