Skip to content

Commit 462a7a7

Browse files
Lut-hubLut-hub
committed
TestExplora🚀
1 parent 3842f0a commit 462a7a7

24 files changed

Lines changed: 5058 additions & 3 deletions

‎README.md‎

Lines changed: 120 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,126 @@
11
# TestExplora
22

3-
## Instructions
3+
This repository is the official implementation of the paper "TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation" It can be used for baseline evaluation using the prompts mentioned in the paper.
4+
5+
## Table of Contents
6+
7+
- [What is TestExplora](#what-is-testexplora)
8+
- [Setup](#setup)
9+
- [How to Deploy TestExplora](#how-to-deploy-testexplora)
10+
- [Test Generation (Inference)](#test-generation-inference)
11+
- [Supported Models](#supported-models)
12+
- [Build Benchmark](#build-benchmark)
13+
- [Contributing](#contributing)
14+
- [Trademarks](#trademarks)
415

516
## What is TestExplora
617

7-
## How to deploy TestExplora
18+
TestExplora is a systematic, repository-level benchmark designed to evaluate the capability of Large Language Models to proactively discover latent software defects by generating tests. It was developed to evaluate the proactive defect discovery capabilities of LLMs at the repository level.
19+
20+
Our dataset is constructed from real-world GitHub pull requests, containing 2,389 test-generation tasks sourced from 1,552 PRs across 482 repositories. Each task is designed such that the model must write test cases capable of triggering a Fail-to-Pass transition between buggy and repaired versions – reflecting true defect detection rather than passive confirmation. The benchmark further includes automatically generated documentation for test entry points to enable scalable evaluation.
21+
22+
## Setup
23+
24+
### Prerequisites
25+
26+
- Python 3.10+
27+
- Docker (for local test evaluation)
28+
- Git
29+
30+
### Installation
31+
32+
```bash
33+
git clone https://github.com/microsoft/TestExplora.git
34+
cd TestExplora
35+
```
36+
37+
Install core dependencies:
38+
39+
```bash
40+
pip install -r requirements.txt
41+
```
42+
43+
## How to Deploy TestExplora
44+
45+
### Test Generation (Inference)
46+
47+
The main entry point is `testexplora/harness/inference.py`. Given the benchmark dataset (JSON format), it drives the target LLM to generate test cases for each task and saves the results as test patches.
48+
49+
```bash
50+
python testexplora/harness/inference.py \
51+
--data_path <path_to_data.json> \
52+
--repo_testbed_dir <output_directory> \
53+
--model <model_name> \
54+
--test_type <whitebox|graybox|blackbox> \
55+
```
56+
57+
#### Output
58+
59+
- `test_patches.json` — Generated test patches per repository and PR.
60+
- `config.yaml` — Experiment configuration for reproducibility.
61+
- `generation.log` — Detailed execution log.
62+
- `trajectory/` — Agent trajectory files (for agent-based models).
63+
64+
### Supported Models
65+
66+
The benchmark supports evaluation across a broad set of LLMs and coding agents. To reproduce or customize results for a specific model, modify the corresponding call file under `testexplora/harness/call_pipeline/`.
67+
68+
**API-based Models (Direct LLM Call)**
69+
70+
| Model Key | Call File |
71+
|---|---|
72+
| `gpt-4o`, `o3-mini`, `o4-mini`, `gpt-5-mini`, `gpt-5`, `r1` | `call_gpt.py` |
73+
| `claude_sonnet` | `call_gpt.py` (Anthropic via Azure) |
74+
| `gemini-2.5-pro`, `gemini-2.5-flash` | `call_gemini.py` |
75+
| `Codellama-34B`, `Qwen3-Coder-30B` | `call_vllm.py` |
76+
77+
**Agent-based Models (Agentic Code Exploration)**
78+
79+
| Model Key | Call File |
80+
|---|---|
81+
| `sweagent-*` | `call_sweagent.py` |
82+
| `traeagent-*` | `call_traeagent.py` |
83+
84+
> **Note:** Agent-based models only support `whitebox` test type.
85+
86+
## Build Benchmark
87+
88+
To construct a benchmark dataset similar to TestExplora from your own set of GitHub repositories, use `testexplora/build_benchmark/process_data.py`. It automates the end-to-end pipeline:
89+
90+
1. **Clone repositories** and iterate over closed pull requests.
91+
2. **Checkout the base commit** (pre-PR state) and extract code structure & dependency graphs.
92+
3. **Apply the PR patch**, then re-extract code structure to obtain the post-PR state.
93+
4. **Identify changed functions/methods** by mapping diff line ranges to AST-level code elements.
94+
95+
```bash
96+
python testexplora/build_benchmark/process_data.py
97+
```
98+
99+
> Before running, update the paths at the bottom of `process_data.py` to point to your repository data JSON directory and a local directory for cloning repos.
100+
101+
The script relies on two helper modules under the same directory:
102+
103+
- **`parse_repo.py`** — AST-based extraction of classes, functions, methods, and their metadata from a Python repository.
104+
- **`build_dependency_graph.py`** — Builds inter-function dependency graphs using NetworkX, including cross-file import resolution.
105+
106+
## Contributing
107+
108+
This project welcomes contributions and suggestions. Most contributions require you to agree to a
109+
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
110+
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
111+
112+
When you submit a pull request, a CLA bot will automatically determine whether you need to provide
113+
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
114+
provided by the bot. You will only need to do this once across all repos using our CLA.
115+
116+
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
117+
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
118+
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
119+
120+
## Trademarks
8121

9-
## Evaluation Pipeline
122+
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
123+
trademarks or logos is subject to and must follow
124+
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
125+
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
126+
Any use of third-party trademarks or logos are subject to those third-party's policies.

‎requirements.txt‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
anthropic==0.79.0
2+
asttokens==3.0.0
3+
azure_storage==0.37.0
4+
coverage==7.13.4
5+
docker_py==1.10.6
6+
matplotlib==3.10.8
7+
numpy==2.4.2
8+
openai==2.20.0
9+
PyYAML==6.0.2
10+
PyYAML==6.0.3
11+
Requests==2.32.5
12+
tabulate==0.9.0
13+
tenacity==8.5.0
14+
tiktoken==0.9.0
15+
tqdm==4.67.1
16+
unidiff==0.7.5

0 commit comments

Comments
 (0)