Supplementary materials for analyzing Automated Program Repair (APR) agents on SWE-bench.
# Install Python 3.9.0 (using pyenv)
pyenv install 3.9.0
pyenv local 3.9.0
# Create and activate virtual environment
python -m venv venv
source venv/bin/activate
# Install dependencies
pip install --upgrade pip
pip install -r requirements.txt
# Set up API keys
export ANTHROPIC_API_KEY="your-api-key-here"
export OPENAI_API_KEY="your-api-key-here"Agent trajectories are available on HuggingFace:
# Make sure git-lfs is installed
git lfs install
# Clone the repository
git clone https://huggingface.co/datasets/zt6c3mxv8q/logs_and_trajs| File | Description |
|---|---|
| T1_solve_rates.md | Solve rates by agent, difficulty, and backbone |
| T2_unique_counts.md | Unique solution counts per agent (Venn analysis) |
| T3_agent_strengths.md | Agent unique strengths by bug type (LLM-derived) |
| T4_clone_detection_similarity.md | Clone detection and patch similarity by agent/difficulty |
| T5_gpt4o_hunk_stats.md | Hunk-level patch statistics (GPT-4o backbone) |
| T6_overfit_rates.md | Overfitting rates by agent and difficulty |
| T7_pass_rate_by_tests.md | Pass rates stratified by number of tests |
| T8_cohens_k.md | Cohen's kappa validation for clone classification |
| T9_error_diversity.md | Error diversity across temperatures |
| File | Description |
|---|---|
| RQ1_motivating_ex.pdf | RQ1 motivating example |
| rq1_patch_analysis_full_table.pdf | Full patch analysis table |
| nebius-50-annotations.pdf | Nebius-50 manual annotation results |
| localization_full.pdf | Full localization analysis |
| samp_vs_temp.pdf | Sampling vs temperature analysis |
| Analysis | README |
|---|---|
| Full pipeline instructions | submission/README.md |
| Reproducer analysis (overfitting, issue reproduction) | submission/code/reproducer_analysis/README.md |
| Reproduction test localization (inference scaling) | submission/code/reproduction_test_localization/README.md |
| Error diversity analysis (temperature effects) | submission/code/reproduction_test_localization/reproduction_test_error_diversity/README.md |
| Patch analysis (hunk stats, edit patterns) | submission/code/patch_analysis_edit_hunks/README.md |
- Clone detection results:
submission/code/backbone_controlled_analysis/data/results/ - Venn diagrams:
submission/code/backbone_controlled_analysis/data/venn_diagrams/ - Reproducer analysis:
submission/code/reproducer_analysis/results/