GrowthHacker is the first benchmark for evaluating LLM and LLM-based agents on automated Off-Policy Evaluation (OPE) code optimization. It autonomously modifies code, runs OPE, and uses the resulting metrics to guide subsequent optimization rounds — achieving 98.1–100% success rate with the
two_agentframework vs. 89.5% for standalone LLMs.
Results from 648 experimental runs across 18 notebooks × 4 frameworks × 3 code-modification methods × 3 repetitions:
| Framework | Success Rate | Positive-Outcome Rate | Median Improvement (positive outcomes) | Zero Extreme-Value Failures |
|---|---|---|---|---|
| two_agent (default) | 98.1–100% | 78% | +4.4% | Yes |
| CrewAI | 90.1% | 63% | +1.6% | Yes |
| AutoGen | 92.6% | 65% | +1.7% | No |
| Default (LLM only) | 89.5% | 65% | +1.9% | No |
—= not reported for that framework in the paper. For full per-notebook breakdowns see Tables 1–7 in the paper.
two_agentachieves the highest reliability and positive-outcome rate; CrewAI achieves the highest average improvement among successful runs.
For full results see Tables 1–7 in the paper.
git clone https://github.com/jie-jw-wu/ope-agent.git
cd ope-agent
bash install.sh # macOS / Linux
# install.bat # Windowsinstall.sh creates all three conda environments, auto-detects Python interpreter paths, and writes them into scripts/config.yml — no manual YAML editing required.
No conda? Install Miniconda (recommended) or use the manual step-by-step setup in the Full Setup Guide below.
Already have
env38/env39_scope?install.shdetects legacy environment names and reuses them automatically — no duplicate environments created.
Option A — .env file (recommended):
echo "GEMINI_API_KEY=your_key_here" > .envOption B — Edit scripts/API_KEYS (created by install.sh from the template):
[{"model": "gemini-2.5-flash", "api_key": "your_key_here", "api_type": "google"}]Get a free Gemini API key at aistudio.google.com. OpenAI and Mistral keys are also supported.
conda activate growth-hacker-agent
python start.py notebooks/obd.ipynb gemini-2.5-flash -fw two_agent -n 1Output is saved to runs/{timestamp}/. A single iteration takes 5–15 minutes.
The table below maps each key paper result to a runnable command. All commands assume conda activate growth-hacker-agent.
| Paper Result | Dataset | Command | Est. Runtime |
|---|---|---|---|
| Table 3 — two_agent on OBD | OBP | python start.py notebooks/obd.ipynb gemini-2.5-flash -fw two_agent -n 7 |
~1 hour |
| Table 3 — Default LLM on OBD | OBP | python start.py notebooks/obd.ipynb gemini-2.5-flash -n 1 |
~15 min |
| Table 3 — CrewAI on OBD | OBP | python start.py notebooks/obd.ipynb gemini-2.5-flash -fw crewai -n 1 |
~15 min |
| Table 5 — RTB Advanced, two_agent | Scope-RL | python start.py notebooks/rtb/rtb_synthetic_discrete_zoo.ipynb gemini-2.5-flash -fw two_agent -n 7 |
~1 hour |
| Full paper benchmark (648 runs) | Both | python tests/enhanced_test_runner.py |
~40 hours |
Scope-RL notebooks use pre-computed artifacts in
artifacts/(TRAINING_BYPASS mode) — no long RL training required. See datasets below.
Cost estimate: Each Gemini-2.5-Flash call costs ~$0.001–0.01. A 7-iteration two_agent run on one notebook uses ~20–50 API calls.
OBP notebooks use the Open Bandit Dataset — real Zoztown fashion e-commerce data (product positions, user clicks, contextual features).
# Download OBD (~200 MB)
bash data/download_obd.shSee data/README.md for details. synthetic.ipynb and multiclass.ipynb generate their own data and require no download.
Scope-RL notebooks use synthetically generated datasets and pre-trained RL policies. No external dataset download is required, but to skip expensive RL training (30 min–2 hours per notebook) you should download the pre-computed artifacts:
bash data/download_artifacts.shThis populates artifacts/ and enables TRAINING_BYPASS mode automatically on first run. Without this step, the first run trains RL policies from scratch and saves artifacts for subsequent runs.
artifacts/
├── basic/ ← 6 notebooks: discrete/continuous, Basic/Zoo/Advanced
├── rec/ ← 3 notebooks: discrete recommendation scenarios
└── rtb/ ← 6 notebooks: real-time bidding, discrete/continuous
The tool uses three separate Python environments due to library constraints:
| Environment | Python | Purpose |
|---|---|---|
growth-hacker-agent |
3.10+ | Run start.py and all agent frameworks |
growth-hacker-obp |
3.8 | Execute OBP notebooks |
growth-hacker-scoperl |
3.9 | Execute Scope-RL notebooks |
install.sh creates all three from the bundled .yml files. The tool automatically selects the correct interpreter for each notebook.
Click to expand
Step 1: OBP environment (Python 3.8)
conda env create -f environment-obp.ymlStep 2: Scope-RL environment (Python 3.9)
conda env create -f environment-scoperl.ymlStep 3: Agent environment (Python 3.10+)
conda env create -f environment-agent.yml
conda activate growth-hacker-agent
pip install -e .Step 4: Configure interpreter paths
Edit scripts/config.yml — or set environment variables:
export GROWTH_HACKER_OBP_PYTHON=$(conda run -n growth-hacker-obp which python)
export GROWTH_HACKER_SCOPE_PYTHON=$(conda run -n growth-hacker-scoperl which python)Step 5: API keys — see Quick Start Step 2 above.
Docker support is not yet implemented. If you need a containerized setup, please open a GitHub issue — contributions welcome.
| Error | Solution |
|---|---|
Building wheel for PyYAML failed |
Use conda env create -f environment-obp.yml instead of pip directly |
ModuleNotFoundError: No module named 'nbformat' |
Activate growth-hacker-agent before running start.py |
No API key found for model '...' |
Create .env with GEMINI_API_KEY=... or edit scripts/API_KEYS |
${GROWTH_HACKER_OBP_PYTHON} appears in errors |
Re-run bash install.sh or set env vars manually |
ModuleNotFoundError: No module named 'scope_rl' |
Run conda activate growth-hacker-scoperl && pip install scope-rl |
RuntimeError: CUDA error: no kernel image is available |
PyTorch 2.0.0 doesn't support your GPU's CUDA version. Run with CUDA_VISIBLE_DEVICES="" python start.py ... to use CPU, or reinstall PyTorch: conda run -n growth-hacker-scoperl pip install torch --index-url https://download.pytorch.org/whl/cu121 |
python start.py <notebook_path> <model> [-fw <framework>] [-n <iterations>] [-opt <method>] [-pv <variant>]Or via the installed CLI:
growth-hacker <notebook_path> <model> [options]Arguments:
| Argument | Options | Default | Description |
|---|---|---|---|
model |
gemini-2.5-flash, gemini-2.5-flash-lite, gpt-4o, gpt-4o-mini, mistral-large-latest |
gemini-2.5-flash |
LLM to use |
-fw |
two_agent, crewai, autogen, random_search |
none (standalone LLM) | Agent framework |
-n |
integer | 1 |
Number of outer optimization iterations |
-ni |
integer | 1 |
Internal iterations per outer call (two_agent only) |
-opt |
whole_code, manual_patch, agent_applies |
whole_code |
Code modification method (use whole_code for highest reliability) |
-pv |
v1, v2 |
v1 |
Prompt variant: v1 = hyperparameter tuning, v2 = algorithmic/structural |
-a |
flag | — | Run all model × framework combinations |
Example commands:
# Most reliable: two_agent with 7 iterations (matches paper default)
python start.py notebooks/obd.ipynb gemini-2.5-flash -fw two_agent -n 7
# Quick single-iteration test
python start.py notebooks/multiclass.ipynb gemini-2.5-flash -n 1
# Scope-RL notebook (discrete actions)
python start.py notebooks/basic/basic_synthetic_discrete_zoo.ipynb gemini-2.5-flash -fw two_agent -n 1
# Algorithmic variant prompt
python start.py notebooks/obd.ipynb gemini-2.5-flash -fw two_agent -n 3 -pv v2
# All frameworks × all models (comprehensive benchmark)
python start.py notebooks/ gemini-2.5-flash -aPerformance notes:
- Single iteration: 5–15 minutes typical
- 7 iterations (two_agent default): ~1 hour
- Advanced Scope-RL notebooks: up to 2 hours for FULL_RUN (artifacts not pre-computed)
LLM / Agent ──► Modified Code ──► Execute OPE ──► Metric Better? ──► Iterate
(OBP / Scope-RL) │
└──► Save best result
The two_agent framework separates concerns into two specialized agents to eliminate the compilation errors and context degradation observed in CrewAI and AutoGen:
- Prompter/Analyzer Agent — examines the original code, identifies optimization opportunities, writes structured instructions to
instruction_i.md - Coder Agent — reads
instruction_i.md+ original code, implements changes, writesnewcode_i.py - Each of 7 iterations starts independently from the original baseline (no sequential dependency → no context accumulation)
- A lightweight LLM selects the best-performing iteration post-hoc from
results.txt
Why this matters for reliability:
- 0% failure rate in the default two_agent configuration (vs. 7.4–10.5% for other frameworks)
- File-based agent communication provides full transparency and debuggability
- Independent iterations avoid the context degradation that caused AutoGen/CrewAI failures
runs/
└── {timestamp}/
├── 0-{notebook}.py # Original baseline
├── 1-{notebook}.py # Iteration 1 modified code
├── ...
├── agent_output.log # Full LLM/agent interaction log
├── {notebook}_results.csv # OPE metrics per iteration
└── diff.txt # Code diff per iteration
For two_agent runs, runs/organized_results/ also contains:
R-doc/Instructions*.md— Analyzer agent instructions per iterationR-doc/newcode*.py— Generated code per iterationResults/final_summary.md— Best iteration selection with metrics
Each row is one iteration (row 0 = baseline, row 1 = iteration 1, etc.). Key columns:
| Column | Meaning |
|---|---|
relative_error |
Primary OPE metric — lower is better |
mse |
Mean squared error of OPE estimators |
iteration |
Iteration number (0 = unmodified baseline) |
Interpreting improvement: Compare relative_error for each iteration against row 0 (baseline). A lower value means the agent's code modification improved OPE accuracy. Percentage improvement = (baseline − iteration) / baseline × 100.
For standalone LLM runs, scan the CSV for the row with the lowest relative_error.
For two_agent runs, Results/final_summary.md does this automatically — it selects the best-performing iteration and explains which hyperparameter changes drove the improvement.
Full transcript of every LLM call, including prompts sent and code generated. Useful for debugging or understanding why the agent made a particular change.
If relative_error in every iteration equals the baseline (no improvement), check agent_output.log for API key errors or code execution failures. A successful run should show at least one iteration with a different value.
| Name | Role | Affiliation |
|---|---|---|
| Jie JW Wu | Co-first author | Michigan Technological University |
| Ayanda Patrick Herlihy | Co-first author | Birmingham City University |
| Ahmad Saleem Mirza | Author | University of British Columbia |
| Ali Afoud | Author | University of British Columbia |
| Fatemeh Fard | Author | University of British Columbia |
If you use GrowthHacker in your research, please cite:
@article{wu2026growthhacker,
title = {GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents},
author = {Wu, Jie JW and Herlihy, Ayanda Patrick and Mirza, Ahmad Saleem and Afoud, Ali and Fard, Fatemeh},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
doi = {10.5281/zenodo.17496869},
url = {https://arxiv.org/abs/2511.00802}
}The replication package (code + data) is permanently archived at Zenodo: doi:10.5281/zenodo.17496869
See CONTRIBUTING.md for how to add new LLM adapters, OPE notebooks, or agent frameworks.
Issues and PRs are welcome. Please open a GitHub issue with the full error message and relevant portion of runs/*/agent_output.log.
- Open Bandit Pipeline (OBP) — OPE library used for bandit notebooks
- Scope-RL — OPE library used for RL notebooks
- pyIEOE — prior tool for interpretable OPE evaluation (requires manual hyperparameter specification; GrowthHacker automates this)