IRO

Inverse rubric optimization (IRO): budgeted hidden-judge testbeds for agent science. This is the minimal codebase behind the experiments in Inverse Rubric Optimization: A testbed for agent science.

In an IRO task, an optimizer agent must learn the preferences of a black-box judge that scores generated samples against a hidden rubric. The agent iterates on a generation policy — e.g. a generation prompt — using a limited budget of judge labels, then submits its best policy for a held-out evaluation. The primitives here (budgeted environments, a generic runner, and a prompt-policy optimizer agent) are domain-agnostic; this repo ships one concrete instance, IRO-Poetry, the setting used in the post.

IRO-Poetry

• the optimizer agent (a ramure agent, e.g. Claude Opus 4.6 or GPT-5.5) iterates on a generation prompt;
• the generator (Claude Haiku 4.5) writes one poem per assignment (topic + form) using that prompt as its system prompt;
• the judge (Claude Opus 4.6) scores each poem 1–10 against a hidden poet-style feature rubric (Milton, Pope, Whitman, Byron, or Browning — see rubrics/);
• each scored poem costs one label from the run's budget.

Scores are normalized per judge against two baselines measured by a probe: generation with no rubric knowledge (blind_mean, normalized 0) and generation with the rubric visible to the generator (rubric_visible_mean, normalized 1).

Requirements

• Python 3.11+ and uv
• API keys for the models you use (ANTHROPIC_API_KEY, OPENAI_API_KEY) in .env
• Docker for the default sandboxed agent runs; the agent image is built from docker/Dockerfile (it packages pi, the agent harness ramure drives, plus tmux and an agent user)
• For --sandbox local runs instead (agent gets bash on your machine): tmux and pi installed on the host (npm i -g @earendil-works/pi-coding-agent)

Setup

uv sync                      # installs litellm + ramure
cp .env.example .env         # add ANTHROPIC_API_KEY / OPENAI_API_KEY
docker build -t iro-agent docker/   # agent sandbox image (skip for --sandbox local)

Generator and judge calls go through LiteLLM, so any LiteLLM-supported model name works. The optimizer agent runs on ramure and needs credentials for its own model as well.

By default run_sweep.py sandboxes the optimizer agent in Docker (--sandbox docker --docker-image iro-agent:latest), matching how the post's experiments were run. --sandbox local runs the agent directly on your machine — convenient for smoke tests, but the agent gets an unsandboxed shell.

Running the experiments

1. Measure baselines (one probe per judge; ~20 eval topics × 2 samples × 2 conditions each):

   python scripts/make_case_manifest.py --out runs/case_manifest.json

2. Run a sweep. One run = one (judge, budget, seed) triple; the optimizer agent gets a sandbox workspace and train_eval / submit_final tools:

   python scripts/run_sweep.py --root runs/sweep \
       --optimizer-model anthropic/claude-opus-4-6 \
       --budgets 10 100 1000 10000 --seeds 0 1 2

   The blog post's comparison used anthropic/claude-opus-4-6, openai/gpt-5.5, and anthropic/claude-fable-5 as optimizer models, with budgets 10/100/1000/10000 and seeds 0–2 (15 runs per budget per model).

3. Aggregate:

   python scripts/aggregate.py --root runs/sweep

   prints normalized score and budget use per budget, and writes per-run rows to runs/sweep/run_scores.csv.

What's in a run directory

runs/sweep/case00_budget1000_seed0/
├── report.json                 # config, final score, per-call budget log
├── run_state.json              # live status while running
├── optimizer_trajectory.json   # prompts tried, scores, file pointers
├── run.log
└── workspace/                  # the agent's sandbox
    ├── README.md               # task description shown to the agent
    ├── sample_assignment.txt
    └── eval_feedback/          # full per-poem rows for every train_eval

Layout

iro/environment.py    # Budget / Environment / EvalResult primitives (general)
iro/llm.py            # generator-model helper (LiteLLM, dry-run support)
iro/agents/optimizer.py  # ramure optimizer agent for any prompt-policy env
iro/envs/poetry.py    # the poetry IRO instance
eval.py               # generic runner: one agent on one budgeted env
scripts/rubric_probe.py       # blind vs rubric-visible baseline probe
scripts/make_case_manifest.py # probe all judges -> case_manifest.json
scripts/run_sweep.py          # poetry sweep: cases x budgets x seeds
scripts/aggregate.py          # normalized scores table + CSV
data/assignments/     # poem topics/forms with train/test split
rubrics/              # the five hidden poet-style feature rubrics

A note on contamination

The five rubrics in rubrics/ are published here, so they are no longer hidden in any strong sense: future models (or any agent with web access) may know them. Treat results on these specific judges as comparable to the post's numbers, not as a clean benchmark going forward. Making fresh judges is cheap: pass poet_style="<any poet>" for an auto-generated style judge, or write a new feature rubric and point judge_rubric_file at it, then re-run make_case_manifest.py to calibrate baselines.

Defining a new IRO instance

A new domain is one file under iro/envs/: subclass Environment, spend budget in train mode, score candidates with a judge of your choice, and expose a factory like make_env(budget=..., **kwargs). Anything whose candidates are prompt strings works with the existing optimizer agent and eval.py unchanged; see iro/envs/poetry.py for the reference implementation.