fix: reuse the auto-build forward pass in evaluate

[YoanSallami]

The problem

evaluate() builds an unbuilt program by calling it on the first batch, then throws those predictions away and lets test_on_batch compute them again:

if not all(module.built for module in self._flatten_modules()):
    for _, data in epoch_iterator:
        self._auto_build(iterator=epoch_iterator, data_batch=data[0])   # runs the program
        break
epoch_iterator.reset()                                                   # rewinds to the same batch
...
logs = await self.test_on_batch(x=x_batch, y=y_batch, ...)               # runs it again

Every sample in that first batch is predicted twice. For a deterministic program that's a wasted pass; for an agent program it's a second full agent run — every LM call, tool call and sandbox turn paid twice for a discarded result.

Why it's permanent, not a first-call toll

built is only ever set by Module.__call__. A module on a path the program doesn't take never gets called, so it never becomes built, so all(module.built ...) never becomes true — and every evaluate pays the extra pass, forever.

Agent programs are full of such modules: a fallback generator that only runs when the loop ends without submitting, a tool the model happens not to call. The better the agent behaves, the more reliably the harness pays double.

Measured on a 30-task ARC-AGI-2 benchmark, top-level program invocations came to exactly test_items + 1 for every task:

test items	invocations
1	2
2	3
3	4

Most tasks have a single test item, so that is close to a 2x on wall-clock and tokens. In the traces you can watch one agent submit, the program return, and a fresh top-level run start 17ms later.

The fix

_auto_build returns the predictions it computed; evaluate hands them to the first test_on_batch instead of recomputing them. The build batch is the first batch scored — the iterator is unshuffled and rewound — so they are the same predictions either way.

test_on_batch gains an optional y_pred for this. Nothing else changes, and no path that previously predicted now skips prediction.

Test

test_evaluate_predicts_each_sample_once_when_unbuilt builds a program holding a module its call() never reaches, then asserts each sample is predicted once. Forward passes for 10 samples:

• before: 21
• after: 11 on the first evaluate, 10 on every one after

It also asserts the second evaluate costs exactly one pass per sample while the program is still unbuilt, which is the part that makes the waste recurring.

The residual +1 on the first evaluate is _auto_build's own spec pass for metric and reward state. It is paid once per program rather than once per sample, and is left alone here.

Verification

Full suite: 2171 passed, 36 skipped, 69 subtests (synalinks/src, sandboxes included).

🤖 Generated with Claude Code