feat(evaluation): report the reasoning-branch intervention delta as an eval metric - #182
feat(evaluation): report the reasoning-branch intervention delta as an eval metric#182gcordova10 wants to merge 1 commit into
Conversation
…n eval metric Wire reasoning_intervention_delta (autowarefoundation#109) into _evaluate_open_loop so every checkpoint reports eval/reasoning_intervention_delta next to ADE/FDE. Additive and a no-op when there is no reasoning head. Thread a fixed initial_noise and the navigation inputs through both forwards so the delta is measured at the real operating point rather than being noise-dominated. Refs autowarefoundation#123, autowarefoundation#109. Signed-off-by: GABRIELA CORDOVA <100548769@alumnos.uc3m.es>
b326080 to
311ceee
Compare
|
Rebased onto main now that #176 has landed, and MERGEABLE again. Both conflicts were additive rather than competing: each side added a keyword to _evaluate_open_loop, and #176 replaced the ADE/FDE computation with the 3-second window plus per-horizon accumulation, which this metric sits alongside rather than duplicates. ruff and mypy clean, 920 passed. The description said I would rebase "so the metric reports inside the composite selector". The rebase makes the two coexist, but it does not put the delta into the selector: the composite score is computed in the per-epoch validation, where this metric is deliberately off. Wiring it there would add a per-epoch cost the current design avoids, so it reports next to ADE/FDE in _run_evaluation instead. Happy to wire it either way if you prefer. On the scoping point: this does not add a reasoning layer or change what is trained — it is skipped when the model has no reasoning head, and the delta is capped at 50 samples, so Reactive-only runs take the path they take today. If you would rather park it until the reasoning layer is picked back up, that is fine by me; the rebase is done either way. |
Problem
reasoning_intervention_delta(added in #109) measures whether the planner actually uses the reasoningbranch: bypass the branch, re-run the same batch, and report how far the trajectory moves. This is the
gate
gdiscussed in #123. But nothing calls it during evaluation, so no checkpoint reports it —today
gis a manual, one-off measurement. #123 asked for the opposite: a number computed percheckpoint that answers "is the reasoning used?" on its own, and rises by itself if the coupling ever
opens with data scale.
One caveat shaped the wiring: the function's forward did not pass the navigation inputs (#161) or a
fixed
initial_noise. So a plain call would measure the delta off the real operating point — and, fora stochastic planner, with different noise in the two runs, which swamps the signal.
Fix
Extend the harness.
reasoning_intervention_delta/horizon_intervention_deltatake**forward_kwargs, threaded identically into both runs — a fixedinitial_noise(so both runs shareit) and the navigation inputs (so the delta is at the real operating point). Additive and
backward-compatible.
Wire it into eval.
_evaluate_open_loopgains areport_interventionflag (default off).evaluate_il_policyturns it on; the training-time validation leaves it off, so training cost isunchanged. When on and the model has a reasoning head, it computes the delta on up to 50 batches
(reusing that batch's stable
initial_noise), averages, and returnsreasoning_intervention_delta;_run_evaluationlogseval/reasoning_intervention_deltanext to ADE/FDE.Guarded. With no reasoning head the metric is skipped, so this is a byte-identical no-op for
non-reasoning checkpoints.
Verification
ruffclean;mypyclean onModel/evaluation/faithfulness.py.Model/tests/test_faithfulness_forward_kwargs.py(new, 3 tests): a plain call is noise-dominated; afixed
initial_noiserecovers only the intervention; the navigation inputs are threaded.test_reasoning_faithfulness.pystays green — 11 passed together.Model/tests/test_workflow_training_lifecycle.py(3 new tests): the metric is reported when opted in,absent without a reasoning head, and absent by default. These reuse the file's existing
flytekit-gated eval harness, so they run in CI alongside the other workflow tests.Coordination
As I flagged in #123, this touches
_evaluate_open_loop, which #176 (feat/rollout-aligned-checkpoint-selection) also restructures. I offered there to base it on that branch or open againstmainand rebase once it lands — opening againstmainhere so it is not blocked, and I will rebase onto #176 when it merges so the metric reports inside the composite selector. Happy to coordinate on the exact wiring point.Refs #123 (the gate this computes) and #109 (the harness it calls).