Skip to content

feat(evaluation): report the reasoning-branch intervention delta as an eval metric - #182

Open
gcordova10 wants to merge 1 commit into
autowarefoundation:mainfrom
gcordova10:feat/gate-eval-metric
Open

feat(evaluation): report the reasoning-branch intervention delta as an eval metric#182
gcordova10 wants to merge 1 commit into
autowarefoundation:mainfrom
gcordova10:feat/gate-eval-metric

Conversation

@gcordova10

Copy link
Copy Markdown
Contributor

Problem

reasoning_intervention_delta (added in #109) measures whether the planner actually uses the reasoning
branch: bypass the branch, re-run the same batch, and report how far the trajectory moves. This is the
gate g discussed in #123. But nothing calls it during evaluation, so no checkpoint reports it
today g is a manual, one-off measurement. #123 asked for the opposite: a number computed per
checkpoint
that answers "is the reasoning used?" on its own, and rises by itself if the coupling ever
opens with data scale.

One caveat shaped the wiring: the function's forward did not pass the navigation inputs (#161) or a
fixed initial_noise. So a plain call would measure the delta off the real operating point — and, for
a stochastic planner, with different noise in the two runs, which swamps the signal.

Fix

  • Extend the harness. reasoning_intervention_delta / horizon_intervention_delta take
    **forward_kwargs, threaded identically into both runs — a fixed initial_noise (so both runs share
    it) and the navigation inputs (so the delta is at the real operating point). Additive and
    backward-compatible.

  • Wire it into eval. _evaluate_open_loop gains a report_intervention flag (default off).
    evaluate_il_policy turns it on; the training-time validation leaves it off, so training cost is
    unchanged
    . When on and the model has a reasoning head, it computes the delta on up to 50 batches
    (reusing that batch's stable initial_noise), averages, and returns reasoning_intervention_delta;
    _run_evaluation logs eval/reasoning_intervention_delta next to ADE/FDE.

  • Guarded. With no reasoning head the metric is skipped, so this is a byte-identical no-op for
    non-reasoning checkpoints.

Verification

  • ruff clean; mypy clean on Model/evaluation/faithfulness.py.
  • Model/tests/test_faithfulness_forward_kwargs.py (new, 3 tests): a plain call is noise-dominated; a
    fixed initial_noise recovers only the intervention; the navigation inputs are threaded.
    test_reasoning_faithfulness.py stays green — 11 passed together.
  • Model/tests/test_workflow_training_lifecycle.py (3 new tests): the metric is reported when opted in,
    absent without a reasoning head, and absent by default. These reuse the file's existing
    flytekit-gated eval harness, so they run in CI alongside the other workflow tests.

Coordination

As I flagged in #123, this touches _evaluate_open_loop, which #176 (feat/rollout-aligned-checkpoint-selection) also restructures. I offered there to base it on that branch or open against main and rebase once it lands — opening against main here so it is not blocked, and I will rebase onto #176 when it merges so the metric reports inside the composite selector. Happy to coordinate on the exact wiring point.

Refs #123 (the gate this computes) and #109 (the harness it calls).

@riita10069

Copy link
Copy Markdown
Collaborator

#168 (comment)

…n eval metric

Wire reasoning_intervention_delta (autowarefoundation#109) into _evaluate_open_loop so every checkpoint
reports eval/reasoning_intervention_delta next to ADE/FDE. Additive and a no-op when
there is no reasoning head. Thread a fixed initial_noise and the navigation inputs
through both forwards so the delta is measured at the real operating point rather than
being noise-dominated.

Refs autowarefoundation#123, autowarefoundation#109.

Signed-off-by: GABRIELA CORDOVA <100548769@alumnos.uc3m.es>
@gcordova10
gcordova10 force-pushed the feat/gate-eval-metric branch from b326080 to 311ceee Compare August 11, 2026 08:49
@gcordova10

gcordova10 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main now that #176 has landed, and MERGEABLE again. Both conflicts were additive rather than competing: each side added a keyword to _evaluate_open_loop, and #176 replaced the ADE/FDE computation with the 3-second window plus per-horizon accumulation, which this metric sits alongside rather than duplicates. ruff and mypy clean, 920 passed.

The description said I would rebase "so the metric reports inside the composite selector". The rebase makes the two coexist, but it does not put the delta into the selector: the composite score is computed in the per-epoch validation, where this metric is deliberately off. Wiring it there would add a per-epoch cost the current design avoids, so it reports next to ADE/FDE in _run_evaluation instead. Happy to wire it either way if you prefer.

On the scoping point: this does not add a reasoning layer or change what is trained — it is skipped when the model has no reasoning head, and the delta is capped at 50 samples, so Reactive-only runs take the path they take today. If you would rather park it until the reasoning layer is picked back up, that is fine by me; the rebase is done either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants