Skip to content

Reward trend is a fragile two-point endpoint diff #12

Description

@nsursock

Problem

reward_delta_pct is a two-point endpoint diff (training.py:169-172): percent change from first to last finite sample of rollout/ep_rew_mean. RL reward curves are noisy; a single outlier final rollout flips the trend (reward_trend = +1/-1/0) and, transitively, the reward_delta_pct plausibility check.

Suggested fix

  • Report a robust slope (Theil-Sen or MA-anchored mean-of-first/last-N) alongside the endpoint value.
  • Keep the endpoint value for API compatibility but do not let it alone drive the verdict; add a CUSUM or at least an MA-delim check.

Impact

Agent-health "improving/degrading" conclusion is high-variance and can flip on one rollout.

Ref: src/dirty_fin_reports/simple/training.py:168-186; report.py:117-123

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions