Fix: reuse training obs normalization stats in ppo_continuous_action eval

[discobot]

Description

Fixes #532.

This is the observation-normalization gap @noahfarr and @pseudo-rnd-thoughts identified in the issue, confirmed at the code level: the save block in ppo_continuous_action.py writes only agent.state_dict(), and cleanrl_utils.evals.ppo_eval.evaluate() rebuilds the envs through make_env with a fresh RunningMeanStd (mean=0, var=1). After a 300k-step HalfCheetah-v4 run the training obs_rms.var reaches ~119 on some dimensions, so the policy evaluates on a completely different observation scale than it trained on. Same root cause as #425.

The fix saves the obs_rms mean/var/count alongside the weights (same model_data dict layout as c51.py), and evaluate() restores them onto the eval envs and freezes updates, so eval uses exactly the training-time scaling. Old checkpoints that are a plain state_dict — including the existing hub models — still load through a legacy path, and training behavior is unchanged (identical training logs before/after on the same seed).

One detail that explains the seed-dependent severity in the issue: gymnasium 0.29.1's NormalizeObservation keeps updating the fresh eval statistics during evaluation, so the error partially self-corrects over long episodes — the wildly negative returns mostly show up on longer (1M+ step) runs.

Validation: A/B on the same HalfCheetah-v4 checkpoint (stats stripped vs restored, 10 episodes) gives eval mean ~822 vs ~872, with the fixed eval matching the run's final training returns (~870 vs ~860). Added to tests/test_classic_control_gymnasium.py (already run by CI): a save+eval smoke test mirroring test_c51_eval, a regression test that synthetic stats round-trip through the checkpoint into the eval wrapper, and a legacy-checkpoint test.

Types of changes

• Bug fix
• New feature
• New algorithm
• Documentation

Checklist:

• I've read the CONTRIBUTION guide (required).
• I have ensured pre-commit run --all-files passes (required).
• I have updated the tests accordingly (if applicable).
• I have updated the documentation and previewed the changes via mkdocs serve.
  • I have explained note-worthy implementation details.
  • I have explained the logged metrics.
  • I have added links to the original paper and related papers.

If you need to run benchmark experiments for a performance-impacting changes:

• I have contacted @vwxyzjn to obtain access to the openrlbenchmark W&B team.
• I have used the benchmark utility to submit the tracked experiments to the openrlbenchmark/cleanrl W&B project, optionally with --capture_video.
• I have performed RLops with python -m openrlbenchmark.rlops.
  • For new feature or bug fix:
    • I have used the RLops utility to understand the performance impact of the changes and confirmed there is no regression.
  • For new algorithm:
    • I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • I have added the learning curves generated by the python -m openrlbenchmark.rlops utility to the documentation.
  • I have added links to the tracked experiments in W&B, generated by python -m openrlbenchmark.rlops ....your_args... --report, to the documentation.

[vercel]

@discobot is attempting to deploy a commit to the Costa Huang's projects Team on Vercel.

A member of the Team first needs to authorize it.