Foundations project: train a PPO policy to control a simulated robot (MuJoCo continuous-control benchmarks) with Gymnasium and stable-baselines3, starting from a random policy and ending with one that can actually hop forward.
My other repos (weekend-getaway, AutoBridge-CAD) all orchestrate a pre-trained LLM through an API. No gradient ever gets computed on my own machine in those. This one does: the policy network starts randomly initialized and gets optimized locally, purely from a reward signal coming out of the simulator.
It's not trying to be novel. Gymnasium + MuJoCo + PPO is a combination thousands of people have run before. The point was to get real hands-on time with the training loop, watch a reward curve for once, and see what it actually looks like when training is working versus when it isn't, before building something more original on top of it (see "What's next").
train.py— trains PPO on any Gymnasium environment id, with periodic evaluation and best-checkpoint saving.evaluate.py— loads a saved policy, runs evaluation episodes, and can record video.tests/test_env_smoke.py— sanity checks (env creation, PPO predict/save/load round-trip) that run in a couple seconds, so a broken setup fails immediately instead of an hour into training.
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pytest # confirms MuJoCo's native bindings actually run on this machineStep 1: confirm the loop works. No real physics, done in seconds.
python train.py --env CartPole-v1 --timesteps 50000
python evaluate.py --model models/ppo-CartPole-v1-final.zip --env CartPole-v1 --recordStep 2: the real result. Continuous control, takes longer.
python train.py --env Hopper-v5 --timesteps 200000 --wandb
python evaluate.py --model models/ppo-Hopper-v5-final.zip --env Hopper-v5 --record--wandb streams training curves to Weights & Biases
(free tier); drop it to just watch the stdout table instead. models/,
videos/, and runs/ are gitignored, so the training artifacts below are
surfaced as a GIF and numbers rather than committed binaries.
On macOS you may see harmless objc[...]: Class ... is implemented in both ...cv2... and ...pygame... warnings when recording video. That's opencv
and pygame both bundling their own copy of SDL2. It doesn't affect the
output.
Trained policy, Hopper-v5, 200k timesteps.
- Training curve: wandb run. Eval reward opens around 92, actually drops to ~58-60 between 20k and 40k timesteps (the policy briefly got worse before it got better, a normal part of on-policy training and not a bug), then climbs steadily from 45k timesteps onward, passing 500 around 160k and settling around 938-940 for the final 15k timesteps.
- Held-out evaluation: 5 deterministic episodes after training, mean reward 1098.2 ± 31.1. The low spread across episodes is what tells you the policy converged on a consistent gait instead of getting lucky once.
- CartPole sanity check (before touching MuJoCo at all): 5 episodes, mean reward 489.4 out of a possible 500, most of them hitting the cap. This is what confirmed the training loop itself was correct before spending time on the harder environment.
- Nothing broke during this run. The smoke tests (
pytest) catching a missing dependency (stable-baselines3[extra]for the progress bar) before the real training run started is exactly why they're there.
Stage 1 of a two-stage plan aimed at embodied AI / robotics work. Stage 2 puts a simulated inspection robot in an environment built from AutoBridge-CAD's generated bridge geometry, with a trained vision model detecting surface defects on it: perception and control in the same project, tied to an actual civil-engineering application instead of a generic benchmark. This repo is the ground floor: get the training loop, evaluation, and reward-curve literacy solid before adding vision and a custom environment on top.
All rights reserved, see LICENSE. Public for portfolio purposes only, not open source.