This project implements a custom reinforcement learning environment for a Spot-inspired quadruped robot (designed by myself), built from scratch in PyBullet and compatible with Gymnasium and Stable-Baselines3.
This work is part of my ongoing effort to make the quadruped robot I personally designed and built learn to walk autonomously.
The robot has 12 actuated joints and learns to follow a target twist command (linear and angular velocity) using Proximal Policy Optimization (PPO).
Observations include joint states and simulated IMU data, enabling rich proprioceptive feedback for control learning.
Developed as a personal research project in robotics and embodied AI, this environment focuses on realistic reward shaping, normalized observations, and reproducible training pipelines.
quadruped_rl/
├── envs/
│ └── simple_quadruped_env.py # Custom Gymnasium environment
├── ressources/
│ └── urdfs/spot/spot_v2.urdf # Spot-inspired URDF model
├── models/ # Saved PPO checkpoints
├── train_quadruped.py # Training entry point (CLI)
└── assets/
├── quadruped_env.png
└── reward_curve.png
- 12-Joint Robot — Simulates a Spot-like quadruped with accurate joint limits
- Custom Reward Function — Encourages forward progress, upright stability, and low slip
- Normalized Observations — 33-dimensional observation vector including joint states, IMU, and target twist
- PID control for joint commands — the policy controls PID-assisted legs, which increases stability
- Stable-Baselines3 — Uses the PPO algorithm to learn the control policy
| Component | Count | Description |
|---|---|---|
| Joint positions | 12 | Normalized joint angles |
| Joint velocities | 12 | Normalized joint speeds |
| IMU linear + angular velocity | 6 | Simulated body motion |
| Target twist | 3 | Linear & angular velocity command |
Total: 33-dimensional observation vector
The reward encourages the robot to walk stably toward the commanded direction:
[ r = -w_v |v - v^| - w_\omega |\omega - \omega^| + w_u \cdot \text{uprightness} + w_p \cdot \text{progress} - w_z |v_z| ]
| Term | Description | Weight |
|---|---|---|
| ( w_v ) | Planar velocity tracking | 1.3 |
| ( w_\omega ) | Yaw rate tracking | 1.3 |
| ( w_u ) | Uprightness | 0.3 |
| ( w_p ) | Progress shaping | 20.0 |
| ( w_z ) | Vertical slip penalty | 0.1 |
# Clone the repository
git clone https://github.com/rantaluca/Quadruped_RL.git
cd Quadruped_RL
# Install dependencies
pip install gymnasium pybullet stable-baselines3 numpy matplotlib tqdmYou can launch training with or without GUI visualization:
export PYTHONPATH=.
# Headless mode (faster)
python script/train.py
# With GUI
python script/train.py --gui
# Testing a policy
python script/test.pyOptional arguments:
--timesteps 10000000 # Total training steps (default: 1e9)
--save-freq 20000 # Save model every N steps
The PPO agent learns a stable forward gait, maintaining upright posture while tracking commanded velocities.
Future work includes terrain adaptation, sensor fusion (LIDAR, camera), and sim-to-real transfer to the real DIY robot.
Robert Antaluca
ÉTS Montréal / Université de Technologie de Compiègne
Website: antaluca.com
MIT License — free to use, modify, and extend for research or educational purposes.