Policy-Optimizations-for-Cartpole

This repository contains a modular, from‑scratch implementation of several state‑of‑the‑art policy optimisation algorithms, applied to the classic CartPole balancing task. The code is written in Python using PyTorch and provides a clean, educational framework for understanding:

• Trust Region Policy Optimization (TRPO) - with both L-BFGS and Conjugate Gradient solvers.
• Proximal Policy Optimization (PPO) - with the clipped surrogate objective.
• Group Relative Policy Optimization (GRPO) - a recent method popularized in the LLM reasoning community, which eliminates the need for a value network.

All algorithms share the same environment and training loop, making it easy to compare their performance, convergence speed, and sensitivity to hyperparameters.

TRPO + CG ~600 episodes	TRPO + L-BFGS ~1500 episodes
GRPO + Adam ~300 episodes	GRPO + L‑BFGS ~700 episodes
PPO + Adam ~500 episodes

---

Features

• 🚀 Custom CartPole environment with a Gym‑like API and Pygame rendering.
• 🧠 Modular policy optimizers - plug in different algorithms and solvers.
• 📊 Empirical comparison - see how each method performs on the same task.
• 🎮 Human play mode and agent demonstration - watch your trained policy in action.
• 🤖 Multi‑agent ensemble - combine multiple policies via voting.

---

Usage

Train an Agent:

python cartpole.py agent_train

This will train the currently uncommented algorithm in main(). To switch algorithms, uncomment the desired policy_optimizer block in cartpole.py (see the file for examples).

Training logs are printed every 100 episodes, and the policy weights are saved to pretrained/.

Play with a Trained Agent:

python cartpole.py agent

You will be prompted to enter the name of the saved policy (e.g., ppo_adam). The agent will then run the CartPole environment and you can watch it balance the pole.

Play as a Human:

python cartpole.py human

Use the LEFT/A and RIGHT/D keys to push the cart. Press R to restart after a failure.

Multi‑Agent Voting:

python cartpole.py multiagent

This loads all .pth files from the pretrained/ folder, lets each policy vote on an action, and takes the majority vote. Useful for ensemble robustness.

---

Results on CartPole

All algorithms were trained for 5000 episodes (max 500 steps each) and evaluated on convergence to the maximum reward of 500. The table below shows the number of episodes required to reliably achieve 500 steps (averaged over 5 seeds).

Method	Episodes to Converge	Notes
TRPO + L‑BFGS	~1500	Sensitive to initialization; often needs multiple runs to escape local minima.
TRPO + CG	~600	Faster and more stable than L‑BFGS.
PPO + Adam	~500	Robust and widely used.
GRPO + Adam	~300	Fastest; no value network, uses group‑relative advantages.
GRPO + L‑BFGS	~700	Slower than Adam; also sensitive to initialisation.

---

Requirements

• Python 3.8+
• PyTorch
• NumPy
• Pygame
• SciPy (only for the multi‑agent mode)

Install with:

pip install -r requirements.txt