This project is a complete, full-stack implementation of the classic board game Monopoly. It features a modern, responsive web frontend built with React, and a powerful Reinforcement Learning (RL) backend built with Python, FastAPI, and PyTorch.
The highlight of the project is the AI bots: agents trained using a combination of Behavioral Cloning (BC) and Proximal Policy Optimization (PPO) to play a deeply strategic game against human players.
The project is split into two distinct halves that communicate via a REST API:
- Located in the root directory.
- Built using Vite, React, and TailwindCSS.
- Game state is managed by a massive, robust Zustand store (
gameStore.ts). - Handles all UI animations, dice rolls, card reveals, player balances, and human interaction.
- Built in Python using FastAPI, PyTorch, and Stable-Baselines3.
- Contains a standalone, headless python
MonopolyEnginethat perfectly simulates the board game rules without a UI. - The FastAPI server (
server/main.py) exposes a/actendpoint. - The Bridge: When it is a bot's turn, the React frontend sends the entire Zustand JSON state to the backend. The backend reconstructs the
MonopolyEnginelocally, calculates exactly which actions are legal, and uses the trained PPO model to select the best move.
Training an RL agent to play Monopoly is notoriously difficult due to the massive observation space, delayed rewards, and strictly zero-sum multi-agent mechanics. We used a two-stage approach:
- We initially used a Large Language Model to play thousands of games against itself.
- We recorded the board states and the actions the LLM chose.
- We trained a baseline neural network (
models/bc/best_model.pt) using supervised learning to simply mimic the LLM's understanding of the game. - This gave the agent a foundational understanding of basic logic (e.g., buying properties is usually good, passing is usually bad) without having to stumble blindly through random exploration.
- We initialized a PPO (Proximal Policy Optimization) model with the weights from the BC model.
- We placed the agent in a custom Gymnasium environment (
MonopolyEnv) where it played millions of steps of self-play. - Reward Shaping: To prevent the agent from playing a "cowardly" or "pacifist" strategy (just walking around the board refusing to buy properties to avoid bankruptcy), we implemented a multi-agent reward function:
- The agent is penalized
-0.005points for every step it takes (to discourage stalling). - The agent receives dense positive rewards whenever an opponent pays it rent or goes bankrupt.
- The agent is penalized
- This forced the agent to become an aggressive capitalist, buying properties and building houses to bankrupt its opponents!
To play the game against the RL bots, you must run both the Python backend and the React frontend simultaneously.
The backend requires Python and PyTorch. It hosts the FastAPI model server.
# Navigate to the backend directory
cd monopoly-rl
# Install dependencies (if you haven't already)
pip install -r requirements.txt
# Start the FastAPI server on port 8765
python server/main.pyNote: Ensure your trained models are located in monopoly-rl/models/ppo/best_model.zip or monopoly-rl/models/bc/best_model.pt.
Open a new terminal window in the root directory of the project.
# Install dependencies
npm install
# Start the Vite development server
npm run dev- Open your browser to
http://localhost:5173. - Click Local Multiplayer.
- In the lobby, use the 👤 HUMAN / 🤖 BOT toggle to set up your game. You can play 1v1 against a bot, 1v3, or even have 6 bots play against each other while you watch!
- Click Start Game and enjoy!
| Slice | Dims | Description |
|---|---|---|
| [0:40] | 40 | Property ownership: −1 opponent, 0 unowned, 1 self |
| [40:80] | 40 | House/hotel count per property (0–5) |
| [80:120] | 40 | Mortgage status per property (0/1) |
| [120:124] | 4 | Each player's normalised cash balance (÷1500) |
| [124:128] | 4 | Each player's board position (÷40) |
| [128:132] | 4 | Each player's jail status (0/1) |
| [132:136] | 4 | Each player's jail-free-card count |
| [136:176] | 40 | Per-property rent owed if landed on (normalised) |
| [176:180] | 4 | Number of complete colour sets owned per player |
| [180:184] | 4 | Each player's bankruptcy status (0/1) |
| [184:188] | 4 | Number of properties owned per player |
| [188:192] | 4 | Rounds remaining (normalised, same for all players) |
| [192:214] | 22 | Current turn context (phase, dice values, pending trade, etc.) |
| ID | Action |
|---|---|
| 0 | Roll dice / end turn |
| 1 | Buy current property |
| 2–41 | Build house on property 0–39 |
| 42 | Sell house (cheapest) |
| 43–52 | Mortgage property 0–9 (grouped by colour) |
| 53 | Unmortgage property (cheapest) |
Actions outside the legal mask are automatically blocked by the environment.
┌─────────────────────────────────────────────────────────────┐
│ Stage 1: Data Collection & Behavioral Cloning │
│ │
│ Groq LLM ──▶ MonopolyEnv ──▶ (state, action) pairs │
│ │ │
│ ▼ │
│ PolicyNetwork (BC) │
│ (supervised training) │
└─────────────────────────────────────────────────────────────┘
│
▼ initialise weights
┌─────────────────────────────────────────────────────────────┐
│ Stage 2: PPO Self-Play │
│ │
│ PolicyNetwork ──▶ SB3 PPO ──▶ self-play + LLM opponents │
│ │ │
│ ▼ │
│ best_model.zip │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ FastAPI Model Server (port 8765) │
│ POST /act { state, legal_actions } ──▶ { action } │
└─────────────────────────────────────────────────────────────┘
The policy network is a deep MLP shared between BC and PPO training:
Input: 214-dim observation
│
├─ Linear(214 → 512) + LayerNorm + ReLU
│
├─ Linear(512 → 512) + LayerNorm + ReLU
│
├─ Linear(512 → 256) + LayerNorm + ReLU
│
├─ Policy Head: Linear(256 → 53) ← action logits (one per action)
│
└─ Value Head: Linear(256 → 1) ← state value (for PPO advantage calculation)
Key design choices:
- Layer Normalization: Stabilizes training and reduces internal covariate shift.
- Orthogonal Initialization: Standard for RL—helps with gradient flow and stability.
- Dual Heads: The shared trunk extracts features; policy head produces action logits, value head estimates discounted future reward.
- Gradient Clipping: Applied during BC training to prevent exploding gradients.
The first step collects human-like gameplay from an LLM (Groq's llama-3.1-70b-versatile):
python collect_data.py --n-games 1000 --n-workers 4What happens:
- The script spawns 4 parallel worker threads.
- Each worker initializes a
MonopolyEnvand lets the LLM play full games. - At each decision point: The current board state (214-dim observation) is sent to the LLM as a structured prompt. The LLM reasons about the best move and returns an action ID.
- All (state, action) pairs are recorded as compressed NumPy archives (.npz) in
data_collected/.
Options:
| Flag | Default | Description |
|---|---|---|
| --n-games | 1000 | Number of full games to simulate |
| --n-workers | 4 | Parallel threads (LLM is the bottleneck) |
| --save-dir | data_collected | Output directory |
| --checkpoint-every | 50 | Save a checkpoint every N games |
Output: Each checkpoint saves ~1,000 (state, action) pairs in compressed NumPy format. For 1,000 games with 4-player matches, expect ~50,000–100,000 total decision steps.
Once data is collected, train a neural network to mimic the LLM's decisions:
python train_bc.py --epochs 50 --batch-size 256 --lr 3e-4Training loop (training/bc_trainer.py):
- Load all collected data from .npz files into a PyTorch Dataset.
- Split into train/val (default 90/10 split).
- For each epoch:
- Forward pass through the network:
obs → trunk → policy_head → logits(53 action logits) - Compute cross-entropy loss:
loss = -log(softmax(logits)[true_action]) - Backprop, gradient clipping (norm 1.0), Adam optimizer step.
- Evaluate on validation set; track both loss and accuracy.
- Forward pass through the network:
- Early stopping: If validation loss doesn't improve for 5 epochs, training stops.
- Save best model as
models/bc/best_model.pt(checkpoint with lowest validation loss).
Key hyperparameters (configurable in config.yaml):
| Key | Default | Description |
|---|---|---|
| epochs | 50 | Maximum training epochs |
| batch_size | 256 | Mini-batch size |
| lr | 3e-4 | Adam learning rate |
| val_split | 0.1 | Fraction of data for validation |
| early_stop_patience | 5 | Epochs of no improvement before stopping |
| hidden | 512 | Hidden layer width |
Expected performance:
- Training typically converges in 10–25 epochs.
- Final validation accuracy: ~70–85% (the LLM's decisions are complex; perfect imitation is impossible).
- Loss decreases smoothly with a cosine annealing learning rate schedule.
W&B Integration (optional):
python train_bc.py --wandbThis logs training curves, accuracy, and hyperparameters to Weights & Biases.
After BC training completes, refine the policy using Proximal Policy Optimization (PPO):
python train_ppo.py --total-timesteps 5000000 --n-envs 8Overview:
- Initializes the PPO policy with BC weights (transfer learning).
- Runs 8 parallel game environments simultaneously.
- Agents play self-play for 5 million timesteps (~500K games with 4 players each).
- Rewards shaped to encourage aggressive, strategic play.
PPO is a policy gradient method that:
- Collects experience by running the current policy in the environment.
- Estimates advantages (how much better an action was vs. expected) using Generalized Advantage Estimation (GAE).
- Updates the policy using a clipped objective to prevent too-large updates.
- Also trains a value network to estimate state values (used for advantage calculation).
In pseudocode:
For each training iteration:
- Collect n_steps=2048 transitions from each of 8 environments (16K transitions total)
- Compute advantages using GAE (λ=0.95)
- For n_epochs=10:
- Split transitions into mini-batches of 64
- Compute policy loss (with clipping) + value loss + entropy bonus
- Backprop and update policy & value networks
| Key | Default | Description |
|---|---|---|
| total_timesteps | 5,000,000 | Total environment interactions |
| n_envs | 8 | Parallel game instances |
| n_steps | 2048 | Rollout length per environment |
| batch_size | 64 | Mini-batch size for gradient updates |
| n_epochs | 10 | Passes over collected experience |
| learning_rate | 3e-4 | Policy optimizer (Adam) learning rate |
| gamma | 0.99 | Discount factor (future rewards) |
| gae_lambda | 0.95 | GAE smoothing parameter |
| clip_range | 0.2 | PPO clipping range (ε in the paper) |
| ent_coef | 0.05 | Entropy bonus coefficient (encourages exploration) |
All these are configurable in config.yaml under the ppo section.
To prevent "cowardly" strategies, the reward function is:
r_t =
- 0.005 * (every timestep penalty to discourage long games)
+ 1.0 * (if opponent goes bankrupt)
+ 0.5 * (if opponent pays me rent)
+ 0.1 * (if I buy a property)
+ 0.05 * (per complete color set I own)
This encourages:
- Fast, aggressive play (penalty for stalling)
- Bankrupting opponents (large +1.0 reward)
- Collecting rent (dense feedback when opponents land on my properties)
- Strategic property development (bonuses for complete sets)
When starting PPO training:
- Load the trained BC network (
models/bc/best_model.pt). - Extract its state dict (all layer weights and biases).
- Load into Stable-Baselines3's MlpPolicy (which has a similar but not identical architecture).
- Partial weight transfer: Only copy weights for layers with matching shapes/names.
- Remaining untrained layers use orthogonal initialization.
This warm-start dramatically reduces training time (PPO doesn't need to learn basic game logic from scratch).
During training, two callbacks monitor progress:
- CheckpointCallback: Saves model snapshots every 100K timesteps to
models/ppo/ppo_checkpoint_*.zip. - EvalCallback: Every 50K timesteps, runs 20 evaluation episodes (deterministic rollouts) and saves the best model as
models/ppo/best_model.zip.
Training output:
| Timestep | Reward | Policy Loss | Value Loss | Entropy | Time |
|------------|--------|-------------|------------|---------|-------|
| 0 | -0.05 | 1.234 | 0.678 | 3.50 | 0.5s |
| 100000 | 0.12 | 0.567 | 0.234 | 2.80 | 45s |
| 1000000 | 0.45 | 0.123 | 0.056 | 1.20 | 450s |
| 5000000 | 0.78 | 0.045 | 0.012 | 0.85 | 2250s |
Expected training time: 4–8 hours on a modern GPU (NVIDIA A100/RTX 3090).
Optional experiment tracking:
# Enable W&B logging (requires WANDB_API_KEY in .env)
python train_ppo.py --wandb
# Or view TensorBoard logs
tensorboard --logdir models/ppo/tb_logs/Logged metrics:
- Episode reward (cumulative return per game)
- Policy loss & value loss
- Entropy (exploration indicator)
- Explained variance (how well the value network predicts returns)
cd monopoly-rl
pip install -r requirements.txtKey dependencies:
torch— Neural network frameworkstable-baselines3,sb3-contrib— PPO and MaskablePPO implementationsgymnasium— RL environment standardpydantic,fastapi— REST APIgroq— LLM inference for data collectionwandb— Experiment tracking (optional)pyyaml— Config file parsing
cp .env.example .env
# Edit .env and add:
# GROQ_API_KEY=sk-...
# WANDB_API_KEY=...monopoly-rl/
├── agents/
│ ├── __init__.py
│ ├── llm_agent.py # Groq-backed LLM agent for data collection
│ └── random_agent.py # Baseline agent (random valid actions)
├── data/
│ ├── __init__.py
│ └── collector.py # DataCollector: loads & aggregates .npz files
├── env/
│ ├── __init__.py
│ ├── game_engine.py # MonopolyEngine: core game logic (no UI)
│ ├── monopoly_env.py # MonopolyEnv: Gymnasium wrapper
│ └── action_masks.py # Legal action computation
├── training/
│ ├── __init__.py
│ ├── policy_network.py # PolicyNetwork: shared MLP (BC + PPO)
│ └── bc_trainer.py # BCTrainer: behavioral cloning training loop
├── server/
│ ├── __init__.py
│ └── main.py # FastAPI server: /act endpoint
├── models/ # Saved checkpoints
│ ├── bc/ # BC trained models
│ └── ppo/ # PPO trained models
├── data_collected/ # Collected game data from LLM (Stage 1)
├── collect_data.py # CLI: run LLM data collection
├── train_bc.py # CLI: behavioral cloning training
├── train_ppo.py # CLI: PPO self-play training
├── config.yaml # All hyperparameters (stages 1 & 2)
├── .env.example # Environment variable template
└── requirements.txt # Python dependencies
To enable experiment tracking and visualization:
- Sign up at https://wandb.ai/
- Set your API key in
.env:WANDB_API_KEY=your_key_here - Enable during training:
python train_bc.py --wandb python train_ppo.py --wandb
Logged:
- Training/validation loss & accuracy (BC)
- Episode rewards, policy loss, value loss, entropy (PPO)
- Hyperparameters used
- System metrics (GPU memory, training time)
Runs are automatically tagged to the monopoly-rl project.
PPO training also logs to TensorBoard:
tensorboard --logdir monopoly-rl/models/ppo/tb_logs/Open http://localhost:6006/ in your browser.
After training completes, serve the model via FastAPI:
python -m server.mainThe server listens on http://0.0.0.0:8765 and exposes:
Endpoint: POST /act
Request body:
{
"state": [0.0, 0.1, ..., 0.5], // 214-dim observation vector
"legal_actions": [0, 1, 3, 5] // list of valid action IDs
}Response:
{ "action": 1 }The server automatically selects the best available model:
- PPO (
models/ppo/best_model.zip) if it exists → preferred - BC fallback (
models/bc/best_model.pt) if PPO not found - Random if neither model exists
This version has breaking changes — APIs, conventions, and file structure may all differ from your training data. Read the relevant guide in node_modules/next/dist/docs/ before writing any code.
MIT