🐍 Snake RL — Reinforcement Learning for the Snake Game

Comparative analysis of PPO and DQN agents in fully and partially observable Snake environments.

Overview

This project trains a reinforcement learning agent to play Snake on a 7×7 grid, comparing performance across:

• Fully observable (MDP) vs. Partially observable (POMDP) environments
• PPO vs. DQN algorithms
• Reward shaping (potential-based distance bonus)
• State representations (frame stacking: 1, 2, 4 frames)

Developed as the course project for Reinforcement Learning 2025/2026 at the University of Padua, under Prof. Gian Antonio Susto.

Results

Agent	Fruits/Episode	vs. Baseline
Heuristic Baseline (Full)	11.68 ± 2.76	—
PPO (Full Obs.)	16.34 ± 1.58	+40%
Heuristic Baseline (Partial)	11.28 ± 3.89	—
PPO (Partial Obs.)	18.22 ± 3.77	+62%

Bonus Experiments (Full Obs.)

Experiment	Fruits/Episode
DQN	11.98 ± 3.26
PPO + reward shaping	19.79 ± 3.74
PPO (n_stack=1)	21.17 ± 5.91
PPO (n_stack=2)	19.73 ± 4.50

Project Structure

├── evaluate.py              # Load weights & evaluate agents
├── train.py                 # Training (PPO, DQN, bonus experiments)
├── baseline.py              # Heuristic baseline evaluation
├── config.py                # Centralized hyperparameters
├── gym_wrapper.py           # Gymnasium wrapper (reward shaping support)
├── snake_env/               # Environment (fully + partially observable)
├── models/                  # Trained weights
├── results/                 # CSV evaluation logs
├── figures/                 # Generated plots
├── utils/                   # Plotting & visualization utilities
└── test/                    # Sanity checks & preflight tests

Quick Start

# Clone
git clone https://github.com/YOUR_USERNAME/snake-rl.git
cd snake-rl

# Install dependencies
pip install -r requirements.txt

# Evaluate pre-trained agents
python evaluate.py

# Run heuristic baseline
python baseline.py

# Train from scratch (~2-3 hours on GPU)
python train.py

Approach

Baseline

Deterministic heuristic with two-tier decision making:

1. Safe action filtering — avoid walls and body
2. Greedy Manhattan — minimize distance to fruit when visible
3. Local freedom — maximize open neighbors when fruit is not visible

RL Agent

• Algorithm: PPO (Proximal Policy Optimization) via Stable-Baselines3
• Architecture: Custom 2-layer CNN (Conv3×3 → Conv3×3 → FC-256) with frame stacking
• Training: 1M timesteps, 16 parallel environments, seed=0

Key Findings

• PPO significantly outperforms the heuristic baseline in both observability settings
• Partially observable agent outperforms fully observable — likely due to implicit attention from the smaller observation window
• Potential-based reward shaping accelerates learning (+21%)
• DQN underperforms PPO due to single-environment training limitation
• Fewer stacked frames achieve higher mean performance but with greater variance

Tech Stack

• RL Framework: Stable-Baselines3
• Deep Learning: PyTorch
• Environment: Custom Gymnasium wrapper over provided Snake environment
• Visualization: Matplotlib, imageio

References

• Schulman et al., Proximal Policy Optimization Algorithms (2017)
• Mnih et al., Human-level control through deep reinforcement learning (2015)
• Ng et al., Policy invariance under reward transformations (1999)
• Raffin et al., Stable-Baselines3 (2021)

License

MIT