Street Fighter II Reinforcement Learning Agent

A PPO-based RL agent that learns to play Street Fighter II: Special Champion Edition (Sega Genesis) using pixel observations.

This is a journey project. Each phase explores a hypothesis, and the results are documented honestly, including the failures.

Phases

Phase	Hypothesis	Outcome
1	PPO + reward shaping vs single opponent	Dominates the training opponent, fails everywhere else
2	Train against all 8 opponents to generalize	Plateaus, then collapses, never learns to win consistently
3	Recurrent policy (LSTM) for temporal reasoning	Planned
4	Self-play	Planned

---

Phase 1: Single Opponent (Ryu)

Trained for 20M timesteps on RTX 4090 (~12.7 hours). The agent specializes in beating Ryu and overfits to his attack patterns.

vs Ryu (WIN)

vs Ken (LOSS)

Opponent	Win Rate
Ryu	5/5 (100%)
Ken, Chun-Li, Guile, Blanka, Zangief, E. Honda, Dhalsim	0/5 (0%)

Lesson: Single-opponent training produces a memorizer, not a fighter.

---

Phase 2: Multi-Opponent Training

Trained for 19M timesteps with the opponent randomized each episode across all 8 characters.

Training plateaued around 10-11M steps (peak eval reward +50.4 at step 11M) and then degraded. By step 19M the policy had collapsed to ~-75 average reward. The "best model" tracked by EvalCallback is from the brief peak period.

vs Ryu (LOSS)

vs Ken (LOSS)

Opponent	Win Rate	Avg Reward
Ryu	0/5	-1
Ken	0/5	-30
Chun-Li	0/5	0
Guile	0/5	-71
Blanka	0/5	-10
Zangief	0/5	-1
E. Honda	0/5	-2
Dhalsim	0/5	-9

Total: 0/40 wins. Even against Ryu, the opponent Phase 1 mastered, Phase 2 loses.

The agent isn't doing nothing. It moves, attacks, and plays differently against different opponents (Guile pressures it hard for -71 reward, Chun-Li ends in a draw at 0). But across the 8 matchups the policy converged to losing routines instead of winning ones.

Lesson: A vanilla CNN policy without memory or per-opponent context struggles to learn 8 different matchups simultaneously. It found a generalist policy that loses to everyone instead of a flexible one that adapts per opponent.

---

What's Next

Phase 3 will replace the CNN-only policy with a recurrent (LSTM) policy. The agent will be able to observe the opponent's behavior over the course of a fight and adapt its strategy, something the current 4-frame stack cannot do.

---

Architecture

• Algorithm: PPO (Stable-Baselines3)
• Policy: CnnPolicy (NatureCNN)
• Observations: 84x84 grayscale, 4-frame stack
• Actions: 24 discrete (movement, attacks, crouch/jump combos)
• Frame skip: 6 frames per decision
• Reward: Asymmetric health-delta (3x damage dealt vs 1x taken) + exponential round bonuses

Project Structure

street-fighter-rl/
├── configs/                    # YAML configs with inheritance
│   ├── base.yaml               # Shared hyperparameters
│   ├── phase1_ppo.yaml         # Single opponent (Ryu)
│   └── phase2_multi_opponent.yaml  # All 8 opponents
├── src/
│   ├── env/
│   │   ├── make_env.py         # SF2Wrapper + vectorized env factory
│   │   └── game_data/          # RAM mappings, save states, reward script
│   ├── training/
│   │   ├── train.py            # Training entrypoint with auto-resume
│   │   └── callbacks.py        # TrainingLogger + checkpoint callbacks
│   └── utils/
│       └── config.py           # YAML loader with inheritance
├── scripts/
│   ├── setup_runpod.sh         # One-command cloud GPU setup
│   └── train.sh                # tmux training launcher
├── notebooks/
│   ├── 01_setup_and_verify.ipynb   # Environment verification
│   ├── 02_phase1_train.ipynb       # Colab training (Phase 1)
│   └── 03_evaluate.ipynb           # Record fights, GIFs, win rates
├── recordings/                 # Fight GIFs
└── results/                    # Checkpoints + best models (gitignored)

Training

RunPod (recommended)

bash scripts/setup_runpod.sh
bash scripts/train.sh configs/phase1_ppo.yaml

Google Colab

Open notebooks/02_phase1_train.ipynb in Colab.

Key Hyperparameters

Parameter	Value	Why
gamma	0.94	Short-horizon for fighting games
frame_skip	6	Prevents button mashing, meaningful decisions
reward_coeff	3.0	Anti-cowardice: attacking 3x more rewarding than defending
reward_scale	0.001	Keeps gradients stable
batch_size	512	Larger batches for stable PPO updates
ent_coef	0.0	No entropy bonus needed

Requirements

• Python 3.10+
• Linux (stable-retro requires Linux)
• Street Fighter II: Special Champion Edition ROM (Genesis)
• GPU recommended (RTX 3090+ for fast training)

ROM disclaimer: This repository does not include the game ROM. You must legally own a copy of Street Fighter II: Special Champion Edition (Sega Genesis) and supply your own ROM file. The expected SHA-1 is in src/env/game_data/rom.sha.

Setup

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
bash scripts/setup_runpod.sh