Reproductions of five reinforcement-learning algorithms on Super Mario Bros (World 1-1), each built as a plug-in for the rl-factory engine. This repo is the what; rl-factory is the how.
The models, in the order they were built — each earning its place by fixing a limitation of the last:
| model | family | idea |
|---|---|---|
random |
off-policy | the performance floor (uniform-random actions) |
dqn |
off-policy | value learning: a Q-network + replay + frozen target |
ppo |
on-policy | policy gradients: optimize the policy directly |
icm |
on-policy | PPO + curiosity from forward-prediction error |
rnd |
on-policy | PPO + curiosity from random-network distillation |
knn |
on-policy | PPO + count-based episodic novelty (can't saturate) |
The write-ups (with the Mario 1-1 comparison curves in logs/) live on the blog; this repo
is the code they describe.
rl_ablations/register.py is the whole architecture in a dozen lines: each model is an acting
policy + a learner + a strategy family. ICM, RND, and kNN all act with the PPO policy and differ
only in their learner — same on-policy loop, a different update. Adding a model never touches the
training loop.
pip install -e ../rl-factory # the engine
pip install -e . # this repo (Mario env + models)
mario --agent dqn --mode train --actors 8
mario --agent ppo --mode evaluate --episodes 5Run the tests with pytest. Requires Python ≥ 3.12 (tested on Apple Silicon / arm64); the Mario
stack is version-pinned in pyproject.toml (gym 0.25.2, numpy<2 — see the notes there).