Reinforcement Learning

Reinforcement learning algorithms with mathematical derivations and reproductions of figures from Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2020).

Theoretical foundations live in Sequential-Decision-Making. Deep-RL methods will live in Deep-Reinforcement-Learning.

Setup

git clone https://github.com/SaiSampathKedari/Reinforcement-Learning
cd Reinforcement-Learning
python -m venv .venv && source .venv/bin/activate
pip install -e .
# or, with uv
uv sync

Requires Python ≥ 3.10.

Layout

• src/rl/: algorithms and environments (see the package map).
• notebooks/: experiments, one folder per environment (see the notebook index).
• reports/: derivations, proofs, and convergence analysis (see the report index).

Algorithms

Category	Algorithms	Status
Model-free prediction	MC prediction, TD(0), n-step TD, TD(λ)	●
Model-free control	Sarsa, Q-learning, n-step Sarsa, Sarsa(λ)	●
Dynamic programming	policy evaluation; policy & value iteration	◐
Model-based & planning	Dyna-Q, prioritized sweeping	○
Value-function approximation	gradient MC, state aggregation; semi-gradient TD / Sarsa, n-step semi-gradient TD / Sarsa, tile coding	◐
Policy gradient	REINFORCE, REINFORCE with baseline	○
Actor-critic	one-step actor-critic, advantage actor-critic (A2C)	○
Trust-region & proximal methods	natural policy gradient, TRPO, PPO	○

_{● implemented  ·  ◐ partial  ·  ○ planned}

Reports — derivations & proofs

Self-contained PDF write-ups, numbered in reading order (see the report index):

• Tabular & approximation — MC control (ES), ε-soft MC control, TD & eligibility traces, Sarsa(0), n-step Sarsa, Sarsa(λ), Q-learning, value-function approximation.
• Policy gradient & actor-critic — policy gradient theorem, average-reward PG, trajectory route, PG preliminaries, REINFORCE, actor-critic, baseline & advantage, GAE.

Environments

Environment	Source	Status	Notebook
Blackjack	S&B §5.1	●	01_blackjack
Windy Gridworld	S&B Example 6.5	●	02_windy_gridworld
Random Walk	S&B Examples 6.2 / 7.1 / 9.1	●	03_random_walk
Cliff Walking	S&B Example 6.6	○
FrozenLake	Gymnasium	○
Dyna Maze	S&B Example 8.1	○
Mountain Car	S&B Example 10.1	●	04_mountain_car
Acrobot	Classic control	○
CartPole	Classic control	○
Short Corridor	S&B Example 13.1	○

_{● implemented  ·  ○ planned}

Related repositories

A sequence from mathematical foundations to deep RL:

• Foundations — Real Analysis · Probability & Distribution Theory · Statistical Inference Theory
• RL theory — Sequential Decision Making
• This repo — Reinforcement Learning: S&B algorithms and figure reproductions
• Next — Deep Reinforcement Learning

Contact

• Email: sampath@umich.edu
• LinkedIn: sai-sampath-kedari

Gallery

Blackjack: optimal policy π* and value V* (S&B Fig 5.2)	Windy Gridworld: value V(s) and 15-step optimal path (S&B Ex. 6.5)
Random Walk: gradient MC with state aggregation vs true vπ (S&B Fig 9.1)	Mountain Car: cost-to-go −maxa q̂(s,a,w) learned by semi-gradient Sarsa (S&B Fig 10.1)
Mountain Car: the learned greedy policy in action — reverse up the left slope to build momentum, then accelerate through to the goal