WINDY GRIDWORLD // TD CONTROL

Q-learning, SARSA, and Expected SARSA from scratch on the Sutton & Barto windy gridworld.

---

This is the windy gridworld from Sutton & Barto (Example 6.5), a 10×7 grid with a column-dependent wind that pushes the agent upward. The agent starts at (0, 3), receives −1 per step, and stops when it reaches (7, 3). The repository compares three tabular temporal-difference control methods:

Method	Backup target	On/off-policy
Q-learning	max_a' Q(s', a')	off-policy
SARSA	Q(s', a') for the actually-taken a'	on-policy
Expected SARSA	E_π[Q(s', a')] under the ε-greedy policy	on-policy

Results

 

• All three methods learn a near-optimal policy. The greedy path reaches the goal in 17 steps by moving above the goal row and accounting for the wind.
• Q-learning has the highest cumulative reward. Expected SARSA produces the smoothest curve because its backup has lower variance. The curves are averaged over 300 independent runs.

Reproduce

pip install numpy matplotlib
python main.py    # trains all three methods -> images/learning_curves.png + agent_path.gif

Grid size, wind profile, start/goal, and episodes/trials are set at the top of main.py.

Repository layout

windy_gridworld.py  environment: wind dynamics, step, reset
algorithms.py       q_learning / sarsa / expected_sarsa + train_q_table
visualizations.py   learning-curve plot + greedy-path animation
main.py             train the three methods and render the figures

Limitations

The environment uses the standard four actions. It does not include diagonal moves or stochastic wind. Both ε and α remain fixed, so the averaged curves settle slightly below the deterministic optimum.

References

Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed.), Example 6.5 · book

MIT. See LICENSE.