Backgammon MCTS + Reinforcement Learning Agent

A self-contained Python implementation of a Backgammon-playing agent that combines Monte Carlo Tree Search (MCTS/UCT) with a learned value function trained via self-play reinforcement learning (TD-learning) — the same recipe used by TD-Gammon and, more recently, AlphaZero-style agents, adapted for a game with real per-turn dice (chance events).

No external ML frameworks are required — only numpy, used for a small linear value model.

Files

File	Purpose
board.py	Full backgammon rules engine: board state, dice, legal move / full-turn generation (hitting, bar, bearing off, doubles), win detection.
value_net.py	Feature extraction + a linear value model V(state) → P(White wins), trained online with TD updates.
mcts.py	MCTSAgent: UCT search over the legal turns for the actual dice just rolled, using value-net-guided Monte Carlo rollouts to score each candidate turn.
train.py	Self-play RL training loop (TD(0), epsilon-greedy exploration) + evaluation harness vs a random-move baseline.
play.py	Watch a full game play out move-by-move, or run silently and just print the result.

Why this design

Backgammon is a game of imperfect determinism: a fresh, random dice roll is drawn every turn, so there's no persistent, explorable multi-turn tree the way there is in chess or Go. The standard, well-established solution (used by strong bots since TD-Gammon) is:

1. Learn a value function V(s) ≈ P(White wins from s) via TD self-play — play fast games against yourself, and after each move nudge the weights so the value of the position before the move moves toward the value after it (bootstrapping), with the real game outcome as the target on the final move of the game.
2. At decision time, run MCTS/UCT over just the legal turns for the dice you actually rolled. Each candidate turn is scored by running many Monte Carlo rollouts (simulated games with fresh random dice each future ply, played with a value-net-guided policy) and averaging the outcome. UCT balances trying the most promising turns more against still sampling uncertain ones, and the value function both drives the rollout policy and evaluates truncated rollouts, so a better value function directly makes the tree search stronger.

This is precisely how MCTS is applied to games with per-turn chance nodes: search per-decision using the actual realized chance outcome, rather than trying to enumerate the whole chance tree in advance.

Quick start

# 1. Train the value function via self-play (few minutes for a couple thousand games)
python3 train.py --games 2000 --save value_weights

# 2. Watch the trained MCTS agent play against a random-move opponent
python3 play.py --weights value_weights.npy --opponent random --sims 150

# 3. Watch two MCTS agents play each other
python3 play.py --weights value_weights.npy --opponent mcts --sims 150

Useful train.py flags:

• --games: number of self-play games (more = stronger value function)
• --eps-start / --eps-end: exploration rate, annealed linearly over training
• --lr: TD learning rate
• --eval-games / --eval-sims: after training, how many games (and MCTS simulations per move) to use to measure win-rate vs a random opponent

Notes / scope

• Implements full standard rules: opening position, doubles (4 moves), hitting/bar/re-entry, blocked points, and bearing off (including the "use a larger die when no checker sits on that exact point" rule).
• Does not implement the doubling cube or gammon/backgammon scoring — only plain win/loss — which keeps the RL objective simple (P(win)) while still being a complete, playable game.
• The value model is a compact linear/logistic model over hand-crafted features (checkers per point, blot/made-point indicators, bar, borne off, pip-count difference, whose turn it is). You can swap in a small MLP in value_net.py for a stronger (if slower) agent — the featurization and the rest of the codebase would not need to change.
• Both train.py and mcts.py are pure Python/numpy, so training speed is dominated by how many self-play games and how many MCTS simulations per move you run; increase --sims for stronger but slower play.