-
Notifications
You must be signed in to change notification settings - Fork 1
Training Guide
Complete guide to training the Orca bot using the AlphaZero-style self-play pipeline.
# Start training with defaults (resumes from latest checkpoint)
python -m orca.train
# Common overrides
python -m orca.train --iterations 50 --games-per-iter 30
python -m orca.train --config orca-transformer --device cuda
python -m orca.train --fresh # ignore existing checkpointsOr from Python:
from orca.train import OrcaTrainer
trainer = OrcaTrainer(iterations=100, games_per_iter=30)
trainer.run()from orca import Orca
Orca.train(iterations=50, games_per_iter=20)For a visual training dashboard with live game replay and charts:
python train_dashboard.pyThen open http://localhost:5001 and click play. See Train Dashboard docs.
Disable specific training features for experimentation:
python -m orca.train --no-curriculum --no-adaptive-lr # fixed sims + fixed LR
python -m orca.train --no-auto-tuner --no-augmentation # no tuner, no data augSee Configuration Reference for all toggles.
Each training iteration follows this cycle:
- Export model -- the current network is exported to ONNX for worker inference
- Self-play -- parallel workers play games against themselves using MCTS
- Augmentation -- collected samples are augmented with hex-valid symmetries
- Training -- gradient descent on the replay buffer
- ELO evaluation -- the current model plays against past generations
- Checkpoint -- model, optimizer, buffer, and metrics are saved
Self-play runs in parallel using ProcessPoolExecutor. Each worker loads the
network weights, creates a search engine, and plays a batch of games independently.
| Setting | Default | Description |
|---|---|---|
MAX_WORKERS |
5 | Maximum parallel self-play processes |
GAMES_PER_FUTURE |
2 | Games per subprocess future |
Workers are auto-sized: min(MAX_WORKERS, cpu_count - 2).
There are two worker implementations:
-
V2 worker (
_self_play_worker_v2): Uses the C game engine (CGameState) withBatchedMCTSorBatchedNNAlphaBeta. Preferred when the C engine is available. -
V1 worker (
_self_play_worker_v1): UsesOnnxPredictorfor CPU inference with the pure-Python MCTS. Fallback when the C engine is unavailable.
Each move during self-play:
- Run MCTS with the current simulation count (set by curriculum)
- Sample a move from the visit count distribution (temperature-based)
- Record the state, policy target, player, and threat label as a
TrainingSample - After the game ends, fill in results and assign priority scores
Temperature control: for the first TEMP_THRESHOLD (35) moves, moves are sampled
proportionally to visit counts. After that, the best move is chosen greedily.
The curriculum dynamically adjusts MCTS simulations and games per iteration based on both wall-clock time and iteration count.
| Time / Iteration | Simulations | Games/iter |
|---|---|---|
| < 0.5h / iter < 10 | 50 | 60 |
| 0.5-1.5h / iter 10-30 | 100 | 50 |
| 1.5-3.0h / iter 30-60 | 150 | 40 |
| > 3.0h / iter 60+ | 200 | 30 |
The actual sim count is max(time_based, iteration_based). More games are played
when sims are low (fast exploration), fewer when sims are high (quality over quantity).
When ELO stalls, the curriculum boosts search depth:
- Threshold: ELO delta < 15 between evaluations
-
Trigger: 10 consecutive stalled iterations (
PLATEAU_ITERS) - Boost: +50 simulations (capped at 400)
# From orca/config.py
PLATEAU_THRESHOLD = 15 # ELO delta to detect plateau
PLATEAU_ITERS = 10 # iterations of stall before boosting sims
PLATEAU_SIM_BOOST = 50 # extra sims on plateau (capped at 400)The AutoTuner class makes rule-based adjustments each iteration:
- Caps MCTS sims at 50 during training (curriculum provides the real count)
- Decays hint blend over time:
max(0.0, 0.3 - iteration * 0.015) - Increases train steps (up to 600) when the buffer is >90% full and loss is decreasing
- Locks game mix to 100% normal self-play
The ModelVault stores compressed (fp16) weights for every evaluated generation.
When the vault exceeds max_models (200), it prunes to keep:
- The first and last models
- The 20 most recent models
- Evenly spaced models across the full history
Every 2 iterations (ELO_EVAL_EVERY), the GenerationalArena runs a mini
round-robin tournament:
- Select up to 6 opponents from the vault (first, last, and evenly spaced)
- Play
ELO_EVAL_GAMES(4) games per opponent, alternating colors - Compute new ELO:
current_elo + 16 * (score - 0.5) * num_opponents
Games use 30 simulations and temperature 0.1 for near-deterministic play.
Checkpoints are saved every 5 iterations (CHECKPOINT_EVERY) as
hex_checkpoint_{iteration}.pt. They contain:
-
model_state_dict-- network weights -
optimizer_state_dict-- Adam optimizer state -
scheduler_state_dict-- cosine annealing LR scheduler state -
iteration-- current iteration number -
metrics-- ELO history, total games, iteration metrics -
auto_tuner-- hyperparameter tuner state
By default, training resumes from the latest checkpoint. The trainer searches for:
-
hex_checkpoint_*.ptfiles, sorted by iteration number - Loads model weights, optimizer, scheduler, metrics, and AutoTuner state
- Restores the replay buffer from
replay_buffer.pklif present
Use --fresh to start from scratch.
Checkpoints are automatically migrated when the architecture changes:
- 5-to-7 channel migration: Old 5-channel models are expanded to 7 channels (adding threat planes) with zero-initialized weights
- Filter migration: Models with different filter counts are resized with padding or truncation
The ReplayBuffer uses priority-weighted sampling. Samples are drawn proportionally
to their priority scores:
| Source | Priority | Rationale |
|---|---|---|
| Normal self-play | 1.0 | Baseline |
Human games (loaded from human_games.jsonl) |
0.8-1.5 | Expert demonstrations |
| Online games (real opponents) | 2.0 | Real human play |
| Late-game positions (last 15 moves) | 2.0 | Stronger learning signal |
| Final 5 positions | 3.0 | Decisive game-ending positions |
| Fork/multi-threat moves (2+ threats) | 3.5 | Tactical patterns |
| Unstoppable forks (3+ threats) | 5.0 | Critical tactical patterns |
| Augmented samples | original * 0.8 | Slightly lower than originals |
| Short games (< 30 moves) | original * 0.5 | Penalized: less mid/late-game signal |
| Spread-out games (spread >= 8) | original * 1.5 | Rewarded: distant play diversity |
After each training step (with 50% probability to save time), the buffer updates priorities based on temporal difference error:
value_err = abs(predicted_value - target_value)
new_priority = value_err + 0.1 # ensures non-zero priorityPositions where the network's value prediction is most wrong get sampled more frequently, focusing learning on the hardest positions.
Default capacity is 400,000 samples (REPLAY_BUFFER_SIZE). When full, oldest
samples are evicted (FIFO via collections.deque).
The augment_sample() function generates up to 7 additional samples per original:
Grid-safe transforms (fast, numpy array ops, 0.8x priority):
- 180-degree rotation: flip both axes
- Transpose: swap q and r axes
- Transpose + 180: combine both
Axial hex rotations (coordinate re-encoding, 0.7x priority): 4. 60-degree: (q,r) -> (-r, q+r) 5. 120-degree: (q,r) -> (-q-r, q) 6. 240-degree: (q,r) -> (r, -q-r) 7. 300-degree: (q,r) -> (q+r, -q)
Axial rotations where all positions fall outside the 19x19 grid are automatically filtered out. Policy targets are remapped to match the transformed board coordinates.
Each training step optimizes three losses jointly:
total_loss = value_loss + policy_loss + 0.5 * threat_loss
| Loss | Function | Description |
|---|---|---|
| Value loss | MSE(predicted_value, game_result) |
How well the network predicts who wins |
| Policy loss | -sum(target_policy * log_softmax(logits)) |
Cross-entropy between MCTS policy and network output |
| Threat loss | BCE_with_logits(threat_pred, threat_label) |
Auxiliary head predicting [my_4, my_5, opp_4, opp_5] in-a-row |
The threat loss weight (0.5) is lower than value and policy because it is an auxiliary signal that improves tactical awareness without dominating the main objectives.
- Adam with learning rate 0.001 and weight decay 1e-4
- CosineAnnealingWarmRestarts scheduler: T_0=50, T_mult=2, eta_min=1e-4
- If LR drops below 1e-4 on resume, it is reset to 0.001
Default training batch size is 1024 (BATCH_SIZE). Training is skipped if the
replay buffer contains fewer samples than the batch size.
Supervised fine-tuning bootstraps the network from expert games before self-play.
from orca.sft import sft_train, import_games, scrape_games
# Scrape games from an online source
scrape_games(source='littlegolem', output='expert_games.jsonl', limit=5000)
# Import and train
samples = import_games('expert_games.jsonl')
net = create_network('standard')
net = sft_train(net, 'expert_games.jsonl', epochs=5, lr=1e-3)
# Then continue with self-play
trainer = OrcaTrainer(iterations=100)
trainer.net = net
trainer.run()SFT trains on cross-entropy policy loss only (no value head) since expert games provide move labels but not reliable value targets. After SFT, switch to the full self-play pipeline which trains all three heads.
python -m orca.sft --games expert_games.jsonl --epochs 5 --lr 1e-3
python -m orca.train --resume # continues from SFT checkpointEnable automatic mixed precision (AMP) for ~2x training speedup on CUDA GPUs.
from bot import train_step
import torch
scaler = torch.amp.GradScaler()
losses = train_step(net, optimizer, replay_buffer, device='cuda', grad_scaler=scaler)The grad_scaler parameter in train_step() enables fp16 forward passes with
fp32 gradient accumulation. Loss scaling prevents underflow in fp16 gradients.
trainer = OrcaTrainer(
iterations=200,
device='cuda',
mixed_precision=True, # auto-creates GradScaler
)
trainer.run()python -m orca.train --mixed-precision --device cudaMixed precision is only effective on CUDA GPUs with Tensor Cores (RTX 20+, A100, etc.). MPS and CPU fall back to fp32 automatically.
Scale training across multiple GPUs or machines.
from orca.distributed import MultiGPUTrainer
trainer = MultiGPUTrainer(net, device_ids=[0, 1, 2, 3])
losses = trainer.train_step(batch)Uses torch.nn.DataParallel to split batches across GPUs. Linear speedup for
large batch sizes.
from orca.distributed import SelfPlayPool
pool = SelfPlayPool(num_workers=16, net=net)
samples = pool.generate(num_games=200)
pool.shutdown()SelfPlayPool manages worker processes with automatic load balancing. Workers
run on CPU; the training loop runs on GPU.
from orca.distributed import RayTrainer
trainer = RayTrainer(net_config='standard', num_actors=32)
trainer.run(iterations=100)RayTrainer distributes self-play across a Ray cluster. Each actor is a
remote process that can run on a different machine. The trainer aggregates
samples and runs gradient updates centrally.
python -m orca.train --distributed ray --num-actors 32The SkillCurriculum replaces the time-based curriculum with a 6-level
progression tied to training milestones.
from orca.curriculum import SkillCurriculum
curriculum = SkillCurriculum(start_level=0)
sims, games = curriculum.settings()| Level | Sims | Games/iter | Focus |
|---|---|---|---|
| 0 | 30 | 80 | Random openings, basic legality |
| 1 | 50 | 60 | Line extension, simple blocking |
| 2 | 100 | 50 | Threat detection, 4-in-a-row patterns |
| 3 | 150 | 40 | Positional evaluation, colony play |
| 4 | 200 | 30 | Complex tactical sequences |
| 5 | 400 | 20 | Full-strength search |
Advancement is triggered by ELO thresholds: when the bot's ELO exceeds the level's target, the curriculum advances automatically.
python -m orca.train --skill-curriculum --start-level 0Defensive moves now receive priority boosts during sample collection. Previously only offensive moves (forks, multi-threats) got boosted priority.
| Move Type | Priority Boost | Description |
|---|---|---|
| Blocks opponent threat | 3.0x (BLOCKING_PRIORITY_BOOST) |
Move successfully prevents an opponent 4+ in-a-row |
| Survives a threat | 2.0x (SURVIVAL_PRIORITY_BOOST) |
Player survives after opponent had a winning threat |
| Fork (2+ threats) | 3.5x | Unchanged from v4.0 |
This produces more balanced training data where the network learns both attack and defense equally well.
Short games are penalized with a tiered priority system instead of the flat 0.5x penalty from v4.0. Long games get bonus priority.
| Game Length | Multiplier | Notes |
|---|---|---|
| < 10 moves | discarded | Not added to replay buffer at all |
| 10-19 | 0.2x | Minimal signal |
| 20-29 | 0.4x | Some signal |
| 30-39 | 0.7x | Decent games |
| 40-44 | 1.0x | No penalty |
| 45-59 | 1.3x | Long game bonus |
| 60+ | 1.8x | Deep strategic games rewarded most |
The alpha-beta pre-check at the MCTS root can now be disabled or tuned.
python -m orca.train --no-ab-hybrid # disable AB pre-check entirely
python -m orca.train --ab-hybrid-depth 6 # deeper AB search (default: 4)Disabling the hybrid lets MCTS see blocking positions that alpha-beta would short-circuit, producing richer training data for defensive play. The trade-off is that forced wins within the AB depth may be missed during self-play.
ELO evaluation now includes matches against fixed-strength anchors for more stable, meaningful ratings.
- Random bot anchored at ~500 ELO
- Heuristic bot anchored at ~1000 ELO
- Blended ELO = 60% generational + 20% random-anchored + 20% heuristic-anchored
Configure with ELO_BASELINE_GAMES (default 4, set 0 to disable).
Data augmentation now produces up to 8x training data per game (up from 4x).
| Transform | Type | Priority |
|---|---|---|
| 180-degree rotation | grid-safe | 0.8x |
| Transpose | grid-safe | 0.8x |
| Transpose + 180 | grid-safe | 0.8x |
| 60-degree axial rotation | axial | 0.7x |
| 120-degree axial rotation | axial | 0.7x |
| 240-degree axial rotation | axial | 0.7x |
| 300-degree axial rotation | axial | 0.7x |
Axial rotations use coordinate re-encoding. Rotations where all mass falls off-grid are automatically filtered out.
Mixed precision was implemented in v4.0 but never wired up. It is now active by
default on CUDA. Uses GradScaler with gradient clipping and pin_memory for
faster CPU-to-GPU transfers.
# config.py defaults
USE_MIXED_PRECISION = True
GRAD_CLIP_NORM = 1.0Automatically applied on CUDA after checkpoint restore. Reduces kernel overhead
for ~2x total speedup on NVIDIA GPUs. No configuration needed -- it activates
when torch.cuda.is_available() returns True.
python -m orca.train [OPTIONS]
Pipeline:
--iterations N Training iterations (default: infinite)
--games-per-iter N Games per iteration (default: from curriculum)
--train-steps N Gradient steps per iteration (default: 200)
--resume Resume from latest checkpoint (default)
--fresh Start fresh, ignore existing checkpoints
--workers N Parallel self-play workers (default: auto)
Network:
--config NAME Architecture: fast, standard, large, hybrid,
orca-transformer (default: standard)
--device DEVICE Device: cuda, mps, cpu (default: auto)
Optimizer:
--lr FLOAT Learning rate (default: 0.001)
--weight-decay FLOAT L2 regularization (default: 1e-4)
--scheduler-t0 N CosineAnnealing T_0 (default: 50)
--scheduler-tmult N CosineAnnealing T_mult (default: 2)
--scheduler-eta-min F CosineAnnealing eta_min (default: 1e-4)
Search:
--mcts-sims N MCTS simulations per move (default: 400)
--mcts-batch N MCTS batch size for NN eval (default: 64)
Replay Buffer:
--buffer-size N Replay buffer capacity (default: 400000)
--batch-size N Training batch size (default: 1024)
Evaluation:
--elo-every N ELO eval frequency in iterations (default: 2)
--elo-games N Games per ELO opponent (default: 4)
Home · Quickstart · Concepts · FAQ · API Reference · GitHub · PyPI
hexbot · MIT licensed · Built for the Hexagonal Tic-Tac-Toe community
Learn
Build
Train
Evaluate & Share
Reference