Chess-based SLM trained exclusively on position (FEN) + best move (UCI/SAN) pairs without move history context.
A baseline for comparison against any claims of emergent LLM "mastery" or "understanding" of chess.
All variants of KinGPT were tested on a random sample of n=100 puzzles from the validation set of puzzles (n=1000).
Training data for Woodpecker/Chimera variants was filtered to remove any shared positions (FEN-level*) between the validation and training set of puzzle positions.
For definitions of accuracy/sanity, please refer to my larger repository benchmarking chess LLMs.
| Variant | Theme | Position Accuracy | Puzzle Accuracy | Sanity |
|---|---|---|---|---|
| Woodpecker | mate-in-1 | 85/100 (85.0%) | 85/100 (85.0%) | 99/100 (99.0%) |
| Woodpecker | mate-in-2 | 163/200 (81.5%) | 71/100 (71.0%) | 197/200 (98.5%) |
| Woodpecker | mate-in-3 | 244/300 (81.3%) | 61/100 (61.0%) | 295/300 (98.3%) |
| Beaver | mate-in-1 | 3/100 (3.0%) | 3/100 (3.0%) | 41/100 (41.0%) |
| Beaver | mate-in-2 | 2/200 (1.0%) | 0/100 (0.0%) | 58/200 (29.0%) |
| Beaver | mate-in-3 | 5/300 (1.7%) | 0/100 (0.0%) | 71/300 (23.7%) |
| Chimera | mate-in-1 | 91/100 (91.0%) | 91/100 (91.0%) | 100/100 (100.0%) |
| Chimera | mate-in-2 | 171/200 (85.5%) | 74/100 (74.0%) | 198/200 (99.0%) |
| Chimera | mate-in-3 | 248/300 (82.7%) | 60/100 (60.0%) | 299/300 (99.7%) |
*This is technically imperfect because castling rights and en passant sometimes do not effect the evaluation of a position. However, checking for piece-location duplicates requires a lot of preprocessing (including likely needing an engine call for the evaluation of each position which is incredibly wasteful) so I did not do it.
All training code for KinGPT was forked directly from karpathy's nanoGPT repository.
I would highly recommend to follow the instructions provided there on how to hack it to your needs instead of reading my config.
I used the instructions for training a mini-Shakespeare model (with char-level tokenization) to train all KinGPT variants.
pip install torch numpy transformers datasets tiktoken wandb tqdmtrain-kingpt-beaver.py - Training config for Beaver variant
train-kingpt-chimera.py - Training config for Chimera variant
train-kingpt-woodpecker.py - Training config for Woodpecker variant
Scroll down ↓ to the KINGPT variant section for more details about size, corpus, train/val loss, etc.
.\GAMBIT-puzzle-evals\eval_kingpt.py - Evaluate all KinGPT variants for the (same) n=300 sample of n=100 mate-in-1, mate-in-2, and mate-in-3 theme puzzles used in GAMBIT.
.\KINGPT-training\sample.py - Forked from karpathy's original directory, contains an example of prompting the model.
KinGPT variants are 25.20M parameter SLMs (~1/5 the size of GPT-2) which use a character-level tokenization (1 char = 1 token vs. BPE which is ~3-4 char = 1 token).
Each variant has a vocab 58 unique characters covering FEN syntax + UCI moves + <|position-start|> and <|position-end|> markers (no padding tokens due to fixed block size).
KinGPT variant trained on 13,341,057 unique puzzle positions (FEN + best move pairs).
Achieved train loss 0.3590, val loss 0.3704 on puzzles corpus after training for ~500B tokens.
KinGPT variant trained on 54,681 unique positions generated from 1050 Stockfish 18 self-play games.
Achieved train loss 0.0974, val loss 1.7554 (overfitting due to small dataset size) on selfplay corpus after training for ~25B tokens.
KinGPT variant trained on combined dataset of 13,395,738 Woodpecker and Beaver variant positions.
Achieved train loss 0.3594, val loss 0.3710 on combined corpus after training for ~500B tokens.
All puzzle positions were sourced from Lichess's open source puzzle database.
Stockfish 18 self-play games were generated by pitting a base Stockfish 18 instance in 50-game matches (25 White, 25 Black) vs. Stockfish instances of varying UCI Skill Levels 0-20 (equivalent to ~1300-3200 Elo according to the Stockfish FAQ), resulting in 1050 total self-play games.
All games were conducted at depth=15 and no opening books were used.
For each self-play game, I filter for only high quality position + best move pairs using the following criteria:
- If White wins, take all position + best move pairs from White's side-to-move only
- If Black wins, take all position + best move pairs from Black's side-to-move only
- If Draw, take all position + best move pairs since opponents are evenly matched (and one of them happens to be a regular instance of Stockfish 18)
The idea behind this is to prevent the model from learning "bad" moves played by weaker versions of Stockfish.
You can also alternatively fix this issue by using a seperate stronger "judge" engine to parse a strong position + best move pair for positions with the weak engine to play.
It is important to note that Stockfish 18 self-play games have significant position overlap!
In the full set of 1050 self-play games, 9772/107018 = ~9.13% of positions overlap across games (present in 2+ games).
Meanwhile, in the subset of games played by Stockfish 18 at FULL POWER, 1070/7069 = ~15.1% of positions overlap across games.
- (Optional) Question if training a chess-focused large language model is even worth it when we have Stockfish (+ the many other chess engines) as external verifiers, which can either be called directly by general LLMs (via a wrapper) or utilized in a LLM-Modulo system (as an external verifier to check LLM generation).
- Train a balanced variant of KinGPT using a mix of puzzle positions, human games, and inter/intra-engine games.
- Use RLHF/RLVR/PPO/GRPO/etc. during training to make the model better.
- Use LLM-Modulo + other verifier-in-the-loop methods during inference to make the model better.
- ...
All code in this repo was initially forked from Andrej Karpathy's nanoGPT repository.
The author acknowledges Research Computing at Arizona State University for providing computing and storage resources to train all variants of KinGPT.