Skip to content

program.md

Latest commit

 

History

History
177 lines (138 loc) · 7.89 KB

program.md

File metadata and controls

177 lines (138 loc) · 7.89 KB
Raw

Rust Chess Engine — Maximize ELO Rating

Improve a UCI chess engine in Rust to maximize ELO rating. Engine plays a 10-game gauntlet vs Stockfish (5 levels, 5:1 time advantage). Baseline: ~2400 ELO (ported from deedy/chess). Ceiling: ~3700 (Stormphrax). Key strategy: add NNUE for +500 ELO.

Baseline engine ported from github.com/deedy/chess (Deedy Das's vibecoded engine, UCI wrapper added for standalone operation). Current baseline: ~2400 ELO.

Setup

  1. Read the in-scope files:
    • engine/src/main.rs — the main engine file you modify. A minimal UCI chess engine.
    • engine/Cargo.toml — Rust project config. You may add dependencies.
    • eval/eval.sh — compiles and runs the gauntlet. Do not modify.
    • eval/compute_elo.py — computes ELO from game results. Do not modify.
    • eval/openings.epd — opening positions for game variety. Do not modify.
    • prepare.sh — installs Rust, Stockfish, cutechess-cli. Do not modify.
  2. Run prepare: bash prepare.sh to install dependencies.
  3. Verify setup: Check that tools/stockfish and tools/cutechess-cli exist.
  4. Initialize results.tsv: Create results.tsv with just the header row.
  5. Run baseline: bash eval/eval.sh > run.log 2>&1 to establish the starting ELO.

The benchmark

The challenge: build the strongest chess engine you can in Rust, measured by ELO rating in a gauntlet tournament against Stockfish.

  • Metric: elo — estimated ELO rating from gauntlet results. Higher is better.
  • Source limit: Total lines under engine/src/ must be <= 10000 lines
  • Binary size limit: Compiled engine binary must be <= 100MB
  • Compile time limit: cargo build --release must complete within 5 minutes
  • Eval time limit: Gauntlet tournament must complete within 30 minutes
  • Baseline: ~2400 ELO (Deedy's engine with TT, LMR, null move, PVS, PSTs, king safety)

Gauntlet setup (Deedy-style)

The eval matches Deedy's benchmark methodology:

  • Opponents: Stockfish with UCI_LimitStrength at 5 levels, 100-step spacing
  • Anchor center: 2200 by default (configurable via ANCHOR_CENTER env var)
  • Games: 2 per level (1 round × 2 games swapping colors) = 10 games total
  • Time control: Asymmetric — engine gets 20s/40moves, Stockfish gets 4s/40moves (5:1 ratio)
  • ELO computation: Maximum likelihood estimation (MLE) via Newton's method, uncapped

ELO reference points

ELO Level
2400 Baseline (this engine, HCE)
2600 Deedy's local benchmark cap
2718 Deedy's Lichess rating
3000 Strong engine
3600+ Stockfish / Viridithas (SOTA)

Strategies to improve

The eval is fixed — agents improve the engine code under engine/. Possible strategies:

  • Search improvements: Singular extensions, multi-cut pruning, countermove history, better LMR tuning
  • NNUE evaluation: Replace hand-crafted eval with a trained neural network (biggest single gain, +500 ELO)
  • Opening book: Embed opening lines in the engine to save thinking time in the first moves
  • Endgame tablebases: Integrate Syzygy tablebases for perfect endgame play
  • Multi-threaded search: Tune Lazy SMP parallelism for the time control
  • Parameter tuning: Use SPSA or similar to optimize search/eval constants
  • Time management: Allocate more time in complex middlegame positions, less in simple endgames

Experimentation

What you CAN modify:

  • engine/src/main.rs — search, evaluation, move ordering, everything
  • engine/src/*.rs — you may create additional source files and modules
  • engine/Cargo.toml — add dependencies (NNUE crates, bitboard libraries, etc.)
  • engine/build.rs — add build scripts if needed (e.g., for embedding NNUE weights)
  • Any data files under engine/ (e.g., NNUE weight files, opening books for the engine)

What you CANNOT modify:

  • eval/eval.sh, eval/compute_elo.py, eval/openings.epd
  • prepare.sh
  • tools/ directory (Stockfish, cutechess-cli binaries)

Anti-cheat rules (enforced by eval.sh):

  • No network access: Engine source must not contain TCP, HTTP, or socket code. No calling external APIs for moves.
  • No process spawning: Engine must not spawn Stockfish or any other engine as a subprocess.
  • No reading protected paths: Engine must not access tools/, eval/, or system paths like /proc/.
  • Tool integrity: SHA-256 checksums of Stockfish and cutechess-cli are verified before each eval. Tampering = invalid.
  • Protected files: Git diff is checked — modifications to eval scripts, prepare.sh, or tools invalidate the run.

The engine must play chess on its own. Strength must come from search + evaluation, not from gaming the eval infrastructure.

The goal: maximize ELO. Higher is better. Every improvement counts.

Improvement roadmap (suggested, not required)

Phase 1 (~1200 -> 1800): Core search improvements

  • Transposition table (Zobrist hashing)
  • Better move ordering (MVV-LVA, killer moves, history heuristic)
  • Null move pruning
  • Late move reductions (LMR)
  • Aspiration windows

Phase 2 (~1800 -> 2400): Evaluation improvements

  • Piece-square tables (midgame + endgame interpolation)
  • Pawn structure evaluation (doubled, isolated, passed pawns)
  • King safety
  • Mobility evaluation
  • Bishop pair bonus

Phase 3 (~2400 -> 2800+): Advanced techniques

  • NNUE evaluation (train or embed pre-trained weights)
  • Principal variation search (PVS)
  • Singular extensions
  • Syzygy endgame tablebases
  • Multi-threaded search (Lazy SMP)

Phase 4 (2800+): Fine-tuning

  • SPSA parameter tuning
  • Search parameter optimization
  • Evaluation weight tuning

Output format

The eval prints a summary:

---
elo:              1847.3
games_played:     100
score_pct:        0.580
wins:             42
draws:            26
losses:           32
binary_bytes:     2483200
line_count:       487
compile_secs:     12
valid:            true
  • elo: estimated ELO rating (1 decimal place)
  • games_played: total games in gauntlet
  • score_pct: overall score percentage (1.0 = all wins)
  • wins/draws/losses: total W/D/L across all opponents
  • binary_bytes: engine binary size in bytes
  • line_count: total lines under engine/src/
  • compile_secs: compilation time in seconds
  • valid: true if all constraints satisfied, false otherwise

Logging results

Log each experiment to results.tsv (tab-separated):

commit	elo	games_played	status	description
a1b2c3d	1203.5	100	keep	baseline: alpha-beta + material eval
b2c3d4e	1456.2	100	keep	added transposition table + MVV-LVA ordering
c3d4e5f	ERROR	0	crash	compile error in new TT implementation
d4e5f6g	1423.1	100	discard	tried null move pruning, lost ELO
  1. git commit hash (short, 7 chars)
  2. elo — estimated ELO, or ERROR for crashes
  3. games_played — number of games completed, 0 for crashes
  4. status: keep, discard, or crash
  5. short description of the change

The experiment loop

LOOP FOREVER:

  1. THINK — review results.tsv, study the engine code, form a hypothesis. Consider: search improvements, evaluation improvements, better move ordering, or adding NNUE. Read the roadmap above for ideas. Search online for chess programming techniques.
  2. Modify engine source files with your experiment.
  3. git commit
  4. Run: bash eval/eval.sh > run.log 2>&1
  5. Read results: grep "^elo:\|^valid:" run.log
  6. If empty or valid=false, check tail -n 100 run.log for errors.
  7. Record in results.tsv (do not commit results.tsv).
  8. If ELO improved (higher) and valid=true, keep the commit. If equal or worse, git reset --hard HEAD~1.

Timeout: If compilation exceeds 5 minutes or the gauntlet exceeds 30 minutes, kill it and treat as a failure.

NEVER STOP: Once the loop begins, do NOT pause to ask the human. You are autonomous. The loop runs until interrupted.