Benchmarking

Benchmarking

The framework ships a comprehensive benchmark that measures every stage of the training pipeline on your hardware. Use it to compare platforms (Mac MPS vs CUDA vs CPU), profile bottlenecks, or sanity-check after upgrading PyTorch or CUDA drivers.

See also: Training Guide | FAQ

---

Quick run

python -m orca.benchmark --quick

About 30 seconds. Prints throughput numbers for the main stages.

Full benchmark

python -m orca.benchmark

A few minutes. Measures every section in detail.

What it measures

Section	What it covers
engine	C game engine: place/undo cycles per second, scored_moves throughput, win detection
nn	Network forward and backward, batch sizes, FP16 vs FP32
search	Pure Python MCTS, C MCTS, alpha-beta, with and without NN
selfplay	End-to-end self-play game generation, positions per second
training	Train step throughput at different batch sizes
augmentation	Hex symmetry augmentation cost per sample

Running a single section

python -m orca.benchmark --section nn
python -m orca.benchmark --section selfplay

Saving results

python -m orca.benchmark --output bench.json

The JSON contains every measurement plus system info (OS, Python, PyTorch, CUDA, GPU model, CPU model, RAM). You can diff two runs to spot regressions.

Reading the output

The benchmark prints a copy-paste-ready markdown table at the end. Sample shape:

| Section      | Metric                | Value         |
|--------------|-----------------------|---------------|
| engine       | place/undo (M/sec)    |          1.42 |
| engine       | scored_moves (K/sec)  |        145.0  |
| nn           | forward (batch=64)    |  12,340 pos/s |
| search       | MCTS (sims=200)       |     880 pos/s |
| selfplay     | positions/sec         |  4,200        |
| training     | steps/sec (batch=512) |        38     |

Numbers are absolute throughput so they compare directly across machines.

When to run it

• First setup on new hardware. Establishes a baseline.
• After driver / PyTorch upgrades. Spot regressions.
• Before a long training run. Confirms the GPU is actually being used.
• Choosing config. Compare --config standard vs --config large throughput before committing to a training run.

Comparing across platforms

The metric positions/sec is the most useful single number for "how fast will my training go?" Use it to compare your machine to friends' or to the numbers in the README.

A typical M-series Mac shows ~3K to 5K positions/sec on selfplay with the standard network. A modern CUDA GPU (e.g. RTX 4080) is usually 15K to 30K positions/sec.