Summary
Create a pipeline that evaluates checkpoints by playing them against previous bests and estimates an Elo (or TrueSkill) score over time.
Motivation
Automated rating helps track progress and decide when to promote a checkpoint.
Acceptance criteria / Definition of Done
- Implement an evaluation runner that:
- Runs N matches between a new checkpoint and current best with configurable time controls.
- Collects match outcomes (win/draw/loss).
- Implement a simple Elo/TrueSkill estimator that converts match results into a rating or rating delta.
- Produce a plot/report (CSV + PNG) showing rating over time for checkpoints.
- Document the evaluation threshold for model replacement (configurable).
Implementation notes
- Use existing libraries (e.g.,
trueskill) for rating if desired.
- Keep evaluation reproducible (fixed seeds/time controls) and make it runnable in CI for small sample tests (but full evaluation should run offline).
Complexity
Medium–High — requires evaluation harness + metrics code.
Summary
Create a pipeline that evaluates checkpoints by playing them against previous bests and estimates an Elo (or TrueSkill) score over time.
Motivation
Automated rating helps track progress and decide when to promote a checkpoint.
Acceptance criteria / Definition of Done
Implementation notes
trueskill) for rating if desired.Complexity
Medium–High — requires evaluation harness + metrics code.