Skip to content

Feature - Elo rating estimation & continuous evaluation pipeline #17

Description

@Codex-Crusader

Summary

Create a pipeline that evaluates checkpoints by playing them against previous bests and estimates an Elo (or TrueSkill) score over time.

Motivation

Automated rating helps track progress and decide when to promote a checkpoint.

Acceptance criteria / Definition of Done

  • Implement an evaluation runner that:
    • Runs N matches between a new checkpoint and current best with configurable time controls.
    • Collects match outcomes (win/draw/loss).
  • Implement a simple Elo/TrueSkill estimator that converts match results into a rating or rating delta.
  • Produce a plot/report (CSV + PNG) showing rating over time for checkpoints.
  • Document the evaluation threshold for model replacement (configurable).

Implementation notes

  • Use existing libraries (e.g., trueskill) for rating if desired.
  • Keep evaluation reproducible (fixed seeds/time controls) and make it runnable in CI for small sample tests (but full evaluation should run offline).

Complexity

Medium–High — requires evaluation harness + metrics code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions