Foundation: evaluation toolkit #66

Goal

Shared, tested metrics so recipes report results the same way and no notebook reimplements a confusion matrix.

Scope

• Choice: accuracy, per-class precision, recall and F1, macro and micro averages, confusion matrix, Cohen's kappa.
• Noul: precision and recall at a threshold, threshold sweep, threshold selection on a validation split, Brier score.
• Multi-label: per-label and micro and macro F1, exact-set match.
• Score and ranking: exact agreement, mean absolute error, nDCG, top-k accuracy, recall at a budget.
• Selective prediction: accuracy against coverage, risk and coverage curve, threshold selection that is frozen before test.
• Calibration: reliability table and expected calibration error.
• Comparison: paired bootstrap confidence interval for the difference between two systems.
• Accounting: request counts and token usage summed from usage.
• Plain numpy. Each function has a docstring that says what it measures in one sentence.

Acceptance criteria

• Every metric has tests against small hand-computed cases, including edge cases (empty class, all one label, ties)
• Functions accept the typed answer objects from the backends issue as well as plain arrays
• No metric silently returns a number for empty input
• Lint and tests pass in CI, or locally with the output pasted in the pull request if CI does not exist yet

Dependencies

• Blocked by Foundation: project scaffolding and Python package skeleton #61 (project scaffolding and Python package skeleton)
• Unblocks Foundation: recipe template and scaffolder #68
• Tracked in Epic: Foundation #74

References

• Recipe contract
• Catalog
• TypeSafe documentation index for tools: https://docs.typesafe.ai/llms.txt
• Python SDK: https://docs.typesafe.ai/sdk/python