Skip to content

Require benchmark coverage declarations for new cuML estimators #8434

Description

@csadorf

Summary

Add an automated test that fails when a new public cuML estimator is introduced without being represented in the benchmark suite.

Motivation

Adding an estimator and adding its benchmark configuration are currently separate manual steps. This allows new estimators to be merged without benchmark coverage.

The benchmark registry and checked-in benchmark manifests should remain synchronized with the set of eligible public cuML estimators.

Proposed behavior

Add a test that:

  • Discovers public, concrete cuML estimators.
  • Verifies that each eligible estimator is registered with the benchmark framework and has at least one entry in the canonical benchmark manifest.
  • Allows estimators that cannot reasonably be benchmarked to appear in a centralized exclusion list with a documented reason.
  • Fails for stale exclusions that no longer correspond to a discovered estimator.
  • Produces an actionable error identifying the uncovered estimator and whether it is missing from the benchmark registry, the manifest, or both.

The test should validate configuration only and should not execute the benchmarks or require benchmark-sized datasets.

Acceptance criteria

  • Introducing a discoverable estimator without benchmark coverage causes the test to fail.
  • Adding the corresponding benchmark registration and manifest entry makes the test pass.
  • Intentional exclusions require an explicit reason.
  • Stale exclusions cause the test to fail.
  • The test runs in existing CI and does not execute performance benchmarks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions