Summary
Add an automated test that fails when a new public cuML estimator is introduced without being represented in the benchmark suite.
Motivation
Adding an estimator and adding its benchmark configuration are currently separate manual steps. This allows new estimators to be merged without benchmark coverage.
The benchmark registry and checked-in benchmark manifests should remain synchronized with the set of eligible public cuML estimators.
Proposed behavior
Add a test that:
- Discovers public, concrete cuML estimators.
- Verifies that each eligible estimator is registered with the benchmark framework and has at least one entry in the canonical benchmark manifest.
- Allows estimators that cannot reasonably be benchmarked to appear in a centralized exclusion list with a documented reason.
- Fails for stale exclusions that no longer correspond to a discovered estimator.
- Produces an actionable error identifying the uncovered estimator and whether it is missing from the benchmark registry, the manifest, or both.
The test should validate configuration only and should not execute the benchmarks or require benchmark-sized datasets.
Acceptance criteria
- Introducing a discoverable estimator without benchmark coverage causes the test to fail.
- Adding the corresponding benchmark registration and manifest entry makes the test pass.
- Intentional exclusions require an explicit reason.
- Stale exclusions cause the test to fail.
- The test runs in existing CI and does not execute performance benchmarks.
Summary
Add an automated test that fails when a new public cuML estimator is introduced without being represented in the benchmark suite.
Motivation
Adding an estimator and adding its benchmark configuration are currently separate manual steps. This allows new estimators to be merged without benchmark coverage.
The benchmark registry and checked-in benchmark manifests should remain synchronized with the set of eligible public cuML estimators.
Proposed behavior
Add a test that:
The test should validate configuration only and should not execute the benchmarks or require benchmark-sized datasets.
Acceptance criteria