benchmark: add canonical cross-validation source and dashboard coverage

[TheHiddenObserver]

Context

The benchmark dashboard already defines a CV presentation contract. PR #111 now supplies the current structured six-family CV source, canonical parser/schema integration, and real dashboard evidence on top of the synchronized PR #76 staging branch.

Version 0.2.4 materially changed cross-validation correctness and transactionality for RidgeCV, ElasticNetCV, LogisticRegressionCV, unified penalized CV, and CoxPHCV. This issue converts those contracts into reproducible benchmark evidence.

Related roadmap: PR #89. Dashboard synchronization is tracked in #90.

Goal

Create, validate, register, parse, and display a current canonical CV benchmark source covering the maintained CV estimator families.

Initial model matrix

• RidgeCV;
• LassoCV;
• ElasticNetCV;
• LogisticRegressionCV;
• PenalizedGLM_CV;
• CoxPHCV.

Required benchmark dimensions

Each canonical run retains, where applicable:

• dataset/case identity and exact scale;
• backend and resolved device;
• dtype;
• fold count and split strategy;
• candidate-grid or path size;
• alpha/lambda/C/l1-ratio grid identity;
• warm-start/path reuse configuration;
• scoring definition and normalization;
• CV evaluation time;
• final full-data refit time;
• total fit time;
• selected hyperparameter(s);
• validation score and final fitted score;
• candidate/fold convergence and failure counts;
• final refit convergence;
• host-to-device transfer policy;
• synchronization policy;
• timing scope;
• peak memory where available;
• source date, environment, seed, repeats, and provenance.

Statistical alignment

• Ridge/Lasso/ElasticNet comparisons align objective scaling and alpha definitions with sklearn or another declared reference.
• Logistic CV aligns scoring and regularization mapping for the representative case.
• PenalizedGLM_CV preserves its declared loss rather than substituting MSE.
• CoxPHCV records held-out partial-likelihood and subject-preserving-fold semantics.
• Unsupported or failed combinations remain explicit rows; they are not dropped silently.

Backend matrix

• NumPy CPU;
• CuPy CUDA;
• Torch CUDA;
• aligned external CPU references where available.

GPU timing synchronizes the resolved backend before and after each measured region. Transfer-inclusive and device-only timing scopes are not mixed.

Canonical source

The immutable P100 runner output is retained at results/cv_benchmark_candidate.json; its byte-identical registered copy is results/benchmark_frontend_sources/cv_benchmark_20260807.json.

source_id: cv-benchmark-20260807-1347184c988d
source_date: 2026-08-07
sha256: 1347184c988d0f9648c8477d64752b646249282978cf28f65c165b391839bad2
measurement_git_sha: ad2cf88d1d443a53eeb5207c33c4ee4f25de2400
GPU: Tesla P100-SXM2-16GB
Python: 3.9.16

The raw source contains 22 framework/backend dispositions: 21 successful measured rows and one explicit LogisticRegressionCV Torch strict-CUDA failure. That failure is preserved as canonical evidence, publishes no fabricated measurements or CPU fallback, and its production fix is tracked separately by #112.

Dashboard integration

• current structured JSON artifact registered under the canonical benchmark source directory;
• SHA256, source date, environment, comparison identity, and measurement provenance registered;
• deterministic parser retains successful and non-success backend dispositions;
• CV scope is enabled by real parsed runs;
• panel exposes status/reason, timing breakdown, selected parameters, scores, and convergence/failure summaries;
• parser, schema, semantic, TypeScript, staleness, build, and real-data E2E regression coverage are present.

Benchmark interpretation

This first canonical package intentionally uses one representative case per family. It establishes reproducible backend ordering for those cases, but it does not support a universal sample-size crossover boundary or a universal GPU-speedup claim. A quantitative crossover boundary should only be stated after a dedicated multi-scale benchmark source exists.

Non-goals

• no exhaustive cross-product of every penalty, solver, grid, and scale in the first source;
• no frontend-only fabricated CV rows;
• no estimator behavior change merely to make the benchmark matrix all-green;
• no universal crossover/speedup claim from a single representative scale.

Acceptance criteria

• A current machine-readable CV source exists with complete auditable provenance.
• All six initial estimator families have at least one representative case.
• NumPy, CuPy, and Torch paths are covered where maintained.
• CV and final-refit timing are distinct fields.
• Selected parameters, scoring definitions, and convergence/failure diagnostics are retained.
• Objective and regularization mappings to external references are documented and tested.
• Parser output retains every eligible source row/disposition.
• The dashboard CV scope is enabled by real data and passes browser tests.
• Strict generator, schema, semantic, staleness, TypeScript, build, and Playwright gates pass.
• Benchmark interpretation records the supported case-level evidence without making unsupported universal crossover or GPU-speed claims.

Completion evidence

PR #111 was squash-merged into feature/benchmark-frontend-dashboard as commit 03141738dfe92da5ba616314fcb2f91bfc325f98.

Post-merge staging validation on that exact commit:

Benchmark Frontend CI #464 — success
Tests #1418 — success
Maintenance compatibility #431 — success
Release notes validation #431 — success
Release package validation #437 — success

Audited inventory after registration:

discovered/classified JSON artifacts: 87
eligible sources: 9
registered sources: 9
available registered sources: 9
parsed registered sources: 9
eligible unregistered sources: 0
unclassified artifacts: 0

The canonical bundle contains 1,796 normalized runs, including 22 CV dispositions.

Follow-up

The LogisticRegressionCV Torch production defect remains tracked by #112. Broader systematic inference, survival-path, and other family-coverage expansions remain separate work packages.

[TheHiddenObserver]

Relationship to benchmark coverage umbrella #100

Issue #100 now owns the audited source catalog, corrected inventory semantics, and repository-wide canonical coverage matrix.

This issue remains the dedicated implementation work package for current structured CV benchmark data. It should not absorb the source-catalog redesign or unrelated method-family gaps.

Expected relationship:

• benchmark: establish audited source catalog and canonical coverage matrix #100 classifies CV as a canonical benchmark-data gap and links here;
• benchmark: add canonical cross-validation source and dashboard coverage #91 creates, validates, registers, parses, and displays the CV source;
• completion of benchmark: add canonical cross-validation source and dashboard coverage #91 updates the benchmark: establish audited source catalog and canonical coverage matrix #100 coverage matrix and catalog disposition;
• CV (0) remains an explicit empty state until benchmark: add canonical cross-validation source and dashboard coverage #91 provides real parsed rows.

The current registered bundle being 8/8 available/parsed does not alter this issue: no current canonical CV source exists.

[TheHiddenObserver]

PR #111 now implements the canonical CV source pipeline and dashboard contract on top of PR #76 staging head 69a3237ac471b55e5614527b9d522dd5ec77b847.

Completed in the draft:

• six-family raw-source JSON Schema and semantic validator;
• explicit NumPy/CuPy/Torch dispositions and non-finite/failure safeguards;
• unified runner with directly observed CV-evaluation/final-refit/total timing, warmups, cache isolation, repeats, and backend synchronization;
• direct PenalizedGLM _compute_cv_scores / _refit_best instrumentation;
• aligned sklearn references for Ridge/Lasso/ElasticNet/Logistic using the exact statgpu fold generator;
• fold-specific Ridge mapping sklearn_alpha = n_fit * statgpu_alpha;
• normalized MSE, binary log-loss, and Cox held-out partial-likelihood score semantics;
• canonical parser/registry integration while preserving the historical Lasso parser;
• structured metrics.cross_validation, dashboard schema/TypeScript contract, and Cross-validation Metrics panel;
• parser, runner, schema, frontend, staleness, and E2E coverage; generated deployment assets are current;
• all temporary patch/diagnostic workflows removed from the final diff.

CPU diagnostics executed all six NumPy families with one warmup and three measured repeats. An aligned-reference diagnostic verified equal selected parameters for all four sklearn-comparable families, final-score agreement within 1e-4, aligned validation scores, and positive separate timing regions. These artifacts were deliberately not registered because the GitHub-hosted runner had no physical CUDA environment; CuPy and Torch were recorded as unavailable.

Remaining completion gate: execute the committed runner on a declared physical CUDA machine with operational CuPy and Torch, validate and audit the immutable source, then register it and refresh the coverage matrix, inventory, generated assets, and real-data CV browser tests. PR #111 remains Draft and #91 remains open until that source exists.

[TheHiddenObserver]

P100 physical-CUDA validation update for PR #111 / #91:

benchmark commit: a9564d510c37f23187d47d91db849324e7a67c31
Python: 3.9.16
CuPy: 13.6.0
Torch: 2.0.0+cu117
GPU: Tesla P100-SXM2-16GB

Backend outcomes:

Family	NumPy	CuPy	Torch	sklearn
RidgeCV	success	success	success	success
LassoCV	success	success	success	success
ElasticNetCV	success	success	success	success
LogisticRegressionCV	success	success	failed	success
PenalizedGLM_CV	success	success	success	n/a
CoxPHCV	success	success	success	n/a

Validation:

benchmark_cv_models.py: exit 0
cv_source.py: exit 0
cv_source.py: OK — canonical CV source contract

The LogisticRegressionCV Torch result is a valid explicit failed disposition under the #91 contract; there was no CPU fallback. Exact failure:

RuntimeError: GPU path failed in _select_logistic_c_cv with backend='torch'; CPU fallback is disabled for strict CUDA execution.

The production estimator/backend defect is now separately tracked in #112. #91 should retain that failed row as benchmark evidence rather than waiting for #112 to be fixed.

The physical-GPU execution gate is therefore satisfied. The remaining #91 blocker is repository integration of the exact raw P100 JSON: it is not currently present under results/benchmark_frontend_sources/ on the PR branch, so it cannot yet be hash-registered, parsed into the canonical bundle, or used for real-data dashboard/E2E coverage. The raw file must be committed unchanged rather than reconstructed from this summary.