From d0ee8f8cd1607eaf9d3cc2d57b80d27bd665e7ab Mon Sep 17 00:00:00 2001 From: Ziqian Lin <51812297+TheHiddenObserver@users.noreply.github.com> Date: Thu, 6 Aug 2026 19:30:43 +0800 Subject: [PATCH] docs: rebuild development roadmap for 0.2.4 --- dev/plans/ISSUES.md | 54 +++++++ dev/plans/README.md | 75 ++++++++++ dev/plans/ROADMAP.md | 291 ++++++++++++++++++++++++++++++++++++++ dev/plans/TO_DO.md | 329 +++++++++++++++++++++---------------------- dev/plans/plan.md | 5 +- 5 files changed, 580 insertions(+), 174 deletions(-) create mode 100644 dev/plans/ISSUES.md create mode 100644 dev/plans/README.md create mode 100644 dev/plans/ROADMAP.md diff --git a/dev/plans/ISSUES.md b/dev/plans/ISSUES.md new file mode 100644 index 000000000..b2570068c --- /dev/null +++ b/dev/plans/ISSUES.md @@ -0,0 +1,54 @@ +# statgpu Roadmap Issue Index + +> Last synchronized: **2026-08-06** +> Roadmap PR: **#89** +> Baseline release: **0.2.4** + +This file maps the canonical roadmap to executable GitHub issues. GitHub issue state is authoritative for execution; repository hard development gates remain authoritative for completion. + +## Active issues + +| Priority | Issue | Work package | Dependencies | +|---|---:|---|---| +| P0 | #90 | Synchronize benchmark dashboard PR #76 with the 0.2.4 `master` baseline | PR #89 planning reference | +| P1 | #91 | Add a canonical cross-validation benchmark source and dashboard coverage | #90 | +| P1 | #92 | Complete dashboard production QA, cross-browser smoke, accessibility, and documentation integration | #90; preferably #91 before final QA | +| P1 | #93 | Complete Panel Tier-1 shared framework, diagnostics, fit statistics, and covariance support | Independent of dashboard lane | +| P2 | #94 | Implement Kaplan-Meier and Nelson-Aalen estimators | Independent; shares survival result design with future work | +| P2 | #95 | Implement initial Weibull, log-normal, and log-logistic AFT family | May proceed independently; sequence after #94 unless resources justify parallel work | +| P2 | #96 | Design and implement unpenalized multinomial logistic regression Phase 1 | Non-tunable base contract; prerequisite for #98 | +| P2 | #98 | Implement the complete penalized multinomial suite with direct-fit/CV closure | #96; sparse input remains blocked on #97 | +| P2 | #97 | Define the shared sparse-array/backend contract with no silent densification | Prerequisite for HDFE, mixed models, sparse multinomial follow-up, and broad sparse estimator support | + +## Recommended sequencing + +### Product and benchmark lane + +```text +#90 → #91 → #92 → propose PR #76 for master integration +``` + +Do not add new benchmark families during #90. Do not perform final dashboard QA on a stale or unsynchronized branch. + +### Statistical workflow lane + +```text +#93 +#94 → #95 +#96 → #98 +#97 → future sparse estimator issues +``` + +#96 is strictly unpenalized and non-tunable. #98 owns the complete penalized multinomial matrix and may not close after only L2 or only direct-fit support. Every tunable penalty exposed by #98 must ship with CV selection and final refit in the same work package. + +These issues may proceed in parallel only when they do not compete for the same backend, inference, solver, or review surface. + +## Issue maintenance rules + +- Keep one primary issue per statistical or product contract. +- Split an issue only when doing so does not produce a partially advertised capability or violate direct-fit/CV closure. +- Add explicit links when an issue blocks or is blocked by another issue. +- Update `ROADMAP.md`, `TO_DO.md`, and this file when priorities change. +- Roadmap and issue scope may narrow work but may not weaken `.claude` or `dev/AGENTS.md` hard gates. +- Close issues only with merged implementation evidence, required CI, external alignment where applicable, physical-GPU validation, and synchronized documentation. +- Do not close an issue solely because a class, function, parser, or frontend control exists. diff --git a/dev/plans/README.md b/dev/plans/README.md new file mode 100644 index 000000000..f402e83d9 --- /dev/null +++ b/dev/plans/README.md @@ -0,0 +1,75 @@ +# statgpu Development Plans + +This directory contains the project roadmap, execution backlog, and historical design notes. + +## Authority by responsibility + +There is no single global precedence order across documents with different responsibilities. Use the authority that matches the question being answered. + +### Hard development and completion gates + +1. Applicable `.claude/workflows/` and `.claude/skills/` protocol. +2. `dev/AGENTS.md`. +3. The mandatory checklist in [`TO_DO.md`](TO_DO.md), which summarizes but does not weaken the two sources above. + +Roadmap priorities, issue scope, and module plans may narrow a task, but they may not weaken or override these hard gates. Any approved exception must follow the explicit approval and deferral contract in the applicable workflow and `dev/AGENTS.md`. + +### Current public capability + +Use the validated implementation and tests together with `docs/en/guides/implemented-methods.md` and the linked maintained model pages. When a capability claim conflicts with validated behavior, correct the stale documentation rather than treating the claim as implementation evidence. + +### Current priority and sequencing + +Use [`ROADMAP.md`](ROADMAP.md). It selects what should be worked on next; it does not redefine development-completion requirements. + +### Executable scope and dependencies + +Use open GitHub issues and active pull requests, summarized in [`ISSUES.md`](ISSUES.md). Issues may split or narrow roadmap packages, but may not declare work complete below the repository hard gates. + +### Research and historical context + +Module plans in this directory provide design, literature, and historical context. Their checklists are not a reliable current capability or priority inventory unless the document states a recent verification release and commit. + +## Verified baseline + +- Last verified release: **statgpu 0.2.4** +- Last verified commit: `0aeeb95b60e3e274053b8f1b6427ae50c8eec015` +- Verification date: **2026-08-06** +- Release workflow, PyPI wheel/sdist publication, clean production installation, and representative model smoke tests passed. + +The release baseline does not imply that every historical plan item is complete. It establishes the code and documentation snapshot from which future work must branch. + +## Document status + +| Document | Status | How to use it | +|---|---|---| +| `ROADMAP.md` | Canonical priority source | Current priorities, sequencing, dependencies, and roadmap-level definition of done. | +| `ISSUES.md` | Canonical navigation | Maps roadmap work packages to executable GitHub issues and dependency order. GitHub issue state remains authoritative for execution. | +| `TO_DO.md` | Mandatory summary checklist | Compact hard-gate checklist plus active queue; subordinate to `.claude` and `dev/AGENTS.md`, not a weaker alternative. | +| `panel_framework_proposal.md` | Active design reference | Shared panel architecture and Tier-1 diagnostics proposal. Validate details against current code before implementation. | +| `plan_survival.md` | Active module reference | Cox Phase 1 status and Survival Phase 2+ scope; last materially updated 2026-07-12. | +| `cran_r_package_mapping.md` | Comparative reference | Method-family gap map. Some individual rows may lag current implementation. | +| `plan_anova.md` | Historical research plan | Its early implementation-status header is stale; use implemented-methods/model docs for current ANOVA support. | +| `plan_covariance.md` | Historical research plan | Its early implementation-status header is stale; current covariance estimators are documented elsewhere. | +| `plan_krr.md` | Historical research plan | Nystroem, KernelPCA, and chi-square kernel status in the old checklist is stale. | +| `plan_spline.md` | Historical research plan | SplineTransformer, cyclic, and thin-plate status in the old checklist is stale. | +| `plan_unsupervised.md` | Historical phase record | Useful for benchmark and algorithm history, not the current priority queue. | +| `plan.md` | Historical delta | Superseded by `ROADMAP.md`. | +| `archive/` | Archive | Completed or superseded planning material. | + +## Planning rules + +A roadmap item becomes executable only after it has a GitHub issue that defines: + +- user or developer problem; +- scope and explicit non-goals; +- public API and failure behavior; +- NumPy, CuPy, and Torch backend contract; +- direct-fit/CV closure for every tunable capability; +- inference and formula implications where applicable; +- external baselines and normalization/alignment settings; +- unit, regression, compatibility, and physical-GPU validation; +- documentation and benchmark deliverables; +- dependencies and completion criteria. + +Do not mark a module complete using only an implementation count or a passing CPU smoke test. Completion is contract-based, evidence-based, and subject to the hard workflow gates. diff --git a/dev/plans/ROADMAP.md b/dev/plans/ROADMAP.md new file mode 100644 index 000000000..c670d89c2 --- /dev/null +++ b/dev/plans/ROADMAP.md @@ -0,0 +1,291 @@ +# statgpu Roadmap + +> Canonical development roadmap +> Last verified release: **0.2.4** +> Last verified commit: `0aeeb95b60e3e274053b8f1b6427ae50c8eec015` +> Last verified: **2026-08-06** + +## 1. Purpose and authority + +This document defines current development priority and sequencing. It is not a public support matrix and does not override repository development gates. + +- Hard development and completion gates come first from the applicable `.claude` workflow/skill and then from `dev/AGENTS.md`. +- For implemented public methods and backend support, use validated implementation/tests together with `docs/en/guides/implemented-methods.md` and the linked model pages. +- For executable scope, use open GitHub issues and pull requests. +- Module-specific plans under `dev/plans/` provide design and literature context but may contain historical checklists. + +Roadmap priorities and issue scope may narrow work, but they may not weaken or override the hard development gates. When a module plan conflicts with current public documentation or tests, update the stale plan rather than reimplementing an already delivered feature. + +## 2. Baseline after 0.2.4 + +Version 0.2.4 established a stable correctness baseline for: + +- public estimator validation and sklearn cloning; +- transactional refits and cross-validation behavior; +- solver/penalty compatibility and narrow numerical fallbacks; +- analytic-weight semantics; +- NumPy/CuPy/Torch finite-input handling; +- binary LogisticRegression and GLM correctness; +- CoxPH/CoxPHCV core contracts; +- release packaging and production installation. + +The next cycle should convert that correctness baseline into maintainable product evidence and complete selected statistical workflows. It should not immediately expand into many unrelated zero-percent modules. + +## 3. Prioritization principles + +Work is ranked by the following criteria: + +1. **Correctness and contract risk:** fix ambiguous or incomplete public behavior before adding breadth. +2. **Workflow completeness:** finish a partially implemented statistical workflow before starting a new module family. +3. **Shared infrastructure leverage:** prefer work that reduces duplication or enables several later features. +4. **Evidence quality:** implementation, external alignment, physical-GPU validation, benchmark provenance, and documentation must move together. +5. **Controlled scope:** avoid PRs that combine framework refactors, multiple new model families, and broad performance work. +6. **Tunable capability closure:** do not expose a direct-fit penalty while leaving its CV path merely planned. + +## 4. Current priority queue + +### P0 — Roadmap and integration control + +#### P0.1 Reconcile planning documents with 0.2.4 + +Deliverables: + +- establish this file as the canonical priority source; +- keep `TO_DO.md` synchronized as the mandatory compact gate checklist and queue; +- classify older module plans as active references, historical research plans, or archive material; +- create GitHub issues for every active work package; +- require future roadmap changes to cite an implementation, test, release, or issue. + +#### P0.2 Synchronize benchmark dashboard PR #76 with current `master` + +PR #76 is the only active product branch at the 0.2.4 baseline, but it was built from an older base and is not currently mergeable. + +The synchronization change must be isolated from new benchmark families: + +- merge or rebase current `master` into the dashboard branch; +- resolve test, workflow, documentation, package-layout, and generated-asset conflicts; +- regenerate the deterministic three-file data bundle and deployment assets; +- rerun Python, TypeScript, build, staleness, and Playwright gates; +- preserve source hashes, canonical identities, and no-fabrication rules. + +### P1 — Benchmark evidence and dashboard readiness + +#### P1.1 Add a canonical cross-validation benchmark source + +The dashboard implements the CV presentation contract but has no current canonical CV source. + +Initial matrix: + +- `RidgeCV`; +- `LassoCV`; +- `ElasticNetCV`; +- `LogisticRegressionCV`; +- `PenalizedGLM_CV`; +- `CoxPHCV`. + +Required dimensions include backend, folds, candidate-grid size, path/warm-start configuration, CV time, final-refit time, selected parameter, score, convergence/failure diagnostics, timing scope, synchronization policy, and peak memory where available. + +#### P1.2 Complete dashboard product QA + +Before PR #76 is proposed for integration into `master`: + +- test the production build from the nested documentation path; +- complete Chrome/Chromium, Firefox, and WebKit/Safari smoke coverage; +- verify filter cascades, chart/table consistency, empty states, and source metadata; +- verify keyboard navigation, visible focus, control labels, and an accessible table path; +- integrate the user guide into documentation navigation; +- keep generated data and deployment assets deterministic and current. + +URL-persisted state, mobile redesign, virtualization, and bundle partitioning remain deferred until supported by measured product need. + +### P1 — Panel workflow completion + +Panel data has substantial estimator coverage but lacks several standard econometric diagnostics and shared infrastructure. + +Implement in three bounded changes: + +1. **Shared panel base and covariance registry** + - consolidate validation, fitted-state handling, summary construction, and covariance dispatch; + - preserve all current numerical behavior with golden regression tests. +2. **Specification tests and fit statistics** + - Hausman FE-vs-RE test; + - pooling F-test; + - Breusch-Pagan LM test; + - within, between, overall, and adjusted R-squared; + - model F-statistic; + - shared structured test-result object. +3. **Extended covariance support** + - robust covariance for RandomEffects; + - HC0/HC2/HC3 where statistically defined; + - Driscoll-Kraay covariance; + - explicit one-way/two-way cluster and bandwidth/kernel contracts. + +External alignment should use `linearmodels`, R `plm`, and R/Python sandwich implementations with explicitly matched formulas, effects, covariance definitions, and degrees-of-freedom corrections. + +Panel IV, high-dimensional fixed-effect absorption, DID/event-study, and dynamic-panel GMM are blocked on this shared foundation. + +### P2 — Survival Phase 2 + +Cox Phase 1 is implemented. The next survival work should complete foundational analysis and prediction before advanced latent-event structures. + +#### P2.1 Nonparametric survival estimators + +Implement Kaplan-Meier and Nelson-Aalen with: + +- right censoring; +- backend-consistent input validation; +- Greenwood or corresponding variance; +- confidence intervals and median survival where defined; +- stratified/grouped output; +- explicit left-truncation follow-up scope; +- alignment with R `survival` and `lifelines`. + +#### P2.2 Parametric AFT models + +Initial distributions: + +- Weibull; +- log-normal; +- log-logistic. + +Required contracts: + +- censored likelihood and parameterization documented explicitly; +- NumPy, CuPy, and Torch paths; +- model-based covariance and summary output; +- survival, hazard, cumulative-hazard, and quantile prediction; +- formula support; +- alignment with R `survreg` and `lifelines`, including scale/sign mappings. + +Frailty, Fine-Gray competing risks, multi-state models, joint models, and survival forests remain deferred until these foundations are complete. + +### P2 — Linear-model API parity and sparse infrastructure + +#### P2.3 Unpenalized multinomial logistic regression + +Issue #96 defines the base multinomial/softmax contract and implements only the unpenalized estimator. + +The work must fix: + +- identifiability convention; +- coefficient, covariance, and probability shapes; +- class and sample weighting; +- unpenalized likelihood and information criteria; +- unpenalized solver support and convergence diagnostics; +- model-based inference; +- formula semantics; +- sklearn compatibility; +- NumPy, CuPy, and Torch backend behavior. + +The Phase-1 implementation includes fit, decision function, probability prediction, hard prediction, likelihood diagnostics, and model-based inference. It must not expose L2 or any other penalty, regularization parameter, or penalized solver. Because the capability is non-tunable, no multinomial CV surface is introduced in #96. + +#### P2.4 Complete penalized multinomial suite + +Issue #98 begins only after #96 is merged and its public contract is stable. + +Penalized multinomial support should be implemented as one coherent capability package rather than exposing L2 first and leaving the remainder fragmented. The declared minimum matrix is: + +- L2; +- L1; +- ElasticNet; +- SCAD; +- MCP. + +Adaptive and group penalties may be included when their initialization and multiclass grouping conventions are mathematically fixed. If excluded, the design review must record the reason, stable unsupported behavior, tests, documentation, explicit approval, and follow-up. + +For every supported penalty, the same work package must close: + +- direct-fit objective, scaling, intercept policy, solver dispatch, warm starts, convergence, and KKT/proximal/LLA checks; +- alpha/lambda/C and mixing-parameter path/grid behavior; +- deterministic folds, scoring, selection, tie breaking, and no-leakage tests; +- backend-preserving final refit and supported final-refit inference; +- NumPy/CuPy/Torch parity and physical-GPU validation; +- external alignment and machine-readable benchmark evidence where performance is claimed; +- EN/CN documentation and changelog synchronization. + +The issue may use a bounded internal PR sequence, but no partial public capability should be advertised as complete, and #98 must not close after only L2 or only direct-fit support. + +#### P2.5 Sparse backend contract + +Define a shared sparse-input policy before adding estimator-specific support: + +- SciPy CSR/CSC; +- CuPy sparse; +- Torch sparse CSR where viable; +- supported operations and solver matrix; +- no silent densification; +- memory-budget and failure tests; +- explicit unsupported combinations. + +This work is a prerequisite for high-dimensional fixed effects, mixed models, sparse multinomial follow-up, and several large-scale algorithms. + +### P3 — Feature-driven technical debt + +Refactor only when a bounded feature or correctness task provides regression coverage. + +Current candidates: + +- split candidate generation, fold execution, selection, and final refit in `_penalized_cv.py`; +- split long FISTA/FISTA-BB solver functions by state update, line search, stopping, and diagnostics; +- unify repeated backend array-copy and scalar-extraction helpers; +- reduce duplicated CPU/CuPy/Torch fit paths where one backend-generic implementation preserves device semantics; +- unify duplicated IRLS coordinate-descent implementations only after objective and stopping contracts are frozen. + +Do not open a single repository-wide “unify all backends and solvers” PR. + +### P4 — Deferred module expansion + +The following remain valid long-term directions but are not in the immediate queue: + +- mixed-effects models and GEE; +- meta-analysis; +- changepoint detection; +- multivariate methods; +- copulas; +- multiple imputation; +- nonlinear least squares; +- advanced ANOVA/repeated-measures workflows; +- advanced robust covariance; +- tensor/adaptive/shape-constrained GAM; +- kernel SVM and broad unsupervised expansion. + +A deferred module can be promoted only with a concrete user need, a scoped design, three-backend feasibility, external baselines, and a clear maintenance owner. + +## 5. Definition of done + +A statistical feature is complete only when all applicable items pass: + +- applicable `.claude` and `dev/AGENTS.md` hard gates are satisfied; +- public API and failure behavior are documented; +- NumPy, CuPy, and Torch execution paths exist, or an explicitly approved exception is recorded; +- explicit device requests do not silently fall back; +- every tunable direct-fit capability has its CV path, selection, and final refit completed in the same declared work package; +- strict inference is implemented or the estimator is explicitly estimation-only; +- formula semantics are tested where the API supports formulas; +- external comparisons use aligned objective normalization, penalties, solvers, ties, tolerances, and feature sets; +- CPU unit/regression/compatibility tests pass; +- physical-GPU validation covers maintained CuPy and Torch paths; +- performance claims use synchronized, provenance-bearing artifacts; +- English and Chinese user documentation and changelog claims remain consistent; +- no stale fitted state, hidden fallback, or untracked diagnostic script is introduced. + +## 6. Issue hygiene + +Each active roadmap package must have one primary GitHub issue. Split implementation into child or follow-up issues only when this does not create a partially advertised public capability or violate direct-fit/CV closure. + +Every issue must include: + +- context and user impact; +- scope and non-goals; +- public API decisions; +- statistical definitions and parameterization; +- backend/device behavior; +- direct-fit/CV status for tunable capabilities; +- inference and formula implications; +- external baseline matrix; +- test and physical-GPU gates; +- documentation and benchmark outputs; +- dependencies; +- acceptance criteria. + +Close issues using evidence from merged commits, CI, external comparisons, and physical-GPU runs. Do not close an issue solely because a class or function name exists. diff --git a/dev/plans/TO_DO.md b/dev/plans/TO_DO.md index 042e80362..901f1e5a1 100644 --- a/dev/plans/TO_DO.md +++ b/dev/plans/TO_DO.md @@ -1,191 +1,176 @@ # statgpu TO DO -> Primary planning document. Last updated: 2026-06-15. -> See also `archive/PLAN_UNIFIED.md` for historical context. +> Compact execution queue and mandatory completion checklist. +> Canonical roadmap: [`ROADMAP.md`](ROADMAP.md) +> Issue index: [`ISSUES.md`](ISSUES.md) +> Development guide: [`../AGENTS.md`](../AGENTS.md) +> Hard automation protocol: [`.claude/workflows/new-module-dev.md`](../../.claude/workflows/new-module-dev.md) +> Last synchronized: **2026-08-06**, release **0.2.4**, commit `0aeeb95b60e3e274053b8f1b6427ae50c8eec015`. -## 开发门禁（必须遵守） +This file is intentionally shorter than `dev/AGENTS.md` and the `.claude` workflows, but it is not a weaker checklist. When wording conflicts, the applicable `.claude` workflow/skill takes precedence, followed by `dev/AGENTS.md`. `ROADMAP.md` controls priority; GitHub issues control executable scope. Roadmap and issue scope may narrow work but may not weaken the hard gates. -### 功能门禁 +## 1. Required task classification -- 每次新增功能，必须同时提供：NumPy (CPU)、CuPy (GPU)、Torch (GPU) 三条路径 -- 每次新增统计功能后，必须补外部框架对标验证（statsmodels、sklearn、R） -- 外部对标时必须显式统一口径（同一特征集合、ties/solver、正则设置） +Before implementation, classify the touched impact axes and record which gates are active: -### 推断门禁 +- public API; +- backend, dtype, device, memory ownership, or fallback; +- loss, penalty, solver, or loss × penalty capability; +- cross-validation; +- inference; +- formula/model-matrix semantics; +- benchmark or performance; +- documentation-only. -- Ridge/Lasso strict 模式必须通过外部对齐阈值：coef 1e-6, bse 1e-3, p-value 5e-2 -- strict 失败策略：默认 raise error +Choose the broader classification when uncertain. Documentation-only work does not activate runtime gates unless it changes a support or performance claim. -### 设备一致性门禁 +Every development report must end with exactly one workflow status: -- strict 模式输出在 CPU/GPU 上对齐 +- `COMPLETE` — all active local blocking gates pass and required docs/artifacts are current; +- `PARTIAL_REMOTE_PENDING` — local work is complete, but specified physical-GPU, R/external, or large-benchmark evidence is unavailable; +- `BLOCKED_NEEDS_USER_APPROVAL` — continuation requires an explicit decision such as a backend deferral, API break, performance caveat, commit, push, merge, release, or publication; +- `FAILED` — a blocking correctness, backend, formula, precision, convergence, fallback, review, or artifact gate remains unresolved. -### 工程门禁 +Do not close work as “mostly complete” or treat `planned` as a completion status. -- 每次提交：lint + type + test -- 每月稳定版：外部矩阵 + benchmark 非回退 + 文档同步 +## 2. Non-negotiable development gates ---- +### 2.1 Public contract -## 模块完成度 (2026-06-17, P2 完成后) +- [ ] Define inputs, outputs, shapes, dtype/device behavior, errors, fallback behavior, statistical parameterization, and explicit non-goals before final implementation. +- [ ] Preserve sklearn-style constructor identity, `get_params` / `set_params`, cloning, fitted-state invalidation, pipeline, and CV behavior where applicable. +- [ ] User-visible unsupported combinations fail early and precisely; they do not optimize an incomplete objective or change behavior silently. -| 模块 | 完成度 | 已实现 | 关键缺失 | -|------|--------|--------|----------| -| **linear_model/** | ~90% | Ridge, Lasso, ElasticNet, Logistic, 7 GLM, Penalized, Ordered, CV | multinomial, sparse input | -| **glm_core/** | ~85% | 6 solvers, 7 families, 5 links | solver 拆分优化 | -| **penalties/** | ~95% | 12 penalties (L1/L2/EN/SCAD/MCP/Adaptive/Group) | 无 | -| **survival/** | ~45% | CoxPH, CoxPHCV, Breslow/Efron, robust SE, cluster, delayed entry | strata, frailty, time-varying | -| **inference/** | ~80% | 15 distributions, p-value adjustment, bootstrap, permutation | 无 | -| **unsupervised/** | ~95% | 12 estimators (PCA, KMeans, DBSCAN, tSNE, UMAP, NMF, GMM...) | sparse input | -| **nonparametric/kernel_methods/** | ~80% | 7 kernels, KernelRidge, KernelRidgeCV, Nystroem, KernelPCA | SVM | -| **panel/** | ~70% | PanelOLS, RE, PooledOLS, BetweenOLS, FDO, FMB, HAC, formula | IV, tests, R² variants | -| **nonparametric/splines/** + **semiparametric/** | ~60% | bspline, natural_cubic, SplineTransformer, cyclic, thin plate, GAM | tensor product, adaptive | -| **covariance/** | ~60% | EmpiricalCovariance, LedoitWolf, OAS, ShrunkCov, MinCovDet, GraphicalLasso | OGK, M-estimator | -| **anova/** | ~60% | f_oneway, f_twoway, f_welch, tukey_hsd, bonferroni, effect sizes | repeated measures, Type II/III | -| **nonparametric/** | ~70% | KDE, kernel regression, bandwidth selection | 无 | -| **feature_selection/** | ~80% | KnockoffSelector, StepwiseSelector | 无 | -| **metrics/** | ~60% | ROC, AUC, confusion matrix | VIF, influence | -| **diagnostics/** | ~50% | RegressionDiagnostics | BP test, DW test | -| **mixed_model/** | 0% | ❌ | lme4/nlme 等效 (LMM, GLMM, GEE) | -| **meta_analysis/** | 0% | ❌ | metafor 等效 (rma, meta-regression, NMA) | -| **changepoint/** | 0% | ❌ | changepoint 等效 (PELT, Bayesian, batch) | -| **multivariate/** | 0% | ❌ | MASS/candisc 等效 (LDA, QDA, MANOVA, CCA, FA) | -| **copula/** | 0% | ❌ | copula 等效 (Gaussian, t, Vine copula) | -| **imputation/** | 0% | ❌ | mice 等效 (MICE, RF imputation, MI pooling) | -| **nonlinear/** | 0% | ❌ | minpack.lm 等效 (NLS, Levenberg-Marquardt) | - ---- - -## 待完成项 +### 2.2 Three backends and device locality -### P0: 进行中 +- [ ] Every new or materially changed statistical method implements NumPy, CuPy, and Torch; CPU-only work is incomplete. +- [ ] A backend deferral requires explicit user approval plus the reason, user-visible failure behavior, deterministic skip condition, and follow-up issue. +- [ ] Explicit `device="cuda"` and `device="torch"` never silently fall back to CPU or another backend; only `device="auto"` may select a backend automatically. +- [ ] Core fitting, prediction, scoring, inference, and validation remain on the selected backend; no hidden full-array GPU-to-CPU transfer is introduced. +- [ ] Fallback, approximate inference, dtype conversion, or device conversion is part of the public contract and is visible through an error, warning, result field, or report. +- [ ] GPU-buffer-owning estimators implement the documented `gpu_memory_cleanup` lifecycle, including cleanup methods and finalization behavior without discarding fit state prematurely. -- [ ] 完善推断严谨性：跨设备一致性（SE/t/z/p/CI、AIC/BIC/LLF） -- [ ] CoxPH Cython 编译版本调试（当前仍需保留 Python fallback） -- [ ] 补 `PenalizedLogisticRegression.predict_proba` smoke test，并修复 wrapper 内 `np` / `_ETA_CLIP` 依赖一致性 - -### P1: API parity / 功能补齐 - -- [ ] LogisticRegression: multinomial/softmax -- [ ] LogisticRegression penalized parity: 将 L1/elastic-net 能力对齐到公开 API、文档和测试矩阵 -- [ ] CoxPH: strata, frailty, time-varying covariates -- [ ] 稀疏输入支持：明确 linear_model 与 unsupervised estimators 的 CSR/CSC 支持范围 -- [ ] CoxPHCV: 跨 CPU/CuPy/Torch 回归验证，覆盖 `entry`、`cluster`、`predict`、`score`、cache key 和文档示例 -- [ ] RidgeCV: 公开/文档化 alpha path 结果，补 sklearn 对标测试；单模型 `Ridge.warm_start` 作为待评估 API +### 2.3 Reuse and architecture -### P2: 新模块扩展 +- [ ] Reuse `BaseEstimator`, `statgpu/backends/`, existing array helpers, solver/penalty registries, `statgpu/cross_validation/`, formula infrastructure, and `statgpu/inference/` before adding private parallel implementations. +- [ ] Model modules do not scatter direct CuPy imports or duplicate backend selection and conversion logic without a documented architectural reason. +- [ ] New inference distribution, p-value, or interval logic checks existing backend-aware inference utilities before adding another implementation. -**anova/** (15% -> 目标 60%): -- [ ] 二因素 ANOVA (with/without interaction) -- [ ] Welch ANOVA (unequal variances) -- [ ] 事后检验: Tukey HSD, Bonferroni -- [ ] 效果量: Cohen's f, partial eta-squared；保留 one-way `eta_squared` 回归测试 +### 2.4 Direct fit and CV closure -**covariance/** (30% -> 目标 60%): -- [ ] GraphicalLasso / GraphicalLassoCV (稀疏逆协方差) -- [ ] MinCovDet (稳健估计) -- [ ] ShrunkCovariance (通用收缩) - -**panel/** (45% -> 目标 70%): -- [ ] FamaMacBeth -- [ ] HAC/Newey-West 协方差 -- [ ] PooledOLS, BetweenOLS, FirstDifferenceOLS - -**nonparametric/splines/** + **semiparametric/** (35% -> 目标 60%): -- [ ] sklearn SplineTransformer API (fit/transform) -- [ ] 循环样条 (cyclic cubic) -- [ ] 薄板样条 (thin plate) - -**nonparametric/kernel_methods/** (60% -> 目标 80%): -- [ ] Nystroem 近似 -- [ ] KernelPCA -- [ ] chi2_kernel - -### P3: 大规模重构 - -- [ ] `_penalized_cv.py` 文件拆分 (2800+ 行) -- [ ] `_solver.py` 函数拆分 (fista_bb_solver 470 行) -- [ ] `_fit_cpu` / `_fit_gpu` / `_fit_torch` 代码重复消除 -- [ ] `_irls_cd` 和 `_irls_cd_gpu` 统一为 backend-agnostic 实现 -- [ ] `_penalized_cv.py` 6 个 FISTA 循环提取为共享 `_fista_cv_step` - -### P4: 性能优化 - -- [ ] Panel 双向 demeaning 批量化（减少 GPU kernel launch） -- [ ] KernelRidgeCV CuPy 路径实现/验证（确认是否仍会回退到 NumPy） -- [ ] 加权 CV 快速路径 - -### P5: 代码质量 - -- [ ] `_array_ops.py` 与 `_utils.py` helper 统一（`_xp_copy` / `xp_copy` 等重复） -- [ ] `_solver.py` 标量提取模式统一（4 种不同方式） -- [ ] `_solver.py` 异常捕获收窄（已部分完成） -- [x] Panel summary() 返回 PanelSummary 结构化对象 ✅ -- [x] PanelOLS.predict() 包含固定效应 (entity_ids/time_ids) ✅ -- [x] ANOVA float32 支持 (dtype 参数) ✅ - -### P6: Loss-as-Plugin 扩展 (详见 `development_priority.md`) - -**核心策略:** 将新方法实现为 `GLMLoss` 子类，接入现有 PenalizedGLM 框架，零改动 solver/penalty 代码。 - -**Phase 1: 新 Loss 函数 (最高 ROI，2026 Q3)** -- [ ] **QuantileLoss** — `quantreg::rq()` — 2-3 周 — 分位数回归 + 所有 penalty -- [ ] **HuberLoss** — `MASS::rlm()` — 2-3 周 — 稳健 M-estimator + 所有 penalty -- [ ] **CoxPH refactor** — `survival::coxph()` — 3-4 周 — 将现有 CoxPH 接入 loss 框架 -- [ ] **BisquareLoss** — `robustbase::lmrob()` — 1 周 — MM-estimator 的 S 步 - -**Phase 2: 推断统一 (2026 Q3-Q4)** -- [ ] 统一 `summary()` 支持所有 loss 类型 -- [ ] 稳健标准误 (HC0-HC3/HAC/cluster) 适用于所有 loss -- [ ] Model selection (AIC/BIC/CV) 适用于所有 loss - -**Phase 3: 新模块 (2026 Q4-2027 Q1)** -- [ ] **混合效应模型** — `lme4::lmer()` — 8-12 周 — 新模块，需稀疏矩阵 -- [ ] **元分析** — `metafor::rma()` — 4-6 周 — 新模块 -- [ ] **GEE** — `geepack::geeglm()` — 3-4 周 — 新模块 -- [ ] **变点检测** — `changepoint::cpt.mean()` — 4-6 周 — 新模块 - -**Phase 4: 高级方法 (2027+)** -- [ ] **多元统计** — `MASS::lda()`, `candisc::cancor()` — 6-8 周 -- [ ] **Copula** — `copula::fitCopula()` — 4-6 周 -- [ ] **SEM** — `lavaan::sem()` — 8-10 周 - -**已有计划扩展:** -- [ ] GAMM (扩展 plan_spline.md) — `mgcv::gamm()` — 广义可加混合模型 -- [ ] 竞争风险 (扩展 plan_survival.md) — `cmprsk::crr()` — Fine-Gray 模型 -- [ ] 变异函数/克里金 (扩展 plan_spatial.md) — `gstat::variogram()` — 空间插值 - ---- - -### CRAN Task View 覆盖审计 (35 个 Task View) - -| 状态 | 数量 | 说明 | -|------|------|------| -| ✅ 已实现 | 3 | Cluster, Distributions, HPC | -| 🟡 部分实现/已有计划 | 20 | Econometrics, Finance, FDA, GraphicalModels, ML, MetaAnalysis, Missing, MixedModels, Multivariate, NumericalMath, Optimization, Psychometrics, Robust, Spatial, SpatioTemporal, Survival, TimeSeries, Causal, ExperimentalDesign(partial) | -| ❌ 缺失但应覆盖 | 2 | DoE(实验设计), DifferentialEquations(ODE求解) | -| ❌ 战略排除 | 1 | Bayesian — Python已有成熟GPU方案(PyMC/NumPyro/Pyro/TFP)，不竞争，提供桥接 | -| ❌ 不适用 | 9 | ChemPhys, MedicalImaging, ModelDeployment, NLP, Phylogenetics, ReproducibleResearch, TeachingStatistics, Tracking, WebTech | - -详见 `cran_r_package_supplement.md` Part F/G。 - ---- - -## 已完成历史 (2026-04 ~ 2026-06) - -> 详细记录见 `archive/PLAN_UNIFIED.md` 和 git history。 - -- RidgeCV / LogisticRegressionCV 完整实现 (2026-04-21) -- CoxPH C-index / Efron ties 修复 (2026-04-20) -- 完整推断体系：LinearRegression / Ridge / Logistic / CoxPH (HC0-HC3/HAC) -- 12 个 Unsupervised estimator -- 5 个新模块：ANOVA, Covariance, Kernel Methods, Panel Data, Splines/GAM -- Panel summary() 返回 PanelSummary 结构化对象 -- PanelOLS.predict() 包含固定效应 (`entity_ids` / `time_ids`) -- ANOVA float32 支持 (`dtype` 参数) -- CoxPHCV 从骨架推进为可拟合实现，仍需跨后端回归验证和文档补齐 -- RidgeCV alpha grid/path 结果可通过 `alphas_`、`cv_results_`、`mean_mse_` 获取，仍需 API 文档和 sklearn 对标测试 -- PR #49: 110+ bug fixes, 428 tests -- PR #48: Panel, ANOVA, Covariance review fixes -- Async FISTA (v22e): 最高 5.41x 加速 -- v23c: 1043/1043 ALL PASS +- [ ] Every public tunable loss × penalty capability supported by direct `fit()` also supports the CV layer: path/grid generation, deterministic folds, fold scoring, best-parameter selection, and final refit. +- [ ] CV preserves the declared loss, weighting, backend, device, dtype, formula alignment, and objective normalization. +- [ ] A capability may omit CV only when it is explicitly non-tunable or the user approves a deferral with failure behavior, tests, docs, and a follow-up issue. +- [ ] Do not advertise a partially completed penalized module: when a roadmap package declares a penalty matrix, direct fit and CV must close for the whole declared matrix before the package is marked complete. + +### 2.5 Inference contract + +- [ ] A model family that exposes `compute_inference`, `summary()`, covariance, SE, p-values, or confidence intervals implements inference or is explicitly documented and tested as estimation-only. +- [ ] Strict inference is the default path; strict failure raises by default. Approximate or downgraded inference requires explicit opt-in and visible status. +- [ ] Inference outputs remain consistent across supported backends, including applicable `coef`, `bse`, `t/z`, `p`, confidence intervals, `AIC`, `BIC`, and `LLF` fields. +- [ ] Current default external-alignment thresholds are recorded where applicable: coefficient error `<= 1e-6`, BSE error `<= 1e-3`, and p-value error `<= 5e-2`; a different tolerance requires a statistical or numerical justification. +- [ ] Direct-fit and final-CV-refit inference use the same declared estimator contract; fold models remain estimation-only only when that behavior is intentional and tested. + +### 2.6 Formula contract + +- [ ] Formula-facing methods test intercept handling, categorical reference levels, interactions, transforms, missing-data row alignment, feature names, and prediction column order. +- [ ] Array and formula paths agree after model-matrix alignment. +- [ ] R-style/Patsy semantics are externally checked where applicable; unsupported syntax has a precise error and documented boundary. + +### 2.7 Objective, penalty, precision, and convergence + +- [ ] State whether the objective uses a sum or average loss and whether the intercept is penalized. +- [ ] Map external regularization scales explicitly; for example, use `lambda_external = n * lambda` when comparing average-loss statgpu objectives with summed-loss references. +- [ ] Do not alter the statgpu objective merely to force agreement with an external package. +- [ ] Validate loss value, gradient, Hessian or Hessian-vector behavior, proximal/KKT conditions, line search, stopping rules, and convergence status for the active component matrix. +- [ ] Precision and convergence are blocking before performance optimization. +- [ ] Numeric recovery catches only recognized numerical-domain or rank failures; OOM, device, shape, index, contract, and programming errors remain fatal. + +### 2.8 External and architecture-specific validation + +- [ ] Use the strongest available baseline: analytic/derivative check, trusted statgpu path, Python reference, R reference, then documented numerical invariants. +- [ ] External comparisons align feature sets, weights, ties, solver, penalty, objective normalization, `alpha` / `C`, `max_iter`, and `tol`. +- [ ] Prefer statsmodels for statistical inference, sklearn for estimator/prediction behavior, and authoritative R packages for key statistical definitions. +- [ ] Activate the relevant architecture matrix: loss, penalty, solver, direct-fit/CV, inference, formula, backend helper, survival, or nonparametric/unsupervised tests. +- [ ] Broad cross-axis changes extend a maintained matrix test rather than relying only on isolated smoke tests. + +### 2.9 Testing, review, and validation tier + +- [ ] Run applicable lint, type, unit, regression, compatibility, formula, external-alignment, and import-order tests. +- [ ] Add deterministic NumPy/CuPy/Torch parity tests and explicit unavailable-backend errors/skips. +- [ ] Complete maintained physical CuPy and Torch validation for a `COMPLETE` claim when those paths are active; otherwise report `PARTIAL_REMOTE_PENDING` with exact commands and missing resources. +- [ ] Record the highest completed validation tier: `local-minimal`, `local-full`, or `remote-full`. +- [ ] Run code review and fix cycles until no unresolved CRITICAL or HIGH issue remains; remaining medium findings require a documented behavior boundary or follow-up issue. +- [ ] Tests must independently calculate expected statistical values where feasible rather than only comparing one statgpu path with another. + +### 2.10 Performance and evidence artifacts + +- [ ] Performance work starts only after correctness, precision, and convergence gates pass. +- [ ] GPU timing synchronizes the correct CuPy/Torch backend before and after each measured region. +- [ ] Record target scale, data shape, dtype, hardware, software environment, timing scope, transfer policy, repeats, seeds, and comparison identity. +- [ ] Store machine-readable benchmark evidence under `results/*.json` and a concise audit summary; do not support public claims with rounded prose alone. +- [ ] Do not claim universal GPU acceleration; report measured crossover and slower regimes. +- [ ] Benchmark and remote evidence must be provenance-bearing and reproducible, with source hashes or equivalent source identity where the workflow requires them. + +### 2.11 Documentation and release surface + +- [ ] Update exports, README/USAGE where applicable, model pages, compatibility matrices, and changelogs in the same feature change. +- [ ] Follow EN-first/CN-follow: update `docs/en/` and English entry points, then synchronize `docs/cn/` and Chinese entry points. +- [ ] Keep root `CHANGELOG.md`, `docs/en/changelog.md`, and `docs/cn/changelog.md` consistent with the actual capability and validation evidence. +- [ ] Model documentation includes applicable objective, estimating equation, covariance/inference, parameters, CPU/CuPy/Torch examples, strict/approx behavior, outputs, FAQ, external validation, and references. +- [ ] Remote or benchmark claims cite auditable artifact paths rather than only verbal conclusions. + +### 2.12 Required completion report + +- [ ] Report impact classification, workflow status, validation tier, files changed, backend matrix, CV status, inference status, formula status, objective/penalty mapping, precision/convergence evidence, tests, external baselines, physical-GPU evidence, benchmark artifacts, review outcome, documentation changes, and any pending remote commands. +- [ ] Commits, pushes, PR creation, merges, tags, releases, and package publication occur only after an explicit user request. +- [ ] Credentials are never read from tracked Markdown or settings files; remote execution uses the maintained untracked/environment configuration path. + +## 3. Active execution queue + +### P0 — planning and integration + +- [ ] Merge roadmap reconciliation PR #89 and use `ROADMAP.md` as the only current priority source. +- [ ] #90 — synchronize benchmark dashboard PR #76 with current `master` without adding new benchmark families in the same change. +- [ ] Regenerate and validate dashboard data, inventory, parse report, and deployed assets during #90. + +### P1 — benchmark evidence and dashboard readiness + +- [ ] #91 — add a canonical CV benchmark source covering RidgeCV, LassoCV, ElasticNetCV, LogisticRegressionCV, PenalizedGLM_CV, and CoxPHCV. +- [ ] Record folds, grid/path size, warm starts, CV time, final-refit time, selected parameter, score, failures, synchronization, and timing scope. +- [ ] #92 — complete production-path browser QA, cross-browser smoke, accessibility checks, and documentation navigation before proposing PR #76 for `master` integration. + +### P1 — panel workflow completion + +- [ ] #93 — refactor panel models onto a shared base and covariance registry while preserving numerical behavior. +- [ ] Add Hausman, pooling F, and Breusch-Pagan LM tests. +- [ ] Add within/between/overall/adjusted R-squared and model F-statistics. +- [ ] Add robust RandomEffects covariance, HC0/HC2/HC3 where defined, and Driscoll-Kraay covariance. +- [ ] Validate against `linearmodels`, R `plm`, and aligned sandwich covariance references. + +### P2 — survival foundations + +- [ ] #94 — implement Kaplan-Meier and Nelson-Aalen estimators with variance, confidence intervals, grouped output, and external alignment. +- [ ] #95 — implement Weibull, log-normal, and log-logistic AFT models with three backends, model-based inference, formula support, and prediction functions. + +### P2 — multinomial and sparse foundations + +- [ ] #96 — define and implement the unpenalized-only multinomial/softmax estimator, including identifiability, shapes, inference, formula semantics, and three-backend parity. +- [ ] #96 must expose no regularization parameter or penalized solver; it is non-tunable and therefore introduces no multinomial CV surface. +- [ ] #98 — after #96, implement the complete penalized multinomial suite as one capability package. +- [ ] #98 must cover at least L2, L1, ElasticNet, SCAD, and MCP across NumPy, CuPy, and Torch. +- [ ] #98 must close direct fit, path/grid, deterministic CV, selection, final refit, supported inference, external alignment, physical-GPU validation, and EN/CN docs for every declared penalty before completion. +- [ ] #97 — define a shared SciPy/CuPy/Torch sparse-input contract with no silent densification. + +### P3 — feature-driven technical debt + +- [ ] Split `_penalized_cv.py` by candidate generation, fold execution, selection, and final refit when #91 supplies regression coverage. +- [ ] Split long FISTA-family solver functions into bounded numerical components without changing objective or stopping contracts. +- [ ] Unify duplicated array-copy and scalar-extraction helpers. +- [ ] Reduce backend duplication only where device behavior remains explicit and fully tested. + +## 4. Deferred + +The following are not immediate priorities: Panel IV, high-dimensional fixed effects, DID/event study, dynamic-panel GMM, frailty, Fine-Gray, multi-state survival, mixed models, GEE, meta-analysis, changepoints, copulas, multiple imputation, nonlinear least squares, and broad new unsupervised families. + +Promote a deferred item only through a scoped GitHub issue satisfying `ROADMAP.md`, this checklist, `dev/AGENTS.md`, and the applicable `.claude` workflow/skill. diff --git a/dev/plans/plan.md b/dev/plans/plan.md index 211872749..fa5068082 100644 --- a/dev/plans/plan.md +++ b/dev/plans/plan.md @@ -1,7 +1,8 @@ # StatGPU Plan Delta (2026-04-05) -> Canonical merged planning entry: `PLAN_UNIFIED.md` in workspace root. -> This file is retained for history and quick delta tracking. +> **Historical document.** This delta is preserved for context and is superseded by [`ROADMAP.md`](ROADMAP.md). Do not use the queue below as a current capability or priority inventory. +> +> Original status: canonical merged planning entry was `PLAN_UNIFIED.md` in the workspace root; this file was retained for history and quick delta tracking. ## Priority Queue (after bootstrap phase)