PCA: cross-validate a block that already has missing cells

[kgdunn]

Found while cross-validating the quality block of the FMC batch dryer case study (kgdunn/pid-book#273), which has 19 missing cells in 46 x 8.

The defect

PCA.fit has always taken a block with missing cells through NIPALS. PCA.select_n_components could not cross-validate the same block:

cv_scheme="ekf"       LinAlgError: SVD did not converge
cv_scheme="row_wise"  RuntimeError: Cross-validation produced NaN PRESS for every component count

The element-wise k-fold routine built its folds from the raw array, so two things broke independently:

1. the per-column centre and scale came from mean() and std() over columns holding NaN, giving NaN constants;
2. only the held-out cells were filled before the SVD, so the caller's own missing cells reached numpy.linalg.svd.

The fix

A cell the caller never measured has no true value to predict, so it is never held out and never scored. It still has to be filled for the SVD, so EM imputes it alongside the held-out cells:

• folds are assigned over the observed cells only, so every fold holds the same number of scorable cells;
• the column centre and scale are fitted on cells that are both in-fold and measured;
• PRESS, the null reference and the per-column split are accumulated over held-out measured cells alone.

A block with no missing cells takes an unchanged path and returns bit-identical numbers.

Measured

Block	Before	After
Pure noise, 46 x 8	Q² [-50.2, -110.3, -183.9, -281.3]	identical
Rank 2, 60 x 8, complete	peaks at 2 components, Q² 84.2%	identical
The same block, 12 cells removed	LinAlgError	peaks at 2 components, Q² 85.0%
FMC quality block, 19 missing	LinAlgError	runs; Q² 30.5, 25.0, −28.4, −79.5 at 5 folds

The third row is the point: the missing cells cost a little accuracy rather than changing the answer.

Tests

Two, both in tests/test_multivariate.py:

• test_pca_select_n_components_ekf_handles_missing_cells punches 12 cells out of a rank-2 block and checks that the rank is still found, that the complete and holed Q² curves peak at the same component count and agree to 0.1 there, and that no NaN reaches PRESS or Q². It fails with LinAlgError without this change.
• test_pca_select_n_components_ekf_q2_is_negative_on_noise pins the property that separates the two schemes. On pure noise the ekf Q² stays negative at every component count, while the row-wise Q² climbs monotonically past 0.5, because that scheme scores a held-out row through its own values and so measures the row's SPE rather than a prediction.

That second test also documents, in a form CI will keep honest, why cv_scheme="row_wise" warns: on data with no structure at all it reports 87% of the variation "predicted" at seven components.

Verification

• 3024 fast tests pass, 3 skipped.
• ruff check ., ruff format --check . and mypy src/process_improve clean.
• Version 1.83.1, with CHANGELOG.md and CITATION.cff in step.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LXthGpHLQFfGubBiKFtGAE

---

Generated by Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!