Skip to content

Multicollinearity diagnostics + GLM χ² (#27 batch 4) - #31

Merged
sipemu merged 1 commit into
mainfrom
feature/issue-27-mc-dispersion
May 28, 2026
Merged

Multicollinearity diagnostics + GLM χ² (#27 batch 4)#31
sipemu merged 1 commit into
mainfrom
feature/issue-27-mc-dispersion

Conversation

@sipemu

@sipemu sipemu commented May 28, 2026

Copy link
Copy Markdown
Contributor

Batch 4 of 4 for #27. Adds 4 multicollinearity + dispersion-related diagnostics.

Implemented

Diagnostic Expression Output
High-VIF mask `ps.high_vif_predictors("x1", "x2", ..., threshold=10.0)` struct{is_high, n_high, n_features}
Generalized VIF `ps.generalized_vif("x1", ..., group_sizes=[2, 3])` struct{gvif, n_groups}
Logistic Pearson χ² `ps.pearson_chi_squared_logistic("y", "x1", ...)` struct{chi_squared, df_resid, n_observations}
Poisson Pearson χ² `ps.pearson_chi_squared_poisson("y", "x1", ...)` same

All four follow the rlib split pattern. The Pearson χ² statistics are just Σ pearson_residual² — reusing batch-2 plumbing without needing the generic GlmFamily trait.

Tests

  • Pythontests/test_mc_dispersion.py, 9 new pytests:
    • VIF mask flags all 3 correlated columns at threshold 5, none of 3 independent columns at threshold 10.
    • GVIF on 5 features split into groups [2, 3] returns 2 GVIF values.
    • χ²/df_resid lands within an order of magnitude of 1.0 for well-specified data on both logistic and Poisson.
  • Rusttests/rust_api.rs: 4 smoke tests appended to diagnostic_fits.

Full Python suite: 457/457 pass (448 prior + 9 new). Rust integration: 12/12 pass.

Issue #27 status after this PR

This completes the user-facing scope I had time for. Remaining items I intentionally deferred:

  • estimate_dispersion_pearson / estimate_dispersion_deviance — trivially computed from χ²/df_resid, so callers can derive them from this PR's output.
  • classify_condition_numberConditionSeverity is already returned by the existing condition_number diagnostic.
  • variance_decomposition_proportions (Belsley-Kuh-Welsch) — returns a full Mat, needs a more complex output schema.
  • Probit / cloglog / negative binomial / Tweedie residual variants — same *_pearson_residuals / *_deviance_residuals / *_working_residuals triple for each family, ~12 more functions.
  • Standardized Pearson / standardized deviance residuals — need GLM leverage, which isn't currently exposed.

Recommend closing #27 with this PR and filing a smaller follow-up issue for the GLM extensions if/when there's demand.

Test plan

  • cargo build --no-default-features clean
  • cargo clippy --all-targets -- -D warnings clean
  • cargo fmt --check clean
  • cargo test --no-default-features — 12/12 pass
  • pytest — 457/457 pass
  • CI matrix

🤖 Generated with Claude Code

Add 4 diagnostic expressions following the established pattern from
batches 1-3:

  - high_vif_predictors_fit + pl_high_vif_predictors: boolean mask of
    features whose VIF exceeds the threshold (default 10.0). Computes
    VIF internally then thresholds.
  - generalized_vif_fit + pl_generalized_vif: VIF generalized to
    grouped predictors (e.g. categorical dummies). Takes a
    comma-separated `group_sizes` string parsed in Rust.
  - pearson_chi_squared_logistic_fit + pl_pearson_chi_squared_logistic:
    Σ pearson_residual² from a logistic fit, plus df_resid and
    n_observations.
  - pearson_chi_squared_poisson_fit + pl_pearson_chi_squared_poisson:
    same for Poisson.

Three new struct output schemas:
  vif_mask_output_dtype{is_high: List<Bool>, n_high, n_features}
  gvif_output_dtype{gvif: List<Float64>, n_groups}
  chi_squared_output_dtype{chi_squared: Float64, df_resid, n_observations}

Python (exprs/regression.py): high_vif_predictors, generalized_vif,
pearson_chi_squared_logistic, pearson_chi_squared_poisson — all
exported from the top-level package.

Tests:
  tests/test_mc_dispersion.py: 9 pytests covering shape, threshold
  behaviour, GVIF group bookkeeping, and χ²/df_resid magnitude on
  well-specified data.
  tests/rust_api.rs: 4 smoke-test blocks added to diagnostic_fits.

Full Python suite: 457/457 pass (448 prior + 9 new).
Rust integration: 12/12 pass.

Deferred to a final small follow-up: estimate_dispersion_pearson,
estimate_dispersion_deviance, classify_condition_number,
variance_decomposition_proportions, probit/cloglog/negative_binomial/
tweedie residual variants, standardized Pearson/deviance residuals.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented May 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.33333% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
python/polars_statistics/exprs/regression.py 93.33% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@sipemu
sipemu merged commit 804f810 into main May 28, 2026
17 of 18 checks passed
@sipemu
sipemu deleted the feature/issue-27-mc-dispersion branch May 28, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant