Skip to content

Add MEDA and oMEDA exploratory diagnostics for latent-variable models #373

Description

@kgdunn

Summary

Add two structure-discovery diagnostics that are standard in established chemometrics toolboxes but missing here:

  • MEDA (Missing-data methods for Exploratory Data Analysis): a K x K variable-relationship map computed through a fitted PCA/PLS model, cleaner than a raw correlation matrix because it uses the model subspace.
  • oMEDA (observation-based MEDA): a per-variable bar diagnostic that quantifies which variables drive the difference between two groups of observations in the model subspace.

Priority 2 of 4 in the exploratory-methods parity set (see also: ASCA/VASCA, parallel-analysis + double-CV, PLS-DA). Low implementation effort, high fit with our existing diagnostics and (downstream) with ScorePilot's brushing-and-linking UI.

Motivation

We already compute per-observation, per-variable contributions (_diagnostics.py) and a correlation-loadings plot (plots.py), but we have nothing that:

  1. Shows the variable-to-variable relationship structure as captured by the model (MEDA). A raw correlation heatmap mixes in directions the model discards; MEDA shows only the relationships the latent subspace actually represents, which is what you want when interpreting loadings.
  2. Answers the very common question "what separates these two groups of points?" directly (oMEDA). Contribution plots explain a single observation against the model center; oMEDA explains group A vs group B.

Both slot naturally into the existing convenience-method pattern (methods on PCA/PLS forwarding to standalone functions in _diagnostics.py).

What is MEDA?

MEDA produces a K x K matrix where entry (i, j) measures how strongly variable i and variable j are related within the A-component model subspace. The method is rooted in missing-data imputation theory: intuitively, for each variable it asks how well that variable is reconstructed from the others through the model, and decomposes that reconstruction by source variable. The result is a symmetric (or near-symmetric) map in [0, 1] (or [-1, 1] for the signed variant) that is then plotted as a heatmap, often seriated so that blocks of co-varying variables are visually adjacent.

Crucially MEDA is computed from the model (loadings / residuals), so increasing the number of components changes the map: it reveals which variable groupings are explained at A components.

Implementation note: implement the index faithfully from the reference below rather than from this paraphrase. The canonical algorithm builds on the model's reconstruction of the data covariance and the leave-one-variable-out imputation used in the missing-data PCA literature.

What is oMEDA?

oMEDA takes the fitted model plus a contrast (dummy) vector d over the observations, with entries +1 for group A, -1 for group B, and 0 for observations not in the comparison. It returns a length-K vector of per-variable weights showing which variables most explain the A-vs-B separation in the model subspace, rendered as a signed bar plot. This is the multivariate analogue of "diff the two groups, but only along the directions the model trusts."

Proposed implementation

  • Add standalone functions to process_improve/multivariate/_diagnostics.py:
    • meda(model, ...) -> K x K DataFrame (variable x variable), with an option for the seriation/ordering and for signed vs squared index.
    • omeda(model, group_a, group_b, ...) (or accept an explicit dummy vector d) -> length-K Series of per-variable weights.
  • Bind both as convenience methods on PCA and PLS (and, where meaningful, MBPLS/MBPCA) using the existing _model_method forwarder pattern documented in CLAUDE.md, so help/inspect.signature stay accurate and the methods are overridable.
  • Plotting in plots.py:
    • meda_plot(model) -> Plotly heatmap (with optional seriation), matching the existing plot styling/Plot wrapper.
    • omeda_plot(model, ...) -> signed bar plot, consistent with the SPE/T2 contribution bar plots already present.
  • Reuse existing model attributes (loadings_, residuals, scores_) and the missing-data machinery in _nipals.py. Do not duplicate imputation code that already exists.

ScorePilot surfacing (downstream)

This is a strong match for ScorePilot's LinkGroup brushing: brush a set of points into group A, brush another into group B, and call oMEDA to get the "what separates them" bar chart - then click a bar to drill into that variable's raw data (the linked-plots and variable-inspector machinery already exist). MEDA becomes a variable-relationship heatmap view alongside the loadings plot. All math stays upstream in process_improve; ScorePilot only adapts the output.

Acceptance criteria

  • meda(model) returns a K x K matrix; on a dataset with two clearly correlated variable blocks, the map shows those blocks; the map changes sensibly as n_components varies.
  • omeda(...) on a dataset with two groups that differ in a known subset of variables ranks those variables highest, with correct signs.
  • Convenience methods present on PCA and PLS; inspect.signature reports the underlying function minus model.
  • Plots render and are covered by the existing Plotly-assertion test style.
  • Tests use real datasets plus synthetic data with a known structure; NumPy-style docstrings with Examples.

References

  • Camacho, J. (2010). "Missing-data theory in the context of exploratory data analysis." Chemometrics and Intelligent Laboratory Systems 103(1):8-18.
  • Camacho, J. (2011). "Observation-based missing data methods for exploratory data analysis to unveil the connection between observations and variables in latent subspace models." J. Chemometrics 25:592-600. (MEDA and oMEDA.)

Notes for the implementing session

  • Read _diagnostics.py (contributions, VIP), plots.py (bar + heatmap styling), and _nipals.py (imputation) first.
  • Keep functions pure over the fitted model; no state added to __init__.
  • Bump the version (MINOR) and update CHANGELOG.md per CLAUDE.md; confirm changelog wording with the maintainer.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions