feat(phenotyping): spatially-aware graph_mi target + opt-in denoising, debias scoring, marker QC

[nceglia]

Summary

Improves coarse cell-type classification in spatial data (scRNA by proxy) while keeping the spirit of genevector. Changes are driven by a benchmark (kept in a gitignored scratch/ lab notebook) over the synthetic dataset (true labels; Poisson-dropout to mimic Xenium sparsity) and GBM/CRC Xenium tiles (reference = grafiti_celltype), using HVG∪SVG gene selection via grafiti's Moran's I.

★ Headline: spatially-aware graph_mi target

Compute the co-expression target on spatially-aggregated expression so neighbour aggregation denoises sparse per-cell counts before estimating MI → a cleaner gene embedding → better coarse typing.

dataset	mi ARI	graph_mi ARI
crc_xenium	0.425	0.440
gbm_xenium	0.141	0.175

graph_mi − mi: mean +0.025 ARI, 7/8 tiles win, worst −0.0004 (never meaningfully hurts). Matches the requested "graph-based target for speedy MI / spatially-motivated computation." Also adds graph_cross_mi (asymmetric ligand→neighbour) for niche/communication directionality.

Multi-kernel graph_mi/graph_cross_mi (all numerically identical, diff ~1e-15; auto-selected: GPU torch on device='cuda' → rayon Rust extension if built → torch CPU → numpy):

• Vectorized torch kernel: one scatter_add per self-gene builds all neighbour joint histograms — O(P) Python loops, not O(P²). Runs on GPU (--device cuda) or CPU.
• rayon Rust kernel (compute_cross_mi_pairs, sibling of the existing compute_mi_pairs): fastest on multi-core CPU.

Benchmark (Apple M4 Max, 16 cores, no CUDA):

panel	numpy	torch-CPU	Rust (16c)
P=400, n=5k	24.9s	0.69s	0.22s
P=1000, n=10k	335s	15.5s	4.4s
P=2000, n=20k	—	129s	72s

→ Rust ~80–110x numpy and ~2–5x torch-CPU; torch-GPU leads for large panels on a CUDA box.

Other changes

CellEmbedding

• Bug fix: dataset_vector was initialised to zeros and never set, silently disabling the contrastive subtraction in get_predictive_genes. It is now the mean cell vector.
• phenotype_probability is vectorised and every regime-dependent lever is an opt-in arg (all default-off → previous behaviour): debias (subtract a fraction of the background vector), contrastive scoring, score_norm (zscore/rank per-phenotype column normalization so a phenotype close to everyone doesn't win by default), scoring-time spatial smoothing (smooth_graph/smooth_alpha/smooth_adaptive, non-mutating), and spatial label propagation (lp_graph/lp_alpha).
• denoise_cell_vectors: opt-in count-adaptive spatial message passing on cell vectors (low-count cells borrow from neighbours; dense cells barely move). +0.2 ARI under severe dropout on the synthetic.
• qc_marker_dict: flags absent / low-specificity (tau index) / off-target markers and thin phenotypes — guidance for building marker matrices. Correctly flags the weak synthetic marker_0, and on GBM flags OLIG2-as-Malignant (peaks in OPC) and broadly-expressed myelin genes.

scripts/phenotype_workflow.py — runnable marker-driven phenotyping CLI: AnnData + JSON markers + output dir → HVG∪SVG selection (grafiti Moran's I when spatial), graph_mi for spatial data, marker QC, optional denoising, assignment. Outputs phenotyped.h5ad, assignments.csv, marker_qc.csv, summary.json.

python scripts/phenotype_workflow.py --input tile.h5ad --markers markers.json \
  --output out/ --target auto --denoise --grafiti-path /path/to/grafiti

Honest negative results (documented; shipped opt-in / default-off)

• Subtracting the dataset vector (Welcome to genevector Discussions! #1) is regime-dependent: helps balanced tissue (GBM +0.03) but hurts epithelial-dominated CRC (−0.09, the mean ≈ epithelial). → debias opt-in, default off.
• Label propagation (Installation via pip fails #4) trades partition-agreement for marker-coherence; net ≤0 vs reference. → opt-in, default off.
• Cell-vector spatial smoothing helps sparse / spatially-pure data but contaminates identity in intermixed tissue → count-adaptive + opt-in.
• Embedding-kNN denoise / hybrid scoring: no robust gain. For coarse typing the absolute-cosine scoring is already near-optimal, so the win came from the embedding/target (graph_mi), not post-hoc scoring tricks.
• marker_auroc as a metric is anti-correlated with true accuracy under dropout (it rewards raw-marker matching) — use true labels.

Tests

tests/test_graph_targets.py (graph_mi/graph_cross_mi) and tests/test_phenotyping.py (dataset_vector fix, debias/contrastive, label-prop, denoising, marker QC). 89 passed (1 pre-existing unrelated failure in test_grafiti_parity — grafiti's generate_synthetic_data now returns AnnData not DataFrame — is independent of this PR).

🤖 Generated with Claude Code