refactor: land the completed refactor in main (PRs #31–#37)

[nceglia]

Lands the completed refactor into main. 34 commits, 71 files, +5635/−4202.

main has been sitting at PR0 (the contract freeze) while PRs #31 → #37 merged branch-to-branch down a stack. Every commit here has already been reviewed and merged in one of those PRs; this is the integration merge that finally moves main. Verified clean against main (no conflicts) and 152 passing at the branch tip.

What lands

PR	scope
#31	PR2 — delete 14 dead / out-of-scope symbols (384 lines)
#32	PR3 — split the model monolith into scvi-style modules + explicit __all__
#33	PR4 — model→AnnData streamline (to_anndata, predict); retire the tcri_manager stash
#34	PR5 — unified joint_distribution engine (tools/ + _compute/), draw-once invariant
#35	PR6–9 — metrics onto the engine · plotting split + pl twins · diag/ PPCs · PGM→docs
#36	model — the phenotype classifier actually trains
#37	contract — enforced model contract + audit-driven correctness/performance fixes

The three things worth knowing

1. The classifier was silently untrained (#36). f_cls never received a gradient, so pure-classifier phenotype recovery sat at 0.200 = chance on a linearly-separable dataset. Two coupled causes: the classifier logits never entered the ELBO, and the alignment target was indexed by the local pyro plate index rather than the global cell index — so every shuffled minibatch trained on the wrong cells' labels. Recovery 0.200 → 1.000.

2. train(lr=...) was a dead knob (#37). UnifiedTrainingPlan overrode scvi's deliberate no-op configure_optimizers shim with a real Adam over every module parameter, which then stepped after SVI.step() had already zeroed the gradients. Consequences: weight decay degenerated into a scale-free ~lr·sign(p) shrink that held the networks ~2.4× small, and Pyro always used scvi's hard-coded lr=1e-3 regardless of what the caller asked for. Now routed through optim_kwargs; lr is live for the first time.

3. Three contracts are now machine-checked. Interface, generative mathematics, and metric definitions each have a manifest + prose + conformance test:

	freezes	test
API	the public interface	test_contract_conformance.py (27)
Model	the generative mathematics	test_model_contract_conformance.py (12)
Metrics	what the metrics compute	test_metrics_contract_conformance.py (12)

The model contract traces the live model()/guide(); it was attacked adversarially and lost three times before being hardened to assert on traced values rather than source text or scalar totals. One escape had reinstated the exact index-scrambling bug above while passing 12/12.

The metrics contract records a source erratum: Supplementary Note 1's eqs 3–4 weight by the marginal (making them cross-entropies) and eq 4's label contradicts its own right-hand side. The code is correct — proven by I(c;φ) = H(c) − E_φ[H(c|φ)], which the literal equations violate by producing a negative mutual information. Confirmed with the author; the code stands and the erratum is pinned by a test.

Also in here

• Deviation [G] — α (global_scale) was never applied to the eq-1 clonotype prior; the concentration was the raw archetype centroid (sum≈1, U-shaped) and inconsistent with the guide. Fixed.
• Deviation [E] — reconstruction_loss_scale re-measured after the phantom optimizer was removed: real-data posterior-predictive library ratio 1.40 → 0.99 at 1e-2, with recovery and latent separation unchanged. Default recalibrated; three drifted defaults unified.
• Knob-test matrix completed (32 tests) with a new wiring layer — lr sat marked "hooked up" for months while dead, because the model still converged and every behavioural test passed.
• Guardrails for footguns that used to fail silently: K > n_clonotypes crashed outright; a second TCRIModel silently continued the first model's fit via Pyro's process-global param store; batch_size ≥ n_obs made every epoch a single optimizer step (the cause of the "9-hour" synthetic notebook); Trainer knobs raised TypeError if you tried to override them.
• Measured optimizations, each verified bit-identical or exact: permutation AUROC 191 s → 0.6 s (~320×) via the Mann-Whitney rank-sum identity; metric path 5.18× by removing a torch→pandas→numpy round-trip; draw materialization 2.8× less peak memory (11 GB → chunked); predict 2.52×.
• Removal Ledger closed — all 20 remaining symbols verified gone.

Not included (deliberately)

Three commits pushed after #37's merge window are not in this branch and follow in a separate PR: the dead-dependency cleanup, a fix for import tcri globally silencing the user's warnings (+1.75× import), and the statistical-recovery test harness.

Also outstanding: DUX-2 (n_steps_kl_warmup counted in steps while max_epochs is epochs) and [F] (in-silico perturbation, out of scope for this release).

🤖 Generated with Claude Code

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.