Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions docs/contract/REFACTOR_AGENDA.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ tracker + running diary for the whole refactor. The detailed spec lives in `tcri
|---|---|---|---|---|---|
| 0 | Contract freeze + CI scaffolding | ✅ | none | — | conformance green |
| 1 | Shared helpers + `_keys` | ✅ | low | 0 | existing tests green |
| 2 | Safe deletions | | very low | 1 | import-graph clean |
| 2 | Safe deletions | | very low | 1 | import-graph clean |
| 3 | Model module split | ☐ | low | 1 | model/pyro tests green |
| 4 | Model→AnnData streamline | ☐ | HIGH | 1,3 | session round-trip |
| 5 | Engine consolidation | ☐ | HIGH | 4 | joint identities |
Expand All @@ -47,12 +47,12 @@ tracker + running diary for the whole refactor. The detailed spec lives in `tcri
## Removal Ledger (the hard bar — every one MUST end deleted)
Tick only when the symbol is gone from source AND `__all__`/imports AND `import tcri` is green.

**Phase 2 (dead / out-of-scope):**
- [ ] `pp.get_latent_embedding` · [ ] `pp.group_small_clones` · [ ] `pp.register_probability_columns`
- [ ] `pp.remove_meaningless_genes` · [ ] `pp.gene_entropy` · [ ] `pp.classify_phenotypes`
- [ ] `pl.polar_plot` · [ ] `pl.probability_distribution` · [ ] `pl.bayesian_mutual_information`
- [ ] `metrics._ent` · [ ] `tl.clone_fraction` · [ ] `metrics.dkl` (→ `_distance.kl_divergence`)
- [ ] `ut.probabilities` (+ its `_plotting.py:18` import, same PR) · [ ] `SankeyNode.hex_to_rgb`
**Phase 2 (dead / out-of-scope):** ✅ ALL DELETED (PR2) — 14 symbols, 384 lines, `import tcri` green.
- [x] `pp.get_latent_embedding` · [x] `pp.group_small_clones` · [x] `pp.register_probability_columns`
- [x] `pp.remove_meaningless_genes` · [x] `pp.gene_entropy` · [x] `pp.classify_phenotypes`
- [x] `pl.polar_plot` · [x] `pl.probability_distribution` · [x] `pl.bayesian_mutual_information`
- [x] `metrics._ent` · [x] `tl.clone_fraction` · [x] `metrics.dkl` (→ `_distance.kl_divergence`)
- [x] `ut.probabilities` (+ its `_plotting.py` import, same PR) · [x] `SankeyNode.hex_to_rgb`

**Phase 4 (folded into `to_anndata` / session):**
- [ ] `pp.register_model` (→ `model.to_anndata`) · [ ] `pp.register_phenotype_key` · [ ] `pp.register_clonotype_key`
Expand Down Expand Up @@ -106,8 +106,15 @@ Template per PR: **Goal · Status · What happened · Issues & fixes · Added
- **Usability:** internal only this step.
- **`K.*` migration (done):** replaced **85** canonical key literals with `K.*` across preprocessing/metrics/plotting/utils via a verified script (model=0; only legacy keys there); added `test_no_canonical_key_literals` guard — none remain. `dkl` reassigned (dead `metrics.dkl`→Phase 2; `flux` inner→Phase 6). Legacy keys left as literals until their removal phases. **PR1 COMPLETE — 35 passed / 1 skipped.**

## PR 2 — Safe deletions · ☐ todo
_(diary to be filled — this is a REMOVAL PR; the ledger Phase-2 block must be fully ticked)_
## PR 2 — Safe deletions · ✅ done (branch `refactor/pr2-safe-deletions`)
- **Goal:** delete the Phase-2 dead / out-of-scope symbols outright (they go to the trash, not to examples). Zero behavior change to the kept surface; `import tcri` stays green.
- **What happened:** verified **every** target has zero in-package call-sites (grep for calls/imports, plus a precise `NAME(` call-site check for the three ambiguous ones — `_ent`, module-level `dkl`, `probabilities` — all 0), then deleted via AST-span (top-level `FunctionDef` line ranges; the `SankeyNode.hex_to_rgb` method inside its `ClassDef`; the dead `from ..utils._utils import probabilities` line). **14 symbols, 384 lines removed** across metrics/plotting/sankey/preprocessing/utils. `import tcri` green; **full suite 35 passed / 1 skipped** (unchanged — nothing referenced them).
- **Why safe (not deferred):** the "dkl" refs that remain are the **string** distance-metric name in `flux` + the `_distance` registry, not the deleted module-level `dkl` function (`flux` keeps its own inner `dkl_func` → Phase 6). The `polar_plot`/`probability_distribution` "callers" were self-referential (own docstring examples / the self-recursion bug). `probabilities` was imported into plotting but never called.
- **Added:** n/a (pure removal PR).
- **Removed (hard bar):** ✅ all 14 Phase-2 ledger items ticked — `pp.get_latent_embedding`, `pp.group_small_clones`, `pp.register_probability_columns`, `pp.remove_meaningless_genes`, `pp.gene_entropy`, `pp.classify_phenotypes`, `pl.polar_plot`, `pl.probability_distribution`, `pl.bayesian_mutual_information`, `metrics._ent`, `tl.clone_fraction`, `metrics.dkl`, `ut.probabilities` (+ import), `SankeyNode.hex_to_rgb`. Confirmed gone: no `def` remains, no explicit re-export/`__all__` names them.
- **Test opportunities:** none new (removal); the existing suite is the regression gate and stayed green.
- **Streamline:** shrinks preprocessing (−127) and plotting (−225) meaningfully ahead of the Phase 3/4/7 splits. Removed the one import this PR orphaned (`cosine_similarity` — sole user was the deleted `classify_phenotypes`); other module-top imports left for the file-split phases (conservative; the PR1 audit already handled the utils ones).
- **Usability:** removes broken/dead public entry points (`probability_distribution` self-recursion, `bayesian_mutual_information` bad kwarg, `polar_plot` undefined-name) from the surface so nobody trips on them.

## PR 3 — Model module split · ☐ todo
## PR 4 — Model→AnnData streamline · ☐ todo
Expand Down
3 changes: 2 additions & 1 deletion docs/contract/tcri_implementation_plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,8 @@ Freeze the map **before** Phase 5 (the first breaking PR). Renames are breaking;
| `mi_compare`, `delta_entropy_table`, `flux_table`, `delta_clonotypic_entropy`, `phenotypic_entropy_delta` | **deleted** — expressed via `groupby=` + `tl.compare_groups` | Phase 6 |
| `tl.phenotypic_entropies` / `tl.clonotypic_entropies` (plural batch forms) | **deleted** — subsumed by `groupby=` on the singular metric | Phase 6 |
| `get_cell_phenotype_probs` | `predict` (scvi/CellAssign idiom; order‑preserving loader, indexed by `obs_names`) | Phase 4 |
| `register_model` (+ `classify_phenotypes`, `register_*_key`) | `model.to_anndata` (thin) | Phase 4 |
| `register_model` (+ `register_*_key`) | `model.to_anndata` (thin) | Phase 4 |
| `classify_phenotypes` | — (DROP; superseded by `REDO_LIST.md`) | Phase 2 ✅ deleted |
| `register_clonotype_key` / `register_phenotype_key` | folded (private) into `to_anndata` | Phase 4 |
| `pl.clonotypic_entropy_by_phenotype` | `pl.clonotypic_entropy` | Phase 7 |
| `plot_pheno_sankey` | `pl.phenotypic_flux` (sankey) | Phase 7 |
Expand Down
20 changes: 0 additions & 20 deletions tcri/metrics/_metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,11 +147,6 @@ def mi_compare(adata, groupby, groups=None, treatment=None, n_samples=50,
}


def dkl(p, q):
epsilon = 1e-10
p = np.clip(p, epsilon, 1)
q = np.clip(q, epsilon, 1)
return entropy(p, q)

import numpy as np, pandas as pd
from scipy.stats import entropy # Shannon entropy
Expand All @@ -165,11 +160,6 @@ def dkl(p, q):
GRN="\x1b[32m"; CYN="\x1b[36m"; MAG="\x1b[35m"; YLW="\x1b[33m"

# small helper ----------------------------------------------
def _ent(p, base=2):
p = np.asarray(p, dtype=float)
eps = 1e-15
p = p.clip(eps) / p.sum()
return entropy(p, base=base)

def clonotypic_entropy_base(
adata,
Expand Down Expand Up @@ -609,16 +599,6 @@ def clonality(adata):
entropys[phenotype] = np.nan_to_num(clonality)
return entropys

def clone_fraction(adata, groupby):
frequencies = dict()
for group in set(adata.obs[groupby]):
frequencies[group] = dict()
sdata = adata[adata.obs[groupby] == group]
total_cells = len(sdata.obs.index.tolist())
clones = sdata.obs[sdata.uns["tcri_clone_key"]].tolist()
for c in set(clones):
frequencies[group][c] = clones.count(c) / total_cells
return frequencies


def mutual_information(
Expand Down
225 changes: 0 additions & 225 deletions tcri/plotting/_plotting.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@
import operator
import itertools

from ..utils._utils import probabilities
from ._sankey import SankeyNode
from ..preprocessing._preprocessing import clone_size, joint_distribution, joint_distribution_posterior
from ..metrics._metrics import clonotypic_entropy as centropy
Expand Down Expand Up @@ -514,26 +513,6 @@ def phenotypic_flux(
if save is not None:
fig.savefig(save)

def probability_distribution(adata, phenotype_order=None, color="#000000", rotation=90, splitby=None, order=None, figsize=(7,5), save=None):
columns = []
if splitby != None:
ncols = len(set(adata.obs[splitby]))
else:
ncols = 1
fig, ax = plt.subplots(1,ncols,figsize=figsize)

if order == None:
order = list(sorted(adata.obs[splitby]))
for i, o in enumerate(order):
zdata = adata[adata.obs[splitby] == o]
pdist = probability_distribution(zdata)
sns.barplot(data=pdist,ax=ax[i],order=phenotype_order,color=color)
ax[i].set_xticklabels(ax[i].get_xticklabels(), rotation=rotation)
ax[i].set_title(o)

fig.tight_layout()
if save != None:
fig.savefig(save)


def top_clone_umap(adata, reduction="umap", top_n=10, fg_alpha=0.9, fg_size=25, bg_size=0.1, bg_alpha=0.6, figsize=(12,5), return_df=False,save=None):
Expand Down Expand Up @@ -1220,210 +1199,6 @@ def mutual_information(adata, splitby=None, temperature=1.0, n_samples=0, normal
return ax


def bayesian_mutual_information(
adata,
*,
group1,
group2,
splitby,
n_samples = 200,
temperature = 1.0,
normalised = True,
normalise_mode = "average",
weighted = False,
posterior = True,
combine_with_logits=True,
seed = 42,
palette = None,
):
np.random.seed(seed)

meta = adata.uns[K.METADATA]
cov_col = meta["covariate_col"]
clone_col = meta["clone_col"]

groups = sorted(adata.obs[splitby].dropna().unique().tolist())
if palette == None:
palette = dict()
for i, g in enumerate(groups):
palette[g] = tcri_colors[i]
print(f"{BOLD}{MAG}──────── MI summary ({group1} → {group2}) ────────{RESET}")
_info("split column", splitby)
_info("# groups", len(groups))
_info("Δ samples / group", n_samples)
_info("weighted", weighted)
print(f"{MAG}────────────────────────────────────────────────────{RESET}")

results = {}

# ---------- iterate over strata --------------------------------
for g in groups:
mask_g = adata.obs[splitby] == g
clones = adata.obs.loc[mask_g, clone_col].unique().tolist()

mi_pre = []; mi_post = []
bar = tqdm(range(n_samples), desc=f"Δ-MI samples ({g})")
for _ in bar:
mi_pre.append( mutual_information_tl(
adata, group1, temperature=temperature, n_samples=1,
clones=clones, weighted=weighted, normalised=normalised,
normalise_mode=normalise_mode, posterior=posterior,
combine_with_logits=combine_with_logits, verbose=False))
mi_post.append( mutual_information_tl(
adata, group2, temperature=temperature, n_samples=1,
clones=clones, weighted=weighted, normalised=normalised,
normalise_mode=normalise_mode, posterior=posterior,
combine_with_logits=combine_with_logits, verbose=False))

mi_pre = np.array(mi_pre)
mi_post = np.array(mi_post)
delta = mi_post - mi_pre

d_mean, d_std = delta.mean(), delta.std()
hdi_low, hdi_hi = np.percentile(delta, [2.5,97.5])
p_gt = (delta>0).mean(); p_lt = 1-p_gt
cohens_d = d_mean/d_std if d_std>0 else 0.0

print(f"\n{BOLD}{g}{RESET} ΔMI = {d_mean:.4f} ± {d_std:.4f} "
f"95 % HDI [{hdi_low:.4f}, {hdi_hi:.4f}] "
f"P(>0)={p_gt:.3f}")

results[g] = dict(delta_samples=delta, mi_pre_samples=mi_pre,
mi_post_samples=mi_post, delta_mean=d_mean,
delta_std=d_std, cohens_d=cohens_d,
p_greater=p_gt, p_less=p_lt,
hdi=(hdi_low, hdi_hi))

fig, ax = plt.subplots(1, 3, figsize=(15, 4),
gridspec_kw=dict(width_ratios=[2, 2, 1]),
constrained_layout=True)

# A ─── Δ-MI KDEs
for i, g in enumerate(groups):
sns.kdeplot(results[g]["delta_samples"],
fill=True, ax=ax[0],
palette=[palette[g]],
alpha=.9, linewidth=1.2, label=g)
ax[0].axvline(0, color="k", ls="--")
ax[0].set(title="Δ MI (post – pre)",
xlabel="Δ normalised MI", ylabel="density")
ax[0].legend(title=splitby)

# B ─── pre / post KDEs
for i, g in enumerate(groups):
sns.kdeplot(results[g]["mi_pre_samples"],
ax=ax[1], palette=[palette[g]],
ls="-", label=f"{g} – {group1}")
sns.kdeplot(results[g]["mi_post_samples"],
ax=ax[1], palette=[palette[g]],
ls="--", label=f"{g} – {group2}")
ax[1].set(title="MI posterior per condition",
xlabel="normalised MI")
ax[1].legend()

# C ─── bar summary of Δ
means = [results[g]["delta_mean"] for g in groups]
errs = [results[g]["delta_std"] for g in groups]
ax[2].bar(groups, means, yerr=errs, capsize=5,
color=[palette[g] for g in groups])
ax[2].axhline(0, color="k", ls="--")
ax[2].set(title="Δ MI summary", ylabel="Δ normalised MI")

fig.suptitle("Bayesian MI Analysis of clonotype ⇄ phenotype coupling",
fontsize=14, weight="bold")
return results



def polar_plot(adata, phenotypes=None, statistic="distribution", method="joint_distribution", splitby=None, color_dict=None, temperature=1.0):
"""
Create a polar plot showing phenotype distributions or entropies.

This function creates a radar/polar chart that visualizes either the distribution of phenotypes
or entropy values across different conditions. It's useful for comparing phenotype proportions
or entropy patterns across experimental groups.

Parameters
----------
adata : AnnData
AnnData object containing the data with TCR and phenotype information
phenotypes : list, optional
List of phenotype names to include in the plot. If None, uses all phenotypes
defined in adata.uns['tcri_metadata']["phenotype_col"]
statistic : str, default="distribution"
Type of statistic to plot, one of "distribution" or "entropy"
method : str, default="joint_distribution"
Method to compute phenotype distributions, either "joint_distribution" (model-based)
or "empirical" (raw cell counts)
splitby : str, optional
Column name to split the data by. If None, uses the covariate column stored
in adata.uns['tcri_metadata']["covariate_col"]
color_dict : dict, optional
Dictionary mapping split categories to colors
temperature : float, default=1.0
Temperature parameter for softening/sharpening distributions

Returns
-------
matplotlib.axes.Axes
The polar plot axis

Examples
--------
>>> import tcri
>>> # Basic phenotype distribution polar plot
>>> ax = tcri.pl.polar_plot(adata, statistic="distribution")
>>>
>>> # Entropy polar plot with custom colors
>>> color_dict = {"Day0": "#FF5733", "Day7": "#33FF57", "Day14": "#3357FF"}
>>> ax = tcri.pl.polar_plot(adata, statistic="entropy", color_dict=color_dict)
"""
if phenotypes is None:
phenotypes = adata.uns[K.METADATA]["phenotype_col"]

if splitby is None:
splitby = adata.uns[K.METADATA]["covariate_col"]

# Get unique splits
splits = adata.obs[splitby].unique()

# Create figure and axis
fig, ax = plt.subplots(figsize=(8, 8), subplot_kw={'projection': 'polar'})

# Calculate angles for each phenotype
angles = np.linspace(0, 2*np.pi, len(phenotypes), endpoint=False)

# Plot for each split
for i, split in enumerate(splits):
if statistic == "distribution":
if method == "joint_distribution":
jd = joint_distribution(adata, split, temperature=temperature)
values = jd.mean().values
else:
subset = adata[adata.obs[splitby] == split]
values = np.zeros(len(phenotypes))
for j, pheno in enumerate(phenotypes):
mask = subset.obs[adata.uns[K.METADATA]["phenotype_col"]] == pheno
values[j] = np.sum(mask) / len(subset)
else: # entropy
values = np.zeros(len(phenotypes))
for j, pheno in enumerate(phenotypes):
values[j] = clonotypic_entropy(adata, split, pheno, temperature=temperature)

# Normalize values
values = values / np.sum(values)

# Plot values
color = color_dict[split] if color_dict else None
ax.plot(angles, values, 'o-', linewidth=2, label=split, color=color)
ax.fill(angles, values, alpha=0.25, color=color)

# Set the labels
ax.set_xticks(angles)
ax.set_xticklabels(phenotypes)

# Add legend
ax.legend(loc='upper right', bbox_to_anchor=(0.1, 0.1))

plt.tight_layout()
return ax
3 changes: 0 additions & 3 deletions tcri/plotting/_sankey.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,6 @@ def plot(self, ax):
ax.add_patch(self.patch)


def hex_to_rgb(self, hex_color):
hex_color = hex_color.lstrip('#')
return tuple(int(hex_color[i:i+2], 16) / 255.0 for i in (0, 2, 4))

def plot_node_connection(self, destination_node, ax, **kwargs):
num_segments = 500
Expand Down
Loading