Skip to content

Benchmark assign_compartments transport pruning and gap-filling - #75

Open
edkerk wants to merge 1 commit into
developfrom
feat/assignment-ablation-benchmark
Open

Benchmark assign_compartments transport pruning and gap-filling#75
edkerk wants to merge 1 commit into
developfrom
feat/assignment-ablation-benchmark

Conversation

@edkerk

@edkerk edkerk commented Jul 17, 2026

Copy link
Copy Markdown
Member

Two assign_compartments features shipped with correctness tests but no performance measurement — flagged as the exposed flank when reviewing what the compartment-assignment work can do. This adds a reproducible ablation benchmark and a study doc measuring both on yeast-GEM (flattened to one compartment and reassigned).

Transport pruning (prune_transports) — a clean win

transports added reaction agreement runtime
pruning on (default) 994 0.721 209 s
pruning off 1268 0.721 74 s

Pruning removes 274 transports (21.6 %) — a strict subset of the unpruned set — at identical accuracy (0.721 to four decimals). It only ever removes genuinely redundant shuttles; it never trades a placement. The cost is runtime (re-certifies after each removal), not quality.

Gap-filling — precise and conservative

It never fires gratuitously. On the natural flattened draft, gap-fill adds 0 reactions with or without a universal — transport addition alone restores growth. The genome-scale confirmation of the test_no_gratuitous_gapfill unit test.

When it fires under real gaps, it adds the right reaction. Ground-truthed knockout-recovery (remove each growth-essential reaction, gap-fill from a universal containing it — the same cobra.flux_analysis.gapfill call _gapfill wraps):

outcome of 60
recovered, re-added the exact removed reaction 45 %
cobra numerical failure (declined) 55 %
recovered with a wrong reaction 0 %
  • 100 % of recoveries are exact; zero wrong additions. When gap-fill answers, it answers correctly.
  • The only failure mode is declining to answer. The 55 % shortfall is entirely cobra.flux_analysis.gapfill's numerical-validation limit, which _gapfill catches — so an unsolvable gap stays uncertified and visible, never silently mis-filled.
  • cobra's error suggests lowering integer_threshold; a sweep shows its default is a sharp optimum (every other value, up or down, collapses recovery to 0 %), so there is no tuning win and _gapfill correctly uses the default.

Verdict

Transport pruning is a strict win; gap-filling is precise, conservative, and fails safe. Both are now measured, not just asserted.

Notes

  • Docs- and script-only; no source change. pytest 827 passed / 4 skipped; ruff and the -W docs build clean. The knockout sample is seeded (deterministic).
  • Study: docs/studies/assignment_ablations.md; reproduce with python scripts/benchmark_assignment_ablations.py.

Both features shipped with correctness tests but no performance
measurement. New scripts/benchmark_assignment_ablations.py and a study
doc measure them on yeast-GEM (flattened and reassigned).

Transport pruning (prune_transports): removes 274 of 1268 provisioned
transports (21.6%), a strict subset, at identical reaction-level accuracy
(0.721) -- a clean win, paid for in runtime (~3x).

Gap-filling, two experiments:
 * Natural draft: 0 additions with or without a universal -- transport
   addition alone restores growth, so the feature never fires
   gratuitously at genome scale (the scaled-up test_no_gratuitous_gapfill).
 * Knockout-recovery (ground-truthed): remove each growth-essential
   reaction and gap-fill from a universal containing it. 100% of
   recoveries re-add the exact removed reaction with zero wrong
   additions; the ~45% recall is bounded entirely by
   cobra.flux_analysis.gapfill's numerical tolerance (a sharp optimum at
   cobra's default -- an integer_threshold sweep collapses to 0% in both
   directions), and the feature fails safe by declining rather than
   mis-filling.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant