Benchmark assign_compartments transport pruning and gap-filling - #75
Open
edkerk wants to merge 1 commit into
Open
Benchmark assign_compartments transport pruning and gap-filling#75edkerk wants to merge 1 commit into
edkerk wants to merge 1 commit into
Conversation
Both features shipped with correctness tests but no performance measurement. New scripts/benchmark_assignment_ablations.py and a study doc measure them on yeast-GEM (flattened and reassigned). Transport pruning (prune_transports): removes 274 of 1268 provisioned transports (21.6%), a strict subset, at identical reaction-level accuracy (0.721) -- a clean win, paid for in runtime (~3x). Gap-filling, two experiments: * Natural draft: 0 additions with or without a universal -- transport addition alone restores growth, so the feature never fires gratuitously at genome scale (the scaled-up test_no_gratuitous_gapfill). * Knockout-recovery (ground-truthed): remove each growth-essential reaction and gap-fill from a universal containing it. 100% of recoveries re-add the exact removed reaction with zero wrong additions; the ~45% recall is bounded entirely by cobra.flux_analysis.gapfill's numerical tolerance (a sharp optimum at cobra's default -- an integer_threshold sweep collapses to 0% in both directions), and the feature fails safe by declining rather than mis-filling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two
assign_compartmentsfeatures shipped with correctness tests but no performance measurement — flagged as the exposed flank when reviewing what the compartment-assignment work can do. This adds a reproducible ablation benchmark and a study doc measuring both on yeast-GEM (flattened to one compartment and reassigned).Transport pruning (
prune_transports) — a clean winPruning removes 274 transports (21.6 %) — a strict subset of the unpruned set — at identical accuracy (0.721 to four decimals). It only ever removes genuinely redundant shuttles; it never trades a placement. The cost is runtime (re-certifies after each removal), not quality.
Gap-filling — precise and conservative
It never fires gratuitously. On the natural flattened draft, gap-fill adds 0 reactions with or without a universal — transport addition alone restores growth. The genome-scale confirmation of the
test_no_gratuitous_gapfillunit test.When it fires under real gaps, it adds the right reaction. Ground-truthed knockout-recovery (remove each growth-essential reaction, gap-fill from a universal containing it — the same
cobra.flux_analysis.gapfillcall_gapfillwraps):cobra.flux_analysis.gapfill's numerical-validation limit, which_gapfillcatches — so an unsolvable gap stays uncertified and visible, never silently mis-filled.integer_threshold; a sweep shows its default is a sharp optimum (every other value, up or down, collapses recovery to 0 %), so there is no tuning win and_gapfillcorrectly uses the default.Verdict
Transport pruning is a strict win; gap-filling is precise, conservative, and fails safe. Both are now measured, not just asserted.
Notes
pytest827 passed / 4 skipped; ruff and the-Wdocs build clean. The knockout sample is seeded (deterministic).docs/studies/assignment_ablations.md; reproduce withpython scripts/benchmark_assignment_ablations.py.