Skip to content

Refresh multi-organism validation on the deterministic placement (stacks on #83) - #84

Open
edkerk wants to merge 2 commits into
developfrom
feat/multiorg-remeasure
Open

Refresh multi-organism validation on the deterministic placement (stacks on #83)#84
edkerk wants to merge 2 commits into
developfrom
feat/multiorg-remeasure

Conversation

@edkerk

@edkerk edkerk commented Jul 20, 2026

Copy link
Copy Markdown
Member

Completes the multi-organism validation for the deterministic placement tie-break in #83.

Stacks on #83 (feat/placement-tiebreak) — merge #83 first; this PR's diff reduces to just the multi-organism doc update once #83 lands.

#83 fixed the reaction-placement co-optimum degeneracy but only re-measured yeast, leaving the Human-GEM / AraCore / iCre1355 reaction-agreement figures in multiorganism_validation.md from the old arbitrary placement (footnoted "await re-measurement"). This re-runs benchmark_certified_multiorg.py on the tie-break code and refreshes those rows.

Result

organism reaction agr gene agr transports blocked
yeast-GEM 72.0 % → 72.5 % 88.7 % (=) 1001 → 967 29.7 % (=)
Human-GEM 52.8 % → 56.9 % 79.4 % (=) 3701 → 3608 10.7 % (=)
AraCore 81.8 % → 82.1 % 93.7 % (=) 215 → 200 0.0 % (=)
iCre1355 53.4 % → 54.1 % 70.5 % (=) 1108 → 1083 19.0 % → 32.9 %
  • Reaction agreement rose or held everywhere; Human-GEM's +4.1 pt is the largest move (AraCore and iCre1355 were already near their ceilings, so the tie-break mostly confirms them).
  • Gene agreement is unchanged per model — direct confirmation, across four kingdoms at genome scale, that the tie-break fixes only the reaction placement and leaves the gene layout untouched.
  • Fewer transports across the board — coherent placement fragments fewer metabolites.
  • One non-obvious shift: iCre1355's blocked fraction rose 19.0 % → 32.9 %. Tighter, less transport-heavy placement leaves more reactions dead-ended (the model still certifies). Reported honestly — the original task assumed blocked was placement-invariant; it isn't, because placement determines connectivity.

Also removes the "await re-measurement" footnote and bumps the prose Human-GEM reaction count to the on-disk model version (12854 → 12877). All three organisms certified; measured with cobra.Configuration().processes = 1 for Windows find_blocked_reactions safety; reproducible run-to-run.

edkerk added 2 commits July 19, 2026 23:59
The placement master maximises a gene-localisation objective that never
mentions the per-reaction placement variable, so each reaction's compartment
was a free co-optimum: the pinned solver returned it reproducibly but
arbitrarily, giving 52.8% reaction agreement with curated yeast-GEM (an earlier
un-pinned build happened to land ~72%).

Add a lexicographic second pass: fix the gene layout to the primary optimum
(fixing the y solution, not the objective value, so there is no tolerance to
tune), then place each reaction in the compartment its own enzymes are
predicted to occupy -- the summed DeepLoc score of the reaction's genes, with a
small default_compartment prior so genes-free and score-tied reactions fall
there deterministically.

Yeast reaction agreement rises to 72.5% (1408/1943) and now rests on the
localisation evidence rather than a solver tie-break; gene agreement is
unchanged (88.7%, 716/807 -- the gene layout is untouched); coherent placement
adds fewer transports (1001 -> 967); growth and blocked fraction unchanged; the
warm-started second solve adds ~4s. Reproducible across independent runs.

Docs updated (yeast_validation, localization_redesign, multiorganism_validation
footnote); the multi-organism reaction-agreement rows await re-measurement.
Re-measure Human-GEM, AraCore, and iCre1355 (scripts/benchmark_certified_multiorg.py)
on the deterministic score-aligned reaction placement, and refresh the yeast row
to match yeast_validation.md. The previous reaction-agreement figures came from
the old arbitrary co-optimum placement.

Reaction agreement rose or held (Human-GEM 52.8% -> 56.9%, the largest move;
AraCore 81.8% -> 82.1%; iCre1355 53.4% -> 54.1%; yeast 72.0% -> 72.5%), and
coherent placement needs fewer transports everywhere. Gene agreement is
unchanged (the tie-break leaves the gene layout untouched), confirmed per model.
One non-obvious shift: iCre1355's blocked fraction rose 19.0% -> 32.9% as tighter,
less transport-heavy placement leaves more reactions dead-ended (still certifies).

Removes the "await re-measurement" footnote; updates the prose reaction counts to
the current Human-GEM version (12854 -> 12877).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant