diff --git a/.gitignore b/.gitignore index 8688e53..c75c4c2 100644 --- a/.gitignore +++ b/.gitignore @@ -144,4 +144,6 @@ efmtool_port/ benchmark_compression.py benchmark_simple.py profile_compression.py -compression_benchmark.png \ No newline at end of file +compression_benchmark.png +# perf-test artefacts +tests/perf_results/ diff --git a/docs/source/developers_guide.md b/docs/source/developers_guide.md index 967f76a..662622d 100644 --- a/docs/source/developers_guide.md +++ b/docs/source/developers_guide.md @@ -11,8 +11,8 @@ linear-algebra and optimization theory), and *why* it is built that way. **Audience.** A scientific programmer comfortable with linear and mixed-integer programming and constraint-based metabolic modeling, but new to this codebase. The chapters are largely self-contained, though the notation is established in [Chapter 1](#ch1) and the LP/duality groundwork in Chapters 2 and 6. Code -is cited as `file.py:line`; line numbers are anchors that drift with edits, so treat them as pointers, -not addresses. +is cited by file and symbol, e.g. `networktools.py`, `compress_ki_ko_cost` -- never by line number, +which drifts with every edit. Grep for the symbol. ## How to read this guide @@ -29,11 +29,11 @@ not addresses. 1. [**Orientation & the strain-design problem**](#ch1) — the MCS problem, SUPPRESS/PROTECT/bilevel semantics, interventions & cost, the binary `z` vector, invocation, and the master notation table. 2. [**The constraint-based foundation**](#ch2) — `Sv=0`, the flux polytope/cone, FBA & FVA as LPs, the internal standard form, and the convex geometry needed for duality. -3. [**Network compression**](#ch3) — why compress; the exact integer/rational nullspace (fraction-free RREF, big-int path); parallel, coupled (kernel-proportionality + bound intersection), conservation-relation, and blocked/zero-flux reductions; the alternating fixpoint; GPR AND/OR propagation; the compression map; and the legacy efmtool Java backend. -4. [**GPR integration**](#ch4) — why gene KOs are encoded as flux structure; `extend_model_gpr` pseudo-metabolite construction (AND/OR), the flux-space-invariance argument, reversible split & `reac_map`; `reduce_gpr`; the two-pass boundary and the regulatory-gene exemption. -5. [**FVA in preprocessing**](#ch5) — the three FVA uses and their rationale; `bound_blocked_or_irrevers_fva` bound relaxation and its MILP effect; size-1 MCS extraction; the `speedy_fva` acceleration algorithm. -6. [**Dualization (the mathematical core)**](#ch6) — LP duality & complementary slackness; Farkas' lemma and the SUPPRESS infeasibility certificate (why the dual ray is unbounded); strong-duality encoding of bilevel problems and *why the one `LP_dualize` operation is reusable* across OptKnock/RobustKnock/OptCouple/DoubleOpt. -7. [**MILP construction & the z-linking**](#ch7) — the seed cost rows, `num_z`, block-diagonal module assembly, `prevent_boundary_knockouts`; `link_z`: per-constraint big-M from a bounding LP vs native indicator constraints, the bound-driven fork, and why indicators give a tighter relaxation. +3. [**Network compression**](#ch3) — exact rational nullspace compression; parallel, coupled, conservation and blocked reductions; lump scaling; GPR propagation and simplification; compression maps; and the legacy efmtool backend. +4. [**GPR integration**](#ch4) — GPR reduction and Boolean simplification, `extend_model_gpr`, reversible splitting, module remapping, and the two-compression-pass boundary. +5. [**FVA in preprocessing**](#ch5) — pre-compression sign classification, desired-region essentiality, final bound/module FVA, the single-classical-module fold, and size-1 MCS extraction. +6. [**Dualization (the mathematical core)**](#ch6) — LP duality, Farkas certificates, and the strong-duality encodings shared by the supported module types. +7. [**MILP construction & the z-linking**](#ch7) — block assembly, per-module sign overrides, bound-derived single-row big-M values, native indicators or the intentional blanket M for multi-variable rows, and free-binary elimination. 8. [**Solving & enumeration**](#ch8) — ANY/BEST/POPULATE objective setups; the iterative loop and superset-excluding integer cuts; solver parameters; the CPLEX-vs-Gurobi gap. 9. [**Decompression & solution semantics**](#ch9) — reverse-map expansion of compressed interventions; size-1 MCS re-injection; `filter_sd_maxcost`; the KI value-0/`(nan,nan)` & `strip_non_ki` encoding; gene↔reaction translation. 10. [**Known issues, gotchas & failure modes**](#ch10) — neutral-gene-KO paths and superset artifacts with mechanism; the in-place dict-mutation footgun; name truncation; numeric-status robustness. @@ -274,10 +274,10 @@ to know that these modules exist, that they set the *global objective* of the co Costs are supplied per-kind: `ko_cost`, `ki_cost` (reactions), `gko_cost`, `gki_cost` (genes), `reg_cost` (regulatory). Defaults: with reaction interventions, every reaction is a KO candidate at cost 1 (`compute_strain_designs.py`); with `gene_kos=True`, every gene is a -KO candidate at cost 1 (`:253-257`). Supplying a partial dict *restricts* candidacy to the +KO candidate at cost 1. Supplying a partial dict *restricts* candidacy to the listed items — anything not listed is simply not knockable. Essential reactions/genes (those whose removal would break a PROTECT or desired region) have their cost entries dropped during -preprocessing so they are never proposed (`:381`, `:494`; [Ch 5](#ch5)). +preprocessing so they are never proposed ([Ch 5](#ch5)). **The binary vector `z`.** After preprocessing, the model has been compressed and GPR-extended; `SDProblem.__init__` allocates **one binary variable per (compressed) reaction**: `num_z = numr` @@ -291,7 +291,7 @@ data is compiled (`strainDesignProblem.py`) into three aligned per-reaction arra - `z_non_targetable[j]` — true iff `j` has neither a KO nor KI cost, so `z_j` is fixed to 0 (`ub[j] = 1 − z_non_targetable[j]`, `strainDesignProblem.py`). -KIs override KOs when both are given (`:143` blanks the KO cost wherever a KI cost exists). The +KIs override KOs when both are given (blanks the KO cost wherever a KI cost exists). The resulting cost vector feeds the two budget rows placed at the very top of the MILP (`strainDesignProblem.py`): a row `Σ cost_j z_j ≤ max_cost` (the `idx_row_mincost` row, `b_ineq[1] = max_cost`) and a companion `−Σ cost_j z_j ≤ 0` row (`idx_row_maxcost`), plus a @@ -319,48 +319,44 @@ lever that keeps the enumeration tractable (the canonical benchmarks all cap it ### 1.6 The end-to-end pipeline at a glance -`compute_strain_designs(model, **kwargs)` (`compute_strain_designs.py`) is the orchestrator. -Its stages, in order, with the chapter that details each: - -1. **Parse & validate** (`:178-304`) — resolve `sd_setup` vs. explicit kwargs, select the - solver, seed the RNG, normalize cost dicts, reject overlapping gene/reaction candidates, - rename genes whose IDs start with a digit, and re-validate each module's constraints against - the chosen solver. (This chapter, §1.7.) -2. **Preprocess** — the bulk of wall-time (measured ~117 s of blocked/irreversible FVA on the - iML1515 gene-MCS benchmark). It interleaves several transformations: - - `remove_ext_mets` and reaction-based regulatory constraints (`:310-330`). - - **Compression pass #1** (`compress_model(..., propagate_gpr=True)`, `:357`): lossless, - *exact integer/rational* network compression on the metabolic model *before* gene - pseudo-reactions exist — [Ch 3](#ch3). - - **FVA #1** (`:373-381`): flux-variability analysis on each desired/PROTECT module to find - reactions essential to those behaviors, and drop them from the knockable set — [Ch 5](#ch5). - - **GPR integration** (`:383-422`, only if `gene_kos`): `reduce_gpr` prunes irrelevant genes, - then `extend_model_gpr` encodes the Boolean gene–protein–reaction rules as *flux structure* - (gene pseudo-metabolites / pseudo-reactions) so that a gene knockout becomes an ordinary - reaction-level constraint in the same MILP; module references are remapped through - `reac_map` — [Ch 4](#ch4). - - **Compression pass #2** (`compress_model(...)`, `propagate_gpr` default, `:434`): compress - the now GPR-extended network — [Ch 3](#ch3)/4. - - **FVA #2** (`bound_blocked_or_irrevers_fva`, `:450`): relax non-binding bounds to ±∞ and pin - blocked/irreversible reactions to 0, which tightens the downstream big-M/indicator - linearization — [Ch 5](#ch5). - - **FVA #3** (knockable-scoped, `:454-494`): find reactions essential to SUPPRESS vs. PROTECT - and, for a classical MCS problem, extract **size-1 MCS** (single reactions whose removal - alone blocks the SUPPRESS region) so they need not be re-discovered by the MILP — [Ch 5](#ch5). -3. **Build the MILP** (`SDMILP(cmp_model, sd_modules, **kwargs_milp)`, `:518`; [Ch 7](#ch7)). Each - module is appended by `addModule` as a block: **SUPPRESS → dualized Farkas infeasibility - rows, PROTECT → raw primal feasibility rows**, bilevel → strong-duality rows ([Ch 6](#ch6)). Then - `link_z` wires the binary `z` to those continuous rows, as **native indicator constraints or - big-M** depending on bound structure ([Ch 7](#ch7)). -4. **Solve / enumerate** ([Ch 8](#ch8)): `compute` (ANY), `compute_optimal` (BEST), or `enumerate` - (POPULATE). Found designs are excluded by iterative **integer cuts** so the next solve returns - a genuinely new design. -5. **Decompress** (`_decompress_solutions`, `:589`; [Ch 9](#ch9)): `expand_sd` reverses the two - compression maps to recover interventions on original reactions, re-injects the size-1 MCS, - filters by `max_cost`, and translates reaction designs to gene designs via the cobra GPR AST. - -Chapters 2–5 cover preprocessing, 6–7 the MILP construction, 8 the solve loop, 9 decompression, -10 known gotchas, and 11 performance and roadmap. +`compute_strain_designs(model, **kwargs)` is the orchestrator. The current order is: + +1. **Parse and validate.** Normalize the setup, costs, solver, seed and module list; reject + incompatible intervention dictionaries; and validate each module's constraints with the selected + solver. +2. **Enter solver-suppressed preprocessing.** Model copies receive a backend-free + `_CarrierSolver`. Compression, FVA and MILP construction read the cobra model's stoichiometry and + bounds but build their own solver objects, so copying or extending a model does not repeatedly + populate an optlang backend. +3. **Prepare the metabolic model.** Remove external metabolites and apply reaction-based regulatory + interventions. Gene-based regulatory constraints are deferred until the gene pseudo-network + exists. +4. **Reversibility pre-tightening and COMPRESS #1** when compression is enabled. + `fast_reversibility` determines which directions are unavailable in the base flux polytope before + compression. Fixing those directions to zero exposes additional exact couplings and avoids + unnecessary reversible GPR splits. `compress_model(..., propagate_gpr=True)` then compresses the + metabolic model while carrying GPR logic through coupled (AND) and parallel (OR) merges. +5. **Pre-GPR desired-region essentiality.** For each non-SUPPRESS module, FVA identifies reactions + that must remain active. Those reactions are removed from the KO candidates and inform + `reduce_model_gprs`. +6. **GPR preprocessing and extension.** `reduce_model_gprs` removes irrelevant/protected genes on the + compressed path; `simplify_model_gprs` performs Boolean-equivalent monotone simplification on both + compressed and uncompressed paths; `extend_model_gpr` translates the remaining rules into flux + gadgets. Deferred gene-regulatory constraints are then attached. +7. **COMPRESS #2.** The GPR-extended model is compressed again and modules and costs are remapped. +8. **Final FVA preprocessing.** Normally, one scoped FVA relaxes non-binding model bounds and a + knockable-scoped FVA for each module supplies essentiality and per-module sign information. If + there is exactly one classical SUPPRESS or PROTECT module and no inner objective, these two jobs + are folded into one constrained FVA over the union of the required scopes. +9. **Build the MILP.** Each module becomes a continuous block. Classical module blocks consume the + stored FVA ranges as sign-only bound overrides. `link_z` connects intervention binaries using + finite bound-derived rows where available and indicators (or the configured blanket M) otherwise. +10. **Solve and decompress.** ANY, BEST or POPULATE finds compressed designs; compression maps, + size-1 MCS and gene/reaction translations restore the original problem space. + +The `dump_preprocessed` path stops between steps 9 and 10 and serializes everything needed to rebuild +and solve the MILP without repeating steps 2–8. + ### 1.7 How the package is invoked @@ -391,24 +387,24 @@ model feasibility. **Constructing an `SDModule`** (`strainDesignModule.py`). Signature: `SDModule(model, module_type, *args, **kwargs)`. `module_type` is one of `'suppress'`, -`'protect'`, `'optknock'`, `'robustknock'`, `'optcouple'`. The constructor: +`'protect'`, `'optknock'`, `'robustknock'`, `'optcouple'`, `'doubleopt'`. The constructor: - parses `constraints` into canonical `[{reac: coeff, …}, op, rhs]` triples via - `parse_constraints` (`:290-291`); the string `"BIOMASS_Ecoli_core_w_GAM >= 0.001"` and the + `parse_constraints`; the string `"BIOMASS_Ecoli_core_w_GAM >= 0.001"` and the list forms `["-EX_o2_e <= 5", "ATPM = 20"]` and `[[{'EX_o2_e':-1},'<=',5], …]` are all - accepted (`:144-152`); + accepted; - parses `inner_objective` / `outer_objective` / `prod_id` from string or dict into - `{reac: coeff}` maps (`:296-308`); + `{reac: coeff}` maps; - validates that the module type has the arguments it needs (OptKnock/RobustKnock require inner - *and* outer objectives, `:248-257`; OptCouple requires an inner objective and `prod_id`, - `:258-268`), and that senses/tolerances are legal (`:277-282`); + *and* outer objectives; OptCouple requires an inner objective and `prod_id`), and that + senses/tolerances are legal; - unless `skip_checks=True`, runs an FBA to confirm the region is feasible in the original model - and (for inner-objective modules) that `v = 0` is excluded (`:311-320`). + and (for inner-objective modules) that `v = 0` is excluded. A `dummy` object with just an `id` may stand in for the model if `skip_checks=True` and -`reac_ids=[…]` are supplied (`:239-242, 284-285`). +`reac_ids=[…]` are supplied. -**Key `compute_strain_designs` kwargs** (docstring `:70-166`, handling `:174-534`): +**Key `compute_strain_designs` kwargs** (docstring, handling): | kwarg | meaning | default | |---|---|---| @@ -423,17 +419,16 @@ A `dummy` object with just an `id` may stand in for the model if `skip_checks=Tr | `reg_cost` | regulatory-intervention constraints → cost | none | | `compress` | run the iterative network compressor | `True` | | `M` | if set (nonzero), use big-M instead of indicator constraints; GLPK forces `M=1000` | `None` (→ `inf` = indicators) | -| `seed` | MILP seed (feeds solver branch-and-bound) | random (`:215-217`) | +| `seed` | MILP seed (feeds solver branch-and-bound) | random | | `time_limit` | MILP solver time limit (s) | `inf` | `M` deserves a note because it silently changes the MILP encoding. With the default `M = None`, -`SDProblem.__init__` sets `self.M = np.inf` (`strainDesignProblem.py`), and `link_z` -attaches each `z` to its continuous rows as a **native indicator constraint** — except GLPK, -which cannot express indicators and is forced to `M = 1000` (`:120-124`). Because SUPPRESS's -dualized rows are unbounded (the Farkas ray) while PROTECT's primal rows are finite-flux, the -*emergent* behavior under `M = inf` is that SUPPRESS rows become indicators and PROTECT rows -become big-M — but this is a consequence of bound structure inside `link_z`, not a hard-coded -per-module switch ([Ch 7](#ch7)). No MIP optimality gap is set anywhere, so both CPLEX and Gurobi run at +`SDProblem.__init__` sets `self.M = np.inf` (`strainDesignProblem.py`). `link_z` derives a +finite relaxation directly for zero- and single-continuous-variable rows; rows with two or more +continuous variables become native indicator constraints. GLPK, which cannot express indicators, +uses the blanket `M = 1000` for those otherwise-indicator rows, and an explicitly +supplied finite M requests the same replacement on other backends. This is a row-structure rule, +not a hard-coded per-module switch ([Ch 7](#ch7)). No MIP optimality gap is set anywhere, so both CPLEX and Gurobi run at their default 1e-4 relative gap ([Ch 8](#ch8), [Ch 11](#ch11)). The call returns an `SDSolutions` object exposing `reaction_sd` (reaction-level designs) and, @@ -454,16 +449,16 @@ attribute in the code the file/field is given. | `lb, ub ∈ (ℝ∪{±∞})^n` | lower / upper flux bounds | `SDProblem.lb`, `.ub` | | `P` | flux polytope `{v : Sv=0, lb≤v≤ub}` (eq. 1.1) | — | | `D⁻`, `D⁺` | undesired (SUPPRESS) / desired (PROTECT) flux region | module `constraints` | -| `z ∈ {0,1}^{num_z}` | binary intervention vector, one per compressed reaction | `SDProblem`, `num_z = numr` (`:144`) | -| `cost ∈ ℝ_{≥0}^{num_z}` | per-reaction intervention cost | `SDProblem.cost` (`:145-151`) | -| `z_inverted` | KI mask (cost paid for *presence*) | `.z_inverted` (`:148`) | -| `z_non_targetable` | non-knockable mask (`z_j` fixed 0) | `.z_non_targetable` (`:149`) | -| `max_cost` | budget: `Σ cost_j z_j ≤ max_cost` | `.max_cost`, `b_ineq[1]` (`:157-160`) | -| `A_ineq z ≤ b_ineq` | MILP inequality block (top rows: budget + objective) | `.A_ineq`, `.b_ineq` (`:156-160`) | -| `A_eq z = b_eq` | MILP equality block | `.A_eq`, `.b_eq` (`:167-168`) | -| `M` | big-M constant (∞ ⇒ indicator constraints) | `.M` (`:120-126`) | +| `z ∈ {0,1}^{num_z}` | binary intervention vector, one per compressed reaction | `SDProblem`, `num_z = numr` | +| `cost ∈ ℝ_{≥0}^{num_z}` | per-reaction intervention cost | `SDProblem.cost` | +| `z_inverted` | KI mask (cost paid for *presence*) | `.z_inverted` | +| `z_non_targetable` | non-knockable mask (`z_j` fixed 0) | `.z_non_targetable` | +| `max_cost` | budget: `Σ cost_j z_j ≤ max_cost` | `.max_cost`, `b_ineq[1]` | +| `A_ineq z ≤ b_ineq` | MILP inequality block (top rows: budget + objective) | `.A_ineq`, `.b_ineq` | +| `A_eq z = b_eq` | MILP equality block | `.A_eq`, `.b_eq` | +| `M` | big-M constant (∞ ⇒ indicator constraints) | `.M` | | `T v ≤ t` | a module's linear region constraints (schematic) | `lineqlist2mat` (`addModule`) | -| `c` | MILP objective coefficients (cost vector for MCS; module objective for bilevel) | `.c` (`:202-212`) | +| `c` | MILP objective coefficients (cost vector for MCS; module objective for bilevel) | `.c` | | `z_map_*` | maps linking `z` to constraint rows / variables | `.z_map_constr_ineq/_eq/_vars` | Two matrix conventions recur. First, "primal" always refers to a flux-space LP over `v` @@ -816,9 +811,9 @@ model, with no error raised. Set `ε` tight and genuine couplings born from larg coefficients (see the 263-bit yeast-GEM case below) are missed. There is no safe `ε`, because the coefficients that arise mid-elimination span many orders of magnitude. The project constraint is therefore absolute: **the nullspace and rank computations are done in exact arithmetic — Python -arbitrary-precision integers and `fractions.Fraction` — and never in float.** `stoichmat_coeff2rational` +arbitrary-precision integers and `fractions.Fraction` — and never in float.** `stoichmat_coeff_to_fraction` (`compression.py`) converts every stoichiometric coefficient to an exact `Fraction`/`sympy.Rational` -before any compression math runs, and `float_to_rational` (`compression.py`) is the one controlled +before any compression math runs, and `float_to_fraction` (`compression.py`) is the one controlled place where a stray float coefficient is turned into a bounded-denominator rational (it first tries `Fraction(val).limit_denominator(100)` and accepts it only if it round-trips to `max_precision` decimals, else falls back to `round(val·10^p)/10^p`). Once inside the engine, no float ever appears. @@ -830,16 +825,16 @@ The exact matrix type is `RationalMatrix` (`compression.py`). It stores a sparse `(i,j)` is `num[i,j] / den[i,j]`. Keeping numerators and denominators as separate scipy `int64` CSR matrices lets the common operations (column iteration, row/column deletion, submatrix extraction) stay in fast compiled sparse code, while every value remains an exact rational. Construction paths: -`from_cobra_model` (`:175`) reads a model's coefficients straight into num/den arrays, preserving -`Fraction`/sympy-`Rational` exactly and only calling `float_to_rational` for genuine floats; -`identity` (`:144`), `from_numpy` (`:155`), and `_from_sparse` (`:130`) cover the rest. +`from_cobra_model` reads a model's coefficients straight into num/den arrays, preserving +`Fraction`/sympy-`Rational` exactly and only calling `float_to_fraction` for genuine floats; +`identity`, `from_numpy`, and `_from_sparse` cover the rest. Two features of `RationalMatrix` matter later: -- **`add_scaled_column`** (`:313`) performs `col[dst] += (num/den)·col[src]` in exact rational +- **`add_scaled_column`** performs `col[dst] += (num/den)·col[src]` in exact rational arithmetic with per-entry GCD reduction — this is the primitive that merges a coupled slave column into its master (§3.4). -- **Batch edit mode** (`begin_batch_edit`/`end_batch_edit`, `:270`/`:276`) switches the backing store +- **Batch edit mode** (`begin_batch_edit`/`end_batch_edit`,/) switches the backing store to LIL for a burst of column mutations and back to CSR afterward, so a whole coupled-group merge does not pay repeated format-conversion costs. @@ -852,7 +847,7 @@ scaling a row of `S` does not change its null vectors. So instead of dividing (w fractions), the algorithm cross-multiplies and then *removes the common integer factor*. **Setup — clear denominators once.** Each input row `r` has its rational entries `num/den` cleared to -integers by multiplying the whole row by the LCM of its denominators (`:527`–`:539`). After this every +integers by multiplying the whole row by the LCM of its denominators. After this every working row is a pure integer row; there are no denominators to track for the rest of the routine — this is the sense in which it is "fraction-free." @@ -863,41 +858,41 @@ target row with entry `ev` in column `c`, the update is new_row[k] = ev_scaled · pivot[k] − pv_scaled · target[k] (conceptually) ``` -where the code (`_eliminate`, `:564`) first divides `pv, ev` by `g = gcd(pv, ev)` to get +where the code (`_eliminate`) first divides `pv, ev` by `g = gcd(pv, ev)` to get `pv_scaled = pv/g`, `ev_scaled = ev/g`, then computes, for the sparse pivot row `prd`, `new_row = {c: v·pv_scaled}` over the target row and subtracts `ev_scaled·prd[c]` on the shared -columns (`:583`–`:589`). This is the classical **fraction-free (Bareiss-style) update**: it keeps +columns. This is the classical **fraction-free (Bareiss-style) update**: it keeps everything integer and makes column `c` vanish in the target, because `ev_scaled·pv − pv_scaled·ev = 0` after the GCD split. **Content reduction (GCD) — why coefficients stay polynomial.** Cross-multiplying integer rows makes entries grow. Without control, the bit-length of coefficients grows *exponentially* down the elimination. The defence is to divide each freshly-computed row by the GCD of all its entries — its -"content" — right after forming it (`:592`–`:595`): `row_gcd = gcd(*new_row.values)` then +"content" — right after forming it : `row_gcd = gcd(*new_row.values)` then `row[c] //= row_gcd`. This is exactly the mechanism (Bareiss / fraction-free Gaussian elimination) that bounds intermediate integers to the size of subdeterminants of the original matrix, i.e. keeps the bit-length **polynomial** rather than exponential. A final content reduction of the pivot rows runs at -`:680`–`:686` as insurance. +– as insurance. **Markowitz pivoting — keep it sparse.** On a genome-scale `S` the elimination is dominated not by arithmetic but by *fill-in* and *pivot search*. Two heuristics keep both small: -- Columns are pre-sorted by ascending nnz (`col_order`, `:510`–`:514`) so that sparse columns — the - likely pivots — are visited first; rows are pre-sorted by ascending nnz (`:544`–`:546`). Results are - translated back to the original column order at the end (`:688`–`:691`). +- Columns are pre-sorted by ascending nnz (`col_order`,–) so that sparse columns — the + likely pivots — are visited first; rows are pre-sorted by ascending nnz. Results are + translated back to the original column order at the end. - At each step the pivot is chosen by the **Markowitz criterion** among the rows that actually contain the current pivot column: sparsest row first, ties broken by smallest absolute pivot value - (`:628`–`:637`). A live `col_rows` index (`:554`–`:562`) maps each column to the set of active rows + . A live `col_rows` index maps each column to the set of active rows containing it, so pivot search visits only the handful of rows that hold the column instead of scanning all active rows (on iML1515 that scan was ~99.9% misses; the index removes it). -**Two-phase echelon, not full Gauss–Jordan.** Phase 1 (`:613`–`:650`) does forward elimination only — +**Two-phase echelon, not full Gauss–Jordan.** Phase 1 does forward elimination only — each pivot is cleared from rows *below* it, leaving already-processed pivot rows sparse. Phase 2 -(`:652`–`:679`) does back-substitution, processing pivots last-to-first and clearing each pivot column +does back-substitution, processing pivots last-to-first and clearing each pivot column from the pivot rows *above* it. Doing it in this order means that when a pivot row is applied during back-substitution, its own later-pivot columns are already cleared, so back-substitution only ever introduces *free-column* fill and only ever *removes* pivot-column entries — enabling the -`pivcol_holders` index (`:664`–`:668`) to be maintained with discards only. The commit comments record +`pivcol_holders` index to be maintained with discards only. The commit comments record the payoff on iML1515: ~0.8M back-substitution ops versus ~9.4M for naive Gauss–Jordan, because full Gauss–Jordan re-reduces every filled row against every later pivot (~99% of the total work). @@ -910,19 +905,19 @@ The routine returns `(rref_data, rank, pivot_columns)` where `rref_data[i]` is p pivots and `cols` columns, the free columns are `free_cols = {0..cols−1} \ pivots` and the nullity is `|free_cols|`. For each free column `f` the basis vector `k_f` is built by the standard RREF rule: -- entry `+1` at row `f` (the free variable is set to 1), `:726`–`:731`; +- entry `+1` at row `f` (the free variable is set to 1),–; - at each pivot row `i` with pivot column `p_i`, entry `−rref[i,f] / rref[i,p_i]`, reduced by GCD to a - clean rational and given a positive denominator (`:734`–`:749`). + clean rational and given a positive denominator. So `k_f` has value `1` in its own free coordinate and `−(free entry)/(pivot value)` in each pivot coordinate. By construction `S·k_f = 0` exactly. The set `{k_f}` is a sparse rational basis of the -right nullspace — one column per free variable — assembled by `_build_from_sparse_data` (`:206`). This +right nullspace — one column per free variable — assembled by `_build_from_sparse_data`. This sparsity is exactly what makes coupling detection cheap in §3.3–§3.4: a coupled reaction shows up as a kernel *row* with a distinctive zero pattern, and sparse kernel rows make that pattern comparison a dictionary lookup. -`nullspace` (`:759`) is the public wrapper; `basic_columns` (`:774`) returns just the pivot columns -(used by conservation removal, §3.5); `sparse_nullspace` (`:785`) is the general-purpose exact-kernel +`nullspace` is the public wrapper; `basic_columns` returns just the pivot columns +(used by conservation removal, §3.5); `sparse_nullspace` is the general-purpose exact-kernel helper that accepts scipy/numpy/`RationalMatrix` input. #### 3.2.5 The big-integer path — when subdeterminants exceed int64 @@ -932,30 +927,30 @@ entries are ratios of subdeterminants of `S`, and on dense, large models those s exceed the 64-bit integers that scipy sparse matrices can hold. The verified extreme is **yeast-GEM, whose exact nullspace needs coefficients up to ~263 bits** — far beyond int64. -The engine handles this transparently. `_INT64_MAX` (`:93`) and `_fits_int64` (`:96`) test whether all -numerators and denominators fit in signed int64. `_build_from_sparse_data` (`:206`) checks this: if -everything fits, it builds the fast dual-`int64`-CSR representation (`:214`–`:217`); if not, it falls -back to a **dict-of-`Fraction`s** store, `_dict_frac : {row: {col: Fraction}}` (`:218`–`:225`), which -uses Python arbitrary-precision integers and bypasses scipy entirely. `is_bigint` (`:407`) reports +The engine handles this transparently. `_INT64_MAX` and `_fits_int64` test whether all +numerators and denominators fit in signed int64. `_build_from_sparse_data` checks this: if +everything fits, it builds the fast dual-`int64`-CSR representation; if not, it falls +back to a **dict-of-`Fraction`s** store, `_dict_frac : {row: {col: Fraction}}`, which +uses Python arbitrary-precision integers and bypasses scipy entirely. `is_bigint` reports which mode a matrix is in. The RREF itself never overflows — it works in Python `int` throughout; only the *storage* of the finished kernel needs the fallback. Because scipy sparse cannot hold >int64 values, the export helpers are mode-aware. `to_sparse_csr` -(`:382`) raises `OverflowError` in big-integer mode (with a message pointing at the exact exports). -`to_coo_exact` (`:412`) is the big-integer-safe export used in both modes: it returns an `ExactCOO` -namedtuple `(rows, cols, data, shape, denom)` (defined `:103`) in which entry `(rows[k], cols[k])` +raises `OverflowError` in big-integer mode (with a message pointing at the exact exports). +`to_coo_exact` is the big-integer-safe export used in both modes: it returns an `ExactCOO` +namedtuple `(rows, cols, data, shape, denom)` (defined) in which entry `(rows[k], cols[k])` equals `data[k]/denom` exactly, with `data` arbitrary-precision Python ints scaled to a common -denominator. `to_sparse_pattern` (`:435`) returns a pure-structure `int8` CSR (1s where nonzero) plus a +denominator. `to_sparse_pattern` returns a pure-structure `int8` CSR (1s where nonzero) plus a `{row: {col: Fraction}}` value map — this is the form coupling detection consumes, and it works identically in int64 and big-integer mode, so the whole compression pipeline runs unchanged on -yeast-GEM. `sparse_nullspace` (`:785`) returns a scipy CSR in the common case and an `ExactCOO` when -`K.is_bigint` (`:820`–`:823`). +yeast-GEM. `sparse_nullspace` returns a scipy CSR in the common case and an `ExactCOO` when +`K.is_bigint`. ### 3.3 The compression working state and the single-kernel pass The nullspace-driven compressor is `StoichMatrixCompressor` (`compression.py`), driven through a -mutable `_WorkRecord` (`:930`). The `_WorkRecord` carries three exact matrices that together record the -entire transformation and satisfy the invariant recorded on `CompressionRecord` (`:896`): +mutable `_WorkRecord`. The `_WorkRecord` carries three exact matrices that together record the +entire transformation and satisfy the invariant recorded on `CompressionRecord`: ``` pre @ stoich @ post == cmp @@ -964,33 +959,33 @@ pre @ stoich @ post == cmp with the flux-space consequence `v_original = post @ v_compressed`. Concretely `pre` is a `RationalMatrix` starting as `identity(m)` (metabolite transformation, tracks row/metabolite operations), `post` starts as `identity(n)` (reaction transformation, tracks column/reaction merges), -and `cmp` starts as a clone of `stoich` and is mutated in place as compression proceeds (`:930`–`:947`). +and `cmp` starts as a clone of `stoich` and is mutated in place as compression proceeds. Every reaction merge is applied *identically to `cmp` and to `post`* so the invariant is preserved and `post` can later expand a compressed flux vector back to the original reaction space ([Ch 9](#ch9)). -The compress driver `StoichMatrixCompressor.compress` (`:1095`) runs a loop (`:1121`–`:1128`): remove +The compress driver `StoichMatrixCompressor.compress` runs a loop : remove all-zero metabolite rows, then call `_nullspace_compress`, and re-iterate only while the previous pass reported a *contradicting* removal (which changes the flux space and can expose new couplings). Note the important design choice: **one nullspace computation drives both zero-flux detection and coupled-group -merging in the same pass.** `_nullspace_compress` (`:1133`) builds the active submatrix, computes -`kernel = nullspace(active)` once (`:1144`), extracts `(kernel_pattern, kernel_values)` via -`to_sparse_pattern` (`:1150`), and hands both to `_handle_compress` (`:1248`). +merging in the same pass.** `_nullspace_compress` builds the active submatrix, computes +`kernel = nullspace(active)` once, extracts `(kernel_pattern, kernel_values)` via +`to_sparse_pattern`, and hands both to `_handle_compress`. -The single kernel yields three kinds of removals in one batch (`_handle_compress`, `:1248`–`:1337`): +The single kernel yields three kinds of removals in one batch (`_handle_compress`,–): 1. **Structural zero-flux reactions** — reactions whose kernel *row is empty*. `_find_zero_flux` - (`:1155`) reports reaction `reac` as zero-flux iff `kernel_pattern.indptr[reac] == + reports reaction `reac` as zero-flux iff `kernel_pattern.indptr[reac] == kernel_pattern.indptr[reac+1]`, i.e. the reaction appears in no null vector. Such a reaction cannot carry any steady-state flux (`Sv=0` forces `v_reac = 0`), so it can never be part of a working pathway and is deleted. This is the *structural* blocked-reaction test, and because it falls out of the kernel it needs no LP/FVA (contrast the bounds-based test in §3.6). 2. **Bounds-blocked reactions** — reactions with `lb = ub = 0` that nonetheless have a nonzero kernel - row are added to the same removal set (`:1266`–`:1271`); they are structurally capable of flux but + row are added to the same removal set; they are structurally capable of flux but pinned to zero by bounds, so removing them here avoids a separate FVA pass. 3. **Coupled-group slaves (and contradicting groups)** — see §3.4. -Everything collected is removed in one `remove_reactions_by_indices` batch (`:1335`), which drops the -columns from `cmp` and `post` together and reindexes names/bounds (`:986`–`:1004`). `_handle_compress` +Everything collected is removed in one `remove_reactions_by_indices` batch, which drops the +columns from `cmp` and `post` together and reindexes names/bounds. `_handle_compress` returns `True` only if a *contradicting* group was removed, which is the sole trigger for another iteration. @@ -1029,32 +1024,32 @@ Both tests are exact equalities on rationals — which is precisely why §3.2's `_find_coupled_groups` (`compression.py`) implements exactly that two-stage test. First it buckets reactions by kernel-row zero pattern: `pattern = tuple(kernel_pattern.indices[start:end])` per reaction, -grouped into a dict, keeping only buckets of size > 1 (`:1181`–`:1188`). Then, within each candidate -bucket, it verifies the constant ratio (`:1201`–`:1244`): pick reaction `a`, take the first nonzero -column `first_col`, compute `ratio = a_val/b_val` there (exact `Fraction` division, `:1218`–`:1226`), -and confirm `a_v/b_v == ratio` for every remaining nonzero column (`:1230`–`:1235`). Reactions that +grouped into a dict, keeping only buckets of size > 1. Then, within each candidate +bucket, it verifies the constant ratio : pick reaction `a`, take the first nonzero +column `first_col`, compute `ratio = a_val/b_val` there (exact `Fraction` division,–), +and confirm `a_v/b_v == ratio` for every remaining nonzero column. Reactions that pass are collected into a group with `ratios[reac_b] = ratio` recorded per slave. The output is `(groups, ratios)`: each group is `[master, slave1, slave2, …]` (master is the first member), and `ratios[slave]` is the exact `Fraction` `v_master / v_slave`. -The `protected_indices` argument (`:1164`, applied at `:1202`/`:1211`) lets specific reactions be kept +The `protected_indices` argument (applied at/) lets specific reactions be kept out of any coupled group — the rest of the group still merges. This is how gene-controlled reactions are held intact through COMPRESS #1 so that gene multiplicity survives into GPR integration (cross-reference [Ch 4](#ch4)); the mapping from protected *names* to current *indices* is done in -`_handle_compress` (`:1275`–`:1276`). +`_handle_compress`. #### 3.4.3 The merge (COLUMN reduction): `_combine_coupled` -Merging is a column operation. `_combine_coupled` (`:1339`) folds each slave column into the master. +Merging is a column operation. `_combine_coupled` folds each slave column into the master. Given `ratios[slave] = v_master/v_slave = λ`, the master flux relates to the slave's own flux by `v_slave = v_master/λ`, so the slave's stoichiometric contribution, expressed in units of the master flux, is `col[slave] · (1/λ)`. The code computes the multiplier as `mult = 1/λ = λ.denominator / -λ.numerator` (`:1350`) and applies `cmp[:,master] += cmp[:,slave]·mult` and the *same* update to -`post[:,master]` (`:1353`–`:1356`), both via the exact `add_scaled_column`. Applying it to `post` +λ.numerator` and applies `cmp[:,master] += cmp[:,slave]·mult` and the *same* update to +`post[:,master]`, both via the exact `add_scaled_column`. Applying it to `post` records that the compressed master reaction expands back to a specific exact linear combination of the original columns — the master column of `cmp` becomes the exact stoichiometry of the lumped pathway, and the master column of `post` becomes the exact expansion recipe. The slaves are then deleted -(`:1326`–`:1327`), so the group of `k` reactions becomes **one** reaction: `k−1` binaries eliminated per +, so the group of `k` reactions becomes **one** reaction: `k−1` binaries eliminated per group. This is a **column (reaction) reduction**. **Worked micro-example.** Take the linear pathway `r1: A→B`, `r2: B→C`, `r3: C→D(ext)` with `A` supplied @@ -1071,23 +1066,23 @@ the constant ratio, carried as an exact `Fraction`, is what makes the cancellati Merging the columns is not the whole story: the slaves' flux *bounds* must be transferred to the master, or the compressed model would silently drop feasibility restrictions. `_handle_compress` -(`:1289`–`:1327`) does this. Because `v_slave = v_master/λ` (with `λ = ratios[slave]`), the slave's +does this. Because `v_slave = v_master/λ` (with `λ = ratios[slave]`), the slave's box `lb_s ≤ v_slave ≤ ub_s` becomes a constraint on `v_master`: -- if `λ > 0`: `lb_s·λ ≤ v_master ≤ ub_s·λ` (`:1302`–`:1305`); -- if `λ < 0`: the inequality flips, `ub_s·λ ≤ v_master ≤ lb_s·λ` (`:1306`–`:1309`). +- if `λ > 0`: `lb_s·λ ≤ v_master ≤ ub_s·λ`; +- if `λ < 0`: the inequality flips, `ub_s·λ ≤ v_master ≤ lb_s·λ`. with `±inf` propagated so that an unbounded slave contributes no restriction. The master's new box is the **intersection** of its own box with all translated slave boxes: `intersected_lb = max(...)`, -`intersected_ub = min(...)` (`:1311`–`:1315`), written back to `work.bounds[master]` (`:1315`). +`intersected_ub = min(...)`, written back to `work.bounds[master]`. **Contradicting groups.** If the intersection is empty (`intersected_lb > intersected_ub`) or collapses to a single point at zero (`intersected_lb == intersected_ub == 0`), the coupled group can carry no nonzero flux in any steady state — a *contradicting* group. Then the master *and all slaves* are removed -(`:1317`–`:1323`) and `contradicting_removed` is set, which is the flag that triggers a re-iteration of -the whole pass (`:1337` → `:1126`): removing a contradicting group changes the flux space and may make +and `contradicting_removed` is set, which is the flag that triggers a re-iteration of +the whole pass (→): removing a contradicting group changes the flux space and may make previously-uncoupled reactions coupled. A consistent (nonempty) group removes only the slaves -(`:1324`–`:1327`). This bound-intersection logic replaced a Java-era behaviour that could drop +. This bound-intersection logic replaced a Java-era behaviour that could drop reactions incorrectly; getting the translate-and-intersect direction right (especially the `λ<0` flip and the `±inf` handling) is exactly the subject of the closed issue #44 cautionary tale in [Ch 10](#ch10). @@ -1103,17 +1098,17 @@ exactly unchanged. It is therefore lossless for fluxes, and it strictly reduces The mechanics use the exact RREF as a rank/independence oracle. The function builds `Sᵀ` (reactions × metabolites) directly from the cobra coefficients as a `RationalMatrix` — deliberately transposed so -that *metabolites become columns* (`:1428`–`:1455`) — and calls `basic_columns` (`:1456`), which runs +that *metabolites become columns* — and calls `basic_columns`, which runs `_rref_integer_sparse` and returns the pivot columns. The pivot columns of `Sᵀ` are a maximal set of **linearly independent metabolite rows**; every non-pivot metabolite is a dependent row, i.e. a -conservation relation. Those dependent metabolites are removed from the model (`:1458`–`:1460`). +conservation relation. Those dependent metabolites are removed from the model. Two design points. First, this is a **row-rank reduction**, complementary to the column reduction of §3.4 — together they push `S` toward full rank (the §3.1 hypothesis). Second, the *ordering* matters: conservation removal runs *before* the expensive coupled step in each cycle (`compress_model`, -`:1906`–`:1910`). Fewer metabolite rows means the nullspace RREF that drives coupling detection operates +–). Fewer metabolite rows means the nullspace RREF that drives coupling detection operates on a smaller matrix, so removing dependent rows first makes the costliest stage cheaper. (There is a -legacy Java oracle, `_remove_conservation_relations_java` at `:1943`, selectable via the +legacy Java oracle, `_remove_conservation_relations_java` at, selectable via the `efmtool_rref` backend; the default `sparse_rref` path uses the pure-Python exact RREF above.) ### 3.6 Blocked and zero-flux removal @@ -1121,30 +1116,30 @@ legacy Java oracle, `_remove_conservation_relations_java` at `:1943`, selectable There are two distinct notions of "carries no flux," removed at two points: - **Bounds-blocked reactions** — `remove_blocked_reactions` (`compression.py`) deletes reactions - whose bounds are exactly `(0, 0)` (`:1701`) with `remove_orphans=True` so metabolites left dangling - go too. This runs once at the very start of `compress_model` (`:1889`), before any rational + whose bounds are exactly `(0, 0)` with `remove_orphans=True` so metabolites left dangling + go too. This runs once at the very start of `compress_model`, before any rational conversion, as a cheap first cut. - **Structural zero-flux reactions** — reactions whose *kernel row is empty* (§3.3, `_find_zero_flux`, - `:1155`). These are reactions that `Sv=0` forces to zero regardless of bounds; they are found for + ). These are reactions that `Sv=0` forces to zero regardless of bounds; they are found for free from the nullspace during each coupled pass and removed in the same batch. The additional check - at `:1266`–`:1271` catches reactions pinned to `(0,0)` by bounds that still have a nonzero kernel row, + at– catches reactions pinned to `(0,0)` by bounds that still have a nonzero kernel row, folding the bounds-blocked case into the kernel pass as well. -`remove_unused_metabolites` (`_WorkRecord`, `:1044`) is the row-side companion: after columns are +`remove_unused_metabolites` (`_WorkRecord`) is the row-side companion: after columns are dropped, any metabolite row that has become all-zero (detected in O(m) via CSR `indptr` diffs, -`:1054`–`:1055`) is removed. It runs at the top and bottom of the compress loop (`:1124`, `:1129`). +–) is removed. It runs at the top and bottom of the compress loop. ### 3.7 The alternating fixpoint `compress_model` (`compression.py`) orchestrates the three reducers into an **alternating -fixpoint** (`:1894`–`:1937`). The order within each cycle is deliberate: +fixpoint**. The order within each cycle is deliberate: 1. **Parallel merge** (`compress_model_parallel`, §3.8) — cheapest: a hash of the (scale-normalized) - stoichiometry row, no RREF (`:1899`). + stoichiometry row, no RREF. 2. **Conservation-relation removal** (§3.5) — shrinks `S`'s rows so the next step's RREF is smaller - (`:1906`–`:1910`). + . 3. **Coupled merge** (`compress_model_coupled`, §3.4) — most expensive: a full exact nullspace/RREF - (`:1920`–`:1935`). + . The loop runs cheap-to-expensive so that each stage feeds the next a smaller network, and the expensive kernel computation only ever runs on an already-thinned matrix. @@ -1156,14 +1151,14 @@ can change the kernel (new couplings); conservation removal changes the row set pass of each is not enough — the pipeline loops. Termination is guaranteed because **every reducer only ever removes reactions or metabolites; none ever adds one.** The reaction count is a non-negative integer that is non-increasing across the loop, so it cannot decrease forever. The explicit stop -condition (`:1916`–`:1918`) is: after at least one full cycle, if *either* the parallel step or the +condition is: after at least one full cycle, if *either* the parallel step or the coupled step found nothing, stop — because a step that changed nothing on the current network will change nothing on re-run unless the *other* step alters the network, and the loop has just established that it did not make progress. `run` counts cycles for the log. In practice on genome-scale models this converges in a handful of cycles. Each productive step appends a record to `cmp_mapReac` — `{"reac_map_exp": reac_map_exp, "parallel": -}` (`:1904`, `:1935`) — the compression map consumed by decompression (§3.10). +}` — the compression map consumed by decompression (§3.10). ### 3.8 Parallel merge @@ -1173,15 +1168,15 @@ factor) *and* have compatible bound topology, e.g. two isozymic reactions with t It never computes a kernel — it groups reactions by an exact hashable key. **Scale-invariant, exact key.** The stoichiometry matrix is taken transposed (`stoichmat_T`, one row -per reaction) and each reaction's key (`_parallel_key`, `:2058`) is its stoichiometry row **normalized -by its first nonzero coefficient in exact rational arithmetic**: `f0 = float_to_rational(vals[0])`, then -`stoich = tuple((col, float_to_rational(v)/f0) …)` (`:2062`–`:2064`). Normalizing by the first +per reaction) and each reaction's key (`_parallel_key`) is its stoichiometry row **normalized +by its first nonzero coefficient in exact rational arithmetic**: `f0 = float_to_fraction(vals[0])`, then +`stoich = tuple((col, float_to_fraction(v)/f0) …)`. Normalizing by the first coefficient makes the key **scale-invariant**: `−1 A → 2 B` and `−3 A → 6 B` both reduce to the tuple `((A,1),(B,−2))` and so share a key, but the division is exact (`Fraction`), so two rows that are only *nearly* proportional get *different* keys — no reaction is ever merged on a rounding coincidence. **Bound topology is part of the key.** The key also carries three bound-derived flags per reaction, -computed at `:2048`–`:2051`: +computed at–: - `fwd`/`rev`: whether the reaction is unbounded in the forward / reverse chemical direction (an `inf` bound on the appropriate side given the sign of the first coefficient); @@ -1193,16 +1188,16 @@ component no other reaction can match, so it is never lumped in parallel.** Para restricted to reactions whose bounds are homogeneous (each side `0` or `±inf`) and whose reversibility matches — i.e. reactions that live in the same cone face. This is the correctness guard that keeps parallel merging from combining reactions with incompatible feasibility. Grouping is a hash pre-filter -(`key_hashes`) followed by an exact full key comparison (`:2073`–`:2085`); `protected_rxns` are forced -into singleton groups (`:2076`–`:2078`). +a single pass appending each reaction index to `groups[key]` under its exact key; `protected_rxns` are forced +into singleton groups. **COLUMN reduction and the flux-split map.** Each group keeps one representative (its id is decorated -with `*`-joined member ids, truncated to `...` past ~220 chars, `:2094`–`:2097`) and the others are -removed (`:2114`–`:2116`) — again a **column reduction**, `k−1` binaries removed per group. The +with `*`-joined member ids, truncated to `...` past ~220 chars,–) and the others are +removed — again a **column reduction**, `k−1` binaries removed per group. The compression map differs from the coupled case in a way that matters for cost accounting: for a parallel group the *compressed* flux is the **total** flux through all members, and each member's share is proportional to its stoichiometric scale `|factor[j]|` (its first-coefficient magnitude). The map is -built (`:2127`–`:2141`) as normalized flux-split fractions: +built as normalized flux-split fractions: ``` rational_map[cmp_id][orig_j] = |factor[j]| / Σ_k |factor[k]| (fractions sum to 1) @@ -1221,34 +1216,29 @@ flux-split map `{r1: ½, r2: ½}` (equal `|factor|`). A knockout of the lump mea knocked out, so its KO cost is the sum — correctly capturing that either isozyme alone still runs the reaction. -### 3.9 GPR propagation through compression - -When compression runs with `propagate_gpr=True` (COMPRESS #1, before gene pseudoreactions exist), each -merge must carry the Boolean gene–protein–reaction (GPR) rules of its members onto the surviving -reaction, so the compressed model still knows which genes control the lumped reaction. This chapter -covers *only the propagation through a merge*; the semantics of encoding GPR as flux structure belongs -to [Ch 4](#ch4) (`extend_model_gpr`), cross-referenced there. - -The rule follows the flux logic of each merge type: - -- **Serial / coupled merges → AND.** A coupled group is an unbranched chain that must run as a unit — - every member's genes are required for the lumped reaction to carry flux — so their GPRs are combined - with **AND**. `_combine_gpr_and` (`compression.py`) is invoked from `compress_model_coupled` - (`:2007`–`:2015`) over the saved GPR ASTs of the contributing reactions. -- **Parallel merges → OR.** Parallel members are alternative routes for the same conversion — *any* of - them suffices — so their GPRs are combined with **OR**. `_combine_gpr_or` (`compression.py`) is - invoked from `compress_model_parallel` (`:2107`–`:2121`). - -Both combiners lift the cobra GPR AST to sympy Boolean expressions (`_gpr_ast_to_sympy`, `:1754`), -combine with `sympy.And`/`sympy.Or` (which auto-flatten and dedupe), and render back to a rule string -(`_sympy_to_gpr_string`, `:1773`). The subtlety is the treatment of an **empty GPR** (a reaction with -no gene requirement, "always active", logically `True`): in an AND-combine an empty GPR is a no-op and -is skipped, and if *all* members are empty the result is empty (`:1815`–`:1822`); in an OR-combine a -single empty member makes the whole lump always-active, so the result is empty (`:1837`–`:1839`). Full -Boolean simplification is deferred to `reduce_gpr` downstream ([Ch 4](#ch4)). Note also that the coupled Python -backend clears gene rules on the raw reactions before the merge (`compress_model_coupled`, `:1996`– -`:1998`) and reinstates the combined rule afterward from the *saved* ASTs (`:1982`–`:1983`, -`:2007`–`:2015`), so the propagation is driven off a clean snapshot rather than the mutated model. +### 3.9 GPR propagation and simplification + +When COMPRESS #1 runs with `propagate_gpr=True`, the Boolean rule of every merged reaction must be +carried to the survivor: + +- Coupled/serial members are joined with **AND** because every member must carry its fixed share. +- Parallel alternatives are joined with **OR** because any member can supply the lumped flux. + +Both paths call `_combine_gprs(gpr_bodies, op)`. Cobra AST nodes are converted into a small nested +expression representation, same-operator children are flattened, duplicates are removed, and the +result is rendered back to a deterministic GPR string. This combination step deliberately avoids +SymPy and does not attempt global minimization. + +An empty GPR means “always active.” It is therefore the identity for an AND merge and absorbing for +an OR merge. Saved AST bodies are used because the compression backend clears the reaction rules while +performing its linear-algebra work. + +After a GPR-propagating compression, `simplify_model_gprs` applies the monotone simplifier in +`compression.py`. It parses the rule, constructs an absorbed sum-of-products representation within a +bounded expansion budget, algebraically factors it, and writes a Boolean-equivalent rule with fewer +gene leaves where possible. The simplifier is also called explicitly by the strain-design pipeline so +the no-compression path receives the same rule minimization. + ### 3.10 The compression map `cmp_mapReac` and back-expansion @@ -1293,7 +1283,7 @@ name `StoichMatrixCompressor` (`compression.py`), and the `CoupledZero`/`Coupled efmtool is a Java library (namespace `ch.javasoft.*`, packaged as `efmtool.jar` alongside the Python sources at `straindesign/efmtool.jar`). straindesign uses only its *compression* half — not its EFM -enumeration — through the classes loaded in `efmtool_cmp_interface.py`–`:179`: +enumeration — through the classes loaded in `efmtool_cmp_interface.py`–: `ch.javasoft.smx.impl.DefaultBigIntegerRationalMatrix` (an arbitrary-precision rational matrix), `ch.javasoft.smx.ops.Gauss` (rational Gaussian elimination), `ch.javasoft.metabolic.compress. StoichMatrixCompressor` and `CompressionMethod`, and `ch.javasoft.math.BigFraction` / @@ -1303,24 +1293,24 @@ in-process JVM, adds `efmtool.jar` to the classpath, and imports the Java classe The routing has three layers. -1. **Import time.** `__init__.py`–`:53` calls `_start_jvm` *eagerly* at `import straindesign`. +1. **Import time.** `__init__.py`– calls `_start_jvm` *eagerly* at `import straindesign`. This is a no-op when jpype1 or a JVM is absent (neither is a package dependency), so a normal install never touches Java. When Java *is* present the JVM must be started here — before NumPy/OpenBLAS spins - up worker threads — or JNI calls later crash with SIGBUS/SIGSEGV (`__init__.py`–`:50`; the code is + up worker threads — or JNI calls later crash with SIGBUS/SIGSEGV (`__init__.py`–; the code is littered with such mitigations, see §3.11.4). 2. **Backend selection.** `compute_strain_designs` reads the kwarg `compression_backend = kwargs.get('compression_backend', 'sparse_rref')` (`compute_strain_designs.py`) and threads it into both `compress_model` calls - (`:357`–`:360`, `:435`). `compress_model` sets `use_java = (compression_backend == 'efmtool_rref')` + . `compress_model` sets `use_java = (compression_backend == 'efmtool_rref')` (`compression.py`). 3. **Dispatch inside the fixpoint.** Crucially, `efmtool_rref` does **not** replace the whole compression pipeline — only two of its three reducers. Inside the alternating fixpoint (§3.7, - `compression.py`–`:1937`): + `compression.py`–): - **Parallel merge** (step 1, §3.8) is **always** the Python hash-based `compress_model_parallel` — efmtool has no equivalent and it is never routed to Java. - - **Conservation removal** (step 2, §3.5) forks on `use_java` (`:1907`–`:1910`): Java goes through - `_remove_conservation_relations_java` (`:1943`), Python through `remove_conservation_relations`. - - **Coupled merge** (step 3, §3.4) forks inside `compress_model_coupled` (`:1985`): Java calls + - **Conservation removal** (step 2, §3.5) forks on `use_java` : Java goes through + `_remove_conservation_relations_java`, Python through `remove_conservation_relations`. + - **Coupled merge** (step 3, §3.4) forks inside `compress_model_coupled`: Java calls `compress_model_java` (`efmtool_cmp_interface.py`), Python calls `compress_cobra_model`. So `efmtool_rref` is really a **hybrid**: Python parallel-merge + Java conservation-removal + Java @@ -1335,23 +1325,23 @@ marshalling lives. It mutates the cobra model in place and returns the same pipeline (module remapping, cost compression, decompression in [Ch 9](#ch9)) is backend-agnostic. **Into Java.** -- `stoichmat_coeff2rational(model)` (`:387`) first converts every stoichiometric coefficient to an +- `stoichmat_coeff_to_fraction(model)` first converts every stoichiometric coefficient to an exact `Fraction`/sympy-`Rational` — the same exactness discipline as §3.2.1, done *before* any Java call. -- All gene rules are cleared, `r.gene_reaction_rule = ''` (`:389`), matching the Python coupled path +- All gene rules are cleared, `r.gene_reaction_rule = ''`, matching the Python coupled path (§3.9); GPR is re-attached afterward (below). -- A `DefaultBigIntegerRationalMatrix(num_met, num_active)` is allocated (`:407`) and filled column by +- A `DefaultBigIntegerRationalMatrix(num_met, num_active)` is allocated and filled column by column. Reactions whose upper bound is `≤ 0` are **flipped** to the forward direction - (`model.reactions[mi] *= -1`, `:412`–`:415`) and their index recorded in `flipped`; efmtool's + (`model.reactions[mi] *= -1`,–) and their index recorded in `flipped`; efmtool's compressor assumes a canonical orientation. Each coefficient `v` is converted by - `sympyRat2jBigIntegerPair` (`:285`) into a Java `BigInteger` numerator/denominator pair — using + `sympyRat2jBigIntegerPair` into a Java `BigInteger` numerator/denominator pair — using `BigInteger.valueOf` for values that fit in 63 bits and `BigInteger(str(...))` otherwise — and set as - a `BigFraction(n, d)` (`:416`–`:418`). This path is **exact**: efmtool's `DefaultBigIntegerRational + a `BigFraction(n, d)`. This path is **exact**: efmtool's `DefaultBigIntegerRational Matrix` is arbitrary-precision, so the Java core does *not* overflow. - A `StoichMatrixCompressor(subset_compression)` is built, where `subset_compression = - [CoupledZero, CoupledCombine, CoupledContradicting]` (`:181`–`:183`): remove structurally + [CoupledZero, CoupledCombine, CoupledContradicting]` : remove structurally zero-flux reactions, combine coupled groups, and drop contradicting groups — the Java analogues of - §3.3's three removal kinds. `smc.compress(stoich_mat, reversible, …, reacNames, None)` (`:423`) + §3.3's three removal kinds. `smc.compress(stoich_mat, reversible, …, reacNames, None)` returns a `comprec` whose `post` matrix is the reaction transformation (the Java counterpart of the Python `post` in §3.3, `v_original = post · v_compressed`). @@ -1364,34 +1354,34 @@ subset_matrix = jpypeArrayOfArrays2numpy_mat(comprec.post.getDoubleRows()) # : The *structure* of the compression (which original reaction maps into which compressed column, and the zero pattern) is read back as a **double-precision** numpy matrix via `getDoubleRows`. The per-reaction merge then: -- flags a reaction zero-flux iff its `subset_matrix` row is all-zero (`:432`–`:434`); -- for each compressed column `j`, gathers members from `subset_matrix[:,j].nonzero` (`:437`), scales +- flags a reaction zero-flux iff its `subset_matrix` row is all-zero; +- for each compressed column `j`, gathers members from `subset_matrix[:,j].nonzero`, scales each member's stoichiometry by the **exact** factor `jBigFraction2sympyRat(comprec.post. - getBigFractionValueAt(ai, j))` (`:445`–`:446`, exact `BigFraction → sympy.Rational`), and **rescales - its bounds by `/= abs(subset_matrix[ai, j])`** (`:447`–`:450`, i.e. by a **double**); + getBigFractionValueAt(ai, j))` (–, exact `BigFraction → sympy.Rational`), and **rescales + its bounds by `/= abs(subset_matrix[ai, j])`** (–, i.e. by a **double**); - merges member reactions into the group representative, concatenating ids with `*` and truncating past - ~220 chars to `...` (`:456`–`:467`) — the same naming convention as the parallel backend (§3.8); + ~220 chars to `...` — the same naming convention as the parallel backend (§3.8); - records `subset_rxns`/`subset_stoich` per representative (negating the stoich for `flipped` - reactions, `:452`–`:455`) and finally assembles `rational_map` from them (`:493`–`:499`). + reactions,–) and finally assembles `rational_map` from them. So the *factors* are exact rationals, but the *pattern detection and the bound rescaling* pass through -double precision. The `suppressed_reactions` argument (`:367`, `:392`) — reaction ids that must survive +double precision. The `suppressed_reactions` argument — reaction ids that must survive because a strain-design module references them — are excluded from the active set entirely and re-added -as standalone identity entries (`:480`–`:485`), a workaround for efmtool's `CoupledContradicting` step, +as standalone identity entries, a workaround for efmtool's `CoupledContradicting` step, which will otherwise delete reactions it deems inconsistent (contrast the Python backend, which keeps them via the exact bounds-intersection of §3.4.4). Back in `compress_model_coupled` the Java branch -then sweeps up any leftover `(0,0)` reactions (`compression.py`–`:1994`) and — identically to the +then sweeps up any leftover `(0,0)` reactions (`compression.py`–) and — identically to the Python branch — re-attaches the **AND-combined GPR** from the pre-merge snapshot -(`compression.py`–`:2015`). GPR propagation is therefore the *same* for both backends on the +(`compression.py`–). GPR propagation is therefore the *same* for both backends on the coupled step. **The conservation path.** `_remove_conservation_relations_java` (`compression.py`) builds `S` as -a LIL matrix, **densifies its transpose** (`stoich_mat.transpose.toarray`, `:1947`), and hands it +a LIL matrix, **densifies its transpose** (`stoich_mat.transpose.toarray`), and hands it to `basic_columns_rat_java` (`efmtool_cmp_interface.py`). That function wraps the dense array into a `DefaultBigIntegerRationalMatrix` via `numpy_mat2jpypeArrayOfArrays` — which builds a **`JDouble[rows, -cols]`** (`:267`) — then runs `Gauss.getRationalInstance.rowEchelon(...)` (`:360`) and returns the +cols]`** — then runs `Gauss.getRationalInstance.rowEchelon(...)` and returns the pivot columns, i.e. the independent metabolite rows; the non-pivot metabolites are dependent -(conservation relations) and removed (`compression.py`–`:1950`). This is the exact-RREF +(conservation relations) and removed (`compression.py`–). This is the exact-RREF independence oracle of §3.5, but computed in Java — and note it marshals the stoichiometry through a **dense double** array, both memory-heavy on genome-scale models and lossy for large coefficients. @@ -1407,9 +1397,9 @@ each a decisive advantage on a genome-scale correctness/performance workload: installs. 2. **Native-crash fragility.** The bridge is defensive to a degree that itself signals the risk: eager JVM startup ordered before OpenBLAS threads (§3.11.1); `gc.disable` wrapped around *every* - JNI block (`efmtool_cmp_interface.py`–`:363`, `:404`–`:426`) because Python's garbage collector + JNI block (`efmtool_cmp_interface.py`–,–) because Python's garbage collector finalizing a JPype proxy mid-call causes Bus error / SIGSEGV; an `atexit` JVM-shutdown hook to dodge - a JPype teardown race (`:150`–`:158`). None of this can occur in a pure-Python engine. + a JPype teardown race. None of this can occur in a pure-Python engine. 3. **Big-integer safety at the interface.** efmtool's Java core is arbitrary-precision (`DefaultBig IntegerRationalMatrix`), so the *internal* arithmetic does not overflow. The hazard is at the **marshalling boundary**: the compression structure and bound rescaling are read back through @@ -1443,11 +1433,11 @@ byte-identical and a few divergences are worth knowing: - **GPR propagation is identical on the coupled step.** Both backends clear gene rules before merging and re-attach the AND-combined GPR from the saved AST snapshot in `compress_model_coupled` - (`compression.py`–`:2015`), and the parallel OR-combine is always the Python + (`compression.py`–), and the parallel OR-combine is always the Python `compress_model_parallel` (§3.9). So GPR handling does *not* diverge between backends. - **Protected reactions are honored only by the Python backend.** `compress_model` passes gene- controlled reactions as `protected_reactions` (`no_coupled_compress_reacs`, `compression.py`– - `:1925`) so they survive COMPRESS #1 un-merged and gene multiplicity is preserved for GPR + ) so they survive COMPRESS #1 un-merged and gene multiplicity is preserved for GPR integration (§3.4.2, [Ch 4](#ch4)). `compress_model_java` **ignores `protected_reactions`** — it reads only `suppressed_reactions`, which `compress_model` never populates on this path. On the Java backend those reactions can therefore be lumped in COMPRESS #1, a genuine semantic divergence in the gene-KO @@ -1459,11 +1449,11 @@ byte-identical and a few divergences are worth knowing: incorrectly" — the cautionary tale of closed issue #44 ([Ch 10](#ch10)). The two backends can thus disagree on which reactions a contradicting group costs you. - **Direction bookkeeping differs.** The Java path physically flips `ub ≤ 0` reactions (`*= -1`) and - negates their recorded stoich (`efmtool_cmp_interface.py`–`:415`, `:452`–`:455`); the Python + negates their recorded stoich (`efmtool_cmp_interface.py`–,–); the Python coupled backend carries sign inside the exact `ratios` (§3.4.3). Same flux space, different maps — which is fine because decompression ([Ch 9](#ch9)) consumes whichever map its backend produced. - **Bound rescaling precision.** Java rescales merged-reaction bounds by a **double** - (`efmtool_cmp_interface.py`–`:450`); the Python backend intersects bounds using exact rationals + (`efmtool_cmp_interface.py`–); the Python backend intersects bounds using exact rationals (§3.4.4). On well-scaled models this is invisible; on large-coefficient models it is another place the Java path can drift. @@ -1494,9 +1484,9 @@ Boolean logic. After extension, "gene *g* is knocked out" becomes the purely lin flux of pseudoreaction *g* to zero," and the MILP's existing reaction-knockout machinery handles it with no separate Boolean-logic layer. We then cover the reversible-reaction split that GPR extension forces (`extend_model_gpr` + the `reac_map` remap in `compute_strain_designs.py`), the -pre-pruning pass `reduce_gpr` (`networktools.py`) that shrinks the work, the delicate ordering of +pre-pruning pass `reduce_model_gprs` (`compute_strain_designs.py`) that shrinks the work, the delicate ordering of the two compression passes around extension (`compute_strain_designs.py`), and the sha256 name -truncation that only fires for Gurobi/GLPK. +sha256 name truncation, which is applied on every backend. ### 4.1 Why encode gene logic as flux structure at all @@ -1534,7 +1524,7 @@ logic. No gene binaries, no second logic layer: a gene knockout is literally a r knockout of the same kind the MILP already handles, so the entire dualization/`link_z` machinery ([Ch 6](#ch6), [Ch 7](#ch7)) applies unchanged. The price is a modest number of extra rows/columns in `S` (one pseudoreaction per surviving gene, plus one pseudo-metabolite/pseudoreaction per Boolean operator), which the second -compression pass (§4.5) then partly reabsorbs. The correctness guarantee that makes this legal is that +compression pass (§4.6) then partly reabsorbs. The correctness guarantee that makes this legal is that **the extension does not change the reachable flux space of the original reactions** (§4.3): all the new structure is "upstream plumbing" whose only effect, when a pseudoreaction is fixed to zero, is to force the guarded reactions to zero exactly when the Boolean rule says the enzyme is absent. @@ -1726,7 +1716,7 @@ reactions whose Boolean rule is now FALSE, forcing `v_r = 0`. That is the intend Two implementation details protect this invariant. First, the pseudoreactions are created **once** and memoized: `created_metabolites` (a set) and the `... not in model.metabolites` guards (e.g. -`networktools.py, 1065, 1094`) ensure a gene shared by many reactions gets a *single* `g_{id}` +`networktools.py`) ensure a gene shared by many reactions gets a *single* `g_{id}` source and metabolite, so all its reactions draw from the same tap — this is what makes a shared gene count once and couple all its reactions. Second, the `and`/`or` metabolite ids are built from the **sorted** child ids (`"_and_".join(sorted(...))`, `"_or_".join(sorted(...))`), so identical @@ -1791,68 +1781,30 @@ i.e. a term `v·(x_k)` becomes `Σ_n (v·w)·(x_n)` over the pieces `n` of `k`, split reversible reaction this turns `v·v_r` into `v·v_fwd − v·v_rev`, faithfully preserving the signed flux the module intended. Objectives (`INNER_OBJECTIVE`, `OUTER_OBJECTIVE`, `PROD_ID`) are single dicts and remapped the same way (`compute_strain_designs.py`). Because `reac_map` contains an entry -for *every* reaction (`{r.id: 1.0}` for the untouched ones, `networktools.py, 1149`), the loop +for *every* reaction (`{r.id: 1.0}` for the untouched ones, `networktools.py`), the loop can blindly remap every key without special-casing which reactions were split. -### 4.5 `reduce_gpr`: pruning before extension - -Extension cost scales with the number of surviving genes and Boolean operators: each gene adds a -pseudoreaction + metabolite, each operator a gadget. Many genes can be proven irrelevant *before* any -of that structure is built, which both shrinks `S` and removes useless binary candidates from the MILP. -`reduce_gpr(model, essential_reacs, gkis, gkos)` (`networktools.py`) does this pruning, returning a -trimmed `gkos` (gene-KO-cost dict); it runs just before `extend_model_gpr` -(`compute_strain_designs.py`). Its steps: - -1. **Blocked reactions lose their GPR** (`networktools.py`). Any reaction with bounds `(0,0)` - is dead anyway; its rule is cleared and genes that end up in no reaction are dropped. No point - encoding logic for a reaction that can never carry flux. - -2. **Protect genes that touch only essential reactions** (`networktools.py`). A gene whose - reaction set is a subset of `essential_reacs` (reactions that *must* stay operational — from the FVA - over PROTECT/desired modules, [Ch 5](#ch5)) can never be a useful KO: knocking it out could only threaten an - essential reaction. It is added to `protected_genes`. - -3. **Protect genes that are individually essential *to* an essential reaction** (`networktools.py`). - Using `is_gene_essential_to_reaction_ast`, which evaluates the reaction's GPR AST with that one gene - set to `False` and checks whether the whole rule collapses to `False`: if deleting the gene alone - would kill an essential reaction, the gene must be protected. (A gene inside an `or` of an essential - reaction is *not* caught here — deleting it leaves the reaction alive — so it stays knockable.) - -4. **Drop protected genes from the KO-cost dict** (`networktools.py`): `[gkos.pop(pg.id) …]` — they - are no longer intervention candidates. - -5. **Everything the user did not list as knockable is also protected** (`networktools.py`): genes - whose id *and* name are absent from `gkos` cannot be knocked out, so they are protected too. - -6. **Genes with knock-in costs are un-protected** (`networktools.py`): a gene in `gkis` is a - *target* (it can be added), so it is removed from the protected set even if the above rules caught it. - -7. **Simplify each GPR rule with protected genes pinned TRUE** (`networktools.py`). - `simplify_gpr_ast` walks the AST setting every protected gene to `True` and applies Boolean - simplification (`apply_gene_protection_to_ast`, `networktools.py`): `True and X → X`, - `True or X → True`, plus absorption (`A or (A and B) → A`, `networktools.py`). If the rule - collapses to `True`, the reaction is no longer knockable-by-gene and its rule is cleared (so it gets - no gadget at all); otherwise the simplified, *smaller* rule replaces the original — fewer operators, - hence fewer gadgets at extension. - -8. **Remove obsolete and protected genes** from the model (`networktools.py`), so - `extend_model_gpr` never sees them. - -The net effect: `extend_model_gpr` is handed a model whose GPR rules mention only genes that are (a) -user-declared knockable or knock-in-able and (b) capable of affecting a non-essential reaction, with -the rules already Boolean-minimized. On genome-scale models this removes a large fraction of genes and -operators before the expensive structure is built. - -**The id-vs-name subtlety.** Genes can be referenced by *either* their id or their (human-readable) -name, and models are inconsistent about which the user supplies in `gkos`/`gkis`. `reduce_gpr` therefore -checks **both**: the protection rule at `networktools.py` protects a gene only if *neither* -`g.id in gkos` *nor* `g.name in gkos`, and the KI un-protection at `networktools.py` collects -`g.id for g in model.genes if (g.id in gkis) or (g.name in gkis)`. Note the asymmetry that this matching -introduces downstream: `extend_model_gpr` names each gene pseudoreaction by id *or* name depending on -the global `has_gene_names` flag (`use_names`, decided at `compute_strain_designs.py` and passed in), -so the id-vs-name choice must stay consistent between the cost dicts and the pseudoreaction ids or the -later cost lookup silently misses (see [Ch 10](#ch10) for the fragility this creates). `reduce_gpr` hedges by -accepting both spellings; the pseudoreaction naming commits to one. +### 4.5 `reduce_model_gprs` and `simplify_model_gprs` + +There are two distinct GPR reductions before extension: + +1. `reduce_model_gprs` in `compute_strain_designs.py` is pipeline-only. It needs the desired-region + essential reactions and the gene KO/KI cost dictionaries. It clears rules on blocked reactions, + protects genes that cannot be valid targets, substitutes protected genes with `True`, removes + obsolete genes and returns the reduced gene-KO cost dictionary. +2. `simplify_model_gprs` in `compression.py` is a model-level Boolean simplifier. It does not know + intervention costs or essential reactions; it only replaces each monotone GPR with a + Boolean-equivalent, leaf-minimized expression. + +Keeping these jobs separate is important. The first is meaningful only inside strain-design +preprocessing, while the second is useful to standalone compression and must also run when +`compress=False`. On the compressed pipeline they run consecutively before `extend_model_gpr`; +re-running the Boolean simplifier is cheap and idempotent. + +Genes may be referenced by ID or name in the intervention dictionaries. The reduction checks both, +while `extend_model_gpr(use_names=...)` commits to one namespace for pseudo-reaction identifiers. +That namespace choice must remain consistent with the compressed cost dictionaries. + ### 4.6 The two-compression-pass boundary and why regulatory genes are exempt from pass #1 @@ -1873,7 +1825,7 @@ compression too. **Why `propagate_gpr` differs.** In pass #1 the metabolic reactions still carry Boolean GPR *strings*. When two reactions are merged, their rules must be combined correctly — an AND-merge for flux-coupled -reactions, an OR-merge for parallel ones (`compression.py, 2040`, the `_combine_gpr_and/or` helpers, +reactions, an OR-merge for parallel ones (`compression.py`, the `_combine_gprs` helper, [Ch 3](#ch3)) — so that after extension the merged reaction's rule still reflects both originals. Hence `propagate_gpr=True`. In pass #2 the rules have *already been consumed* by `extend_model_gpr` (converted to flux structure) and the reactions' `gene_reaction_rule` strings are no longer the source of truth — @@ -1896,7 +1848,7 @@ The remedy is to **exempt exactly the reactions controlled by a deferred-regulat in pass #1. The block scans each deferred regulatory constraint string for tokens matching a gene id or name (`compute_strain_designs.py`), collects that gene's reactions into `no_coupled_compress_reacs`, and passes them to `compress_model` so they are *not* coupled-merged; it -also adds them to `no_par_compress_reacs` (`:353`) so they are not parallel-merged and their **names stay +also adds them to `no_par_compress_reacs` so they are not parallel-merged and their **names stay stable** across the two passes (the pass-#1 exemption matches them by name, so a rename would break the matching). These same reactions *do* merge safely in **pass #2**, once `extend_model_gpr` has created the `g_gene` metabolite and `extend_model_regulatory` has hung the bound on the gene pseudoreaction — at that @@ -1907,510 +1859,124 @@ only *regulatory* genes, whose bound is a finite scaled quantity, are sensitive rescaling. (This exemption logic is the fix for closed issue #44's class of bound-scaling bugs; see [Ch 3](#ch3) for the compression bound-intersection mechanics and [Ch 10](#ch10) for the cautionary history.) -### 4.7 Name truncation (sha256), Gurobi/GLPK only +### 4.7 Deterministic name truncation -Extension generates pseudo-metabolite and pseudoreaction ids by *concatenating* child ids with `_and_` -/ `_or_` separators. Nested rules over long gene ids can produce names hundreds of characters long. -**Gurobi and GLPK impose a 255-character limit on variable/constraint names**; CPLEX and SCIP do not. -The code sets `MAX_NAME_LEN = 230` (`networktools.py`) and, *only when the active solver is in -`{GUROBI, GLPK}`* (checked at every id-construction site, e.g. `networktools.py, 1043, 1059, 1072, -1088, 1103, 1144`), truncates: +GPR gadgets construct identifiers by combining gene and child-metabolite names. Any generated +identifier longer than `MAX_NAME_LEN` is shortened for every solver, not only Gurobi or GLPK. The +short form keeps a readable prefix and appends the first 20 hexadecimal digits of a SHA-256 digest. +Applying one deterministic rule across all backends keeps cost lookup, module remapping, +decompression and cross-solver comparisons in the same identifier space. -```python -def truncate(id): - h = hashlib.sha256(id.encode()).hexdigest()[:20] - return id[0:MAX_NAME_LEN - 21] + "_" + h -``` - -i.e. it keeps the first `209` characters and appends `_` + a 20-hex-char sha256 digest of the full id, -yielding a ≤230-char name. The digest suffix preserves uniqueness (two long ids sharing a 209-char -prefix still differ in hash) so distinct pseudo-metabolites do not accidentally collide after -truncation. A `warning_name_too_long` message (`networktools.py`) is logged once per truncated -name, suggesting the user switch to CPLEX or simplify gene names to avoid it. - -Two properties matter for a maintainer. First, **truncation is solver-conditional**: the *same model* -produces different pseudoreaction ids under Gurobi/GLPK than under CPLEX/SCIP. Any code that matches -these ids by string (cost-dict lookups, module remapping, decompression) must therefore see the *same* -truncated names — which is why the id is truncated at the single point of creation and reused, not -re-derived elsewhere. Second, the sha256 rewrite is a **known fragility, adjacent to open issue #43**: -because the truncated name is not human-meaningful and because the truncation depends on solver -identity, a mismatch between where a name is generated and where it is later looked up can silently drop -a gene knockout from the reported solution. The mechanism and the concrete failure are owned by **[Ch 10](#ch10)**; -here we only flag that the `{GUROBI, GLPK}`-gated sha256 truncation is the code path involved. +The warning recommends simplifying GPR rules or source identifiers. Switching solver does not change +the truncation policy. (ch5)= ## 5. FVA in preprocessing -Flux Variability Analysis (FVA) — the pair of LPs that, for every reaction *j*, compute -`min v_j` and `max v_j` over the steady-state polytope `{v : Sv = 0, lb ≤ v ≤ ub}` (see -[Ch 2](#ch2) for the LP formulation) — appears **three times** in `compute_strain_designs`'s -preprocessing, at three different points in the pipeline, on three different versions of the -model, each time answering a different question and feeding a different downstream consumer. -None of the three is "just diagnostics": each one *removes work from the MILP* that the solver -would otherwise have to do, and one of them (the second) is the single largest slice of -genome-scale wall-time. This chapter dissects all three, then the accelerated FVA engine -(`speedy_fva`) that all of them call, and closes by explaining why FVA #2 costs ~117 s. +FVA-related work now occurs at several deliberately different points. Counting only calls named +`fva` is misleading because the pre-compression pass is a specialized sign-only implementation and +the final two jobs can be folded. -The three uses, at a glance: +### 5.1 Reversibility pre-tightening before COMPRESS #1 -| # | Call site (`compute_strain_designs.py`) | Model state | Scope | Question answered | Consumer | -|---|------------------------------------------|-------------|-------|-------------------|----------| -| 1 | ~L373–381 | after COMPRESS #1, **pre-GPR** | whole model | Which reactions are *essential* for a PROTECT/desired behaviour? | drop from `ko_cost`; feed `reduce_gpr` | -| 2 | `bound_blocked_or_irrevers_fva`, ~L450 (→ `networktools.py`) | after GPR extension + COMPRESS #2 | whole model | Which bounds never bind? Which reactions are blocked/irreversible? | rewrite model bounds → shrink/condition the MILP | -| 3 | ~L460–491 | after COMPRESS #2 | **knockable only** (`reaction_list`) | Which knockable reactions are essential per module? Which are size-1 cut sets? | drop essentials + size-1 MCS from `ko_cost`; re-inject MCS at decompression | +When compression is enabled, `fast_reversibility` runs on the metabolic model before COMPRESS #1. It +asks only whether each reaction can carry positive and negative flux; magnitudes are not retained. +Directions classified unavailable are fixed to zero before compression. This is what lets +one-directional reactions form larger coupled groups and prevents unnecessary forward/reverse GPR +splits. -All three ultimately dispatch to `fva` in `lptools.py`, which is a thin wrapper that -immediately calls `speedy_fva` (`lptools.py`). The legacy brute-force implementation -`fva_legacy` (`lptools.py`) is retained only as a debugging fallback. +The implementation combines: -### 5.1 The essentiality test — geometry of `min(abs(range)) > 1e-10 and prod(sign(range)) > 0` +0. a zero-objective feasibility preflight that fails loudly on an empty polytope and seeds every + incumbent from its flux vector, so no later warm-started optimum can contradict an + already-witnessed achievable flux; +1. a sound structural producer/consumer and dead-end sweep; +2. one temporary coupled compression; +3. warm-started per-direction LPs on the compressed model; +4. co-option scans that use a feasible optimum to witness directions of other reactions; and +5. exact expansion of the sign results through the compression map. -Both FVA #1 and FVA #3 classify a reaction as *essential* (for a given module's constraint -set) using the identical predicate, at `compute_strain_designs.py` and again at `:465`: +The scan threshold is only a shortcut for a clear *witness* of nonzero flux. It must not be interpreted +as proof that smaller fluxes are zero. Likewise, a nonoptimal solve is numerical uncertainty, not a +blockedness certificate. These distinctions are important because the result changes model bounds. -```python -if np.min(abs(limits)) > 1e-10 and np.prod(np.sign(limits)) > 0: # find essential - essential_reacs.add(reac_id) -``` - -Here `limits` is the two-element vector `[v_min, v_max]` returned by FVA for reaction *j*, -i.e. the endpoints of the attainable flux interval `[v_min^j, v_max^j]` under that module's -constraints. Read the predicate geometrically: - -- **`np.prod(np.sign(limits)) > 0`** — `sign(v_min)·sign(v_max) > 0` — is true iff `v_min` - and `v_max` have the **same, nonzero sign**. That is exactly the statement *the interval - `[v_min, v_max]` does not contain 0*. (If either endpoint were 0 the product would be 0; - if the interval straddled 0 the signs would differ and the product would be negative.) -- **`np.min(abs(limits)) > 1e-10`** — `min(|v_min|, |v_max|) > 10⁻¹⁰` — is the *numerical - guard* that the endpoint closest to zero is a strict, non-noise distance away from it, so - the "does not contain 0" conclusion is not an artifact of solver tolerance. - -Together they assert: **every feasible flux state that satisfies the module's constraints -routes a strictly nonzero, sign-definite flux through reaction *j*.** Geometrically, the flux -polytope of that module lies entirely on one side of the hyperplane `v_j = 0` and does not -touch it. Consequently, the constraint `v_j = 0` (which is precisely what a knockout imposes) -is *inconsistent* with the module: **knocking out *j* makes the module infeasible.** - -Why that matters depends on the module type, and this is the whole point of running FVA #1/#3 -separately per module (`for m in sd_modules:`): - -- If the module is **PROTECT/desired** (a behaviour that must remain *possible*), a reaction - essential to it can never appear in a valid design — knocking it out would violate the - PROTECT requirement. Such a reaction is therefore useless as a knockout candidate and is - stripped from `ko_cost` (removing its binary `z_j` from the MILP entirely). -- If the module is **SUPPRESS** (a behaviour that must be made *impossible*), a reaction - essential to it is, by itself, a valid intervention: deleting it kills the behaviour. That - is the size-1 MCS observation exploited by FVA #3 (§5.4). - -A tiny worked example. Two reactions, `R1: A→B`, `R2: B→C`, sink `EX_C`, with a PROTECT -module requiring `EX_C ≥ 1`. FVA over `{Sv=0, v≥0, EX_C≥1}` yields `v_R1 ∈ [1, 1000]`, -`v_R2 ∈ [1, 1000]`: both intervals sit strictly above 0, `sign(1)·sign(1000)=+1`, and -`min(|1|,|1000|)=1 > 10⁻¹⁰`. Both are flagged essential — correctly, since either KO drops -`EX_C` to 0 and breaks the PROTECT. - -### 5.2 FVA #1 — essential reactions in PROTECT/desired modules (pre-GPR) - -FVA #1 runs immediately after COMPRESS #1 and *before* GPR integration -(`compute_strain_designs.py`), so it sees a purely metabolic, compressed network with -no gene pseudoreactions yet (see [Ch 4](#ch4) for the COMPRESS #1/GPR boundary). It iterates only over -non-SUPPRESS modules: +### 5.2 Desired-region essentiality before GPR extension -```python -for m in sd_modules: - if m[MODULE_TYPE] != SUPPRESS: # essentiality only meaningful for desired / opt-/robustknock - flux_limits = fva(cmp_model, solver=..., constraints=m[CONSTRAINTS], compress=False) - for (reac_id, limits) in flux_limits.iterrows(): - if np.min(abs(limits)) > 1e-10 and np.prod(np.sign(limits)) > 0: - essential_reacs.add(reac_id) -[cmp_ko_cost.pop(er) for er in essential_reacs if er in cmp_ko_cost] -``` - -**Rationale (why drop from `ko_cost`).** As argued in §5.1, a reaction essential for a -required (PROTECT/desired) behaviour can *never* be part of any feasible design — its knockout -would violate a PROTECT constraint that the MILP is required to keep feasible. Every candidate -design that includes it is infeasible *a priori*. Popping it from `cmp_ko_cost` -removes its binary variable `z_j` from the intervention set the MILP will branch over: the -solver never even considers it, and no infeasible node is generated to reject it. This is a -pure model-size reduction with zero effect on the solution set. - -**Second consumer: `reduce_gpr`.** The `essential_reacs` set computed here is passed straight -into GPR reduction (`compute_strain_designs.py`): +After COMPRESS #1, every non-SUPPRESS module is analyzed with its constraints. A reaction is treated +as essential when its FVA interval stays strictly on one side of zero: ```python -uncmp_gko_cost = reduce_gpr(cmp_model, essential_reacs, uncmp_gki_cost, uncmp_gko_cost) +np.min(abs(limits)) > 1e-10 and np.prod(np.sign(limits)) > 0 ``` -`reduce_gpr` (`networktools.py`) simplifies the Boolean gene–protein–reaction rules before -they are compiled into flux structure ([Ch 4](#ch4)). Knowing which reactions are essential lets it -also drop the *genes* that only ever control essential reactions from the knockable gene set: -if a reaction can never be knocked out, a gene whose only role is to (be required to) enable -that reaction is likewise non-knockable, and pruning it shrinks both the GPR encoding and the -gene KO cost dictionary. Thus one FVA pass feeds two reductions — reaction-level and, through -`reduce_gpr`, gene-level. +Such a reaction cannot be knocked out while preserving the desired region, so it is removed from the +reaction KO candidates. The same set is supplied to `reduce_model_gprs`, allowing genes that can only +damage required reactions to be removed before their gadgets are built. A SUPPRESS region is not used +for this early protection: reactions essential to the undesired behavior may be exactly the desired +single-reaction cut sets. -**Why `compress=False` here.** The model is *already* compressed (COMPRESS #1 just ran), so -`speedy_fva`'s own internal coupled-compression pass is switched off to avoid re-compressing an -already-compressed, rational-bound network. FVA #1 is comparatively cheap: it runs on the small -pre-GPR metabolic network and typically for a single PROTECT module. +### 5.3 Final model-bound FVA -### 5.3 FVA #2 — `bound_blocked_or_irrevers_fva`: relaxing non-binding bounds +After GPR extension and COMPRESS #2, `bound_blocked_or_irrevers_fva` computes ranges and mutates the +stored cobra bounds: -FVA #2 runs *after* GPR extension and COMPRESS #2, so that **all** reactions — including the -gene pseudoreactions added by `extend_model_gpr` — are processed -(`compute_strain_designs.py`): +- a lower or upper model bound that is not reached is relaxed to `-inf` or `+inf`; +- a direction whose optimum is zero is pinned to zero according to the solver-specific numerical + policy; and +- the DataFrame is returned so the same result can serve downstream consumers. -```python -bound_blocked_or_irrevers_fva(cmp_model, solver=kwargs[SOLVER], compress=False) -``` +The call is scoped by `_fva_scope`. Reactions already at `(0,+inf)` are omitted because their only +possible additional tightening is blockedness; retaining `(0,+inf)` does not enlarge the actual flux +space when stoichiometry already forces zero, and a blocked target cannot occur in a minimal cut set. -Its body (`networktools.py`) runs one whole-model FVA and then rewrites each -reaction's *stored* bounds (`r._lower_bound` / `r._upper_bound` directly, to make the change -permanent and bypass cobra's optlang synchronisation) according to **four independent -branches**. With CPLEX/Gurobi the tolerance `tol` is `0.0`; with SCIP/GLPK it is `1e-10` -(`networktools.py`). Let `[v_min, v_max]` be the FVA interval and `[lb, ub]` the -current bounds. +### 5.4 Per-module FVA and size-1 MCS -```python -if r.lower_bound < 0.0 and limits.minimum - tol > r.lower_bound: # (A) redundant lb → −inf - r._lower_bound = -np.inf ; n_lb_to_inf += 1 -if limits.minimum >= tol: # (B) min ≥ 0 → lb = 0 - r._lower_bound = max([0.0, r._lower_bound]) ; n_tightened_zero += 1 -if r.upper_bound > 0.0 and limits.maximum + tol < r.upper_bound: # (C) redundant ub → +inf - r._upper_bound = np.inf ; n_ub_to_inf += 1 -if limits.maximum <= -tol: # (D) max ≤ 0 → ub = 0 - r._upper_bound = min([0.0, r._upper_bound]) ; n_tightened_zero += 1 -``` - -Decoding the four branches: - -- **(A) redundant lower bound → −∞.** The reaction *can* go negative (`lb < 0`), yet the - achievable minimum flux `v_min` is strictly greater than `lb`. The lower bound therefore - never binds — the network's stoichiometry constrains `v_j` more tightly than the box bound - does. Relaxing `lb` to `−∞` discards a constraint that is provably slack everywhere. -- **(B) min ≥ 0 → lb = 0.** FVA proves `v_j` cannot be negative under steady state. The - reaction is effectively **irreversible in the forward direction**, so its lower bound is - pinned at 0 (`max(0, lb)`). Note the interaction with (A): a reaction with `lb = −1000` but - `v_min = 2` first has `lb` set to `−∞` by (A), then *overwritten* to `0` by (B) because the - branches are evaluated in sequence on the same reaction. The net effect is `lb = 0` - (irreversible), not `−∞`. Detecting irreversibility this way lets the MILP omit the negative - half-space entirely. -- **(C) redundant upper bound → +∞.** Symmetric to (A): `ub > 0` but the achievable maximum - `v_max` is strictly below `ub`, so the upper box bound never binds and is relaxed to `+∞`. -- **(D) max ≤ 0 → ub = 0.** Symmetric to (B): the reaction cannot carry positive flux, so it - is irreversible in the backward direction and `ub` is pinned at 0. - -A reaction that is fully **blocked** (`v_min = v_max = 0`) triggers (B) *and* (D): `lb` and -`ub` are both pinned to 0, freezing it out of every flux state. - -**Decoding the real log line.** `bound_blocked_or_irrevers_fva` emits, on iML1515 after GPR -extension (`networktools.py`): - -``` -FVA bounds: 4 lb→inf, 1825 ub→inf, 2258 tightened to 0, 2150 stayed finite -``` - -- `4 lb→inf` = branch (A) fired 4 times: only 4 reactions had a genuinely reversible, slack - lower bound. (Almost all reactions in a curated model are already forward-irreversible, so - few have a slack negative lower bound to relax.) -- `1825 ub→inf` = branch (C) fired 1825 times: for 1825 reactions the upper bound was slack - and is relaxed to `+∞`. This is the large one — most reactions' nominal upper bound (e.g. - the default 1000) never binds; the true maximum is limited by network stoichiometry. -- `2258 tightened to 0` = **the combined count of branches (B) and (D)** — the same counter - `n_tightened_zero` is incremented in both (`networktools.py` and `:1621`). It therefore - aggregates "lower bound pinned to 0 (forward-irreversible)" and "upper bound pinned to 0 - (backward-irreversible / blocked)". It is *not* a count of distinct reactions: a single - reaction that triggers both (B) and (D) — i.e. a blocked reaction — is counted twice, and a - reaction that triggers (A) then (B) contributes to both `n_lb_to_inf` and `n_tightened_zero`. -- `2150 stayed finite` is computed independently at `networktools.py` as the number - of reactions with **at least one finite bound after all rewrites**: - `sum(1 for r in model.reactions if not isinf(r.lower_bound) or not isinf(r.upper_bound))`. - These are the reactions that were *not* fully relaxed to `(−∞, +∞)`. - -Because the four counters overlap (a reaction can increment several), they do **not** sum to -the reaction count; only "stayed finite" is a clean per-reaction tally. This subtlety is easy -to misread as an inconsistency — it is intentional (each counter reports how often a *branch* -fired), not a bug. - -#### Why relaxing a provably non-binding bound to ±∞ shrinks and conditions the MILP - -This FVA is not cosmetic — it directly determines the size and numerical quality of the MILP -built next (`SDMILP`, [Ch 6](#ch6)–7). The mechanism has two prongs. - -**(1) Only genuinely finite (binding) bounds become knockable constraints.** In the MILP, a -reaction knockout is enforced by tying its binary `z_j` to the reaction's flux-bound rows so -that `z_j = 1 ⇒ v_j = 0`; and in the dualized SUPPRESS block every finite reaction bound -becomes a *dual variable* with its own row and its own coupling to `z` (see [Ch 6](#ch6) for the -Farkas dualization and [Ch 7](#ch7) for `link_z`). A bound relaxed to `±∞` is, by definition, *no -constraint at all*: it contributes no row to the primal, hence no dual variable to the -dualized problem, and nothing for `z` to switch on that side. So every `lb→−∞` (branch A) and -`ub→+∞` (branch C) *deletes* a constraint row and, in the dual, a variable. On the numbers -above that is `4 + 1825 = 1829` bound rows removed. Conversely, the 2150 reactions that -"stayed finite" are exactly the ones whose remaining binding bound *does* need an -indicator/big-M linkage in the MILP — the relaxation has narrowed the set of reactions that -require this machinery to the ones that genuinely constrain flux. - -**(2) The remaining big-Ms get tighter.** Where a knockout linkage is realised as a **big-M** -constraint (PROTECT's finite-flux primal rows; the big-M vs indicator fork is emergent from -bound structure, [Ch 7](#ch7)), the constant `M` must be a valid over-estimate of `|v_j|`. `link_z` -derives each `M` from a bounding LP over the reaction's flux range. By replacing the loose -nominal box bounds (e.g. `±1000`) with (a) the *tight, FVA-proved* range or (b) an honest -`±∞` where the bound is slack, FVA #2 feeds `link_z` sharper information: reactions with a -proved finite range get a smaller, tighter `M` (better LP relaxation, faster branch-and-bound), -and reactions whose bound is genuinely non-binding are steered toward the **indicator** -formulation (which has no `M` at all and yields a tighter relaxation) rather than a -meaningless huge `M`. Both outcomes improve the MILP: fewer rows, tighter continuous -relaxation, better conditioning. (See [Ch 7](#ch7) for the exact `self.M`/bounding-LP fork.) - -The important invariant: because branches (A) and (C) only relax bounds that FVA has *proved* -never bind, and (B)/(D) only pin bounds the reaction can provably never cross, **the feasible -flux set is unchanged.** No design is added or lost; only the *description* of the polytope is -made leaner and better-conditioned. - -### 5.4 FVA #3 — knockable-scoped essentials and size-1 MCS extraction - -FVA #3 (`compute_strain_designs.py`) runs on the final, fully GPR-extended and -COMPRESS #2-compressed model, but — unlike #1 and #2 — it is **scoped to knockable reactions -only** via `speedy_fva`'s `reaction_list` kwarg: +For the general path, each final module receives an FVA constrained to its region and scoped to the +knockable reaction IDs. The returned ranges have two consumers: -```python -knockable_ids = list(set(cmp_ko_cost.keys()) | set(cmp_ki_cost.keys())) -for m in sd_modules: - flux_limits = fva(cmp_model, solver=..., constraints=m[CONSTRAINTS], - compress=False, reaction_list=knockable_ids) - ... - if m[MODULE_TYPE] != SUPPRESS: - essential_reacs.update(essentials_in_module) # essential for a PROTECT/desired module - else: - suppress_essential.update(essentials_in_module) # essential for the SUPPRESS module -``` +- essentiality classification and classical size-1 MCS extraction; and +- `SDProblem._module_bound_override`, which carries only blockedness or sign into that module's + continuous block. -Essentiality of a *non-knockable* reaction is irrelevant here — the MILP will never toggle its -`z` — so restricting FVA to `knockable_ids` avoids computing `2n` LPs and instead computes only -`2·|knockable|`. The same essentiality predicate from §5.1 is applied, but now the results are -**split by module type** into two sets: `essential_reacs` (essential for some PROTECT/desired -module) and `suppress_essential` (essential for the SUPPRESS module). +The override is not written to the shared cobra model. This is essential for multi-module problems: +a reaction can be one-sided or blocked in one module and unrestricted in another. -**Size-1 MCS: the core observation.** A Minimal Cut Set is a smallest set of knockouts that -makes the SUPPRESS behaviour infeasible while keeping PROTECT feasible ([Ch 1](#ch1)). A reaction that -is **essential for the SUPPRESS behaviour but NOT essential for any PROTECT behaviour** is, -all by itself, a valid cut set of size one: deleting it makes SUPPRESS infeasible (essential ⇒ -`v_j = 0` breaks it, §5.1), and — because it is *not* PROTECT-essential — deleting it leaves -PROTECT feasible. This is computed by a set difference -(`compute_strain_designs.py`): +For one classical SUPPRESS plus any PROTECT modules, a reaction essential to SUPPRESS but not to any +PROTECT is a size-1 MCS. It is removed from the KO search and re-injected during decompression. +Reactions essential to both undesired and desired regions are non-targetable. -```python -is_classical_mcs = (len([m for m in sd_modules if m[MODULE_TYPE] == SUPPRESS]) == 1 and - all(m[MODULE_TYPE] == PROTECT for m in [... non-SUPPRESS ...])) -if is_classical_mcs and suppress_essential: - size1_mcs = suppress_essential - essential_reacs # SUPPRESS-essential, not PROTECT-essential - size1_mcs_knockable = {r for r in size1_mcs if r in cmp_ko_cost} - if size1_mcs_knockable: - cmp_size1_mcs = [{r: -1} for r in size1_mcs_knockable] - both_essential = suppress_essential & essential_reacs # essential for BOTH → non-knockable - essential_reacs.update(both_essential) - for r in size1_mcs_knockable: - cmp_ko_cost.pop(r, None) # remove from KO candidates -``` - -**The `is_classical_mcs` guard.** The size-1-MCS shortcut is *only* valid for a classical MCS -problem: **exactly one SUPPRESS module and every remaining module a PROTECT** -(`compute_strain_designs.py`). The guard exists because the "essential-for-SUPPRESS ⇒ -valid single cut" argument relies on there being a single, well-defined behaviour to suppress -and only feasibility-preservation (not optimization) requirements to respect. In bilevel -problems (OptKnock/RobustKnock/OptCouple, which carry inner/outer objectives) or multi-SUPPRESS -problems, a reaction that is SUPPRESS-essential is *not* guaranteed to be a self-contained -minimal intervention — the objective coupling or a second SUPPRESS can make the "singleton" -either non-minimal or insufficient — so the shortcut is disabled and those reactions flow into -the ordinary MILP. - -**Why pull size-1 MCS out of `ko_cost`.** Once a reaction *r* is known to be a size-1 cut set, -any larger design that *contains* *r* is **non-minimal** — it is a superset of the already-known -minimal cut `{r}`. Leaving *r*'s binary `z_r` in the MILP would invite the solver to enumerate -exactly those non-minimal supersets, wasting branch-and-bound effort and (in POPULATE mode) -polluting the solution pool with dominated designs that would only be filtered out later. So -each such *r* is `pop`ped from `cmp_ko_cost`, deleting `z_r` from the MILP. The -size-1 cuts themselves are stashed in `cmp_size1_mcs` as `[{r: -1}]` entries (the `-1` encodes -"knock this reaction out") and are **re-injected as standalone solutions at decompression** -(`_decompress_solutions`, [Ch 9](#ch9)), so they still appear in the final result set — they are simply -solved by inspection instead of by the MILP. - -Two guard details worth noting: - -- The filter `size1_mcs_knockable = {r for r in size1_mcs if r in cmp_ko_cost}` - restricts extraction to reactions that are *pure KO candidates*. Reactions carrying a KI or - regulatory intervention are left in place (comment at `:486–489`), because they may still - participate in non-KO solutions that the singleton-KO shortcut does not represent. -- `both_essential = suppress_essential & essential_reacs`: a reaction essential for - BOTH the SUPPRESS and a PROTECT behaviour cannot be knocked out at all (it would break - PROTECT), and is therefore folded into `essential_reacs` and removed from `ko_cost` by the - final sweep at `compute_strain_designs.py`. - -### 5.5 The `speedy_fva` acceleration engine - -Every FVA above calls `fva` → `speedy_fva` (`speedy_fva.py`). Understanding its algorithm -is essential because it is where the wall-time is spent, and its behaviour depends sharply on -the `reaction_list` scoping and `compress` flags the three call sites pass. - -The naive FVA (`fva_legacy`, `lptools.py`) solves **`2n` independent LPs**: for each of the -`n` reactions it sets objective `+e_j` and `−e_j` and solves to get `v_min^j` and `v_max^j`. -`speedy_fva` produces the identical result but replaces most of those `2n` solves with a small -number of *global scan LPs* whose single optimal vertex simultaneously resolves the min or max -of many reactions at once. It is a **two-phase** algorithm. - -#### Bookkeeping and the "resolved" mask - -`speedy_fva` maintains, for the `n` reactions, boolean masks `res_max`, `res_min` and -incumbent vectors `incumbent_max`, `incumbent_min` (`speedy_fva.py`). A reaction's max -(resp. min) is "resolved" when its true `v_max` (resp. `v_min`) is known. Three cheap -pre-resolutions run before any LP: - -- **Fixed reactions** (`|ub − lb| < 10⁻¹²`): `v_min = lb`, `v_max = ub` with no - LP. -- **`reaction_list` scoping**: every reaction *not* in the requested list is - marked resolved with `NaN` incumbents. This is how FVA #3's `reaction_list=knockable_ids` - collapses the problem — non-knockable reactions are simply never scanned or solved, and come - back as `NaN` in the returned DataFrame. -- **`v = 0` feasibility shortcut**: if `0` is a feasible flux vector — which - holds when no lower bound is strictly positive, no upper bound strictly negative, and there - are no extra constraints (`not np.any(lb > tol) and not np.any(ub < -tol) and not - has_constraints`) — then for every reaction whose `lb = 0`, the minimum is provably `0` - (it cannot go below `lb=0`, and `0` is attainable), and symmetrically every reaction with - `ub = 0` has maximum `0`. These are resolved for free, no LP. This single check typically - clears a large fraction of an irreversible-heavy genome-scale model's bounds. - -#### Phase 1 — global scan LPs - -**(1b) The `min Σ|x|` scan LP.** The first real LP minimizes the total absolute flux -`Σ_j |v_j|` subject to `Sv = 0`, the extra constraints, and the bounds (`_build_abssum_lp`, -`speedy_fva.py`). Absolute values are linearized by **variable splitting**: reactions are -classified as forward-only (`lb ≥ 0`, so `|v_j| = v_j`, objective coeff `+1`), backward-only -(`ub ≤ 0`, so `|v_j| = −v_j`, coeff `−1`), or truly reversible (`lb < 0 < ub`). For each -reversible reaction the variable is split `v_j = p_j − n_j` with `p_j, n_j ≥ 0` and an -auxiliary equality row `v_j − p_j + n_j = 0`, and both `p_j` and `n_j` carry objective coeff -`+1` so the objective equals `p_j + n_j = |v_j|` at optimum (`speedy_fva.py`). -Infinite bounds are clamped to `±BIG (=1000)` purely so the *push* objective is bounded; this -does not alter feasibility. - -The optimal vertex of this LP is the flux state with the least total flux. Its virtue is that -it drives most reactions **to zero**: any reaction sitting exactly at a `lb = 0` or `ub = 0` -bound at this vertex is resolved by the vectorized *bound scan* `_bound_scan` -(`speedy_fva.py`), which marks `res_max`/`res_min` wherever `|x_j − ub_j| < 10⁻⁹` or -`|x_j − lb_j| < 10⁻⁹`. In one LP this resolves the min/max of every reaction that touches a -zero bound at the min-flux vertex. Simultaneously the vertex's flux values update the -incumbents (`np.maximum(incumbent_max, x_scan)`, `np.minimum(incumbent_min, x_scan)`, -): even a reaction not *proved* extreme has its known range widened by this -witness — **co-optimization**, one LP contributing evidence about `n` reactions at once. - -**(1c) Iterative push-to-bounds with warm-started dual simplex.** The remaining unresolved -maxima are attacked collectively: a single objective `c` puts `−1` on *every* reaction whose -max is still unresolved (`speedy_fva.py`) and the LP is re-solved — pushing all of -them toward their upper bounds at once. Whatever lands on its `ub` is resolved by `_bound_scan`; -incumbents update for the rest. The symmetric objective with `+1` on unresolved-min reactions - pushes toward lower bounds. This alternation repeats -(`while True: ... if resolved_this_round < 5: break`) until a round resolves -fewer than 5 new bounds — i.e. until the cheap global pushes stop paying off. - -The critical performance ingredient is that the scan LP object is **reused** across all these -re-solves — only the objective vector changes (`scan_lp.set_objective(...)`), never the -constraint matrix — and the solver is set to **dual simplex** (`set_lp_method(LP_METHOD_DUAL)`, -). Changing only the objective keeps the previous basis *primal*-feasible but -dual-infeasible, which is exactly the situation dual simplex resumes from cheaply: each -re-optimization is a warm-started handful of pivots rather than a cold solve. Dozens of push -LPs therefore cost a small multiple of one LP. - -#### Phase 2 — individual LPs for the residual - -Whatever Phase 1 could not resolve (`n_remaining = 2n − n_done`) is finished with -individual per-objective LPs, dispatched one of two ways (`speedy_fva.py`): - -- **Parallel** (`n_remaining ≥ 1000 and threads > 1`): the unresolved objective - indices (even = max, odd = min, via `idx2c`) are farmed to an `SDPool` of workers, each - holding its own persistent LP (`fva_worker_init`/`fva_worker_compute`), with a NaN-retry - loop for any solve that returns NaN. -- **Sequential** (`0 < n_remaining < 1000`, or `threads == 1`): a single warm-started - LP is stepped through the residual objectives with `set_objective_idx`, periodically rebuilt - every 200 solves to limit warm-start basis degeneration. Each solved vertex is - *also* run through `_bound_scan` and the incumbent update, so even in Phase 2 - one LP can opportunistically resolve *other* pending reactions — the same co-optimization - trick. A correctness guard detects when a warm-started optimum is *worse* than - the incumbent (a sign of a degenerate/stale basis) and rebuilds the LP and re-solves from - scratch for that objective. - -`threads` auto-selects to `Configuration.processes` only when the model has `≥ 1000` -reactions, else `1`. Note the asymmetry that drives §5.6: the parallel path is -gated on **`n_remaining ≥ 1000`**, i.e. on how many objectives *survive Phase 1*, not on the -model size. - -#### Internal compression (`compress`) and result expansion - -When `compress` is `None`/`True` and the model has `≥ 200` reactions, -`speedy_fva` first lumps flux-coupled reactions and removes conservation rows -(`_compress_for_fva`) — a *single* nullspace pass (no recursive fixpoint), since FVA -needs only first-order couplings — runs FVA on the smaller compressed model, then expands the -results back via `_expand_fva`, scaling lumped reactions by their coupling factor -(with a min/max swap when the factor is negative) and filling blocked reactions -with `0/0`. **All three preprocessing call sites pass `compress=False`**, because the model is -already compressed by the pipeline's own COMPRESS passes; this is the key fact for §5.6. - -#### Contrast with `fva_legacy` - -`fva_legacy` (`lptools.py`) always solves the full `2n` LPs (parallel over an `SDPool` when -`processes > 1 and numr > 300`, else a serial warm-started loop), with no scan phase, no `v=0` -shortcut, no co-optimization, and no `reaction_list` scoping. On genome-scale models -`speedy_fva`'s Phase 1 typically resolves well over half of the `2n` objectives with a handful -of scan LPs, so the residual handed to Phase 2 is a fraction of `2n`. The two return identical -DataFrames (both post-process `|value| < 10⁻¹¹ → 0`); `fva_legacy` exists purely as a -debugging oracle. - -### 5.6 Why FVA #2 is the ~117 s genome-scale bottleneck - -On the canonical iML1515 gene-MCS run (SUPPRESS biomass ≥ 0.001, POPULATE, `max_cost=3`, -`gene_kos`), preprocessing's blocked/irreversible FVA — **FVA #2** — measures at **~117 s**, -the single largest preprocessing slice ([Ch 11](#ch11)). Every structural reason for this is visible in -the three call sites and in `speedy_fva`'s control flow: - -1. **It is whole-model — no `reaction_list`.** FVA #2 (`bound_blocked_or_irrevers_fva`, - `networktools.py`) forwards its kwargs to `fva` with *no* `reaction_list`, so - `speedy_fva` must resolve **all `2n` objectives** — every bound of every reaction — because - the bound-relaxation logic in §5.3 needs the true range of *every* reaction, not just - knockable ones. FVA #1 is also whole-model but runs on the smaller pre-GPR network; FVA #3 - is scoped to `knockable_ids` and so solves only `2·|knockable|` objectives. FVA #2 is the - only one paying the full `2n` on the *large* model. - -2. **It runs on the GPR-extended model, which is much larger.** FVA #2 executes *after* - `extend_model_gpr`, which injects a gene pseudoreaction per gene and additional - pseudoreactions/pseudo-metabolites to encode the Boolean AND/OR structure ([Ch 4](#ch4)). On - iML1515 this roughly doubles the reaction count relative to the metabolic-only network FVA #1 - saw. The log line's totals (`1825` + `2258` + `2150` + …) reflect a network of several - thousand reactions. More reactions ⇒ more objectives *and* larger per-LP factorizations. - -3. **Internal compression is disabled (`compress=False`).** Because the model is already - compressed by COMPRESS #2, FVA #2 passes `compress=False`, so `speedy_fva` does **not** run - its own coupled-lumping pass — it solves LPs at the full GPR-extended dimension rather than a - reduced one. This is correct (re-compressing the rational-bound model would be wasteful and - the caller needs bounds on the *actual* reactions), but it means no dimension reduction - cushions the LP cost. - -4. **Phase 2 likely drops below the parallel threshold.** `speedy_fva` parallelizes Phase 2 - only when `n_remaining ≥ 1000`. Phase 1's scan LPs are very effective at resolving - the many trivially-bounded reactions of a GPR-extended model (huge numbers of forward-only - reactions with `lb=0`, resolved by the `v=0` shortcut and the `min Σ|x|` scan), so the - *residual* handed to Phase 2 can fall **below 1000** — at which point Phase 2 runs the - **sequential, single-threaded** path, grinding through the residual individual LPs - one at a time. A residual of a few hundred genome-scale LPs solved serially, each on a - several-thousand-variable model, accounts for the bulk of the 117 s. (Phase 1's own push LPs - are cheap thanks to dual-simplex warm-starting; the cost concentrates in the serial Phase 2 - tail.) - -This makes FVA #2 a concrete, high-value **performance lever** ([Ch 11](#ch11)). Candidate mitigations -that follow directly from the analysis above: force Phase 2 onto the parallel path even for -`n_remaining < 1000` (or lower the threshold) so the residual LPs use all cores; or restrict -FVA #2's objectives to the reactions whose bounds can actually matter downstream — although, -unlike FVA #3, it genuinely needs *all* reactions' ranges to relax bounds correctly, so a -`reaction_list` restriction is not directly applicable and any scoping must be justified against -the bound-relaxation semantics of §5.3. The safe, immediately-available win is parallelism on -the Phase 2 tail. +### 5.5 The single-module fold + +If and only if all of the following hold: + +- there is exactly one module; +- its type is SUPPRESS or PROTECT; and +- it has no inner objective, + +one constrained call to `bound_blocked_or_irrevers_fva` replaces the final model-bound FVA and the +module FVA. Its reaction list is the union of `_fva_scope` and the knockable IDs. The returned table +both updates the model bounds and supplies `module['fva_bounds']`. + +The gate matters. With multiple modules, ranges from one region cannot safely be written into the +shared model because they may remove flux used by another module. Bilevel modules also require their +existing construction path. + +### 5.6 `speedy_fva` + +Public `fva` dispatches to `speedy_fva`. It avoids blindly solving `2n` independent LPs by combining +bound-resolved directions, scan LPs, warm-started residual solves and optional internal compression. +Only the objective changes between solves. Large residual sets may be distributed through `SDPool`; +smaller ones remain sequential to avoid process overhead. `fva_legacy` remains the simple `2n`-LP +reference implementation. + +### 5.7 Current preprocessing profile + +On the current PR branch, the canonical iML1515 single-SUPPRESS gene-MCS preprocessing run with +Gurobi measured about **19.6 s**. The two largest pieces were reversibility pre-tightening +(about **6.3 s**) and the folded final FVA (about **5.8 s**), followed by the two compression passes +(about **5.0 s** total). See Chapter 11 for the detailed, nested profile. These numbers are a +machine/solver-specific benchmark, not constants of the algorithm. (ch6)= @@ -2534,10 +2100,10 @@ finite nonzero lower/upper bound is not left on the variable; it is appended to explicit row so it acquires its own dual multiplier: ``` -lb_j finite, ≠ 0: −x_j ≤ −lb_j (row in LB, line 1111) -ub_j finite, ≠ 0: x_j ≤ ub_j (row in UB, line 1112) -A_ineq_p ← [A_ineq_p ; LB ; UB] (line 1113) -b_ineq_p ← b_ineq_p + [−lb_j…] + [ub_j…] (line 1114) +lb_j finite, ≠ 0: −x_j ≤ −lb_j (row in LB) +ub_j finite, ≠ 0: x_j ≤ ub_j (row in UB) +A_ineq_p ← [A_ineq_p; LB; UB] +b_ineq_p ← b_ineq_p + [−lb_j…] + [ub_j…] ``` Zero bounds and `±∞` bounds are skipped (an `x_j ≥ 0` reaction contributes no LB row; its @@ -2572,8 +2138,8 @@ read off the columns of `[A_eq ; A_ineq]`. variables are ordered `[λ (one per A_eq row) ; μ (one per A_ineq row)]` with bounds ``` -lb = [−∞]·(#A_eq rows) + [0]·(#A_ineq rows) (line 1124) -ub = [+∞]·(#A_eq rows + #A_ineq rows) (line 1125) +lb = [−∞]·(#A_eq rows) + [0]·(#A_ineq rows) +ub = [+∞]·(#A_eq rows + #A_ineq rows) ``` So an **equality primal constraint → free dual variable** (`λ_i ∈ ℝ`), an **inequality primal @@ -2601,11 +2167,11 @@ The maps are transposed accordingly: ``` # a knockable primal VARIABLE (reaction flux) becomes a knockable dual CONSTRAINT -z_map_constr_ineq ← [ z_map_vars_p[:, x_geq0] , z_map_vars_p[:, x_leq0] ] (line 1130) -z_map_constr_eq ← z_map_vars_p[:, x_eR] (line 1131) +z_map_constr_ineq ← [ z_map_vars_p[:, x_geq0], z_map_vars_p[:, x_leq0] ] +z_map_constr_eq ← z_map_vars_p[:, x_eR] # a knockable primal CONSTRAINT becomes a knockable dual VARIABLE -z_map_vars ← [ z_map_constr_eq_p , z_map_constr_ineq_p , 0(for the new LB/UB rows) ] (line 1132-1133) +z_map_vars ← [ z_map_constr_eq_p, z_map_constr_ineq_p, 0(for the new LB/UB rows) ] ``` Reading it in words: reaction `j`'s flux variable maps, after dualization, onto its *reduced-cost @@ -2616,7 +2182,7 @@ directly (their knockout is handled through the flux variable they bound). The o variable *and* a constraint in the same block, which would make the transpose ambiguous. **Step 6 — `reassign_lb_ub_from_ineq`** (`strainDesignProblem.py`, defined at -`:1207`). After transposing, many dual `A_ineq` rows are single-entry (a reduced-cost row on a dual +). After transposing, many dual `A_ineq` rows are single-entry (a reduced-cost row on a dual variable with no metabolic coupling). This helper folds single-variable inequality rows back into `lb/ub` on the dual variables, *except* where the row is flagged knockable (`z_map_constr_ineq` nonzero), because a knockable row must remain an explicit constraint for `z` to switch. This keeps @@ -2700,7 +2266,7 @@ infeasible. Making the undesired region infeasible *after knockouts* therefore r this dual system feasible after the same knockouts — which is a set of ordinary linear rows the MILP can hold, with `z` switching the rows that correspond to knocked reactions (via the transposed `z_map` from §6.2.3). This is the `SUPPRESS` branch: `addModule` calls `farkas_dualize` at -`strainDesignProblem.py` and sets a zero module objective `c_i` at `:670`. +`strainDesignProblem.py` and sets a zero module objective `c_i` at. #### 6.3.3 Why the certificate is unbounded by nature, and the normalization row @@ -2724,10 +2290,12 @@ A direct performance consequence follows from the unboundedness: **FVA-style bou bound these dual variables.** The preprocessing FVA ([Ch 5](#ch5)) tightens variable ranges by maximizing/minimizing each variable over the polytope; for a Farkas dual variable that range is `(−∞, +∞)` by construction (the feasible set is a cone, scale-free), so FVA returns `±∞` and buys -nothing. In `link_z` ([Ch 7](#ch7)) this is exactly why the SUPPRESS dual rows end up as **indicator -constraints** rather than big-M: the per-constraint bounding LP that would supply a finite `M` -returns `±∞`, and the code's `self.M = inf` default routes an unbounded row to a native indicator. -This is emergent from the cone geometry, not a hard-coded "SUPPRESS ⇒ indicator" switch. +nothing. In `link_z` ([Ch 7](#ch7)) these rows therefore remain **indicator constraints** for a +native-indicator backend under the default `M = inf` policy. MILP construction deliberately does +not solve per-row bounding LPs. GLPK, which has no native indicator constraints, uses the configured +blanket finite M (1000 by default); an explicitly supplied finite M requests the same formulation on +other backends. This is a compatibility formulation, not a claim that the Farkas cone has useful +finite coordinate bounds. #### 6.3.4 The `b^T y ≠ 0` caveat @@ -2826,7 +2394,7 @@ A PROTECT or SUPPRESS module may itself carry an *outer* objective to be optimiz inner-optimal set (`strainDesignProblem.py`). The already-assembled bilevel `_p` (region primal ⊕ inner dual) is dualized *again* by `LP_dualize` with the outer objective `c_out` (`strainDesignProblem.py`), and coupled by the same strong-duality equality -(`strainDesignProblem.py` exact, `:604-631` relaxed with a reference copy of the whole `_p`). +(`strainDesignProblem.py` exact, relaxed with a reference copy of the whole `_p`). Nesting `LP_dualize` on an already-dual system is possible precisely because it returns its output in the same standard container it consumes (§6.2.3) — the transform is closed under composition. @@ -2866,7 +2434,7 @@ $\max_z \max_{v \in \arg\max c_{\text{inner}}^\top v} c_{\text{out}}^\top v$. Co (`strainDesignProblem.py`), and the outer problem `_r` is joined to that second dual by a further strong-duality equality (`strainDesignProblem.py`). Bounds are reassigned (`strainDesignProblem.py`) and the outer objective set (`strainDesignProblem.py`, - and the final MILP objective at `:675-685`). + and the final MILP objective at). The max-min is thus two `LP_dualize` calls: one to characterize the inner-optimal face, one to turn the maximization *over* that face into flat rows. @@ -2929,465 +2497,129 @@ Every row is a stacking of "assert an LP's optimum via primal + dual + strong-du `LP_dualize`. That is what makes the dualization machinery reusable: the metabolic content changes, the linear-algebra primitive does not. -### 6.6 Boundary with [Chapter 7](#ch7) - -Everything above produces **continuous rows only**: dual variables `y = (λ, μ)`, dual-feasibility -constraints, strong-duality equality rows, Farkas normalization rows, and the primal blocks they are -paired with — together with the `z_map_vars`, `z_map_constr_ineq`, `z_map_constr_eq` matrices that -record *which reaction's knockout removes which row or variable* after all the transposition. What is -**not** done here is attaching the binary intervention variables `z` to those rows. That is -`link_z` (`strainDesignProblem.py`), [Ch 7](#ch7): it reads the `z_map_*` matrices, splits knockable -equalities into directional inequalities, tries to bound each row with an LP to obtain a valid -big-M, and — where the bounding LP returns `±∞`, as it always does for the scale-free Farkas dual -rows (§6.3.3) — falls back to native indicator constraints. The emergent split noted throughout this -chapter (SUPPRESS's unbounded Farkas rows → indicators; PROTECT's finite-flux primal rows → big-M) -is a *consequence* of the bound structure this chapter's dualization produces, decided in [Ch 7](#ch7)'s -`self.M`/bounding-LP fork, not a per-type switch. Read this chapter for *what the rows mean*; read -[Ch 7](#ch7) for *how `z` turns them on and off*. - - -(ch7)= -## 7. MILP construction & the z-linking - -By the time this chapter's code runs, every strain-design *module* has been turned into a -self-contained linear (in)equality block — a Farkas infeasibility certificate for **SUPPRESS**, a raw -primal feasibility system for **PROTECT**, or a strong-duality sandwich for the bilevel types ([Ch 6](#ch6) -owns that content). What remains is *assembly*: stacking those blocks into one matrix, attaching the -seed rows that account for intervention cost, and — the substance of this chapter — **wiring the binary -intervention variables `z` to the continuous rows** so that flipping `z_j` genuinely removes reaction -`j` from the flux system. That wiring is done two ways, native **indicator constraints** or **big-M** -linearization, and the choice between them is made per-constraint by a bound-computing LP. Getting it -right is what separates a correct, numerically well-behaved MILP from one that either admits phantom -solutions (M too small) or grinds through a useless LP relaxation (M too large). - -All line references are to `strainDesignProblem.py` unless noted; the indicator container lives in -`indicatorConstraints.py`. - -### 7.1 Notation and the shape of the master problem - -The MILP variable vector is partitioned as - -``` -x = [ z ; y ] z ∈ {0,1}^{num_z}, y ∈ ℝ^{n_cont} -``` - -with the `num_z` binaries occupying the *leading* columns (`self.idx_z = [0..numr-1]`, -`SDProblem.__init__`:164) and all continuous module variables `y` appended afterward. The final -`self.vtype = 'B'*num_z + 'C'*(z_map_vars.shape[1]-num_z)` simply records that split. - -`z_j = 1` means "intervention `j` is applied". For a **knockout** that is removal of reaction `j`; for -a **knock-in** the meaning is inverted (`z_inverted[j] = True`, set from `ki_cost`), and -the sign machinery of §7.6 flips the coupling so that `z_j = 1` still reads as "the intervention is -made". One binary per *compressed* reaction: `self.num_z = numr` (`numr = -len(model.reactions)`), because at this point the model has already been through both compression -passes and GPR extension ([Ch 3](#ch3), [Ch 4](#ch4)), so a "reaction" may be a lumped subnet or a gene -pseudoreaction. There is deliberately **no** separate binary per constraint or per variable — a single -`z_j` fans out to *all* rows and variables that reaction `j` controls, tracked by the three maps -introduced below. - -Throughout, the master inequality system is `A_ineq · x ≤ b_ineq`, the equality system `A_eq · x = -b_eq`, with variable box `lb ≤ x ≤ ub`. - -#### The three z-maps - -Coupling bookkeeping is carried in three sparse matrices, each with `num_z` rows (one per binary) and -one column per constraint/variable of the system being tracked: - -| map | shape | entry `(j, k)` meaning | -|---|---|---| -| `z_map_constr_ineq` | `num_z × #ineq` | `z_j` knocks inequality row `k` | -| `z_map_constr_eq` | `num_z × #eq` | `z_j` knocks equality row `k` | -| `z_map_vars` | `num_z × #vars` | `z_j` knocks variable `k` (forces its flux to 0) | - -The stored value encodes *both* which binary and the coupling polarity: **`+1` = knockout** (this row -disappears when `z_j = 1`), **`−1` = knock-in / addition** (the row is present only when `z_j = 1`), -`0` = no coupling. These are the maps `link_z` reads to decide, for every row, which `z` column to -write into and with which sense. They are the single source of truth linking the *combinatorial* layer -(`z`) to the *continuous* layer (fluxes, dual variables). - -### 7.2 `SDProblem.__init__` — the seed rows, `num_z`, and the M switch - -Before any module is added, `__init__` lays down a 3-row skeleton over the `z` columns only. - -#### The three fixed seed rows - -```python -self.A_ineq = sparse.csr_matrix([[-i for i in self.cost], # row 0: idx_row_maxcost - self.cost, # row 1: idx_row_mincost - [0 for _ in range(num_z)]]) # row 2: idx_row_obj -self.b_ineq = [0.0, max_cost_or_sum, np.inf] -``` - -with `self.cost` the per-reaction intervention weight (KO cost, overwritten by KI cost where a KI is -defined;, `nan`→`0`). The three rows and their right-hand sides: - -- **Row 0, `idx_row_maxcost`**: $-\sum_j \text{cost}_j \cdot z_j \le 0$, i.e. $\sum_j \text{cost}_j z_j \ge 0$. With - non-negative costs this is slack at construction, but it is a *live lower bracket* on total - intervention cost: the enumeration/optimization layer ([Ch 8](#ch8)) raises its RHS to force the solver past - cost levels already exhausted, turning it into $\sum \text{cost}_j z_j \ge \kappa$. Keeping it as a permanent row - means that lower bound can be tightened in place without restructuring the matrix. - -- **Row 1, `idx_row_mincost`**: $\sum_j \text{cost}_j z_j \le b$, the **budget cap**. Its RHS is - `self.max_cost` when the user supplied one, else $\sum_j |\text{cost}_j|$ — the latter is a - vacuous cap (no design can cost more than the sum of all weights), present so the row always exists - and can be tightened later. This is the constraint that makes "minimal" cut sets minimal-*enough*: - no design exceeding the budget is admitted. - -- **Row 2, `idx_row_obj`**: an all-zero placeholder with RHS $+\infty$. For a pure MCS problem - the objective is *minimize intervention cost* and lives in the objective vector `self.c` (lines - 202–205: `c[j] = cost[j]`), so this row stays inert. For **bilevel** problems (OptKnock, OptCouple, - …) the outer objective is a flux expression, not a cost sum; the row is then overwritten with the - objective coefficients and used by `fixObjective` (`strainDesignMILP.py`:239–241) to - pin $c \cdot x \le \text{value}$ during the BEST search. Reserving row 2 up front lets that pin be a single - `set_ineq_constraint` call rather than a matrix resize. - -The naming (`maxcost` on the `≥ 0` row, `mincost` on the `≤ budget` row) reads backwards against the -RHS values and is best treated as an internal label; the *mathematics* is: row 0 lower-brackets and -row 1 upper-brackets the weighted intervention sum, and row 2 is the swappable objective slot. - -The companion `z_map_constr_ineq` is initialised to `(numr × 3)` **zeros**: the seed rows -are *not knockable* — they constrain `z`, they are not part of any flux subsystem, so no `z` ever -"removes" them. - -#### `self.M` — the master indicator/big-M switch - -```python -bound_thres = max(|cobra_conf.lower_bound|, |cobra_conf.upper_bound|) -if self.M is None and solver == 'glpk': self.M = bound_thres # GLPK: no indicators -elif self.M is None: self.M = np.inf # default -# else: user-supplied M kept as-is -``` - -`self.M` is the *fallback* big-M used only when the per-constraint bounding LP (§7.5) cannot produce a -finite bound. Its three regimes: - -- **`inf` (default).** Rows with no finite bound get **no** big-M row; they fall through to native - **indicator constraints** (§7.7). This is the preferred, numerically clean path. -- **cobra bound (GLPK).** GLPK has no indicator-constraint API, so `self.M` is forced finite (the - cobra default bound, typically 1000) and *every* unbounded row becomes a big-M row with that - constant. A warning is logged. This is the escape hatch that lets the open-source solver - run at all, at the cost of a loose, uniform M. -- **user override.** Passing `M=` in kwargs pins the fallback explicitly (for a solver that - supports indicators, this forces big-M everywhere a bound is missing). - -So `self.M` decides what happens to the rows the bounding LP *cannot* bound; the bounding LP decides -everything else. The emergent SUPPRESS→indicator / PROTECT→big-M split (§7.8) is a downstream -consequence of this, not a separate branch. - -### 7.3 `addModule` — block-diagonal assembly - -Each module produces its own block `(A_ineq_i, b_ineq_i, A_eq_i, b_eq_i, lb_i, ub_i, c_i)` plus its -own three z-maps `z_map_*_i` (the [Ch 6](#ch6) dual/primal machinery; here we only care about *how* the block -joins the master). The join is: - -```python -self.z_map_constr_ineq = hstack((self.z_map_constr_ineq, z_map_constr_ineq_i)) # 688 -self.z_map_constr_eq = hstack((self.z_map_constr_eq, z_map_constr_eq_i)) # 689 -self.z_map_vars = hstack((self.z_map_vars, z_map_vars_i)) # 690 -self.A_ineq = sparse.bmat([[self.A_ineq, None], - [None, A_ineq_i]]).tocsr() # 691 -self.b_ineq += b_ineq_i -self.A_eq = sparse.bmat([[self.A_eq, None], [None, A_eq_i]]).tocsr() # 693 -self.b_eq += b_eq_i -self.c += c_i; self.lb += lb_i; self.ub += ub_i -``` - -The constraint matrices grow **block-diagonally**: the new module's rows occupy new rows *and* new -columns, with explicit `None` (zero) off-diagonal blocks. The z-maps, in contrast, grow **only in -columns** (`hstack`) — they keep their `num_z` rows. - -#### Why block-diagonal for the continuous part - -Each module owns a **private set of continuous variables**. A SUPPRESS module's block is a Farkas dual -living in *dual* space (one dual variable per primal constraint of that module's flux system); a -PROTECT module's block is a *primal* flux vector `v`; a bilevel module carries primal flux *and* dual -variables. These variable sets are semantically disjoint — the flux that must stay feasible in a -PROTECT module has nothing to do with the dual ray that certifies infeasibility in a SUPPRESS module, -and two SUPPRESS modules certify infeasibility of two *different* behaviors, each needing its own ray. -Sharing continuous columns between them would impose spurious equalities (module A's flux = module B's -flux) that are simply wrong. Block-diagonal placement gives each module an independent copy of flux -space; the modules never see each other's continuous variables. - -#### Why the z-columns are shared - -The *only* thing all modules must agree on is **which reactions are cut** — that is the design, and it -is global. Those are the `z` columns, columns `0..num_z-1`, which are *not* re-created per module: the -seed skeleton put them there once, and every module's z-maps are `hstack`-ed onto the same `num_z` -rows. When `link_z` later writes a big-M coefficient into `A_ineq[row, z_j]`, it writes into that -shared leftmost block — filling the bottom-left "`None`" corner that `bmat` left as zeros. So the -architecture is: **block-diagonal in the continuous variables, dense-shared in the `z` variables**. -The design vector `z` is the coupling backbone; every module hangs off it. This is exactly the -structure that makes a *single* set of `num_z` binaries enforce *all* modules simultaneously — a -knockout that satisfies the SUPPRESS certificate is the *same* `z` that must leave the PROTECT flux -feasible. - -`z_map_constr_ineq_i / z_map_constr_eq_i / z_map_vars_i` carried in with each module record precisely -which of that module's *new* rows/variables reaction `j` controls, so after the `hstack` the master -maps know, for every row in the assembled system, which `z` (if any) knocks it and with what polarity. - -### 7.4 `prevent_boundary_knockouts` — why nonzero-sign bounds must be moved - -This runs inside `build_primal_from_cbm`, before dualization, on every primal flux -system. It repairs a specific incompatibility between the KO encoding and reactions whose flux is -*forced away from zero*. +### 6.6 Boundary with Chapter 7 -#### The KO encoding and the failure +Dualization produces continuous rows and the `z_map_*` bookkeeping that says which intervention gates +which row or variable. It does not itself choose a big-M or construct an indicator constraint. -A knockout of reaction `j` is ultimately realized (link_z, §7.5–7.6) by driving its flux `v_j` to 0. -The mechanism *tightens the reaction's box toward 0*: for a variable with `ub_j > 0` it adds the row -`v_j ≤ 0` gated by `z`; for `lb_j < 0` it adds `−v_j ≤ 0`. This is valid **iff `0 ∈ [lb_j, ub_j]`** — -the KO row merely collapses the box onto a value the box already contains. +`link_z` owns that final encoding. Zero- and single-continuous-variable rows can obtain a finite +relaxation directly from variable bounds. Rows with two or more continuous variables are intentionally +routed to a native indicator when `self.M = inf`; GLPK or an explicit finite `M` uses that configured +blanket value instead. No per-row bounding LP is run during MILP construction. -Now suppose the reaction has a **nonzero-sign bound**: `lb_j > 0` (obligatorily forward) or `ub_j < 0` -(obligatorily reverse). Then `0 ∉ [lb_j, ub_j]`. The variable's *own box bound* — which is a property -of the variable, not a constraint row, and is therefore **never multiplied by `z`** — keeps forcing -`v_j ≥ lb_j > 0` even when the KO row `v_j ≤ 0` is active. The two are contradictory: the "knockout" -does not remove the reaction, it renders the subsystem infeasible. Equivalently, in the -bound-multiplication view the docstring uses (multiply the bound by `z` to simulate the KO): -multiplying a bound that lies strictly on one side of 0 can never *reach* 0, so **the residual bound -still forces flux**. +This is especially relevant to SUPPRESS. Farkas-certificate variables are commonly unbounded, so their +rows naturally land on the indicator/configured-M path. The behavior follows the row structure and +the global M policy rather than a hard-coded module-type test. -#### The transformation -For each knockable column (`col_has_z`, from `z_map_vars`): +(ch7)= +## 7. MILP construction & the z-linking -``` -if lb_j > 0: add row -v_j ≤ -lb_j (i.e. v_j ≥ lb_j), then set lb_j := 0 -if ub_j < 0: add row +v_j ≤ ub_j (i.e. v_j ≤ ub_j), then set ub_j := 0 -``` +By this point each strain-design module has become a continuous linear block: a Farkas certificate for +SUPPRESS, a primal feasibility system for PROTECT, or a strong-duality system for a bilevel module. +`SDProblem` stacks those blocks and connects the shared intervention vector `z`. -The obligation is *moved out of the variable box and into an explicit inequality row*, and the box is -reset so that `0 ∈ [lb_j, ub_j]`. Concretely, `lb_j > 0` becomes box `[0, ub_j]` plus a standalone row -`v_j ≥ lb_j`. The new rows are appended with **zero z-columns** (: `hstack([z_map_constr_ineq, -zeros(numz, new_z_cols)])`) — they are **non-knockable**. That is the crucial point: the obligation is -now a fixed property of the flux system that survives into the dual as an ordinary constraint with an -unconditioned multiplier, rather than a variable bound that the z-machinery would try (and fail) to -multiply. The KO machinery can now cleanly collapse the (0-containing) box, and the moved row, carrying -no `z`, cannot be corrupted by the coupling. +### 7.1 Seed rows and shared binaries -(It moves the nonzero-sign bounds — `lb > 0` and `ub < 0`, the ones that exclude 0, since those are what break the encoding.) +The first `num_z` variables are binary intervention indicators. Three fixed inequality rows represent +the lower cost bracket, the `max_cost` budget and the objective placeholder. Continuous module blocks +are appended block-diagonally, while their `z` mappings share the same binary columns. This is how one +reaction intervention acts in every module simultaneously. -In practice this fires rarely, because FVA preprocessing ([Ch 5](#ch5)) has already relaxed non-binding bounds -to `±∞` and pinned irreversible/blocked reactions to 0; the survivors are the genuinely -obligatory-flux reactions, and this function is what keeps them knockable. +`self.M` selects the fallback: -### 7.5 `link_z` — the heart of the chapter +- GLPK with no user value uses the cobra default bound, normally `1000`; +- Gurobi, CPLEX and SCIP with no user value use `inf`, enabling native indicators; and +- an explicit finite `M` requests the blanket big-M formulation. -`link_z` transforms the assembled but *unlinked* system — where `z`-columns are still zero in every -module row — into a fully coupled MILP. Six steps. +The finite fallback is intentionally global. It is not presented as an automatically valid or tight +row bound; users choosing big-M accept its known numerical sensitivity. -#### Step 1: knockable equalities → ± inequality pairs +### 7.2 Per-module sign overrides -You cannot "relax an equality with a big-M" in one row: `a·x = b` gated off needs both `a·x ≤ b` and -`a·x ≥ b` to disappear. So each knockable equality (a nonzero column of `z_map_constr_eq`) is split: +For classical SUPPRESS and PROTECT modules without an inner objective, +`_module_bound_override` reads `module['fva_bounds']` and creates a targeted subset of sign-only +overrides: -``` -a·x = b → a·x ≤ b and −a·x ≤ −b -``` - -Both new inequalities are gated by the *same* `z` (`z_eq = z_map_constr_eq[:, tuple(idx)*2]`, - — the column is duplicated). The originals are deleted from `A_eq`. When the -gate is *inactive*, the pair re-imposes the equality exactly; when active, both directions relax. (If -this equality later lands on the indicator path with both directions unbounded, §7.7's lumping step -fuses the pair *back* into a single `'E'` indicator — the split is undone once it is no longer needed.) +- blocked in the module: `(0,0)` -- but only where the margin is exact, i.e. on gurobi and cplex. On + SCIP and GLPK the margin is `_MODULE_OVERRIDE_TOL` (1e-8, ten times their feasibility tolerance), and + since `lo` then requires `minimum >= 1e-8` while `hi` requires `maximum <= -1e-8`, the two are mutually + exclusive: a solver-reported blocked reaction yields *no* override there. A range with + `minimum > maximum` beyond tolerance is logged and skipped on every backend; +- nonnegative in the module: lower bound `0`; and +- nonpositive in the module: upper bound `0`. -#### Step 2: variable-KOs → inequality rows +No magnitude is tightened and no bound is relaxed to infinity. The override is passed to +`build_primal_from_cbm` for this module block only, preserving the semantics of other modules that +share the same reaction binary. -A knockable *variable* (nonzero column of `z_map_vars`) is translated into an inequality that pins its -flux to 0 on the relevant side: +### 7.3 `prevent_boundary_knockouts` -``` -if ub_j > 0: row +1·v_j ≤ 0 (knock the positive side toward 0) -if lb_j < 0: row −1·v_j ≤ 0 (knock the negative side toward 0) -``` +A hard variable bound cannot be disabled by a binary. Therefore, the knockable side of a nonzero-sign +bound is moved into an inequality row before dualization. The associated `z_map_constr_ineq` column +records which binary owns that row. Non-knockable bounds remain in the variable box. -A reversible reaction (`lb_j<0 0 +max(a*x) = a*lb if a < 0 ``` -Dropping *every* knockable row is what makes `P_relaxed` a superset of every actually-reachable knocked -polytope (any real design drops only *some* rows), so `max a·x` over `P_relaxed` upper-bounds `a·x` -over any knocked subsystem — hence a **valid** M — and taking the exact max makes it **tight**. - -Because solving one LP per knockable row is expensive, rows are triaged by sparsity: - -- **`nnz == 0`** (empty row): `max = 0`. (`n_zero`) -- **`nnz == 1`** (single variable `coeff·v_c`): `max = coeff·ub_c` if `coeff>0` else `coeff·lb_c`, - read straight off the box; `∞` if that bound is infinite. (`n_single`) -- **`nnz ≥ 2`**: needs an actual LP, $\max a \cdot x = -\min(-a \cdot x)$ over `P_relaxed`. (`n_lp`) - -logged as `Bounding MILP: N constraints (X zero, Y single-var, Z need LP)`. Only the `n_lp` -rows hit the solver, optionally across a worker pool (`worker_compute` maximises `a·x` by minimising -`−a·x` and negating). Finite results are rounded *up* to 5 digits (`ceil(M·1e5)/1e5`, -) to stay safely on the valid side; **infinite** results are replaced by `self.M` — -the point where §7.2's switch takes effect. +If the required bound is infinite, the row follows the indicator/configured-M path. A row containing +two or more continuous variables is deliberately assigned `inf` without solving a bounding LP. -#### Step 4: the fork at the M value +For a finite relaxation value `M`, a KO-style gate uses: -For each knockable inequality row, `Ms[row]` is now either a finite number or `self.M` (which may be -`inf`). The loop: - -```python -for row in ...: - if not isinf(Ms[row]) and not isnan(Ms[row]): # finite M → big-M row - z_i = z_map_constr_ineq[:, row].nonzero()[0][0] - sense = z_map_constr_ineq[z_i, row] - if sense > 0: # z_i = 1 knocks out (KO) - A_ineq[row, z_i] = -Ms[row] + b_ineq[row] - else: # z_i = 0 knocks out (KI convention) - A_ineq[row, z_i] = Ms[row] - b_ineq[row] - b_ineq[row] = Ms[row] +```text +a*x + (b-M) z <= b ``` -Rows with `isinf(Ms[row])` are **skipped** here and picked up by the indicator path in step 5. The two -sense cases, written out (let `a·x ≤ b` be the row, `M = Ms[row]`): +so `z=0` enforces the original row and `z=1` relaxes it to `a*x <= M`. The inverse polarity is used +for knock-ins. -- **`sense > 0` (KO, active when `z=1`)** — coefficient `b − M` in the z-column gives the row - $a \cdot x + (b - M) \cdot z \le b$: - - `z = 0`: $a \cdot x \le b$ — **enforced**. - - `z = 1`: $a \cdot x \le M$ — relaxed to the tight maximum, hence **non-binding** (since $M = \max a \cdot x$). +### 7.5 Indicators and the blanket-M fallback - This is exactly tight: at the knocked state the bound equals the reachable maximum, not the looser - `b + M` a naive formulation would use. +With `self.M = inf`, the remaining rows are represented by `IndicatorConstraints` and passed to +Gurobi, CPLEX or SCIP. GLPK has no native indicator implementation and therefore receives the +configured finite M, normally 1000. Passing an explicit finite `M` requests the same blanket +substitution on every backend. -- **`sense < 0` (KI, active when `z=1`, absent when `z=0`)** — coefficient `M − b`, and `b` reset to - `M`, giving $a \cdot x + (M - b) \cdot z \le M$: - - `z = 1`: $a \cdot x \le b$ — **enforced** (reaction present). - - `z = 0`: $a \cdot x \le M$ — relaxed, **non-binding** (reaction absent). +This behavior is intentional. Automatic per-row M estimation was removed because it was expensive +and did not make the big-M formulation reliable: values that are too small can miss designs, while +very large values can introduce numerical artifacts and spurious designs. Native indicators remain +the preferred formulation for these rows. -Both cases realize the same logic — "constraint holds in the active state, evaporates in the knocked -state" — with the polarity dictated by the `z_map` sign. The finite-M rows are now permanently part of -`A_ineq`; only their `z`-column entries changed. - -#### Steps 5–6: indicators and cleanup - -Every row still carrying `isinf(Ms[row])` (`knockable_constr_ineq_ic`) becomes a **native -indicator constraint**. First, a **lumping** pass undoes the step-1 split where it is -no longer useful: rows are canonicalised by the sign of their first nonzero entry, grouped -by an exact `(indices, data)` key, and pairs found to be identical up to a global sign -flip (`ident_rows` product `−1`) — i.e. an `a·x ≤ b` and an `a·x ≥ b` on the same `z` — are fused into -a single equality indicator; exact duplicates (product `+1`) drop one copy. The -survivors are packaged into an `IndicatorConstraints` object and *removed* from the -static `A_ineq`/`A_eq`, because an indicator row is enforced by the solver's logic -engine, not by the LP matrix. - -### 7.6 Indicator constraints (`indicatorConstraints.py`) - -`IndicatorConstraints(binv, A, b, sense, indicval)` is a thin container (constructor) for -rows of the form - -``` -z_{binv[k]} = indicval[k] ⇒ A[k]·x b[k] -``` - -with `sense ∈ {'L','E','G'}` (≤, =, ≥). The container is populated in `link_z`: - -- **`binv`** — the `z` index gating each row, read from the nonzero of the row's `z_map` column. -- **`A, b`** — the surviving knockable inequality rows first (`'L'`), then the lumped equality rows - (`'E'`): `sense = 'L'*n_ineq + 'E'*n_eq`. -- **`indicval`** — *which* value of the binary triggers enforcement, derived from the `z_map` polarity -: `[0 if d == 1 else 1 for d in data]`. So a `z_map` entry of **`+1` (KO) → `indicval = 0`** - (the constraint is enforced while the reaction is *present*, `z=0`, and released on knockout), and - **`−1` (KI/addition) → `indicval = 1`** (enforced only when the reaction is *added*, `z=1`). The code - comment states this mapping directly. This is the exact combinatorial analogue of the - big-M sense cases in §7.5 step 4. - -Semantically, $z = \text{indicval} \Rightarrow A \cdot x \;\{\le,=\}\; b$ and, when $z \ne \text{indicval}$, the constraint is simply *not -present* — there is no slack variable, no large constant, nothing in the LP relaxation. The solver -enforces the implication by branching/logic. - -### 7.7 Why indicators give a tighter LP relaxation than big-M - -Take the KO row from §7.5, $a \cdot x + (b - M) \cdot z \le b$, and relax the binary to $z \in [0,1]$ (what every LP -node in branch-and-bound actually sees). Rearranged: - -``` -a·x ≤ b + (M − b)·z -``` - -At a *fractional* `z` the right-hand side floats up proportionally to `z`: the relaxation lets `a·x` -exceed its true bound `b` by up to `(M−b)·z`. The feasible region of the relaxation is therefore -**enlarged**, and the enlargement grows *linearly with M*. A loose (large) M produces a weak -relaxation: the LP bound at each node is poor, branch-and-bound explores more nodes, and the wide -spread between M and the unit-scale flux coefficients degrades numerical conditioning (`FeasibilityTol` -/ `IntFeasTol` interactions, ill-scaled bases). This is the concrete cost of a bad M. - -The indicator constraint has *no* continuous relaxation of the implication: at fractional `z` the -solver does not manufacture a proportional slack; it enforces `z=indicval ⇒ a·x ≤ b` combinatorially. -The relaxation it presents is at least as tight as the big-M one and usually strictly tighter, with no -M to condition on. That is why indicators are the default whenever the solver supports them, and why -the per-constraint tight M matters when it does *not*: the bounding LP of §7.5 exists precisely to -make each finite M as small as validly possible. This is also the payoff of [Ch 5](#ch5)'s FVA bound -relaxation — by pushing non-binding bounds to `±∞`, FVA makes the corresponding `max a·x` *infinite*, -which routes those rows to indicators (the tightest option, no M at all) instead of leaving them with a -finite-but-large M. Tight preprocessing and tight linearization are the same fight. - -### 7.8 The emergent SUPPRESS→indicator / PROTECT→big-M split - -A frequently observed pattern under the default `M = inf`: SUPPRESS modules end up almost entirely on -**indicator** constraints, PROTECT modules almost entirely on **big-M**. This is *emergent from bound -structure*, not a per-type branch anywhere in the code. - -- A **SUPPRESS** module is a **Farkas dual** (`farkas_dualize`, [Ch 6](#ch6)). Its variables are the components - of an unbounded *dual ray*; the dual feasible set is a **homogeneous cone**, so the dual variables - are unbounded above. The knockable rows are constraints on these unbounded dual variables, so their - bounding LP returns `max a·x = +∞` → `Ms = self.M = inf` → **indicator**. +### 7.6 Duplicate consolidation and free binaries -- A **PROTECT** module is a **raw primal** flux system (`reassign_lb_ub_from_ineq`, [Ch 6](#ch6)). Its - variables are fluxes with **finite FVA bounds**; the knockable rows are ordinary flux constraints, - so their bounding LP returns a **finite** `max a·x` → **big-M** with that tight constant. +Opposite indicator inequalities with the same normalized sparse row can be represented as one +equality indicator; same-direction duplicates are removed. Hashing exact sparse `(indices, data)` +keys avoids the previous quadratic row comparison. -So the fork is decided entirely by whether `max a·x` over the relaxed polytope is finite — a property -of the *bounds*, funneled through the single `self.M`/bounding-LP mechanism in `link_z`. Change the -bound structure (e.g. cap the dual variables, or lose FVA relaxation on the primal) and the split -moves. On GLPK it collapses entirely: `self.M` is finite, so even the unbounded SUPPRESS rows get a -big-M, and there are no indicators at all. This is the mechanistic content behind the memory note that -SUPPRESS means *"cannot"* (make a behavior infeasible — certified by an unbounded dual ray, hence -indicators) and PROTECT means *"can"* (keep a behavior feasible — a bounded primal flux, hence big-M). +After all links are built, a targetable KO binary that appears only in the cost/budget rows and gates +no finite-M row, equality or indicator cannot affect feasibility. Such a binary cannot occur in a +minimal design and its upper bound is fixed to zero. Non-targetable variables, knock-ins and essential +knock-ins are excluded from this cleanup. -### 7.9 Final consolidation and the binary block +### 7.7 Numerical consequences -After `link_z`, the master problem is: - -- **`A_ineq`** — seed rows 0–2, then the block-diagonal module rows, plus the eq→ineq rows (step 1) - and var-KO rows (step 2), with finite-M `z`-column coefficients written in place; indicator rows have - been *removed* (they live in `self.indic_constr`). -- **`A_eq`** — the non-knockable equalities (stoichiometry `S·v = 0`, fixed module equalities) plus any - lumped equalities that stayed on the big-M path; indicator equalities removed. -- **`self.indic_constr`** — the `IndicatorConstraints` bundle. -- **`self.c`** — for a pure MCS problem, `c[j] = cost[j]` on the `z` block, 0 elsewhere (minimize - intervention cost, `is_mcs_computation = True`); for bilevel, `c` on `z` is 0 and the - outer objective sits in seed row 2. `self.c_bu` backs it up. -- **`self.vtype = 'B'*num_z + 'C'*(z_map_vars.shape[1]-num_z)`**: the binary block is the - leading `num_z` columns — the design variables `z`, which every module's coupling was wired into — - and everything after is the continuous module variables (fluxes, dual rays) that hang off them - block-diagonally. - -The `ContMILP` snapshot stores the continuous projection (all columns except `idx_z`) -together with the three z-maps, so that a candidate design `z*` can be validated by substitution -without re-solving the full MILP (used by `verify_sd`, [Ch 8](#ch8)). At this point the problem is a complete, -solver-ready MILP: binaries coupled to continuous rows through tight per-constraint big-Ms where -bounds are finite and native indicators where they are not. +Indicators avoid choosing an M but remain subject to each solver's indicator implementation and +feasibility tolerances. The explicit/GLPK big-M path is a compatibility mode whose completeness and +specificity must be checked against known designs for the model at hand. Changing M is a formulation +change, not merely a performance tune. (ch8)= @@ -3437,7 +2669,7 @@ def fixObjective(self, c, cx): self.set_ineq_constraint(self.idx_row_obj, c, cx) # row 2 := (c·x ≤ cx) ``` -`resetObjective` (`:243-245`) restores the *vector* to `c_bu`; `setMinIntvCostObjective` (`:247-250`) +`resetObjective` restores the *vector* to `c_bu`; `setMinIntvCostObjective` clears the vector and installs the intervention-cost objective $\sum cost_i z_i$ over targetable `z`; `clear_objective` (`solver_interface.py`) zeroes the vector. @@ -3495,7 +2727,7 @@ ANY vs BEST vs POPULATE. The user wants *some* valid design, not necessarily the smallest. Each outer iteration does two solves. -**Solve 1 — zero-objective feasibility (`:443-446`).** +**Solve 1 — zero-objective feasibility.** ```python self.resetTargetableZ() # all candidate z free again (ub=1) @@ -3515,7 +2747,7 @@ problem**: "find any `(z, x)` satisfying all constraints". Why do this first? feasible `z` the solver stumbles onto is typically *far* from minimal (it may knock out dozens of reactions), but that is fine — we only wanted a foothold. -**Solve 2 — minimize intervention cost within the found subspace (`:470-492`).** +**Solve 2 — minimize intervention cost within the found subspace.** ```python cx = np.sum([c*x for c,x in zip(self.c_bu, x)]) # objective value at the found point @@ -3527,7 +2759,7 @@ while ...: ... ``` -`setTargetableZ(z)` (`:256-258`) sets `ub=0` on every candidate `z_i` that the feasibility solve left +`setTargetableZ(z)` sets `ub=0` on every candidate `z_i` that the feasibility solve left at 0. This **restricts the search to the subspace spanned by the reactions the first design already touched** — the support of `z` and its subsets. Inside that tiny subspace the solver now *minimizes* $\sum cost_i z_i$: it finds the cheapest sub-design that still satisfies all modules. @@ -3550,7 +2782,7 @@ precisely the guarantee BEST adds. #### 8.3.2 BEST — `compute_optimal` (`strainDesignMILP.py`): global optimum, then fix and iterate The user wants the **globally cheapest** design(s), in nondecreasing cost order. The first solve is *not* -a feasibility solve; it is a genuine global optimization (`:335-338`): +a feasibility solve; it is a genuine global optimization: ```python self.resetTargetableZ() @@ -3564,12 +2796,12 @@ close the gap between the best incumbent and the lower bound. That is inherently feasibility solve (the whole tree may need pruning to certify no cheaper design exists), which is the price of the stronger guarantee. -For a pure MCS problem (`is_mcs_computation`, `:342-351`) the objective *is* the intervention cost, so +For a pure MCS problem (`is_mcs_computation`) the objective *is* the intervention cost, so the optimal `z` is already a minimal design; BEST verifies it, records it, adds the exclusion cut, and loops — each iteration returns the next-cheapest design because the accumulated cuts push the solver to progressively higher cost. -For a bilevel problem (OptKnock etc., `is_mcs_computation == False`, `:352-373`) the primary objective +For a bilevel problem (OptKnock etc., `is_mcs_computation == False`) the primary objective is a *production* objective, not cost, so BEST does the same fix-and-reminimize trick as ANY but around the **global** optimum: `fixObjective(c_bu, opt)` pins the optimal production value, `setMinIntvCostObjective` switches to minimizing knockouts, `setTargetableZ(z)` restricts to the found subspace, and the inner @@ -3580,10 +2812,10 @@ loop enumerates minimal-intervention designs that all achieve the optimal produc The user wants **all equally-optimal designs at each cost level** — the exhaustive enumeration used for the correctness gates (e_coli_core = 455 MCS, iML1515 393 gene-MCS). The objective setup is the same as BEST (optimize, then fix the optimal value), but instead of extracting one solution per solve it calls -the solver's **native solution pool** via `populateZ` (`:221-237`) → `populate` (`solver_interface.py`). +the solver's **native solution pool** via `populateZ` → `populate` (`solver_interface.py`). -For pure MCS (`:571`), the cost objective is already installed, so `enumerate` goes straight to -`populateZ(remaining)`. For bilevel (`:571-580`) it first optimizes the production objective, fixes it, +For pure MCS, the cost objective is already installed, so `enumerate` goes straight to +`populateZ(remaining)`. For bilevel it first optimizes the production objective, fixes it, and swaps to the cost objective — then populates. ```python @@ -3595,7 +2827,7 @@ for i in range(z.shape[0]): self.add_exclusion_constraints(z[i]) # drop invalid, still exclude ``` -`populateZ` (`:221-237`) collects the whole pool, rounds the binary blocks, and **deduplicates by +`populateZ` collects the whole pool, rounds the binary blocks, and **deduplicates by support** (two pool members with identical `z.indices` are the same design even if their continuous tails differ — the same cut set can be certified by different Farkas rays / flux distributions). The pool is configured to contain **only equally-optimal** members (pool gaps set to ~0, §8.6), so one @@ -3613,12 +2845,12 @@ All three modes are iterative: find a design, exclude it, repeat until infeasibl the **minimality** and **distinctness** guarantees are actually enforced, via two different exclusion constraints chosen by whether the found design is valid. -#### 8.4.1 The superset-excluding cut — `add_exclusion_constraints` (`:162-181`) +#### 8.4.1 The superset-excluding cut — `add_exclusion_constraints` Given a found binary design `z*` with support $K = \{i : z^*_i = 1\}$, $|K| = k$, this routine handles three cases: -**Case $k \ge 2$ (the classic no-good / integer cut, `:177-181`):** +**Case $k \ge 2$ (the classic no-good / integer cut):** $$\sum_{i \in K} z_i \le k - 1$$ @@ -3636,7 +2868,7 @@ the minimality guarantee. (The PROTECT constraints mean a superset is not *autom the MILP, but excluding it is still correct and keeps the enumeration to minimal designs; the inner subspace minimization is what ensures we found the *minimal* member of that up-set before cutting it.) -**Case $k = 1$ (single-reaction cut, `:172-175`):** +**Case $k = 1$ (single-reaction cut):** ```python interv_idx = int(z[i].indices[0]) @@ -3651,13 +2883,13 @@ superset containing `i*`* — same up-set semantics as the `k≥2` cut, but impl than a row, so it does not grow the constraint matrix. A size-1 MCS means reaction `i*` alone suffices; no design containing `i*` can ever be minimal-and-new, so banning `i*` outright is exactly right. -**Case $k = 0$ (empty design, `:166-170`):** adds the row $\sum_i z_i \le -1$, which is **infeasible** for +**Case $k = 0$ (empty design):** adds the row $\sum_i z_i \le -1$, which is **infeasible** for any nonnegative `z`. This deliberately makes the MILP infeasible to force clean termination. It is only reachable in degenerate setups (the "no interventions needed" case is caught earlier by the `verify_sd` -of the all-zero design at `:322`/`:429`/`:548`); the guard is defensive — some solvers reject genuinely +of the all-zero design at//); the guard is defensive — some solvers reject genuinely empty constraint rows, so a `-1` rhs is used rather than an empty row. -#### 8.4.2 The exact-pattern cut — `add_exclusion_constraints_ineq` (`:183-198`) +#### 8.4.2 The exact-pattern cut — `add_exclusion_constraints_ineq` Sometimes we must exclude *exactly* `z*` but **not** its supersets: @@ -3680,7 +2912,7 @@ both **complete** (no valid design lost) and **minimal** (no non-minimal design | valid, minimal-in-subspace | `verify_sd` ✓ | `add_exclusion_constraints` | `z*` **and all supersets** | | invalid (relaxation artifact) | `verify_sd` ✗ | `add_exclusion_constraints_ineq` | **exactly** `z*` | -You can see the branch explicitly in `compute` (`:484-490`) and `compute_optimal` (`:365-371`): valid → +You can see the branch explicitly in `compute` and `compute_optimal`: valid → superset cut + record; invalid → exact cut, no record. ### 8.5 `verify_sd`: re-checking validity in the true continuous subsystem @@ -3721,7 +2953,7 @@ feasible" (`slim_solve` not NaN). edge cases, drop a knockout the certificate needed, producing a `z` the MILP's relaxation still accepts but that is not truly valid. Re-verification is the guard that routes such a `z` to the exact-pattern cut. -3. **The all-zero pre-check.** At the top of each mode (`:322`, `:429`, `:548`) `verify_sd` is called on +3. **The all-zero pre-check.** At the top of each mode `verify_sd` is called on the empty design `csr_matrix((1, num_z))`; if the untouched strain already satisfies the modules, no interventions are needed and the mode returns `[{}]` immediately. @@ -3755,7 +2987,7 @@ of gap). Do not confuse this with the `1e-9` values that *are* set: those are `O #### 8.6.2 The solution-pool parameters are inert for single `solve` The CPLEX pool parameters `mip.pool.intensity = 4`, `mip.pool.absgap = 0`, `mip.pool.relgap = 0` -(`cplex_interface.py`), and the Gurobi `PoolGap`/`PoolGapAbs = 1e-9` (`:162-163`), only take +(`cplex_interface.py`), and the Gurobi `PoolGap`/`PoolGapAbs = 1e-9`, only take effect during pool generation (`populate_solution_pool` / `PoolSearchMode = 2`). During an ordinary `solve` — which is all ANY and BEST ever call — the pool stays empty and these settings do nothing. They matter **only for POPULATE**, where `intensity = 4` (CPLEX's most aggressive pool search) and @@ -3767,7 +2999,7 @@ additionally flips `PoolSearchMode = 2`, `NumericFocus = 2` on entry and resets #### 8.6.3 Seed → branch-and-bound tree shape → why speed needs a distribution The `seed` flows from the SD problem to the backend and lands on `randomseed` (CPLEX, -`cplex_interface.py`), `Params.Seed` (Gurobi, `:157`), and `randomization/randomseedshift` (SCIP, +`cplex_interface.py`), `Params.Seed` (Gurobi), and `randomization/randomseedshift` (SCIP, `scip_interface.py`). If the user gives no seed, each backend draws one from `[0, 2^16)` and logs it — so *even an unseeded run is reproducible after the fact*, given the logged seed. @@ -3789,47 +3021,32 @@ The `_trim_z_variables` step (`strainDesignMILP.py`) is a determinism-adjacent o worth noting: it physically removes non-knockable (`ub=0`, `cost=0`) binary columns from the matrices before the solver sees them, shrinking the binary count and keeping the B&B tree from carrying dead variables. Solutions are expanded back to the original `z`-space afterward (`_expand_z_to_orig`, -`:151-160`). +). -### 8.7 Verified performance: the phase timeline and CPLEX vs Gurobi +### 8.7 Enumeration performance and the preprocessing boundary -For the canonical **iML1515 gene-MCS** problem (SUPPRESS biomass ≥ 0.001, POPULATE, `max_cost = 3`, -gene KOs) yielding **393 MCS** (package v1.18): +An older end-to-end run of the canonical **iML1515 gene-MCS** problem (SUPPRESS biomass ≥ 0.001, +POPULATE, `max_cost = 3`, gene KOs) returned **393 MCS** and showed that exhaustive solution-pool +search can dominate total runtime. That run recorded CPLEX at 1241 s and Gurobi at 280 s, but it +predates the current preprocessing implementation and used only one seed. Treat those values as +historical evidence about the importance of pool enumeration, not as a current solver ratio or +preprocessing benchmark. -| Phase | Time | Notes | -|---|---|---| -| Preprocessing: blocked/irreversible FVA | **~117 s** | solver-agnostic, one-time | -| MILP build | **~4 s** | matrix assembly + `link_z` | -| Populate (enumeration) | **~1101 s** (CPLEX) | dominates | -| **Total** | **CPLEX 1241 s / Gurobi 280 s (≈4.4×)** | | - -For **e_coli_core** (455 MCS) the whole thing is **~1.2 s** on CPLEX — small enough that phase structure -is irrelevant. - -**Interpretation.** On iML1515, preprocessing FVA (~117 s) and build (~4 s) are essentially fixed costs -independent of the MILP solver; they are ~10 % of the CPLEX total. The remaining **~89 %** is the -**pool search** inside `populate`. So the thing that dominates genome-scale enumeration is *not* solving -a single MILP to optimality — a single feasibility or optimality solve is comparatively quick — it is -**exhaustively filling the solution pool at each cost level**: the solver must, after finding the optimal -cost, keep branching to enumerate *every* tied design and prove there are no more. That is intrinsically -harder than a single optimize, and it is where CPLEX and Gurobi diverge: Gurobi's pool search -(`PoolSearchMode = 2`) closes this instance ~4.4× faster than CPLEX's `populate_solution_pool` at -`intensity = 4`. The preprocessing FVA ([Ch 5](#ch5)) is the second-largest lever and, being solver-agnostic, is -where portable speedups live; the pool search is a solver-quality question. - -Because this 4.4× is a **single-seed** figure, per §8.6.3 it should be read as "Gurobi is materially -faster here", not as a precise constant — reproduce across seeds before quoting it as a benchmark. +The current preprocessing-only profile is maintained in [Ch 11](#ch11): approximately 19.6 s on +the profiled Gurobi setup, dominated by sign/FVA queries and compression, with `SDMILP` construction +below one second. End-to-end solver comparisons must report preprocessing and enumeration separately, +use the same preprocessed problem, verify the decompressed MCS set, and run multiple seeds. **The discredited "big-M / indicators-catastrophic" dead-end.** An earlier performance hypothesis held that native **indicator constraints** were catastrophically slow at genome scale and that forcing a global **big-M** reformulation would fix it. This was investigated and **discredited** — do not repeat -it. Two reasons: (1) The dominant cost is pool enumeration (~89 % above), *not* the LP relaxation of the -z-linking, so swapping the linking mechanism cannot address the actual bottleneck. (2) Indicator +it. Two reasons: (1) exhaustive pool enumeration can dominate an end-to-end run, so swapping the +linking mechanism does not address that cost. (2) Indicator constraints give a **tighter** LP relaxation than big-M ([Ch 7](#ch7)) — a valid big-M must be large enough to never spuriously bind, which loosens the relaxation and generally *hurts* branch-and-bound, the opposite -of the hypothesis. Recall also (CONTEXT §, [Ch 7](#ch7)) that under the default `M = inf`, SUPPRESS's unbounded -Farkas-dual rows *become* indicator constraints and PROTECT's finite-flux primal rows *become* big-M -**emergently** from the bound structure in `link_z` — there is no per-module type switch to "fix". The +of the hypothesis. Under the default `M = inf`, multi-continuous-variable rows become indicators, +while zero- and single-variable rows can use a finite relaxation read directly from their bounds; +there is no per-module type switch to "fix". The lever that actually moves genome-scale time is faster pool search (solver choice) and cheaper preprocessing FVA, not the linking encoding. @@ -3865,7 +3082,7 @@ proved unstable. The MILP does not run on the model the user handed to `compute_strain_designs`. By the time `SDMILP` is built ([Ch 7](#ch7)), the network has passed through two lossless compression rounds (COMPRESS #1 before GPR integration, COMPRESS #2 after — [Ch 3](#ch3)), an optional GPR extension that turned genes -into pseudoreactions ([Ch 4](#ch4)), and three FVA passes that pruned essential reactions and pulled out +into pseudoreactions ([Ch 4](#ch4)), and several sign/FVA jobs that prune essential reactions and pull out size‑1 minimal cut sets ([Ch 5](#ch5)). The binary intervention variables `z` therefore index **compressed reactions of the GPR‑extended model**, not the original reactions or genes the user cares about. @@ -3966,7 +3183,7 @@ Parallel reactions carry flux in fixed proportion because their `S`‑columns ar one another; they are, metabolically, redundant routes for the same conversion. To *suppress* the group you must remove **every** knockable member — leaving any one open leaves the conversion possible. So expansion produces **one** design that knocks out all knockable members -(`networktools.py‑1504`): +(`networktools.py`): ```python if par_reac_cmp: @@ -3988,7 +3205,7 @@ Coupled (flux‑coupled) reactions must all carry flux together in every steady = 0` for members of the group. Therefore killing **any single** member forces the whole group to zero. Cutting the group is not "cut them all"; it is "cut *one*, your choice." Each choice is a distinct, minimal strain design, so expansion **branches** — it emits one new design per knockable -member (`networktools.py‑1510`): +member (`networktools.py`): ```python else: # coupled @@ -4009,7 +3226,7 @@ one representative) is what lets the user pick the intervention that is easiest Knock‑ins mirror the KOs with parallel/coupled swapped, because "adding capability" is dual to "removing it": -- **Parallel KI** (`networktools.py‑1520`): the parallel members are interchangeable routes, so +- **Parallel KI** (`networktools.py`): the parallel members are interchangeable routes, so adding *any one* suffices. Expansion branches — one design per KI‑able member — and, in each branch, explicitly marks the *other* members as **not added** with value `0.0`: @@ -4027,7 +3244,7 @@ Knock‑ins mirror the KOs with parallel/coupled swapped, because "adding capabi The `0.0` tags are not cosmetic — they carry the "this KI candidate existed and was deliberately left out" information that §9.5 and `strip_non_ki` depend on. -- **Coupled KI** (`networktools.py‑1526`): coupled members only carry flux together, so a +- **Coupled KI** (`networktools.py`): coupled members only carry flux together, so a functional insertion must add **all** of them; expansion emits **one** design that knocks in every KI‑able member. @@ -4035,7 +3252,7 @@ Knock‑ins mirror the KOs with parallel/coupled swapped, because "adding capabi A compressed id may appear in the design with value `0` — a KI candidate the solver decided *not* to use (§9.5). Expansion propagates that "not added" verdict to every member of the group -(`networktools.py‑1532`): +(`networktools.py`): ```python elif val == 0: # KI that was not introduced @@ -4070,14 +3287,14 @@ decision in §9.4. ### 9.3 Size‑1 MCS re‑injection -Recall from [Ch 5](#ch5) that FVA #3 (`compute_strain_designs.py‑491`) finds reactions that are +Recall from [Ch 5](#ch5) that the final module FVA finds reactions that are **essential for the SUPPRESS behaviour but not for any PROTECT behaviour** — i.e. reactions whose sole knockout already makes the undesired flux infeasible while keeping the desired flux feasible. These are size‑1 minimal cut sets. They are deliberately **removed from the knockable set before the MILP is built** (`cmp_ko_cost.pop(r, None)` at `compute_strain_designs.py`) and stored separately: ```python -cmp_size1_mcs = [{r: -1} for r in size1_mcs_knockable] # compute_strain_designs.py:481 +cmp_size1_mcs = [{r: -1} for r in size1_mcs_knockable] # compute_strain_designs.py ``` The rationale ([Ch 5](#ch5)) is twofold: they need no search, and — more importantly — leaving them in the @@ -4085,9 +3302,9 @@ MILP would let the enumerator report every *superset* that contains a size‑1 M non‑minimal. Pulling them out keeps the MILP's minimal‑cut‑set guarantee clean. But they are still real solutions, so decompression must add them back. Note this happens only for **classical MCS problems** (exactly one SUPPRESS + only PROTECT modules — the `is_classical_mcs` gate at -`compute_strain_designs.py‑475`); bilevel problems (OptKnock etc.) never populate `cmp_size1_mcs`. +`compute_strain_designs.py`); bilevel problems (OptKnock etc.) never populate `cmp_size1_mcs`. -Re‑injection runs after the MILP designs have been expanded (`compute_strain_designs.py‑712`). +Re‑injection runs after the MILP designs have been expanded (`compute_strain_designs.py`). Each stored size‑1 MCS `{r:-1}` is itself a compressed design — `r` is a compressed reaction id — so it goes through the *same* `expand_sd` + `filter_sd_maxcost` pipeline (one size‑1 compressed cut can still fan out to several originals if `r` is a lumped reaction). It is then de‑duplicated against the @@ -4117,7 +3334,7 @@ details: fan‑out back to "one decision per group" for display. - **Status promotion.** If the MILP itself found nothing (INFEASIBLE) but size‑1 MCS exist, the status is lifted to OPTIMAL so the result is not reported as "no solution" (`compute_strain_designs.py`). - The `dump_preprocessed` early‑return path (`compute_strain_designs.py‑592`) uses the same + The `dump_preprocessed` early‑return path (`compute_strain_designs.py`) uses the same expand→filter→postprocess sequence to return size‑1 MCS even when the MILP solve is skipped entirely. ### 9.4 `filter_sd_maxcost`: why a post‑expansion cost re‑check is mandatory @@ -4127,7 +3344,7 @@ details: can change a design's effective cost, in both directions, so a compressed design that was within budget can expand into original‑model designs that are not — and vice versa. -The reason is that `compress_ki_ko_cost` (`networktools.py‑1410`) does not preserve cost +The reason is that `compress_ki_ko_cost` (`networktools.py`) does not preserve cost additively; it collapses a group's member costs to a single number using rules that are correct for the *group* decision but lossy about the *members*: @@ -4135,7 +3352,7 @@ the *group* decision but lossy about the *members*: costs only as much as cutting its cheapest member (you only need one). - **parallel KO cost** = `sum` of member KO costs (`networktools.py`) — because you must cut them all. -- **coupled KI cost** = `sum`; **parallel KI cost** = `min` (`networktools.py,1409`) — the duals. +- **coupled KI cost** = `sum`; **parallel KI cost** = `min` (`networktools.py`) — the duals. Now cross this against §9.2's expansion. A **coupled KO** was compressed at cost `min`, but expansion branches into one design *per member*, and each branch's true cost is *that member's* KO cost — which @@ -4147,7 +3364,7 @@ survives — the cost‑5 sibling is filtered out. Without the re‑check we wou design. `filter_sd_maxcost` recomputes the true cost in original space and keeps designs within a small -tolerance of the budget (`networktools.py‑1554`): +tolerance of the budget (`networktools.py`): ```python if max_cost: @@ -4163,7 +3380,7 @@ count toward cost.** A KI candidate left un‑made carries value `0` and is free `(nan,nan)` / value‑0 encoding of §9.5, and it is why that encoding must survive expansion rather than being stripped early. Second, it costs each *original* reaction independently with the *uncompressed* cost dicts `uncmp_ko_cost` / `uncmp_ki_cost` (assembled in the orchestrator and, for gene problems, -merged with gene costs at `compute_strain_designs.py‑422`) — never the compressed dicts. Third, +merged with gene costs at `compute_strain_designs.py`) — never the compressed dicts. Third, the surviving designs are **sorted by ascending true cost** via a throwaway `'**cost**'` key, so the cheapest realisations surface first; in the lazy path (below) this ordering is what makes `expanded[0]` the "cheapest representative" of a group (`compute_strain_designs.py`). @@ -4173,7 +3390,7 @@ cheapest realisations surface first; in the lazy path (below) this ordering is w For problems where the fan‑out is enormous — many deep coupled groups multiplying together — materialising every expanded design would exhaust memory even though the search itself finished (this is issue #47, noted in `SDSolutions.save`). `_decompress_solutions` guards against this -(`compute_strain_designs.py,654‑681`): +(`compute_strain_designs.py`): ```python LAZY_EXPANSION_THRESHOLD = 100_000 @@ -4187,10 +3404,10 @@ if estimated > LAZY_EXPANSION_THRESHOLD: `_build_lazy_representatives` (`compute_strain_designs.py`) expands each compressed group *just far enough* to keep **one** representative — the cheapest survivor of `expand_sd` + `filter_sd_maxcost` — and records the machinery (the compressed designs, the map, the uncompressed cost dicts, the model) -in an `_expansion_meta` dict on the `SDSolutions` (`compute_strain_designs.py‑677`). The result +in an `_expansion_meta` dict on the `SDSolutions` (`compute_strain_designs.py`). The result reports `get_num_sols` as the *estimated total* while only a handful are materialised (`get_num_materialized`), and the user can force any group's full expansion on demand via -`expand_group` / `expand_all` (`strainDesignSolutions.py,520`), which run the identical +`expand_group` / `expand_all` (`strainDesignSolutions.py`), which run the identical expand→filter→translate pipeline lazily. This is a pure space/time optimisation — the eager and lazy paths compute the same designs; lazy just defers the combinatorial blow‑up until (if ever) the user asks for it. @@ -4209,7 +3426,7 @@ numeric value with a fixed meaning: | `False` | regulatory intervention not added | — | | *(absent)* | reaction never a candidate | — | -The value originates in `sd2dict` (`strainDesignMILP.py‑213`), which reads the solved binary +The value originates in `sd2dict` (`strainDesignMILP.py`), which reads the solved binary vector. A `z` variable is *inverted* iff it is a KI candidate — `z_inverted[i] = not isnan(ki_cost[i])` (`strainDesignProblem.py`). For a non‑inverted (KO) variable, `z=1` means "apply the cut", written as `-sol = -1`; for an inverted (KI) variable, `z=1` means "insert", written as `+sol = +1`. The @@ -4233,7 +3450,7 @@ several steps downstream need to tell them apart: group's members, so that a compressed un‑made KI does not silently reappear as made after expansion. 2. **Cost correctness.** `filter_sd_maxcost` charges only `v != 0` interventions; an un‑made KI must be present‑but‑free, which requires it to be present with value `0`, not absent. -3. **Bounds semantics.** `_compute_costs_and_bounds` (`strainDesignSolutions.py‑255`) turns value +3. **Bounds semantics.** `_compute_costs_and_bounds` (`strainDesignSolutions.py`) turns value `0` into bounds `(nan, nan)` — a deliberate "no bound change; this capability was considered and declined" marker, distinct from a KO's `(0,0)` and from an added KI's real bounds. @@ -4245,7 +3462,7 @@ def strip_non_ki(sd): return {k: v for k, v in sd.items() if v not in (0.0, False)} ``` -The public accessors `get_reaction_sd` and `get_gene_sd` (`strainDesignSolutions.py,330`) pass +The public accessors `get_reaction_sd` and `get_gene_sd` (`strainDesignSolutions.py`) pass every design through `strip_non_ki`, so the user sees only interventions that were *actually made*. The un‑stripped forms remain available through `get_reaction_sd_mark_no_ki` / `get_gene_sd_mark_no_ki` for callers that need the full picture. This "internal representation keeps @@ -4268,7 +3485,7 @@ disabled. A reaction is governed by its **gene–protein–reaction (GPR) rule** expression over genes (e.g. `(b0001 and b0002) or b0003`). The previous implementation re‑parsed these rules into disjunctive normal form and evaluated a hand‑rolled `gpr_eval`. The current code instead reuses cobra's already‑parsed GPR abstract syntax tree and its evaluator -(`strainDesignSolutions.py‑161`): +(`strainDesignSolutions.py`): ```python rxn_gpr = {r.id: r.gpr for g in model.genes for r in g.reactions} @@ -4281,7 +3498,7 @@ convention**: `eval` treats every gene *listed* in `knockouts` as off and **ever present/active**. So you drive it entirely through which genes you place in the knockout set. The translation exploits this by evaluating each reaction's GPR under three different knockout sets, to -answer three distinct phenotype questions (`strainDesignSolutions.py‑195`): +answer three distinct phenotype questions (`strainDesignSolutions.py`): ```python ko_off = gene_ko | gene_no_ki # KOs applied; un-made KIs off; made KIs on @@ -4313,14 +3530,14 @@ Reading the three comparisons: else (typically an un‑made KI it depended on) → reaction "not added" (`reac_no_ki`, value `0`). Only reactions attached to an intervened gene are examined (`candidate_reacs` is built from the union -of the gene KO/KI/no‑KI sets, `strainDesignSolutions.py‑185`) — every other reaction is untouched +of the gene KO/KI/no‑KI sets, `strainDesignSolutions.py`) — every other reaction is untouched by definition, so evaluating it would waste time and could only return "unchanged." The output preserves the §9.5 encoding on the reaction side: `-1.0` for `reac_ko`, `+1.0` for `reac_ki`, `0.0` for `reac_no_ki`, plus `True`/`False` for regulatory interventions -(`strainDesignSolutions.py‑200`). The gene‑level view (`gene_sd`) is kept verbatim from the raw +(`strainDesignSolutions.py`). The gene‑level view (`gene_sd`) is kept verbatim from the raw solution dicts (`strainDesignSolutions.py`), including any gene‑name→gene‑id normalisation -(`strainDesignSolutions.py‑154`), so the two views stay linkable via `get_gene_reac_sd_assoc` +(`strainDesignSolutions.py`), so the two views stay linkable via `get_gene_reac_sd_assoc` (the association is typically many gene sets → one reaction phenotype, since different gene KOs can disable the same reactions). @@ -4361,15 +3578,15 @@ mechanisms, either of which can leave a knockable-but-inert gene in the problem. > that match the reported id/name signature and remain worth hardening — not as a bug with a known fixing > commit. The issue stays open awaiting the reporter's exact failing `gene_sd`. -#### Mechanism 1 — `reduce_gpr` pops protected/essential genes by **id only** +#### Mechanism 1 — `reduce_model_gprs` pops protected/essential genes by **id only** -`reduce_gpr` (`networktools.py`) is the pre-GPR-integration pass that removes genes which cannot +`reduce_model_gprs` (`compute_strain_designs.py`) is the pre-GPR-integration pass that removes genes which cannot usefully be knocked out — genes that only touch essential reactions, or that are essential to an essential reaction — so they never become MILP binary variables (see [Ch 4](#ch4) for the full GPR-reduction role). It builds a `protected_genes` set (steps 2–3), and then, in step 4: ```python -# line 904 +# compute_strain_designs.py, protected-gene KO-cost drop [gkos.pop(pg.id) for pg in protected_genes if pg.id in gkos] ``` @@ -4382,25 +3599,25 @@ The asymmetry is visible one line later. Step 5 protects "all genes that are not and *this* line is name-aware: ```python -# line 907 — note: id OR name +# compute_strain_designs.py, name-aware protected-gene handling — note: id OR name [protected_genes.add(g) for g in model.genes if (g.id not in gkos) and (g.name not in gkos)] ``` Likewise step 6 restores knock-in candidates by matching *either* `g.id in gkis` or -`g.name in gkis`. So `reduce_gpr` knows perfectly well that `gkos`/`gkis` may be name-keyed — every +`g.name in gkis`. So `reduce_model_gprs` knows perfectly well that `gkos`/`gkis` may be name-keyed — every membership *test* checks both id and name — but the one place it *mutates* `gkos`, the `.pop` at line 904, uses `pg.id` alone. That is the fragility: a single un-mirrored key access in an otherwise id-or-name-tolerant function. -The downstream effect compounds through the rest of `reduce_gpr`. `protected_genes_dict` is keyed by +The downstream effect compounds through the rest of `reduce_model_gprs`. `protected_genes_dict` is keyed by `pg.id` and fed to `simplify_gpr_ast`, which rewrites each reaction's GPR treating protected genes as constant-`True` and **deletes them from the Boolean rule**; then step 8 removes -protected genes from `model.genes` entirely. So after `reduce_gpr` a name-keyed essential +protected genes from `model.genes` entirely. So after `reduce_model_gprs` a name-keyed essential gene can be in an inconsistent state: still present as a cost entry in `gkos` (because the pop missed it), but scrubbed out of the GPRs and the gene list. When `extend_model_gpr` then builds gene pseudoreactions from `model.genes` ([Ch 4](#ch4)), that gene has no pseudoreaction to attach a `z` to — the intervention is declared but wired to nothing, i.e. a neutral gene KO. **Fix direction:** pop by id *and* name, -mirroring the membership tests already used at 907/910. +mirroring the membership tests already used for the protected-gene drop. #### Mechanism 2 — `_translate_genes_to_reactions` evaluates the GPR only over solution-present genes @@ -4412,7 +3629,7 @@ and its `.eval` (`rxn_gpr = {r.id: r.gpr ...}`; the AST evaluator replaced the o `gpr_eval`, per PR #51): ```python -# lines 187–195 (paraphrased structure) +# paraphrased structure if gpr_r.eval(ko_off): # reaction still possible under the interventions if not gpr_r.eval(all_off): # ... only because of a knock-in → it's an effective KI reac_ki.add(r) @@ -4440,7 +3657,7 @@ pruning that is supposed to remove inert genes upstream. #### The id-vs-name fragility, end to end -Beyond `reduce_gpr`, the id/name split threads through several stages and is the reason "names break, ids +Beyond `reduce_model_gprs`, the id/name split threads through several stages and is the reason "names break, ids work" is a plausible signature: - **Pseudoreaction vs. pseudometabolite naming diverge.** In `extend_model_gpr`, when `use_names=True` the @@ -4449,13 +3666,14 @@ work" is a plausible signature: - **Name→id remap happens inside the translator, not before.** `_translate_genes_to_reactions` builds `gene_name_id_dict` and rewrites name keys to id keys on its *working copy*, but `gene_sd` keeps the original (possibly name) keys. Two dicts, two key spaces, kept only loosely in sync. -- **Truncation is solver-dependent** (§10.5b): long lumped names are sha256-truncated for Gurobi/GLPK but - not CPLEX, so a name that is a valid key on CPLEX can be a *different* (hashed) key on Gurobi — id-keyed - runs sidestep this because ids are short. +- **Truncation applies on every backend** (§10.5): `extend_model_gpr` truncates any generated name past + `MAX_NAME_LEN` to a sha256-suffixed form regardless of solver, so a long name is the *same* hashed key + everywhere — id-keyed runs sidestep the length problem entirely because ids are short. The practical takeaway: whenever you touch gene-keyed logic, test with `gko_cost` keyed **both** ways and -assert the two runs produce identical designs. That equivalence is precisely the regression assertion the -investigation recommended and that no existing test yet enforces. +assert the two runs produce identical designs. That equivalence is enforced by +`test_gene_names_equivalent_to_ids_no_neutral_kos` in `tests/test_10_gene_design_validity.py`, which also +asserts that no neutral gene KOs appear. ### 10.2 Issue #38 (OPEN) — superset/subset (non-minimal) solutions @@ -4473,7 +3691,7 @@ dicts (and as `(nan, nan)` bounds in `itv_bounds`); a made KI is `+1`, a KO is ` value/`strip_non_ki` semantics). The user-facing accessors hide the value-0 entries: ```python -# strainDesignSolutions.py:768 +# strainDesignSolutions.py def strip_non_ki(sd): return {k: v for k, v in sd.items() if v not in (0.0, False)} ``` @@ -4551,14 +3769,17 @@ PROTECT-violating designs on the reporter's setup; current code produces 0 acros 2. **The blind spot: the existing tests were cardinality-only.** `test_05` (`mcs_gpr`) and `test_08` asserted the *number* of solutions, never that each returned design actually satisfies its PROTECT modules on the original model. A bug that returns the right *count* of *wrong* designs sails straight - through. The guard that would have caught #44 — and must be added as a standing regression test — is: + through. The guard that would have caught #44 is: **re-evaluate every returned design against every PROTECT module on the ORIGINAL (uncompressed, un-extended) model**, by re-applying the gene/reaction interventions via cobra's own GPR knockout and solving, and assert feasibility. This is a different assertion class from cardinality, and it is the - single test most likely to catch any regression of the whole "compressed phantom flux" family. Note the - coupled-merge fix `d6f3d28` shipped without a *targeted* unit test for the bound-intersection / - contradicting-group logic (its test additions were unrelated), so this coverage gap is still open at - both the compression-unit level and the end-to-end validation level. + single test most likely to catch any regression of the whole "compressed phantom flux" family. That + end-to-end guard now exists as `test_gene_kos_designs_satisfy_protect_and_suppress` in + `tests/test_10_gene_design_validity.py`, which re-reads the SBML, applies each design via cobra's + `knock_out()`, and asserts the PROTECT and SUPPRESS conditions. The *unit*-level gap remains: the + coupled-merge fix `d6f3d28` shipped without a targeted test for the bound-intersection / + contradicting-group logic (its test additions were unrelated), and `tests/test_07_compression.py` + still has none. ### 10.4 Gotcha (a) — `compute_strain_designs` mutates the caller's `reg_cost`/module dicts in place @@ -4569,7 +3790,7 @@ PROTECT-violating designs on the reporter's setup; current code produces 0 acros passed are safe. The cost dicts are **not** copied — they are aliased: ```python -# lines 225–234 +# compute_strain_designs.py, cost-dict aliasing if key == KOCOST: uncmp_ko_cost = value if key == KICOST: uncmp_ki_cost = value if key == REGCOST: uncmp_reg_cost = value # <-- the caller's dict, by reference @@ -4586,7 +3807,7 @@ mutates its argument dict in place to use those generated names. The orchestrato immediate (reaction-based) regulatory constraints: ```python -# lines 329–330 +# compute_strain_designs.py, regulatory-cost reset uncmp_reg_cost.clear() uncmp_reg_cost.update(_immediate_reg) ``` @@ -4604,31 +3825,13 @@ so they misroute (deferred as if gene-regulatory) or raise. The same aliasing me surface — there is a code comment acknowledging the in-place mutation, but the fix (copy the caller's dict on entry, as is already done for modules) has not been applied. -### 10.5 Gotcha (b) — Gurobi/GLPK-only name truncation (sha256; CPLEX exempt) +### 10.5 Gotcha (b) — deterministic truncation of long GPR gadget names -`extend_model_gpr` can generate very long pseudo-metabolite/pseudoreaction names, especially after -compression lumps many reactions into one ([Ch 3](#ch3)/[Ch 4](#ch4)): the lumped id is a `*`-joined concatenation of the -member ids and gene tags, easily exceeding a few hundred characters. To stay within solver name-length -limits, names longer than `MAX_NAME_LEN = 230` are hashed: +Long generated gene/metabolite/reaction identifiers are truncated for every solver using the same +prefix-plus-SHA-256 rule. This avoids the former cross-solver identifier mismatch. Downstream code +must still treat generated IDs as opaque: reconstructing them independently or matching only the +human-readable prefix can break cost lookup and decompression. -```python -# networktools.py:1001,1012–1014 -MAX_NAME_LEN = 230 -def truncate(id): - h = hashlib.sha256(id.encode()).hexdigest()[:20] - return id[0:MAX_NAME_LEN - 21] + "_" + h -``` - -The crucial detail is the **guard**: every truncation site fires only for `solver in {GUROBI, GLPK}` -. **CPLEX is exempt.** The consequence is that the *same -input model* produces *different reaction/metabolite identifiers* depending on which solver is selected: a -long name is preserved verbatim under CPLEX but replaced by `_` under Gurobi/GLPK. -That changes reaction/metabolite identity in logs and in any downstream lookup keyed by name — which is why -it is #43-adjacent: a name-keyed gene/reaction lookup that works on CPLEX can miss on Gurobi because the -key was hashed out from under it, and the reporter of #43 saw exactly the truncation warning. It also means -solver-to-solver diffs of the extended model are not name-comparable without accounting for truncation. -Ids, being short, never hit `MAX_NAME_LEN`, so id-keyed workflows are immune — a second reason the #43 -signature is "names break, ids work". ### 10.6 Gotcha (c) — solver numeric-status robustness (Gurobi 12 NUMERIC; CPLEX 5/6 unscaled-infeasibilities) @@ -4638,9 +3841,9 @@ gracefully. **Why these MILPs hit the numeric statuses.** The SUPPRESS blocks are Farkas infeasibility certificates ([Ch 6](#ch6)) whose dual variables are unbounded by nature and are anchored only by a normalization row, and the -`z`-linking mixes big-M rows with indicator rows ([Ch 7](#ch7)). Big-M constants derived from bounding LPs on an -ill-conditioned genome-scale network can span many orders of magnitude (the MILP-conditioning workstream -measured a ~9-order big-M range), giving the LP relaxation a badly scaled constraint matrix. Under such +`z`-linking mixes big-M rows with indicator rows ([Ch 7](#ch7)). A blanket finite M on an +ill-conditioned genome-scale network can be too large for some rows and too small for others, giving +the LP relaxation a badly scaled constraint matrix or changing the feasible set. Under such scaling the simplex/barrier can reach a point it believes optimal or feasible but whose *unscaled* residuals exceed tolerance — that is precisely CPLEX status 5/6 ("optimal/best with unscaled infeasibilities") and Gurobi status 12 (`NUMERIC`). These are not logic bugs; they are the expected @@ -4653,8 +3856,10 @@ problems (e.g. `ko_cost` on ~1600 reactions) rather than on small models. `NumericFocus = 3`**, restoring the previous value afterward. If the retry yields a solution it is accepted as `OPTIMAL`; if it yields an incumbent under time-limit-like status it is returned as `TIME_LIMIT_W_SOL`; otherwise it reports no solution (`TIME_LIMIT`) — never a crash. -- *CPLEX* (`cplex_interface.py`, and `slim_solve` at 250–255): status `5`/`6` is accepted with a +- *CPLEX* (`cplex_interface.py`): in `solve`, status `5`/`6` is accepted with a warning and mapped to `TIME_LIMIT_W_SOL` (the solution is used but flagged), rather than raising. + `slim_solve` treats the same statuses separately and more quietly: it returns the objective as a plain + float, with no warning and no status mapping, because it has no status channel to report on. The philosophy is *degrade, don't crash*: a numerically-imperfect incumbent is far more useful to an enumeration loop than an exception that discards the whole run. Note one residual rough edge: the SCIP/GLPK interfaces were flagged as likely to have analogous unhandled-status gaps that have not all been audited. Also relevant to @@ -4666,7 +3871,7 @@ fix trades a crash for occasionally accepting a marginally non-minimal design. - **Hard-coded essentiality tolerance `1e-10`.** Both essential-reaction FVA passes classify a reaction as essential with `np.min(abs(limits)) > 1e-10 and np.prod(np.sign(limits)) > 0` - (`compute_strain_designs.py` and `:465`) — the flux range must exclude zero by more than `1e-10` + (`compute_strain_designs.py` and) — the flux range must exclude zero by more than `1e-10` with a fixed sign. This absolute threshold has no relation to model scaling: a reaction that is biologically essential but whose minimal required flux is below `1e-10` will be missed (and remain wrongly knockable), while the ~`4e-7` growth-coupling boundary of §10.2 sits *above* the threshold and is @@ -4699,387 +3904,95 @@ fix trades a crash for occasionally accepting a marginally non-minimal design. (ch11)= ## 11. Performance, benchmarking & roadmap -This chapter is forward-facing. The rest of *StrainDesign Internals* explains how the pipeline works; -this one is a map for the developer who wants to make it **faster** without making it **wrong**. It -does three things: (1) pins down where wall-time actually goes at genome scale, with numbers, so that -optimization effort lands on real bottlenecks and not folklore; (2) enumerates the performance levers, -each grounded in that profile and in the mathematics of the formulation (see [Ch 6](#ch6), [Ch 7](#ch7)); and (3) lays -out the benchmarking discipline and the roadmap. Throughout, the governing constraint is -**completeness** — a Minimal Cut Set (MCS) computation must never silently drop a valid design ([Ch 8](#ch8), -[Ch 9](#ch9)), so every speedup is a claim that has to be gated against a known answer. +Performance work must preserve the complete design set. A faster preprocessing or MILP formulation is +not accepted on timing alone; it must pass a known-answer gate after decompression. -Two numbers to keep in your head, both measured on the canonical iML1515 gene-MCS run -(SUPPRESS `BIOMASS_Ec_iML1515_core_75p37M ≥ 0.001`, POPULATE, `max_cost=3`, `gene_kos=True`): -**CPLEX 1241 s, Gurobi 280 s**, both returning the identical 393 MCS. That ≈4.4× solver gap, and the -internal split of those seconds, is the spine of everything below. +### 11.1 Current iML1515 preprocessing profile -### 11.1 The verified bottleneck profile +The current PR branch was profiled on the canonical iML1515 single-SUPPRESS gene-MCS setup with +Gurobi, compression enabled and preprocessing dumped before enumeration. Total preprocessing was +about **19.6 s**. -All timings here were measured against the real solver APIs (package v1.18, CPLEX 22.1.2 / Gurobi 13.0.1) -on the canonical iML1515 393-MCS problem. State them as given; -re-measure before trusting anything not on this list. +| Phase | Time | Detail | +|---|---:|---| +| reversibility pre-tightening | ~6.32 s | 1281 LP solves ~4.72 s; temporary compression ~1.17 s; structural sweep ~0.05 s | +| folded final FVA | ~5.81 s | about 702 LP solves ~5.61 s | +| main compression passes | ~5.00 s | coupled work ~3.75 s; parallel ~0.81 s; conservation removal ~0.33 s | +| suppressed model copies | ~0.75 s | four `_CarrierSolver` copies, counted here rather than inside the phases that trigger them | +| `extend_model_gpr` | ~0.62 s | gadget construction without a live backend | +| `SDMILP` construction | ~0.48 s | includes `link_z`; no per-row bounding LP | +| module validation FBA | ~0.33 s | selected-solver validation | +| dump serialization | ~0.16 s | preprocessed pickle | +| GPR reduction/simplification | ~0.17 s | small relative to FVA/compression | -#### 11.1.1 Where the seconds go (canonical iML1515, CPLEX) +The rows are disjoint slices and sum to the stated total. The important result is the ordering: +**FVA/sign classification and compression dominate preprocessing; model copying and MILP construction +do not.** -| Phase | What it is | Time | -|---|---|---| -| Prepare/parse | modules, solver, costs, seed | ~7 s | -| COMPRESS #1 | 2712 → 1237 reactions (parallel + coupled, 5 iters) | 3.4 s | -| GPR preprocessing | 1516 genes → `extend_model_gpr` (model → 3448 reac) | ~1 s | -| COMPRESS #2 | after GPR extension, 3448 → 2152 reactions | 4.3 s | -| **`bound_blocked_or_irrevers_fva`** | whole-model bound-classifying FVA (the ~4300-LP sweep) | **117.4 s** | -| FVA essential + size-1 MCS | 88 size-1 MCS extracted via SUPPRESS-scoped FVA | 3.5 s | -| MILP build | Farkas dual assembly 2.7 s + `link_z` (536 indicators) 0.9 s | **3.7 s** | -| **Solve (POPULATE)** | pool search → 84 compressed solutions | **1101 s** | -| Decompress | `expand_sd` + maxcost filter + phenotype → 393 | ~1 s | -| **Total** | | **1241 s** | - -Three facts fall straight out of this table, and each one redirects a class of optimization effort: - -1. **The two costs that matter at genome scale are the preprocessing FVA (~117 s) and the solve/pool - search (~1101 s).** Together they are 98% of wall-time. Everything else — parse, both compressions, - GPR extension, size-1 MCS extraction, decompression — is single-digit seconds. Optimize the two big - phases; leave the rest alone unless it becomes structurally coupled to them. - -2. **MILP *construction* is now cheap (~4 s).** This was not always true: before PR #55 the build was - ~70 s, dominated by a scalar-loop `prevent_boundary_knockouts` (~51 s) and a non-deduplicated - `link_z` (~16 s). Vectorizing `prevent_boundary_knockouts` and hashing the `link_z` bounding-LP - dedup collapsed it to ~7 s, byte-identical output, and the exact-nullspace/build refinements since - have trimmed it further. **The lesson for the next optimizer:** the build phase has already been - wrung out; do not spend effort shaving milliseconds off matrix assembly. The money is in FVA and the - solve. - -3. **The 117 s FVA is a genuinely preprocessing cost, not a solve cost** — it is the whole-model - `bound_blocked_or_irrevers_fva` call (see [Ch 5](#ch5), §3.3), roughly `2n` single-reaction LPs with no - `reaction_list` scoping and no extra constraints. That structure is what makes it CPLEX's per-LP - overhead multiplied by ~4300, and it is why it is separately attackable from the pool search. - -#### 11.1.2 The CPLEX-vs-Gurobi ≈4.4× gap and its *true* causes - -The same 393-MCS problem runs in **CPLEX 1241 s vs Gurobi 279.8 s**. Decomposing both runs by phase -localizes the entire gap to exactly two places: - -- **Preprocessing FVA: ~117 s on CPLEX.** This is CPLEX's per-LP construction/solve overhead paid ~4300 - times over. Gurobi's per-LP overhead on the same sweep is materially lower. This is a *fixed tax per - LP*, so the fix is architectural (fewer LPs, parallelism, cheaper backend for the sweep — §11.2.5), - not a solver-parameter tweak. -- **Pool search (POPULATE): ~1101 s on CPLEX vs a small fraction of that on Gurobi.** CPLEX's - solution-pool enumeration runs ~4–7× slower than Gurobi's on this MILP. This is the dominant term and - the dominant contribution to the 4.4×. - -Everything else — the branch-and-bound on the incumbent-finding solves, the MILP build — is at rough -parity between the two solvers. So the correct one-sentence statement of the gap is: **the CPLEX -disadvantage is per-LP preprocessing overhead plus pool-search speed, and nothing else.** - -Three things the gap is emphatically **NOT**, each of which cost prior investigation time and is now -closed: - -- **NOT the indicator constraints.** Under the default `M = inf`, SUPPRESS's Farkas-dual rows become - indicator constraints and PROTECT's finite-flux primal rows become big-M rows — but this is emergent - from the bound structure via the `self.M`/bounding-LP fork in `link_z` (`strainDesignProblem.py`, the - finite-vs-`inf` `max_Ax` test around line ~853), **not** a per-module-type switch ([Ch 7](#ch7), §3.2). Both - solvers get the *same* formulation with the same indicators, and both handle those indicators fine. - The indicators are not the gap. - -- **NOT the pool parameters.** CPLEX sets `mip.pool.absgap=0`, `mip.pool.relgap=0`, - `mip.pool.intensity=4` at solver construction (`cplex_interface.py`), and Gurobi sets - `PoolGap=PoolGapAbs=1e-9` (`gurobi_interface.py`). These have been dated by `git blame` to - 2022 (CPLEX line `b87d49c1`, 2022-04-18 — not a recent regression) and, more importantly, **verified - inert for single `solve`**: after a feasibility solve at `intensity=4`, `pool.get_num==0`, - identical to `intensity=0`. CPLEX does not populate the pool during a plain `optimize`; the pool - params only bite inside `populate` (POPULATE). They are architecturally misplaced (they belong - inside `populate`), but they are **not a performance bug for ANY/BEST**. Do not re-derive this — it - was tested three ways. - -- **NOT a big-M conditioning catastrophe.** A discredited earlier reading claimed "CPLEX 400 s / - indicators catastrophic / use big-M." That number came from calling `backend.solve` on the MILP's - *construction* objective — a global optimization that no production path ever runs — on a self-made - iML1515/1,4-BDO/`max_cost=40` dump with 2228 indicators and a loose cardinality bound. It is not - representative of any real run and has been thrown out. **The dead-end to remember:** there is no - 9.4-order big-M range in the built MILP to fix. As the MILP roadmap verified (§0–§1), the shipped - formulation carries only a few dozen big-M rows, all at the loose default ±1000 (e.g. iMLcore: 34 - big-M / 388 indicators), because the wide-flux-span reactions all relax to ±inf bounds and become - *indicators*, not tiny big-M's. Equilibration of a big-M range that does not exist is moot. - -The practical upshot: **do not chase the solver gap through solver knobs or the indicator/big-M -dichotomy.** The gap lives in the *number of LPs* in preprocessing and in *pool-search throughput*. -Fix those structurally. - -### 11.2 The performance levers - -The levers below are grouped and ordered to match the profile: compression (cuts the problem before it -is built), formulation/conditioning (shapes the MILP the solver sees), skipping hopeless work, the -Farkas-dual pre-bounding problem, the preprocessing FVA, and the enumeration strategy. This list -reflects informed intuition, not a ranked plan — argue with it, and measure before committing effort. -Phil's standing prior: the biggest *suspected* structural win is a better MILP formulation/conditioning -(group 2), solver parameters (group 4-adjacent) are a fragile secondary bet, and the "good compression -≈ MCS2" insight (group 1) is a **hypothesis to verify**, not a foundation to build on. - -#### 11.2.1 Compression depth = rank / z-count reduction (the structural lever) - -The binary variable count `num_z = numr` — one `z` per compressed reaction (`strainDesignProblem.py` -`__init__`, `num_z` set around line ~144) — is the dominant complexity driver of the MILP. Branch and -bound over `z` is combinatorial; halving `numr` is worth far more than any constant-factor solver tune. -Network compression ([Ch 3](#ch3)) is the mechanism that reduces `numr` losslessly and exactly, and it is -therefore the single largest structural lever available. - -The reasoning is that compression is a **rank/dimension reduction of the flux system done for free**: -parallel merge, coupled/flux-coupled merge, conservation-relation (row) removal, and blocked/zero-flux -removal each shrink `S` while preserving the exact set of steady-state flux distributions (the exact -integer/rational nullspace guarantees this — never float; see [Ch 3](#ch3) and the hard constraint). Every -reaction removed is a `z` never created, an LP row never linked, a branch never taken. On the canonical -run, COMPRESS #1 takes 2712 → 1237 and COMPRESS #2 takes 3448 → 2152 (after GPR extension inflates the -count); pushing either merge closer to a true fixpoint directly removes binaries. - -Concrete sub-levers, in decreasing certainty: - -- **Scaled-parallel merging** (shipped, PR #54): merge reactions whose stoichiometry is identical *up - to any rational scalar* and that share reversibility/bound topology. This is strictly more merging - than exact-equality parallel detection, and it is exact (the merge factor is a flux-split share). -- **Push the coupled+parallel alternation to a genuine fixpoint.** The compression loop alternates - parallel-merge → conservation-removal → coupled-merge until a step stops reducing ([Ch 3](#ch3)). Confirming - we reach *maximal* exact reduction — that no additional pass would remove one more reaction — is the - cleanest way to guarantee the `z`-count is minimal for a given model. -- **Order interactions** between blocked/dead-end removal, conservation-relation removal, and coupling: - removing dead ends first can expose new couplings and vice versa; the order the fixpoint visits them - affects how quickly it converges and, at the margin, what it finds. - -The deeper claim attached to this lever is the **"good compression ≈ MCS2" hypothesis** (Phil). -MCS2 (doi:10.1093/bioinformatics/btz393) computes minimal coordinated supports over the nullspace; -its structural benefit is essentially working in a full-rank coordinate system. The hypothesis is that -*a sufficiently good compression already reduces the MILP to (near) full rank, producing a problem -almost identical to MCS2's* — so maximizing exact compression captures most of the MCS2 advantage -without importing MCS2's method. Two pieces of evidence bear on it: a standalone MCS2-style nullspace -approach was tried and gave **no speedup** (solid compression already captured the structural benefit), -and the exact-nullspace PR #60 lifted compression ~1.6× and made yeast-GEM compress at all. But this -remains a **hypothesis, not a fact**, and the way to settle it is stated in §11.3: complete-enumerate -(ALL, not BEST/ANY) reaction MCS up to ~6 KOs on a couple of genome-scale models and compare -head-to-head with MCS2. If the hypothesis holds, compression depth is the whole game for competitiveness -and the MILP-formulation work is secondary; if it fails, the reverse. - -#### 11.2.2 MILP formulation & conditioning - -Compression decides *how many* binaries; formulation decides *how hard the solver's job is per binary*. -The relevant machinery is `link_z` ([Ch 7](#ch7)), which wires each binary `z` to the continuous rows either as -a native indicator constraint or as a big-M row, choosing per-row on the sign of a bounding-LP maximum -`max_Ax` (finite ⇒ big-M with that constant; `inf` ⇒ indicator). The levers: - -- **Prefer native indicators; use big-M only where forced.** Gurobi, CPLEX, and SCIP all support native - indicator constraints; only GLPK forces everything to big-M (its `self.M` is a finite cobra bound). - A loose big-M gives a weak LP relaxation, and a weak relaxation hurts CPLEX more than Gurobi. The - shipped formulation already leans indicator-heavy by construction (536 indicators on the canonical - run), which is why the indicator/big-M split was ruled *out* as the cause of the solver gap - (§11.1.2). But the audit is still worth doing on new model classes: verify we never hand CPLEX a - structurally weaker formulation than Gurobi on the same problem. - -- **Tighten every big-M to its smallest valid bound.** `link_z` already computes a per-row - `max_Ax` = max of the constraint over the LP-relaxed feasible region, which is the tightest *valid* - M given the bounds (an LP-tight, not MILP-tight, heuristic — the true MILP-tight max-min is as hard as - SUPPRESS itself). The gap here: the few dozen *functional* big-M rows that survive are written at the - loose default ±1000, not at their tighter FVA maxima (MILP roadmap §0: iMLcore = 34 big-M all ≈1000). - Tightening those 34 from 1000 to their FVA-computed maxima strengthens the relaxation. The honest - caveat is that 34 ≪ 388 indicators, so the impact is likely small and *must be measured across models* - before it earns effort. - -- **Cut the `z` count at the formulation boundary, not just in compression.** Beyond compression - (§11.2.1), drop structurally-non-knockable reactions and essential reactions *before* they become - `z` variables: FVA #1 removes reactions essential to a desired/PROTECT module from the knockable set, - and FVA #3 pulls size-1 MCS out entirely (re-injected at decompression so the MILP never enumerates - their supersets; [Ch 5](#ch5), [Ch 9](#ch9)). Every reaction kept out of `cmp_ko_cost` is one fewer binary. - -- **The trace-cofactor ill-conditioning and the 9.4-order big-M range — a note, now largely closed.** - The MILP roadmap initially diagnosed a chain: stoichiometry spanning 7.6 orders of magnitude → - FVA flux spans of 9.4 orders → tiny big-M's from trace-cofactor pathways (biotin flux ~1e-6, etc.). - Following the actual pipeline showed **that chain does not exist in the built MILP**: the tiny-flux - reactions relax to ±inf bounds and become *indicators*, never tiny big-M's, so there is no 9.4-order - big-M range to condition (§11.1.2). Exact row+col equilibration of the stoichiometry (7.6 → ~3.8–4.0 - orders, exact via `D·N·v=0 ⟺ N·v=0`) remains a *possible* lever on the primal/dual matrix - conditioning that the SUPPRESS-indicator path sees — but whether stoich conditioning of 4.0 vs 7.6 - orders changes the indicator solve at all is **unproven and is the correct experiment to run**, not an - assumption. Combined stoich + big-M equilibration is a genuine conflict (`s_j·M_j` spans ~9.7 orders; - one column scaling can fix stoich·α *or* big-M/α but not both when `s·M ≉ 1`), so it is off the table - for the big-M range and only live for the (separate, unproven) stoich angle. - -#### 11.2.3 Skip hopeless big-M / dual work - -The cheapest work is work not done. When a knockable constraint's reaction is provably always-zero, or -its bound provably never binds, the entire big-M/indicator machinery for that row can be skipped rather -than computed and added. Two concrete pieces: - -- **The `link_z` sparse short-circuit** (on `hpc_benchmark`): before running the bounding LP, inspect - the row's nonzero count. `nnz==0` ⇒ `M=0` directly; `nnz==1` (a plain reaction KO) ⇒ M is just - `coeff·bound` (∞ if that bound is ∞) — no LP needed, because a single-variable row's maximum over a - box is read straight off the bound. Only `nnz≥2` rows (module/dual constraints) go to an actual LP - (parallelized via `SDPool` above ~1000 rows). This is what makes the build cheap; promote it and keep - it. The corollary lever, from MILP roadmap §0, is that `max_Ax` for single-var KOs is *redundant* — it - reproduces the bound `bound_blocked_or_irrevers_fva` just set — so the LP pool can be restricted to - multi-variable rows with no behavior change and a measurable preprocessing saving. - -- **Substituting out or removing binaries after a target is found** is the uncertain end of this lever. - Once a synthetic-lethal single (`DBTS`) or a specific double (`AOXSr2, DBTS`) is identified, it is - unclear whether anything beyond removing the binary variable helps — branch-and-bound may already - prune those paths. This is problem-structure-dependent and may require a MILP rebuild; treat wins here - as speculative until measured. - -#### 11.2.4 The Farkas-dual pre-bounding problem (the known hard lever) - -This is the deepest formulation lever and the one with the most headroom, because it is the one the -current architecture *cannot* address with its existing tools. - -The asymmetry: PROTECT modules embed the raw primal (the desired flux state must stay feasible), so -their reaction variables carry **finite flux bounds** that FVA can pre-bound and tighten. SUPPRESS -modules instead build a **Farkas infeasibility certificate**: `farkas_dualize` (`strainDesignProblem.py` -~1141) dualizes the primal with a zero objective and appends the normalization row `c_d·y ≤ −1` -(verified: `A_ineq_f = vstack(A_ineq_d, c_d)`, `b_ineq_f = b_ineq_d + [-1]`), which encodes "the -undesired flux state is infeasible after the knockouts" ([Ch 6](#ch6)). The knockouts act on **dual variables**, -and those duals are **unbounded by nature** — one-sided `[0,∞)` for inequality duals or free for -equality duals — pinned only by the `≤ −1` anchor. There is no finite flux bound to read off, so -**FVA pre-bounding does not help the SUPPRESS rows at all.** This is *why* they fall to `inf` `max_Ax` -and become indicators (§11.1.2): not a design choice, a mathematical fact about Farkas rays. - -Because SUPPRESS is the "cannot" half of every classical MCS problem, this is not a corner case — it is -the core. Three redesign options, in increasing ambition, each a *different exact encoding of the same -problem* ([Ch 6](#ch6) owns the dual math; these are pointers for the optimizer): - -1. **Split the compressed network into forward/reverse before Farkas construction.** Constructing the - certificate over a sign-definite (fwd/rev-split) network changes which dual components are free vs - one-sided and can expose bounds that the un-split formulation hides. This is the lowest-risk of the - three because it operates on the network before dualization. -2. **Slack variables tied to global binaries.** Replace the pure dual-ray encoding with slacks that are - directly linked to the intervention binaries, so the "infeasibility after KO" condition is carried by - bounded slacks rather than unbounded duals — giving FVA something finite to bound. -3. **Branch on the indicator constraints directly** rather than routing through the dual ray at all. - -A related, concrete M-dimensioning idea for the Farkas certificate (MILP roadmap R2, untested): run FVA -at *all combinatorial cases of the few inhomogeneous bounds* (PROTECT biomass, glucose uptake, ATPM), -take the smallest nonzero flux a reaction can carry, and use `1/v_min` as that reaction's M in the -certificate (or 1000 if every case gives 0). This would give tight-but-valid Farkas M's for the trace -reactions without the exponential max-min — but it must be prototyped and checked for **completeness** -(no missed solutions) before it is trusted. - -#### 11.2.5 The whole-model preprocessing FVA - -`bound_blocked_or_irrevers_fva` ([Ch 5](#ch5), `networktools.py`) is ~117 s and the entire preprocessing -bottleneck. It runs one whole-model FVA — passing *no* `reaction_list` and *no* extra constraints, so it -does the full `2n` objectives — and then classifies each reaction's bounds: redundant bound (FVA never -reaches it) → ±inf; `min≥0` → irreversible-forward (`lb=0`); `max≤0` → blocked/reverse (`ub=0`); and it -mutates `_lower_bound`/`_upper_bound` in place. It *needs* every bound to do the classification, so it -genuinely cannot be scoped to knockable reactions only. The levers are therefore about the *cost of the -sweep*, not its scope: - -- **Parallelize the Phase-2 residual.** `speedy_fva` ([Ch 5](#ch5), `speedy_fva.py`) already avoids most of the - `2n` LPs via a `v=0`-feasibility pass, a `min Σ|x|` scan, and iterative warm-started push-to-bounds, - falling to individual LPs only for the residual reactions Phase-1 did not resolve. The likely win: on - this whole-model call Phase-1 resolves so much that the Phase-2 residual drops *below* the ~1000-LP - parallelization threshold and runs **serially** — so it pays CPLEX's per-LP tax one reaction at a time. - Forcing the residual to parallelize (or lowering the threshold for this call) directly attacks the - 117 s. -- **A cheaper backend for the LP sweep.** The 117 s is dominated by CPLEX's ~2 s/LP construction - overhead × ~4300 LPs. Nothing about a bound-classification FVA needs CPLEX specifically; running the - sweep on a lighter LP backend (or `slim_fba`/`slim_solve`-style reduced solves) sidesteps the per-LP - tax that is the whole cost. -- **Amortize across seeds.** `dump_preprocessed` + `compute_strain_designs_from_preprocessed` (shipped) - lets one preprocessing run feed many seeded solves — essential for the multi-seed benchmarking below, - since it turns a per-seed 117 s tax into a one-time cost. -- **FVA relocation** (on `hpc_benchmark`): moving/reordering the FVA relative to COMPRESS #2 and snapshotting - `pre_fva_bounds` is prototyped; its real speedup must be measured rigorously head-to-head, not assumed. - -#### 11.2.6 Enumeration & pooling strategy - -The ~1101 s pool search is the largest single term, and it is the one place where the enumeration -*strategy* (as opposed to the formulation) is the lever. The solve loop rebuilds and re-solves, -excluding each found design with `add_exclusion_constraints` (integer cuts that exclude a design *and -its supersets*; [Ch 8](#ch8)). Levers: - -- **Integer cuts as lazy constraints.** Adding the exclusion constraints as solver-native *lazy* - constraints, and reusing the branch-and-bound tree / basis across iterations, avoids rebuilding the - model for every solution found. This is the natural fit for the iterative enumerate loop and is where - a warm-started, incremental architecture would pay off most against the 1101 s. -- **Warm starts.** Reuse the previous solve's basis and incumbent when adding the next cut, rather than - cold-starting each populate iteration. -- **A cross-solution minimality/dedup pass** on pooled `sd.ANY` results — removes the residual ~2% - non-minimal supersets (issue #38) that arise from value-0 KI markers and from pooling many seeds, and - is cheap relative to the search itself. - -Solver-parameter tuning of the pool (CPLEX emphasis/numeric-emphasis, indicator-API usage) is a -**fragile bet** and belongs strictly *after* the formulation is confirmed identical across solvers: -leaning on parameter defaults makes the package vulnerable to solver-version updates that change those -defaults or add better internal routes. Confirm the formulation first, tune params only to *confirm* a -hypothesis, never to carry one. - -### 11.3 Benchmarking discipline - -Speed claims about a branch-and-bound MILP are worthless without discipline, because B&B is chaotic in -ways that a naive timing hides. Four rules. - -**Multi-seed distributions — single-seed timing is meaningless.** The seed is fully plumbed -(`compute_strain_designs(seed=)` → `kwargs_milp[SEED]` → the backend constructor → CPLEX -`parameters.randomseed` / Gurobi `Params.Seed`). The B&B tree *shape* is seed-dependent: the order in -which the solver branches, and therefore how quickly it finds and proves solutions, changes with the -seed. A single-seed run is one sample from a wide distribution, and comparing two configurations on one -seed each can invert the true ordering. **Every speed comparison — ANY, BEST, and POPULATE alike — needs -≥5 seeds** and is reported as a distribution (median + spread), never a single point. This is why the -`dump_preprocessed` amortization (§11.2.5) matters operationally: it makes a 5-seed sweep affordable by -paying the 117 s preprocessing once. - -**Known-answer gates — completeness is the gate, not a nicety.** Two canonical counts are the regression -oracle: **e_coli_core = 455 MCS** (CPLEX ~1.2 s) and **iML1515 = 393 gene-MCS** (the canonical run -above). No MIP optimality gap is ever set, so both solvers run at their default 1e-4 relative gap, which -for integer intervention-cost objectives is effectively exact. Any change to bounds, big-M values, -Farkas M-dimensioning, compression depth, or enumeration strategy **must reproduce these counts -exactly**. A speedup that returns 392 MCS is not a speedup; it is a correctness regression. The -non-negotiable phrasing from the MILP roadmap: any M/bound change must not drop a valid MCS, and every -experiment must re-verify the known-answer counts. The test class that enforces this — re-evaluating -*every* returned design against all PROTECT modules on the original model — is precisely the gate that -would catch a completeness regression (and would have caught the historical #44). - -**Head-to-head against the real competitors, on both solvers.** The target is competitiveness with -**MCS2** (doi:10.1093/bioinformatics/btz393, code at `github.com/RezaMash/MCS`) and **gMCSpy** -(doi:10.1093/bioinformatics/btae318, code + benchmark at `github.com/PlanesLab/gMCSpy`), measured on -**both Gurobi and CPLEX** — because the whole point of the Direction-A work is that Gurobi is currently -much faster than CPLEX on the same straindesign problem, and a fair comparison must not hide behind one -solver. The benchmark set is iML1515 / Yeast-GEM 8.7 / Human-GEM 1.16. The harness lives locally on the -`hpc_benchmark` branch (gitignored), with `benchmarks/tools/MCS2/` reconstructed and its MEX -Octave-recompiled. A caution learned the hard way: prior bound-config experiments (the P-A/B/C, F-A–E -configs in `bench_bound_configs.py`) produced almost no actual MILP change and *insignificant* perf -differences — the amount of real headroom is unknown, so **measure before committing effort**, and do -not mine old JSON in place of a fresh, correctly-distinct experiment. - -**Never drop a valid MCS.** Restated because it is the one rule that overrides all others: completeness -is not traded for speed. The complete-enumeration (ALL, not BEST/ANY) runs up to ~6 KOs that would -settle the "good compression ≈ MCS2" hypothesis (§11.2.1) are themselves the strongest completeness -test, because they force the machinery to produce *every* MCS in a size band and expose any silent drop. - -### 11.4 Roadmap & directions - -**Direction A — compute performance & MCS2/gMCSpy competitiveness (the live thrust).** This is the -active work. Shipped so far: MILP build cut ~70 s → ~7 s (PR #55) and the CPLEX-populate configuration -win. The measured gap stands at CPLEX 1241 s vs Gurobi 280 s ≈ 4.4× on the canonical -iML1515 393, split into preprocessing FVA ~117 s and pool search ~1101 s — so the two real levers are -the whole-model bound FVA (§11.2.5) and the pool-enumeration strategy (§11.2.6), **not** indicators and -**not** the pool params (both verified inert). The near-term milestones are: (1) MCS2/gMCSpy -head-to-heads on iML1515 / Yeast-GEM 8.7 / Human-GEM 1.16; (2) push compression depth to a true fixpoint -(§11.2.1) and settle the "good compression ≈ MCS2" hypothesis by complete enumeration; (3) redesign the -Farkas-dual pre-bounding (§11.2.4); (4) clean up the solver-agnostic `internal_other` remnant. Hexaly is -an optional extra backend target. - -**The exact-nullspace compression thread.** The exactness constraint is upstream and settled: the -nullspace/compression stays integer/rational (never float — small numeric deviations introduce -irreparable compression errors), and PR #60 folded the exact integer/rational sparse nullspace into -`compression.py` as public `straindesign.nullspace`/`sparse_nullspace`, delivering ~1.6× compression on -iML1515/Human-GEM and making **yeast-GEM compress at all** (it previously crashed on scipy's int64 -ceiling; the fix routes >64-bit coefficients through a dict-of-Fractions mode + `ExactCOO`). This is the -shared building block under the compression-depth lever: better exact compression is more `z`-count -reduction, which §11.2.1 argues is the largest structural win. - -**Adjacent efforts (pointers only).** Two prototypes share the exact-nullspace core but are not part of -the straindesign performance work: **SENUS** (`VonAlphaBisZulu/SENUS`) is the standalone exact -integer/rational sparse nullspace lifted out of `compression.py` — a longer-shot Direction-B play whose -next speedup is a Bareiss fraction-free elimination to bound coefficient growth; and **Kimonu** -(`VonAlphaBisZulu/Kimonu.py`) is an *independent* kinetic-module (COCOA-style) analyzer that reuses the -same nullspace core but is not a straindesign component. Both are mentioned here only so a reader tracing -the nullspace code across repos knows where it went; neither is on the straindesign performance critical -path. +### 11.2 Solver-suppressed model copies + +`suppress_lp_context` patches solver-touching cobra/optlang methods during preprocessing. A copied +model receives `_CarrierSolver`, which retains the solver interface identity but has empty +constraint/variable containers and is never optimized. LP and MILP consumers construct their own +`MILP_LP` from stoichiometry and bounds. + +This removes two costs: + +- deep-copying and reconstructing a populated optlang backend; and +- populating an otherwise empty live backend while GPR metabolites and reactions are added. + +In the profile, emulating the previous live-empty-solver copy increased the measured copy/GPR +components by roughly 0.25 s. More importantly, the carrier makes the intended lifetime explicit: +preprocessing copies are data carriers, not cobra models to optimize through `model.solver`. + +The patch is process-global while active and is designed for the serial preprocessing pipeline. +Nested calls are no-ops and the outer context owns restoration. The source model's live solver is +either left untouched or rebuilt and repopulated once on outer-context exit when the structural +reaction set changed; intermediate mutations are deliberately not mirrored into it. Carrier copies +stay backend-free for their preprocessing lifetime. + +### 11.3 Where optimization effort belongs + +The current levers, in priority order, are: + +1. **Reduce LP count without changing the queried polytope.** The single-module fold is an example: + one constrained FVA serves two consumers. +2. **Reduce each FVA LP structurally.** Temporary exact compression and the structural sign sweep are + useful only if their own setup cost remains below the saved solve time. +3. **Improve exact compression.** Fewer reactions reduce both later LP objectives and MILP binaries. + Scaling choices must preserve biologically meaningful finite bounds and avoid pushing them below + solver resolution. +4. **Keep solver work out of model mutation.** Carrier copies and batched solver reconstruction avoid + optlang bookkeeping that does not contribute to the mathematical problems being solved. +5. **Treat big-M as a compatibility formulation.** The blanket M is intentional for GLPK or an + explicit user request; native indicators are preferred for multi-variable rows. + +### 11.4 Benchmarking discipline + +Every performance comparison should record: + +- commit, solver/version, license mode, thread count and seed; +- exact model/setup and whether bounds were converted to a cone; +- compressed model dimensions and number of targetable interventions; +- per-phase timings and FVA LP counts; +- compressed and decompressed design counts; and +- set identity against a trusted run. + +Preprocessing experiments should normally stop at `dump_preprocessed`; enumeration is required only +when the changed preprocessing or formulation could affect the design set. Timing a full population +search to compare two byte-equivalent MILPs only adds solver variance. + +### 11.5 Numerical policy + +There are two different uses of tolerances: + +- a **witness tolerance** lets an observed nonzero flux resolve another direction without a dedicated + LP; a false negative merely costs an extra solve; +- a **zero/tightening policy** changes a model bound and therefore must be conservative. + +These must not be represented by one threshold. Nonoptimal LP statuses are uncertainty and should +preserve the direction as possible rather than convert it to zero. Small-flux regression models should +be part of the correctness suite alongside genome-scale known-answer tests. (ch12)= @@ -5496,7 +4409,7 @@ c[0][new_reac] = np.sum([c[0].pop(k) * old_reac_val[k] for k in lumped_reacs]) `c[0]` is the coefficient dict; `old_reac_val` is `{old: factor}`; each merged term is popped and its coefficient times its factor is accumulated onto `new_reac`. Objectives (`INNER_OBJECTIVE`, `OUTER_OBJECTIVE`, `PROD_ID`) are linear expressions and get the identical treatment. -Coefficients are first converted to exact rationals (`modules_coeff2rational`) so the +Coefficients are first converted to exact rationals (`modules_coeff_to_fraction`) so the factor multiply-and-sum stays exact — the same integer/rational discipline compression itself insists on ([Ch 3](#ch3)): never let a merge introduce float drift into a constraint that the MILP will treat as hard. @@ -5509,7 +4422,7 @@ factor multiply-and-sum stays exact — the same integer/rational discipline com reactions referenced in any module are **protected from parallel merging** in the first place: `_collect_no_par_compress_reacs` (`compute_strain_designs.py`) gathers every reaction id named in a module's constraints/objectives and passes them as `no_par_compress_reacs` to `compress_model` -(`compute_strain_designs.py, 433`), which exempts them from the parallel compressor. A +(`compute_strain_designs.py`), which exempts them from the parallel compressor. A module-referenced reaction therefore never appears on the `old` side of a parallel `reac_map_exp`, so there is nothing to remap for those steps — and if the code *did* try, it would still be correct but redundant. (Coupled merges are not exempted this way; a module reaction may be coupled-merged, which is @@ -5609,7 +4522,7 @@ module is just a validated specification. `SDModule` is declared as ```python -class SDModule(Dict): # strainDesignModule.py:29 +class SDModule(Dict): # strainDesignModule.py def __init__(self, model, module_type, *args, **kwargs): ``` @@ -5696,9 +4609,9 @@ strings are kept in the module docstring for historical reference only). The constructor's validation (`strainDesignModule.py`) runs in this order: -1. **Type whitelist** (`:245`). Unknown `module_type` → exception. +1. **Type whitelist**. Unknown `module_type` → exception. -2. **Bilevel objective presence & senses** (`:248-268`). +2. **Bilevel objective presence & senses**. - For OPTKNOCK/ROBUSTKNOCK/DOUBLEOPT: default `inner_opt_sense`/`outer_opt_sense` to `MAXIMIZE` if unset; both must be `'minimize'` or `'maximize'`; **both** `inner_objective` and `outer_objective` must be non-`None`, else raise. @@ -5706,22 +4619,22 @@ The constructor's validation (`strainDesignModule.py`) runs in this order: require `inner_objective` **and** `prod_id`. (No `outer_objective` — the outer objective is implicitly the growth-coupling potential.) -3. **MCS-with-inner-objective wrinkle** (`:269-276`). PROTECT/SUPPRESS normally take no outer +3. **MCS-with-inner-objective wrinkle**. PROTECT/SUPPRESS normally take no outer objective, but *if one is supplied*, an `inner_objective` becomes mandatory and `outer_opt_sense` is defaulted/validated. This supports the "optimal-yield-at-max-growth" pattern the docstring describes. -4. **Optimality tolerances** (`:277-282`). `inner_opt_tol`/`outer_opt_tol`, if given, must lie in +4. **Optimality tolerances**. `inner_opt_tol`/`outer_opt_tol`, if given, must lie in `(0, 1]` — a fraction of the optimum (`1.0` = exact, `0.95` = "within 95 % of optimal"). These feed the inner/outer LP as an ε-optimality band. -5. **`reac_ids` fallback** (`:284-285`). If no explicit reaction-id list was passed, it is taken +5. **`reac_ids` fallback**. If no explicit reaction-id list was passed, it is taken from `model.reactions.list_attr('id')`. This is why a *dummy* model works: pass `skip_checks=True` and `reac_ids=[...]` and the constructor never touches `model.reactions` - (see the guard at `:239-242`, which errors only if *both* `reac_ids` and `model.reactions` are + (see the guard at, which errors only if *both* `reac_ids` and `model.reactions` are empty). -6. **Parsing to matrix/dict form** (`:290-308`). This is where free-form user input is normalized +6. **Parsing to matrix/dict form**. This is where free-form user input is normalized (all via `parse_constr.py`, [Ch 12](#ch12)): - `constraints` → a list of `[coeff_dict, sign, rhs]` triples via `parse_constraints`. So `'growth >= 0.1'` becomes `[[{'growth': 1.0}, '>=', 0.1]]`. `None` becomes `[]`. @@ -5732,11 +4645,11 @@ The constructor's validation (`strainDesignModule.py`) runs in this order: **Both string and dict forms are accepted for every expression field** — a deliberate convenience so the same module can be written terse (strings) or programmatic (dicts). -7. **Feasibility checks** (`:311-339`, skipped when `skip_checks=True`): +7. **Feasibility checks** (skipped when `skip_checks=True`): - The constraints alone must leave the *original* model feasible: `fba(model, constraints=self[CONSTRAINTS]).status != INFEASIBLE`. This catches contradictory or mistyped constraints at construction time. - - **The zero-vector exclusion** for SUPPRESS/PROTECT-with-inner-objective (`:316-320`): the + - **The zero-vector exclusion** for SUPPRESS/PROTECT-with-inner-objective: the constructor pins *every* reaction to 0 (`[[{k:1},'=',0] for k in reactions]`) and checks that the constraint region is then infeasible. If the all-zero flux vector satisfies the module's constraints, the module is ill-posed (an MCS can never exclude the trivial @@ -5744,10 +4657,10 @@ The constructor's validation (`strainDesignModule.py`) runs in this order: suppress constraint is written `'growth >= 0.01'` (excludes 0) rather than `'growth >= 0'` (includes 0). - Every reaction referenced in `inner_objective`/`outer_objective`/`prod_id` must exist in - `reac_ids` (`:322-331`), and `min_gcp` must be numeric (int is coerced to float, `:333-339`). + `reac_ids`, and `min_gcp` must be numeric (int is coerced to float). `skip_checks=True` bypasses items 7 entirely — used internally when a module is reconstructed -from already-validated data (see `SDModule.copy`, `:341-359`, which rebuilds via a `DummyModel` +from already-validated data (see `SDModule.copy`,, which rebuilds via a `DummyModel` carrying only `.id` and passes `skip_checks=True`). #### 13.1.4 Construction examples @@ -5790,7 +4703,7 @@ optknock = SDModule(model, 'optknock', ``` Here `inner_objective`/`outer_objective` become coefficient dicts, `inner_opt_sense` and -`outer_opt_sense` default to `'maximize'` (`:250-252`), and the constructor verifies that both +`outer_opt_sense` default to `'maximize'`, and the constructor verifies that both objectives reference real reactions and that the growth-≥-0.2 constraint is satisfiable. For OptCouple you would instead pass `inner_objective='BIOMASS...'` and `prod_id='EX_etoh_e'` (no outer objective), optionally with `min_gcp=0.05`. @@ -5805,8 +4718,8 @@ users."* The orchestrator builds it; the user reads it. #### 13.2.1 What a "design" is: the intervention dict The atomic unit is an **intervention set**: a plain `dict` mapping a reaction/gene/regulatory -identifier to an integer-valued marker. The constructor docstring (`:47-54`) defines the -encoding, and `_compute_costs_and_bounds` (`:246-281`) turns it into bounds: +identifier to an integer-valued marker. The constructor docstring defines the +encoding, and `_compute_costs_and_bounds` turns it into bounds: | Value in dict | Meaning | Reaction bounds produced (`itv_bounds`) | |---------------|---------|------------------------------------------| @@ -5823,26 +4736,26 @@ that are literally "not a number". The `-1`/`1`/`0` trichotomy exists precisely not simply the absence of a KO: the same reaction can be a KO candidate in one design and a not-added KI candidate in another, and the object must distinguish them. -`itv_bounds` is computed once at construction (`:246-281`) and cached; `get_reaction_sd_bnds` +`itv_bounds` is computed once at construction and cached; `get_reaction_sd_bnds` just returns it. For a KO you get `(0,0)`; for an added KI you get the reaction's real bounds (so the caller can re-impose them on a model); regulatory `True` entries with a *simple* -single-reaction constraint are folded into a bound (`:256-281`), while complex multi-reaction +single-reaction constraint are folded into a bound, while complex multi-reaction regulatory constraints set `has_complex_regul_itv = True` and are left as symbolic strings. #### 13.2.2 Internal storage -The fields set by `__init__` (`:72-105`): +The fields set by `__init__`: - **`reaction_sd`** — `list[dict]`, the designs at *reaction* level. Always present. - **`gene_sd`** — `list[dict]`, the designs at *gene* level. Present **only** when the computation used gene knockouts/knock-ins (i.e. `GKOCOST` or `GKICOST` in `sd_setup`); the - flag `is_gene_sd` records this (`:91-99`). In gene mode, the raw solution dicts are - gene-keyed, so the constructor calls `_translate_genes_to_reactions` (`:134-201`) to derive + flag `is_gene_sd` records this. In gene mode, the raw solution dicts are + gene-keyed, so the constructor calls `_translate_genes_to_reactions` to derive `reaction_sd` from `gene_sd` via cobra's parsed GPR AST (`reaction.gpr.eval`, [Ch 9](#ch9) owns this translation). In reaction mode `reaction_sd` *is* the raw input and `gene_sd` does not exist. - **`sd_cost`** — `list[float]`, one total cost per design, summed over the applicable cost dictionaries (`KOCOST`/`KICOST`/`GKOCOST`/`GKICOST`/`REGCOST`) in `_compute_costs_and_bounds` - (`:217-243`). An entry contributes its cost only when present *and non-zero* in the design + . An entry contributes its cost only when present *and non-zero* in the design (`if k in s and s[k] != 0`), so a not-added KI (value 0) costs nothing — consistent with the bounds table above. - **`itv_bounds`** — `list[dict]`, the per-design bound overrides described in 13.2.1. @@ -5861,7 +4774,7 @@ The fields set by `__init__` (`:72-105`): #### 13.2.3 The public accessor contract The methods differ along two axes: **level** (reaction vs gene) and **whether not-added KIs are -shown**. The rule for the "clean" accessors is `strip_non_ki` (`:768-770`): +shown**. The rule for the "clean" accessors is `strip_non_ki`: ```python def strip_non_ki(sd): @@ -5895,7 +4808,7 @@ to `[i]` internally. Two contract subtleties to note: gives you the *raw* (unstripped) lists; the `get_*` methods are the curated view. `itv_bounds` has no stripping variant — `get_reaction_sd_bnds` returns it as-is. -`get_gene_reac_sd_assoc` (`:366-388`) deserves a note: gene-level designs are frequently +`get_gene_reac_sd_assoc` deserves a note: gene-level designs are frequently degenerate — several distinct gene-knockout sets collapse to the *same* reaction-level phenotype (because different genes gate the same reactions through the GPR). This method deduplicates the reaction-level designs by hashing `json.dumps(s, sort_keys=True)` and returns @@ -5914,32 +4827,32 @@ The mechanism lives across `_decompress_solutions` (`compute_strain_designs.py`) `SDSolutions`. When the orchestrator's `estimate_expansion_size` exceeds `LAZY_EXPANSION_THRESHOLD` (`= 100_000`, `compute_strain_designs.py`), it builds **one representative expanded design per compressed group** via `_build_lazy_representatives` -(`:721-756`, taking `expanded[0]`, the cheapest, per group) and constructs the solution with a +(taking `expanded[0]`, the cheapest, per group) and constructs the solution with a `_lazy_init` payload: ```python sd_solutions = SDSolutions(orig_model, sd, status, setup, _lazy_init=lazy_meta) ``` -`lazy_meta` (`:667-676`) carries everything needed to expand a group on demand later: +`lazy_meta` carries everything needed to expand a group on demand later: `compressed_sd`, `compression_map`, the uncompressed cost dicts, `max_cost`, the live `model`, -and `estimated_total`. In lazy mode (`self._lazy == True`, `:75`): +and `estimated_total`. In lazy mode (`self._lazy == True`): - **`get_num_sols`** returns `self._estimated_total` (the *estimated* full count), not the - number materialized (`:284-288`). `get_num_materialized` returns the actual count in + number materialized. `get_num_materialized` returns the actual count in `reaction_sd`. -- **`get_representative_sd`** (`:431-444`) returns one stripped design per compressed group — +- **`get_representative_sd`** returns one stripped design per compressed group — the cheap, canonical answer. If there is no `group_map` it falls back to `get_reaction_sd`. -- **`get_group(i)`** / **`get_num_groups`** (`:414-429`) expose the group structure: which +- **`get_group(i)`** / **`get_num_groups`** expose the group structure: which materialized indices share a compressed origin, and how many distinct compressed designs exist. -- **`expand_group(grp_idx)`** (`:446-518`) does the on-demand work: it calls `expand_sd` + +- **`expand_group(grp_idx)`** does the on-demand work: it calls `expand_sd` + `filter_sd_maxcost` ([Ch 9](#ch9)) for that one group, re-runs the regulatory post-processing and the GPR translation + cost/bounds computation, then **splices** the results into `reaction_sd`, `sd_cost`, `itv_bounds`, `group_map` (and `gene_sd`) in place, replacing the single representative. It requires a live `self._model` — if the object was loaded without one it raises with an actionable message pointing at `load(..., model=True)` or `attach_model`. -- **`expand_all(n_per_group=None)`** (`:520-542`) expands every not-yet-expanded group, +- **`expand_all(n_per_group=None)`** expands every not-yet-expanded group, optionally capping to `n_per_group` designs per group, then clears `self._lazy`. The design contract for a developer: **treat a fresh `SDSolutions` as possibly lazy.** Call @@ -5950,36 +4863,36 @@ only while a model is attached. #### 13.2.5 Save / load and model embedding `SDSolutions` is designed to be a **self-contained, portable record** of a computation -(`save`/`load`, `:553-687`). The pickled state already includes the full problem specification +(`save`/`load`). The pickled state already includes the full problem specification via `sd_setup` (§13.3); embedding a model snapshot closes the remaining gap. The central complication is that the live `cobra` model carries an un-picklable solver interface (and would tie the pickle to specific cobra/optlang/solver versions), so the model is never pickled live. Instead: -- `__getstate__` (`:107-120`) strips `_model`, `_cmp_model`, and the `model` entry inside the +- `__getstate__` strips `_model`, `_cmp_model`, and the `model` entry inside the lazy `_expansion_meta` before pickling. -- `save(filename, embed_model=True)` (`:553-612`) embeds *portable, solver-less snapshots* of +- `save(filename, embed_model=True)` embeds *portable, solver-less snapshots* of both the full model and the compressed (GPR-extended) model, produced by StrainDesign's **rational-safe** `networktools.model_to_dict`. Rational-safety matters: the compressed model's bounds/coefficients are exact rationals ([Ch 3](#ch3)), and a naive float round-trip would corrupt them. The two snapshots (`_embedded_model_dict`, `_embedded_cmp_model_dict`) are written only for *this* pickle and then restored off the live object so a subsequent - `embed_model=False` save stays lean (`:597-612`). -- `save` **does not force expansion** of lazy/compressed results (`:565-571`) — it pickles them + `embed_model=False` save stays lean. +- `save` **does not force expansion** of lazy/compressed results — it pickles them as-is, precisely to avoid the memory blow-up of issue #47. To persist a fully-expanded set, call `expand_all` first. -- `load(filename, model=None, cmp_model=None)` (`:638-687`) rebuilds models only on request: +- `load(filename, model=None, cmp_model=None)` rebuilds models only on request: `None` attaches nothing, `True` rebuilds the embedded snapshot via `model_from_dict`, and a - passed `cobra.Model` attaches that object directly. `_resolve` (`:678-683`) implements this + passed `cobra.Model` attaches that object directly. `_resolve` implements this three-way choice independently for the full and compressed model. `get_model` / - `get_compressed_model` / `attach_model` (`:614-636`) are the retrieval/attachment accessors. + `get_compressed_model` / `attach_model` are the retrieval/attachment accessors. The compressed model is offered separately because analysing `compressed_sd` in the *small* compressed model is far faster than in the full one. -Finally, `SDSolutions` supports **merging** (`__iadd__`/`__add__`, `:704-765`): two result sets +Finally, `SDSolutions` supports **merging** (`__iadd__`/`__add__`): two result sets over the same model can be combined, deduplicating at the compressed-design level (via `frozenset(s.items)`) when compression info is present, or at the expanded level otherwise, -with `OPTIMAL` status winning. `_check_merge_compatible` (`:689-702`) refuses to merge across +with `OPTIMAL` status winning. `_check_merge_compatible` refuses to merge across different models, across gene/reaction levels, or across incompatible compression maps. This is what lets the benchmarking harness stitch together the outputs of several seed runs into one solution set. @@ -6022,13 +4935,13 @@ one `sd_setup` dict) are interchangeable descriptions of the same problem. Note that the `sd_setup` *stored on a result object* is not byte-identical to the input one: the orchestrator rebuilds it from the *original* (uncompressed) modules and cost dictionaries at -decompression time (`:606-609`, `:837-840`) so that the record refers to the user's model, not +decompression time so that the record refers to the user's model, not the internal compressed one (see §13.3.3). #### 13.3.2 Role 1 — `sd_setup` as INPUT `compute_strain_designs(model, **kwargs)` lets a caller pass the **entire** configuration as one -`sd_setup=` argument instead of spelling out every parameter (docstring `:75-78`). The handling +`sd_setup=` argument instead of spelling out every parameter (docstring). The handling is at `compute_strain_designs.py`: ```python @@ -6041,7 +4954,7 @@ if SETUP in kwargs: ``` Two accepted forms: the value may be an **in-memory dict**, or a **path to a JSON file** — the -latter is how CNApy stores problems as `.sd` files (docstring `:63-65`), which are then loadable +latter is how CNApy stores problems as `.sd` files (docstring), which are then loadable and re-runnable from Python. Either way the setup becomes the working `kwargs` for the rest of the function. @@ -6049,7 +4962,7 @@ the function. keyword arguments; the `else` branch **replaces `kwargs` wholesale** with the setup dict, so any explicit kwargs passed alongside `sd_setup` (other than `model`, which is a separate positional) are silently discarded. The docstring states this as a hard rule: *"sd_setup and other arguments -(except for model) must not be used together"* (`:77-78`). So the contract is "all-or-nothing," +(except for model) must not be used together"*. So the contract is "all-or-nothing," not "defaults-plus-overrides": use *either* individual kwargs *or* one `sd_setup`, never both. (This is unlike `compute_strain_designs_from_preprocessed`, §13.4.2, whose keyword arguments genuinely *override* the dumped configuration.) @@ -6067,8 +4980,8 @@ Every `SDSolutions` stores the setup it was produced under: `self.sd_setup = sd_ carries not just the answers but the full question. The orchestrator builds this record from the *original* model/modules/costs right before constructing the solution: it `deepcopy`s the setup returned by the MILP layer and overwrites the module/cost keys with the uncompressed originals -(`compute_strain_designs.py` in the normal path, `:837-840` in the from-preprocessed -path, and `:570-573` in the dump early-return), adding `GKOCOST`/`GKICOST` when in gene mode. The +(`compute_strain_designs.py`, in the normal path, the from-preprocessed +path, and the dump early-return), adding `GKOCOST`/`GKICOST` when in gene mode. The `deepcopy` is deliberate: the record must be an immutable snapshot, decoupled from any later mutation of the live cost dictionaries. @@ -6081,7 +4994,7 @@ to the original call site**: `KOCOST`/`KICOST`/`GKOCOST`/`GKICOST`/`REGCOST` *straight out of `sd_setup`* to total each design's cost. Because the cost model lives in the record, `sd_cost` can be recomputed for any (e.g. lazily expanded, §13.2.4) design without the caller re-supplying the cost dictionaries — - `expand_group` (`:493-494`) does exactly this, passing `self.sd_setup` back into + `expand_group` does exactly this, passing `self.sd_setup` back into `_compute_costs_and_bounds`. - **Re-expansion.** The same setup drives on-demand decompression of compressed groups; the gene-vs-reaction branch and the cost lookups both key off it. @@ -6097,129 +5010,71 @@ makes the pickle a fully self-contained, reproducible record. ### 13.4 The preprocessed-dump workflow -The single most expensive part of a strain-design run is **preprocessing**, not the MILP solve: -the compression passes and — dominantly — the blocked/irreversible FVA. On the canonical -iML1515 gene-MCS problem the preprocessing FVA alone is ~117 s, while MILP *construction* is -~4 s ([Ch 11](#ch11)). If you want to sweep the MILP solve across many configurations — different random -seeds, different solvers, different solution approaches, different pre-FVA bound settings — you -should pay the ~117 s **once** and replay the cheap part. That is exactly what `dump_preprocessed` -+ `compute_strain_designs_from_preprocessed` provide. This is the workhorse of the benchmarking -harness. - -#### 13.4.1 Dumping: `dump_preprocessed` - -`dump_preprocessed` is a kwarg to `compute_strain_designs` (whitelisted at -`compute_strain_designs.py`); its value is a path. The orchestrator runs the *entire* -preprocessing pipeline normally — compression #1/#2, GPR integration, all three FVA phases, -size-1 MCS extraction, essential-reaction removal, and MILP-kwarg assembly — and then, just -before it would solve the MILP (`:534-592`), if `dump_preprocessed` is set it pickles a -dictionary and returns early (with any size-1 MCS already found, but *without* running the -MILP). The dumped dict (`:540-562`) contains: - -| Key | What it is | Why it's needed on replay | -|-----|-----------|----------------------------| -| `cmp_model` | the **compressed, GPR-extended** cobra model (exact-rational bounds) | the model the MILP is built on — the expensive artifact | -| `sd_modules` | the modules **remapped to compressed reaction space** | `SDMILP` construction consumes these | -| `kwargs_milp` | solver, `max_cost`, `M`, `seed`, threads, **compressed** ko/ki costs, `essential_kis` | the exact MILP-build arguments | -| `kwargs_computation` | `max_solutions`, `time_limit`, `show_no_ki` | passed to `compute`/`compute_optimal`/`enumerate` | -| `solution_approach` | `'any'`/`'best'`/`'populate'` | which solve method to call | -| `cmp_mapReac` | the compression map | needed to decompress the eventual solutions | -| `uncmp_ko_cost`, `uncmp_ki_cost`, `uncmp_reg_cost` | uncompressed cost dicts | decompression + `filter_sd_maxcost` | -| `orig_model`, `orig_sd_modules`, `orig_*_cost`, `orig_g*_cost` | the pristine originals | building `sd_setup` and the returned `SDSolutions` | -| `gene_kos` | bool flag | selects gene vs reaction decompression | -| `max_cost`, `cmp_size1_mcs` | cost cap and the size-1 MCS found in preprocessing | decompression/filtering | -| `pre_fva_bounds` | `{reac_id: (lb, ub)}` **before** the blocked/irrevers FVA | lets you *re-run* the bound-relaxation with a different config, or study its effect, without recompressing | - -`pre_fva_bounds` (captured at `:449`, immediately before `bound_blocked_or_irrevers_fva`) is the -key enabler of **bound-configuration experiments**: the compressed model is snapshotted with its -bounds *as they were before* the redundant-bound relaxation, so a downstream experiment can -apply a different bound policy to the already-compressed model rather than re-deriving the whole -compression. The dump thus amortizes not just the FVA but the entire compression + GPR chain. - -On dump the function logs a copy-pasteable resume line and returns an `SDSolutions` holding only -the size-1 MCS (or infeasible/empty), with `compressed_sd`/`compression_map`/`group_map` and -`_cmp_model` populated (`:568-592`). - -#### 13.4.2 Replaying: `compute_strain_designs_from_preprocessed` - -`compute_strain_designs_from_preprocessed(dump, seed=None, solver=None, solution_approach=None, -max_solutions=None, time_limit=None)` (`:759-851`) is the cheap replay. Its signature *is* the -sweep interface: every keyword is an **override** applied on top of the dumped configuration. - -- `dump` may be a **path** (unpickled) or the **dict itself** (`:776-781`) — the latter lets you - unpickle once, mutate the dict in a loop (e.g. rewrite `cmp_model` bounds using - `pre_fva_bounds`, or swap `sd_modules`), and feed each variant in without touching disk. -- Overrides (`:803-813`): `seed` → `kwargs_milp[SEED]`; `solver` → - `kwargs_milp[SOLVER]` (via `select_solver`); `max_solutions`/`time_limit` → - `kwargs_computation`; `solution_approach` replaces the dumped approach. -- The compressed model was pickled while its LP/solver was suppressed (its solver is a stub), so - the replay re-enters `suppress_lp_context(cmp_model)` (`:817-818`) before building the - `SDMILP`, so that `SDMILP` can safely touch variables without triggering a solver build. -- It then rebuilds the MILP (`SDMILP(cmp_model, sd_modules, **kwargs_milp)`, `:824`), solves via - the chosen approach, and — crucially — runs the **identical** `_decompress_solutions` path - (`:842-845`) as the normal orchestrator, so the returned `SDSolutions` (lazy expansion, costs, - bounds, gene translation, `_cmp_model`) is indistinguishable from one produced end-to-end. - -#### 13.4.3 The developer workflow - -The typical benchmarking loop: +`dump_preprocessed` separates deterministic preprocessing/MILP construction from enumeration. On the +current canonical Gurobi profile, preprocessing is about 19.6 s and is dominated by reversibility +classification, the final folded FVA and compression. Reusing the dump is therefore useful for seed, +solver and enumeration comparisons. -```python -from straindesign import (compute_strain_designs, - compute_strain_designs_from_preprocessed) +#### 13.4.1 Dumping + +`compute_strain_designs(..., dump_preprocessed=path)` runs normal preprocessing, including: + +- optional reversibility pre-tightening and both compression passes; +- desired-region essentiality and GPR reduction/extension; +- either the folded single-classical-module FVA or the general final bound/module FVA route; +- size-1 MCS extraction; +- per-module `fva_bounds`; and +- MILP argument assembly. + +It then serializes the compressed model and returns before solving the MILP. The dictionary contains +the compressed model/modules, MILP and computation kwargs, compression map, original and compressed +cost information, pristine model/setup, gene-mode metadata, size-1 MCS and `pre_fva_bounds`. + +`pre_fva_bounds` is captured immediately before the final bound-relaxation FVA. It supports controlled +bound-policy experiments without rerunning compression and GPR extension. + +#### 13.4.2 Carrier solver in the dump -# 1. Pay preprocessing ONCE (~117 s on iML1515). Returns early; writes the dump. -compute_strain_designs(model, sd_modules=[suppress], - gene_kos=True, max_cost=3, - solution_approach='populate', - dump_preprocessed='iml1515_gmcs.pkl') +The compressed cobra model is pickled with a backend-free `_CarrierSolver`, not a populated optlang +model. The carrier preserves the solver interface needed by selection and model metadata, but it is +not itself solved. `compute_strain_designs_from_preprocessed` re-enters `suppress_lp_context` while +constructing `SDMILP`; the latter builds its own backend from the serialized matrices and bounds. -# 2. Sweep the cheap MILP solve — e.g. a seed sweep for solver-variance study: -results = [] -for s in range(10): - sol = compute_strain_designs_from_preprocessed('iml1515_gmcs.pkl', seed=s) - results.append(sol) +#### 13.4.3 Replaying -# 3. Or a solver comparison (the CPLEX-vs-Gurobi story, Ch 11): -gu = compute_strain_designs_from_preprocessed('iml1515_gmcs.pkl', solver='gurobi') -cp = compute_strain_designs_from_preprocessed('iml1515_gmcs.pkl', solver='cplex') +`compute_strain_designs_from_preprocessed` accepts either the pickle path or an already loaded +dictionary. Optional arguments override seed, solver, solution approach, maximum solutions and time +limit. It rebuilds the MILP, runs ANY/BEST/POPULATE, and passes the compressed result through the same +decompression and filtering path as the end-to-end function. -# 4. Or a bound-config experiment using the in-memory dict form: -import pickle -d = pickle.load(open('iml1515_gmcs.pkl', 'rb')) -for cfg in bound_configs: - apply_bounds(d['cmp_model'], d['pre_fva_bounds'], cfg) # mutate compressed model - results.append(compute_strain_designs_from_preprocessed(d)) # pass the dict +```python +compute_strain_designs( + model, + sd_modules=[suppress], + gene_kos=True, + max_cost=3, + solution_approach="populate", + dump_preprocessed="iml1515_gmcs.pkl", +) + +sol = compute_strain_designs_from_preprocessed( + "iml1515_gmcs.pkl", seed=42, solver="gurobi" +) ``` -Because each replay reuses the same compressed model, module remapping and cost translation, the -*only* variable across runs is the MILP itself — which is precisely the isolation a benchmark -wants. And because the returned `SDSolutions` objects are merge-compatible (same model, same -compression map), a seed or solver sweep can be folded into a single deduplicated solution set -with `sum(results, results[0])`-style `__iadd__` (13.2.5). This is the object-level plumbing -that makes the benchmarking harness ([Ch 11](#ch11)) fast and reproducible. +For a parameter sweep, load the dictionary once and pass it directly. Keep preprocessing fixed unless +the experiment explicitly changes a stored model bound or module; otherwise the comparison no longer +isolates the MILP/solver phase. (ch14)= ## 14. The solver-interface layer (`MILP_LP` + backends) -Every LP and MILP that `straindesign` ever solves — the three preprocessing FVA sweeps, the -size-1 MCS probes, the bounding LPs that compute big-M values, and the central strain-design -MILP with its integer-cut enumeration — passes through a single class, `MILP_LP` in -`solver_interface.py`. `MILP_LP` is a thin, uniform façade over four numerically and API-wise -very different solvers (CPLEX, Gurobi, SCIP/SoPlex, GLPK). This chapter is about the physical -handoff: how the abstract problem `(c, A_ineq, b_ineq, A_eq, b_eq, lb, ub, vtype, indic_constr, M)` -built upstream ([Ch 7](#ch7)) becomes a live solver object, how `solve` / `slim_solve` / `populate` map onto -each backend's very different notion of "solve," how indicator constraints are handed over natively -or reduced to big-M, how each solver's status codes are collapsed into one canonical vocabulary, -and where — physically — the ~4.4× CPLEX-vs-Gurobi runtime gap on the canonical iML1515 gene-MCS -benchmark lives. - -Boundaries: **[Ch 7](#ch7)** owns the *decision* of which continuous rows get a big-M encoding versus a -native indicator constraint (the `link_z` fork) and the mathematics of a valid/tight `M`. **[Ch 8](#ch8)** -owns the *solve loop* — the ANY / BEST / POPULATE objective setups and the integer-cut enumeration -that repeatedly calls the methods described here. This chapter owns only the layer in between: the -abstraction and the four backend translations. +Every LP and MILP that `straindesign` solves passes through `MILP_LP`: module validation, +public FBA/FVA, reversibility classification, final preprocessing FVA and the strain-design MILP. +MILP construction no longer launches per-row big-M bounding LPs. This chapter describes the common +status vocabulary and the backend-specific implementations for CPLEX, Gurobi, SCIP/SoPlex and GLPK. + ### 14.1 Why an abstraction layer exists @@ -6346,14 +5201,13 @@ An `IndicatorConstraints` object (`indicatorConstraints.py`) stores a *batch* of `A` a sparse matrix (one row per constraint), `b` the right-hand sides, `sense ∈ {'L','E','G'}`, and `indicval ∈ {0,1}`. This is a solver-neutral container; each backend translates it. -Recall the [Ch 7](#ch7) result stated as given in CONTEXT: under the default `M = inf`, `link_z` emits the -**SUPPRESS Farkas-dual rows as indicator constraints** (their fluxes are unbounded, so no finite `M` -exists) and the **PROTECT finite-flux primal rows as big-M rows already baked into `A_ineq`**. This -split is emergent from bound structure, not a per-module switch. Consequently, by the time a problem -reaches this layer, the big-M rows are *ordinary inequality rows* — no backend does anything special -with them — and the `indic_constr` block carries only the genuinely indicator-encoded implications. -The one exception is GLPK, which cannot represent indicators and must convert that block to big-M -here, using the `M` value the abstraction passed it. +Recall the [Ch 7](#ch7) rule: under the default `M = inf`, `link_z` derives finite relaxations for +zero- and single-continuous-variable rows and emits multi-variable rows as indicator constraints. +The split follows row structure, not module type. Consequently, by the time a problem reaches this +layer, finite-M rows are ordinary inequality rows and `indic_constr` carries the remaining +indicator-encoded implications. GLPK cannot represent indicators and receives the configured blanket +M (1000 by default) for those rows; explicitly passing a finite M requests this replacement on the +other backends too. **CPLEX** (`cplex_interface.py`). The batch is reshaped to CPLEX's format — each row becomes `[[col indices],[coeffs]]` — and handed to `self.indicator_constraints.add_batch` with @@ -6605,42 +5459,22 @@ The common design principle: a numerically caveated but present solution is retu `TIME_LIMIT_W_SOL` and left for the outer verification to accept or reject, never crashing the enumeration mid-run. -### 14.9 Where the CPLEX-vs-Gurobi performance story physically lives - -The interface choices in this chapter are the physical substrate of the headline benchmark -(CONTEXT): the canonical iML1515 gene-MCS run (SUPPRESS biomass ≥ 0.001, POPULATE, `max_cost = 3`, -gene KOs) finds **393 MCS** in **Gurobi 280 s vs CPLEX 1241 s (≈ 4.4×)**, with the split -preprocessing FVA ~117 s, MILP build ~4 s, populate ~1101 s. Reading that against the code: - -1. **The gap is in `populate`, not construction.** Both backends receive the *same* abstract MILP - with the *same* native indicator constraints and the *same* default `1e-4` MIP gap; construction - is ~4 s either way. The ~1101 s populate phase is a single native pool search on each solver, and - the 4.4× difference is the two solvers' pool-search engines exploring the design space at - different rates — not a formulation asymmetry this layer introduces. This is why the CPLEX pool - parameters, though set since 2022, are *not* the culprit: they are inert during `solve` and, in - `populate`, they configure the pool identically in spirit to Gurobi's `PoolGap`/`PoolSearchMode`. - -2. **Per-LP overhead in preprocessing goes through this layer.** The ~117 s of blocked/irreversible - FVA is thousands of small LPs, each a `slim_solve` on a freshly constructed backend object. - Gurobi mitigates the per-object cost by sharing **one quiet `Env`** across all models - (`gurobi_interface.py`, `_get_quiet_env`) — creating a Gurobi environment per model would - spin up a licence session each time, which on a node-locked HPC licence is expensive. CPLEX - constructs a fresh `Cplex` per object (and sizes `workmem` to 75 % RAM each time). For a run - that instantiates the interface thousands of times, this fixed per-solve overhead — object - creation, parameter setting, matrix load — is real and is paid inside `MILP_LP.__init__` and the - backend constructors, which is exactly why `slim_solve` (no solution-vector extraction) and - `skip_checks` exist as fast paths. - -3. **The abstraction does not tax the hot path with translation.** Matrices are handed to each solver - in its preferred bulk form (CPLEX `set_coefficients` on COO triplets, Gurobi `addMConstr` on the - sparse matrix directly, GLPK a single `glp_load_matrix`), so the per-call cost is solver-native - assembly, not a Python re-encoding loop — with the exception of SCIP, whose term-by-term `Expr` - assembly (`scip_interface.py`) is inherently slower and compounds its lack of a native - pool. This is the mechanical reason SCIP and GLPK, while correct, are validation backends rather - than the engines behind the benchmark numbers. - -For the enumeration-loop mechanics that drive these calls and the deeper benchmark analysis, see -[Ch 8](#ch8) and [Ch 11](#ch11); for the conditioning that provokes the Section 14.8 numeric states, see [Ch 11](#ch11). +### 14.9 Where preprocessing performance reaches the solver layer + +The current preprocessing profile creates many small LPs in two places: +`fast_reversibility` before COMPRESS #1 and the final bound/module FVA after COMPRESS #2. Each phase +reuses one `MILP_LP` while changing objectives, with periodic rebuilds to limit warm-start +degeneration. Solver-specific model setup and objective-update costs are therefore multiplied by the +number of residual directions. + +Gurobi and CPLEX provide native indicator constraints for the multi-variable `link_z` rows. SCIP also +has an indicator path; GLPK uses the finite blanket M. There is no bounding-LP phase in `link_z`, so +MILP construction is now a sub-second component in the canonical profile. + +Status normalization is correctness-critical during preprocessing. `OPTIMAL` supplies a bound or +flux witness, `UNBOUNDED` proves the corresponding direction is available, and any other status is +uncertainty. A caller that changes model bounds must handle that uncertainty conservatively. During +enumeration, `TIME_LIMIT_W_SOL` can still expose an incumbent for outer verification. (ch15)= @@ -7129,5 +5963,5 @@ cProfile.run("compute_strain_designs(model, sd_modules=[...], solver='glpk')", ' pstats.Stats('profile_out').sort_stats('cumulative').print_stats(30) ``` -The hot spots are typically the preprocessing FVA, `link_z` (its per-constraint LP bounding), and the +The hot spots are typically the preprocessing FVA, `link_z`, and the solver's enumeration loop ([Ch 11](#ch11)). diff --git a/straindesign/compression.py b/straindesign/compression.py index d9bed1c..262a308 100644 --- a/straindesign/compression.py +++ b/straindesign/compression.py @@ -18,6 +18,7 @@ import ast import copy import logging +import re import numpy as np from enum import Enum from functools import reduce @@ -28,7 +29,6 @@ from fractions import Fraction from scipy import sparse from scipy.sparse import csr_matrix, csc_matrix -from sympy import Rational, Symbol as SympySymbol, And as SympyAnd, Or as SympyOr from cobra import Configuration from cobra.util.array import create_stoichiometric_matrix @@ -38,12 +38,10 @@ # function bodies to avoid the circular dependency (networktools re-exports # compression symbols). -# ============================================================================= # Utility Functions -# ============================================================================= -def float_to_rational(val, max_precision: int = 6, max_denom: int = 100) -> Fraction: +def float_to_fraction(val, max_precision: int = 6, max_denom: int = 100) -> Fraction: """Convert float to Fraction with bounded denominators.""" if isinstance(val, Fraction): return val @@ -85,9 +83,7 @@ def _lcm_list(numbers: List[int]) -> int: return reduce(lcm, numbers, 1) if numbers else 1 -# ============================================================================= # Rational Matrix with Sparse Storage -# ============================================================================= _INT64_MAX = (1 << 63) - 1 @@ -122,9 +118,7 @@ def _invalidate_cache(self): if not self._batch_mode: self._csc_cache = None - # ------------------------------------------------------------------------- # Construction - # ------------------------------------------------------------------------- @classmethod def _from_sparse(cls, @@ -160,7 +154,7 @@ def from_numpy(cls, arr: np.ndarray, max_precision: int = 6, max_denom: int = 10 for c in range(cols): val = arr[r, c] if val != 0: - frac = float_to_rational(val, max_precision, max_denom) + frac = float_to_fraction(val, max_precision, max_denom) row_idx.append(r) col_idx.append(c) num_data.append(frac.numerator) @@ -190,7 +184,7 @@ def from_cobra_model(cls, model, max_precision: int = 6, max_denom: int = 100) - elif hasattr(coeff, 'numerator'): frac = Fraction(coeff.numerator, coeff.denominator) else: - frac = float_to_rational(coeff, max_precision, max_denom) + frac = float_to_fraction(coeff, max_precision, max_denom) row_idx.append(i) col_idx.append(j) @@ -223,9 +217,7 @@ def _build_from_sparse_data(cls, row_indices: List[int], col_indices: List[int], result._dict_frac = dic return result - # ------------------------------------------------------------------------- # Size queries - # ------------------------------------------------------------------------- def get_row_count(self) -> int: return self._rows @@ -233,9 +225,7 @@ def get_row_count(self) -> int: def get_column_count(self) -> int: return self._cols - # ------------------------------------------------------------------------- # Iteration - # ------------------------------------------------------------------------- def iter_column_fractions(self, col: int) -> Iterator[Tuple[int, Fraction]]: """Iterate over non-zero entries in column as (row, Fraction) pairs.""" @@ -262,9 +252,7 @@ def get_signum(self, row: int, col: int) -> int: return -1 return 0 - # ------------------------------------------------------------------------- # Batch edit mode - # ------------------------------------------------------------------------- def begin_batch_edit(self): """Enter batch edit mode - delays cache invalidation.""" @@ -279,9 +267,7 @@ def end_batch_edit(self): self._den_sparse = self._den_sparse.tocsr() self._csc_cache = None - # ------------------------------------------------------------------------- # Matrix operations - # ------------------------------------------------------------------------- def clone(self) -> 'RationalMatrix': """Create a deep copy.""" @@ -363,9 +349,30 @@ def add_scaled_column(self, dst_col: int, src_col: int, scalar_num: int, scalar_ self._invalidate_cache() - # ------------------------------------------------------------------------- + def scale_column(self, col: int, scalar_num: int, scalar_den: int) -> None: + """Multiply a column by a scalar: col[i] *= scalar_num/scalar_den.""" + if scalar_num == 0 or scalar_num == scalar_den: + return + num_lil, den_lil = self._num_sparse, self._den_sparse + num_csc = num_lil.tocsc() if num_lil.format != 'csc' else num_lil + den_csc = den_lil.tocsc() if den_lil.format != 'csc' else den_lil + entries = [(num_csc.indices[i], int(num_csc.data[i]), int(den_csc.data[i])) + for i in range(num_csc.indptr[col], num_csc.indptr[col + 1])] + for row, cur_num, cur_den in entries: + if cur_num == 0: + continue + new_num, new_den = cur_num * scalar_num, cur_den * scalar_den + if new_den < 0: + new_num, new_den = -new_num, -new_den + g = gcd(abs(new_num), new_den) + if g: + new_num //= g + new_den //= g + num_lil[row, col] = new_num + den_lil[row, col] = new_den if new_num != 0 else 0 + self._invalidate_cache() + # Conversion - # ------------------------------------------------------------------------- def to_numpy(self) -> np.ndarray: """Convert to numpy float array.""" @@ -475,9 +482,7 @@ def __repr__(self) -> str: return f"RationalMatrix({self._rows}x{self._cols})" -# ============================================================================= # Sparse Integer RREF for Nullspace Computation -# ============================================================================= def _rref_integer_sparse(rm: RationalMatrix) -> Tuple[Dict[int, Dict[int, int]], int, List[int]]: @@ -504,7 +509,7 @@ def _rref_integer_sparse(rm: RationalMatrix) -> Tuple[Dict[int, Dict[int, int]], rows = rm.get_row_count() cols = rm.get_column_count() - # --- Column sorting: sparse columns first --- + # Column sorting: sparse columns first # col_order[sorted_pos] = original_col nnz_per_col = np.diff(rm._num_sparse.tocsc().indptr) col_order = np.argsort(nnz_per_col, kind='stable').tolist() @@ -539,7 +544,7 @@ def _rref_integer_sparse(rm: RationalMatrix) -> Tuple[Dict[int, Dict[int, int]], if row_data: data[r] = row_data - # --- Row sorting: sparse rows first (better initial pivot candidates) --- + # Row sorting: sparse rows first (better initial pivot candidates) if data: sorted_row_keys = sorted(data.keys(), key=lambda r: len(data[r])) data = {new_r: data[old_r] for new_r, old_r in enumerate(sorted_row_keys)} @@ -609,7 +614,7 @@ def _eliminate(prd, pivot_val, pivot_col, targets, index): for c in old_cols: col_rows[c].discard(elim_row) - # ---- Phase 1: forward elimination to row-echelon form ---- + # Phase 1: forward elimination to row-echelon form # Eliminate each pivot only from rows BELOW its pivot row, so already-processed pivot rows stay # sparse. Full Gauss-Jordan (eliminating upward too) re-reduces those filled rows with every later # pivot — ~99% of the total work on iML1515. The reduced form is recovered in phase 2. Rows are not @@ -648,7 +653,7 @@ def _eliminate(prd, pivot_val, pivot_col, targets, index): targets = [(r, data[r][pivot_col]) for r in list(col_rows.get(pivot_col, ()))] _eliminate(pivot_row_data, best_val, pivot_col, targets, True) - # ---- Phase 2: back-substitution to reduced row-echelon form ---- + # Phase 2: back-substitution to reduced row-echelon form # Process pivots last-to-first, clearing each pivot column from the pivot rows ABOVE it. In this # order each pivot row's later-pivot-column entries are already cleared, so back-substitution only # introduces free-column fill — far less than Gauss-Jordan (iML1515: ~0.8M ops vs ~9.4M). @@ -750,9 +755,7 @@ def _nullspace_sparse(matrix: RationalMatrix) -> RationalMatrix: return RationalMatrix._build_from_sparse_data(row_indices, col_indices, numerators, denominators, cols, nullity) -# ============================================================================= # Linear Algebra Functions -# ============================================================================= def nullspace(matrix: RationalMatrix) -> RationalMatrix: @@ -822,9 +825,7 @@ def sparse_nullspace(matrix): return csr -# ============================================================================= # Configuration -# ============================================================================= class CompressionMethod(Enum): @@ -846,9 +847,7 @@ def standard(cls) -> List['CompressionMethod']: return [cls.NULLSPACE, cls.RECURSIVE] -# ============================================================================= # Statistics -# ============================================================================= class CompressionStatistics: @@ -887,9 +886,7 @@ def __repr__(self): f"coupled={self.coupled_count})") -# ============================================================================= # Compression Record -# ============================================================================= class CompressionRecord: @@ -913,9 +910,7 @@ def __init__(self, self.stats = stats -# ============================================================================= # Working State (Internal) -# ============================================================================= class _Size: @@ -1080,9 +1075,7 @@ def get_truncated(self) -> CompressionRecord: return CompressionRecord(pre_trunc, cmp_trunc, post_trunc, meta_names_trunc, self.stats) -# ============================================================================= # Core Algorithm -# ============================================================================= class StoichMatrixCompressor: @@ -1283,6 +1276,9 @@ def _handle_compress(self, work: _WorkRecord, kernel_pattern, kernel_values) -> work.post.begin_batch_edit() for group in groups: + # Count nonzeros per member here; used later to pin the lump's scale to the member + # with the most coefficients. + nnz = {r: sum(1 for _ in work.cmp.iter_column_fractions(r)) for r in group} self._combine_coupled(work, group, ratios) # Check bounds intersection to detect contradicting groups. @@ -1324,6 +1320,22 @@ def _handle_compress(self, work: _WorkRecord, kernel_pattern, kernel_values) -> # Consistent: only remove slaves (merged into master) for idx in group[1:]: reactions_to_remove.add(idx) + # Pick the member whose units the lump keeps. nnz was counted pre-merge above; + # co-locate the small-bound test with it here so the whole decision reads in one + # place. Prefer a reaction with a small finite bound (e.g. Biomass, ATP + # maintenance) whose own ratio is near 1; else the member with the most + # coefficients. Master bounds are already intersected at this point. + def _small_bound(r): + fin = [abs(x) for x in work.bounds[r] if not isinf(x) and x != 0 and abs(x) < 100] + return min(fin) if fin else None + + def _lam(r): + return 1.0 if ratios[r] is None else float(abs(ratios[r])) # master's own ratio is 1 + + bounded = [(b, r) for r in group for b in [_small_bound(r)] + if b is not None and 0.1 <= _lam(r) <= 10] + keep = min(bounded)[1] if bounded else max(group, key=lambda r: nnz[r]) + self._restore_group_scale(work, group, ratios, keep) # End batch edit mode work.cmp.end_batch_edit() @@ -1335,6 +1347,30 @@ def _handle_compress(self, work: _WorkRecord, kernel_pattern, kernel_values) -> return contradicting_removed + def _restore_group_scale(self, work: _WorkRecord, group: List[int], + ratios: List[Optional[Fraction]], keep: int) -> None: + """Express a merged group in the units of one of its members. + + A lump's ratios are fixed but its overall scale is free, and merging into ``group[0]`` can + yield an extreme scale (the iML1515 biomass lump comes out 4484x, pushing ``biomass >= 0.001`` + below LP feasibility tolerance). ``keep`` names the member whose units to re-express in + (chosen by the caller from the nnz / small-bound criteria); ``cmp``, ``post`` and the bounds + are scaled together, so the change of units is exact. + """ + master = group[0] + if keep == master: + return + lam = abs(ratios[keep]) # |.| so the reaction keeps its orientation + if lam == 0 or lam == 1: + return + # v_master = ratios[keep] * v_keep, so re-expressing the lump in v_keep multiplies the + # column by that ratio and divides the bounds by it. + work.cmp.scale_column(master, lam.numerator, lam.denominator) + work.post.scale_column(master, lam.numerator, lam.denominator) + lb, ub = work.bounds[master] + f = float(lam) + work.bounds[master] = (lb if isinf(lb) else lb / f, ub if isinf(ub) else ub / f) + def _combine_coupled(self, work: _WorkRecord, group: List[int], ratios: List[Optional[Fraction]]) -> None: """Combine coupled reactions into master reaction. @@ -1357,9 +1393,7 @@ def _combine_coupled(self, work: _WorkRecord, group: List[int], ratios: List[Opt work.stats.inc_coupled_reactions_count(len(group)) -# ============================================================================= # COBRA Interface -# ============================================================================= class CompressionResult: @@ -1442,7 +1476,7 @@ def remove_conservation_relations(model) -> None: elif hasattr(coeff, 'numerator'): frac = Fraction(coeff.numerator, coeff.denominator) else: - frac = float_to_rational(float(coeff)) + frac = float_to_fraction(float(coeff)) row_idx.append(j) # reaction → row (transposed layout) col_idx.append(i) # metabolite → column num_data.append(frac.numerator) @@ -1577,7 +1611,7 @@ def _apply_compression_to_model(model, compression_record, original_reaction_nam reaction_map[main_rxn.id] = {original_reaction_names[main_idx]: Fraction(1)} continue - # --- Merged group (2+ contributing reactions) --- + # Merged group (2+ contributing reactions) # Store subset info main_rxn.subset_rxns = [idx for idx, _ in contributing] @@ -1696,9 +1730,7 @@ def _apply_compression_to_model(model, compression_record, original_reaction_nam return reaction_map -# ============================================================================= # Preprocessing Functions -# ============================================================================= def remove_blocked_reactions(model) -> List: @@ -1731,17 +1763,20 @@ def remove_dummy_bounds(model) -> None: rxn.upper_bound = np.inf -def stoichmat_coeff2rational(model) -> None: - """Convert stoichiometric coefficients to rational numbers.""" +def stoichmat_coeff_to_fraction(model) -> None: + """Convert stoichiometric coefficients to exact fractions.Fraction.""" for rxn in model.reactions: for met, coeff in rxn._metabolites.items(): - if isinstance(coeff, (float, int)): - rxn._metabolites[met] = float_to_rational(coeff) - elif not hasattr(coeff, 'p'): # Not sympy.Rational - if hasattr(coeff, 'numerator'): # fractions.Fraction - rxn._metabolites[met] = Rational(coeff.numerator, coeff.denominator) - else: - raise TypeError(f"Unsupported coefficient type: {type(coeff)}") + if isinstance(coeff, Fraction): + continue # already exact + elif isinstance(coeff, (float, int)): + rxn._metabolites[met] = float_to_fraction(coeff) # -> Fraction + elif hasattr(coeff, 'p'): # sympy.Rational -> Fraction + rxn._metabolites[met] = Fraction(int(coeff.p), int(coeff.q)) + elif hasattr(coeff, 'numerator'): # other Rational -> Fraction + rxn._metabolites[met] = Fraction(coeff.numerator, coeff.denominator) + else: + raise TypeError(f"Unsupported coefficient type: {type(coeff)}") def stoichmat_coeff2float(model) -> None: @@ -1751,108 +1786,310 @@ def stoichmat_coeff2float(model) -> None: rxn._metabolites[met] = float(coeff) -# ============================================================================= # GPR Propagation Helpers -# ============================================================================= -def _gpr_ast_to_sympy(node): - """Convert a cobra GPR AST node to a sympy boolean expression. +def _gpr_ast_to_expr(node, op=None): + """Convert a cobra GPR AST node to a nested expression, or join expressions under ``op``. - Returns None for empty GPR (node is None), meaning the reaction has - no gene requirement and is always active. + A gene is its name (str) and a boolean node is ``(op, (children...))``; None means no gene + requirement (always active). Passing ``op`` joins the given expressions instead of + converting a node, applying only associativity (same-op children are flattened) and + idempotence (duplicates dropped) -- real simplification is done by simplify_model_gprs, which + compress_model runs at the end of a propagate_gpr pass. """ - if node is None: - return None - if isinstance(node, ast.BoolOp): - children = [_gpr_ast_to_sympy(v) for v in node.values] - if isinstance(node.op, ast.And): - return SympyAnd(*children) + if op is None: + if isinstance(node, ast.BoolOp): + op = 'and' if isinstance(node.op, ast.And) else 'or' + node = [_gpr_ast_to_expr(v) for v in node.values] + elif isinstance(node, ast.Name): + return node.id else: - return SympyOr(*children) - elif isinstance(node, ast.Name): - return SympySymbol(node.id) - return None + return None + flat = [] + for e in node: + if e is None: + continue + flat.extend(e[1] if isinstance(e, tuple) and e[0] == op else [e]) + uniq = list(dict.fromkeys(flat)) + return (op, tuple(uniq)) if len(uniq) > 1 else (uniq[0] if uniq else None) -def _sympy_to_gpr_string(expr): - """Convert a sympy boolean expression to a GPR rule string. +def _expr_to_gpr_string(expr): + """Render an expression as a GPR rule string, '' for None. - Produces correctly parenthesised output with sorted gene names for - deterministic results. Returns '' for None input. + ``expr`` is one of the nested-expression forms produced by ``_gpr_ast_to_expr``: ``None`` (no + gene requirement, renders to ''), a gene-id string (e.g. ``'g1'``), or an ``(op, args)`` tuple + such as ``('and', ('g1', 'g2'))`` -> ``'g1 and g2'`` or, more nested, + ``('and', ('g1', ('or', ('g2', 'g3'))))``. + + Operands are sorted so equivalent inputs give identical rules, and a nested clause of the + opposite operator is parenthesised. """ if expr is None: return '' - if isinstance(expr, SympySymbol): - return str(expr) - if expr.func == SympyAnd: - parts = [] - for arg in sorted(expr.args, key=str): - s = _sympy_to_gpr_string(arg) - if hasattr(arg, 'func') and arg.func == SympyOr: - s = f'({s})' - parts.append(s) - return ' and '.join(parts) - if expr.func == SympyOr: - parts = [] - for arg in sorted(expr.args, key=str): - s = _sympy_to_gpr_string(arg) - if hasattr(arg, 'func') and arg.func == SympyAnd: - s = f'({s})' - parts.append(s) - return ' or '.join(parts) - return str(expr) - - -def _combine_gpr_and(gpr_bodies): - """Combine GPR AST bodies with AND logic (for coupled/serial reaction merge). + if isinstance(expr, str): + return expr + op, args = expr + other = 'or' if op == 'and' else 'and' + parts = [f'({s})' if isinstance(a, tuple) and a[0] == other else s + for a, s in sorted(((a, _expr_to_gpr_string(a)) for a in args), key=lambda p: p[1])] + return f' {op} '.join(parts) - Args: - gpr_bodies: list of AST nodes (reaction.gpr.body), may include None - An empty/None GPR means the reaction has no gene requirement (always active), - which acts as True in boolean logic. AND with True is a no-op, so empty GPRs - are skipped. Returns '' if all inputs are empty (no gene restriction). +def _combine_gprs(gpr_bodies, op): + """Combine GPR AST bodies (reaction.gpr.body, may include None) under ``op``, as a rule string. - Uses sympy And constructor which automatically flattens nested ANDs and - deduplicates terms. Full simplification is deferred to reduce_gpr downstream. + An empty GPR is True: under 'and' it is dropped, under 'or' it makes the whole rule + unrestricted (''). Used to merge the GPRs of reactions that compression lumps together -- + 'and' for coupled/serial merges, 'or' for parallel ones. """ - sympy_exprs = [_gpr_ast_to_sympy(b) for b in gpr_bodies] - non_empty = [s for s in sympy_exprs if s is not None] - if not non_empty: + exprs = [_gpr_ast_to_expr(b) for b in gpr_bodies] + if op == 'or' and (not exprs or any(e is None for e in exprs)): return '' - if len(non_empty) == 1: - return _sympy_to_gpr_string(non_empty[0]) - combined = SympyAnd(*non_empty) - return _sympy_to_gpr_string(combined) + return _expr_to_gpr_string(_gpr_ast_to_expr(exprs, op)) -def _combine_gpr_or(gpr_bodies): - """Combine GPR AST bodies with OR logic (for parallel reaction merge). +# High-Level Compression API - Args: - gpr_bodies: list of AST nodes (reaction.gpr.body), may include None - If any input is None (reaction always active regardless of genes), the - combined reaction is also always active, so the result is '' (no restriction). +# Monotone (positive-unate) GPR-rule simplification +# +# Pipeline: parse -> minimal SOP (DNF + absorption) -> algebraic factoring. +# Cubes are int bitmasks (bit i == variable i): subset = (a & b) == a, union = a | b. +# Output is inverter-free by construction and boolean-EQUIVALENT to the input, so replacing a +# reaction's GPR with its factored form leaves flux/knockout semantics -- and strain designs -- +# unchanged, while shrinking the GPR gadget built by extend_model_gpr. `factor_auto(node, budget)` +# guards the only source of DNF blow-up (an AND of large ORs) by AND-splitting over-budget +# conjuncts (exact, near-optimal since complexes sit on ~disjoint genes). `simplify_model_gprs(model)` +# is the entry point; compress_model calls it when propagate_gpr is set so standalone compression +# emits already-simplified rules. - Uses sympy Or constructor which automatically flattens nested ORs and - deduplicates terms. Full simplification is deferred to reduce_gpr downstream. - """ - sympy_exprs = [_gpr_ast_to_sympy(b) for b in gpr_bodies] - if any(s is None for s in sympy_exprs): - return '' - if not sympy_exprs: - return '' - if len(sympy_exprs) == 1: - return _sympy_to_gpr_string(sympy_exprs[0]) - combined = SympyOr(*sympy_exprs) - return _sympy_to_gpr_string(combined) +# popcount: C-level int.bit_count() on Python 3.10+, else the bin().count fallback +_popcount = getattr(int, 'bit_count', None) or (lambda c: bin(c).count('1')) -# ============================================================================= -# High-Level Compression API -# ============================================================================= +def _gpr_tokenize(s): + for m in re.finditer(r'\(|\)|\*|\+|[^\s()*+]+', s): + yield m.group() + + +def _gpr_parse(s): + """Parse a GPR string into an AST. + + Accepts both ``and``/``or`` and ``*``/``+`` operators, and is robust to any gene id, + including digit-leading or dotted names. + """ + toks = list(_gpr_tokenize(s)); pos = 0 + def peek(): return toks[pos] if pos < len(toks) else None + def eat(): + nonlocal pos; t = toks[pos]; pos += 1; return t + def p_or(): + n = [p_and()] + while peek() in ('or', '+'): eat(); n.append(p_and()) + return ('OR', n) if len(n) > 1 else n[0] + def p_and(): + n = [p_atom()] + while peek() in ('and', '*'): eat(); n.append(p_atom()) + return ('AND', n) if len(n) > 1 else n[0] + def p_atom(): + if peek() == '(': eat(); e = p_or(); eat(); return e + return ('VAR', eat()) + return p_or() + + +# variable <-> bit mapping (reset per rule via simplify_gpr_string) +_GPR_VMAP = {}; _GPR_VINV = [] +def _gpr_bit(v): + i = _GPR_VMAP.get(v) + if i is None: + i = len(_GPR_VINV); _GPR_VMAP[v] = i; _GPR_VINV.append(v) + return 1 << i +def _gpr_lits_of(mask): + out = [] + while mask: + l = mask & -mask; out.append(('VAR', _GPR_VINV[l.bit_length() - 1])); mask ^= l + return out + + +# cover algebra (cubes = ints) +def _gpr_absorb(cubes): + uniq = set(cubes) + buckets = {} + for c in uniq: + buckets.setdefault(_popcount(c), []).append(c) + keep = [] + for pc in sorted(buckets): + smaller = keep[:] + for c in buckets[pc]: + if not any((k & c) == k for k in smaller): + keep.append(c) + return keep + + +def _gpr_to_dnf(node): + t = node[0] + if t == 'VAR': return [_gpr_bit(node[1])] + if t == 'CONST': return [] if not node[1] else [0] + if t == 'OR': + cov = [] + for ch in node[1]: cov += _gpr_to_dnf(ch) + return _gpr_absorb(cov) + if t == 'AND': + cov = [0] + for ch in node[1]: + sub = _gpr_to_dnf(ch) + cov = _gpr_absorb([a | b for a in cov for b in sub]) + return cov + raise ValueError(t) + + +def _gpr_common(cubes): + it = iter(cubes); c = next(it) + for x in it: c &= x + return c + + +def _gpr_lit_counts(F): + cnt = {} + for c in F: + m = c + while m: + l = m & -m; cnt[l] = cnt.get(l, 0) + 1; m ^= l + return cnt + + +def _gpr_one_kernel(F, l): + Q = [c & ~l for c in F if c & l] + cc = _gpr_common(Q) + if cc: Q = [c & ~cc for c in Q] + Q = _gpr_absorb(Q) + cnt = _gpr_lit_counts(Q) + reps = [x for x, n in cnt.items() if n >= 2] + if not reps: return Q + return _gpr_one_kernel(Q, max(reps, key=lambda x: cnt[x])) + + +def _gpr_candidate_divisors(F): + F = _gpr_absorb(F) + if len(F) < 2: return [] + cnt = _gpr_lit_counts(F) + reps = sorted((x for x, n in cnt.items() if n >= 2), key=lambda x: -cnt[x]) + seen = set(); out = [] + for l in reps: + K = tuple(sorted(_gpr_one_kernel(F, l))) + if len(K) >= 2 and K not in seen: + seen.add(K); out.append(list(K)) + return out + + +def _gpr_divide(F, D): + """Exact algebraic division: (Q, R) with D*Q disjoint-union R == F (correctness guaranteed + regardless of divisor quality -- a quotient cube is accepted only if D*Q stays inside F).""" + Fs = set(F); quo = None + for d in D: + vd = {c & ~d for c in F if (c & d) == d} + quo = vd if quo is None else (quo & vd) + if not quo: return [], list(F) + Q = list(quo) + DQ = {dc | qc for dc in D for qc in Q} + if not DQ <= Fs: return [], list(F) + return Q, list(Fs - DQ) + + +def _gpr_factor(F): + F = _gpr_absorb(F) + if not F: return ('CONST', False) + if F == [0]: return ('CONST', True) + if len(F) == 1: + lits = _gpr_lits_of(F[0]) + return lits[0] if len(lits) == 1 else ('AND', lits) + cc = _gpr_common(F) + if cc: + rem = [c & ~cc for c in F] + return ('AND', _gpr_lits_of(cc) + [_gpr_factor(rem)]) + best = None + for D in _gpr_candidate_divisors(F): + Q, R = _gpr_divide(F, D) + if not Q or len(D) >= len(F) or len(Q) >= len(F): + continue + clean = 1 if not R else 0 + pulled = sum(_popcount(c) for c in D) + cand = (clean, pulled, D, Q, R) + if best is None or cand[:2] > best[:2]: + best = cand + if best is None: + return ('OR', [_gpr_factor([c]) for c in F]) + _, _, D, Q, R = best + dq = ('AND', [_gpr_factor(D), _gpr_factor(Q)]) + return dq if not R else ('OR', [dq, _gpr_factor(R)]) + + +def _gpr_est_cubes(node): + """Upper bound on DNF cube count (product across ANDs, sum across ORs); cheap, no expansion.""" + t = node[0] + if t == 'VAR': return 1 + if t == 'CONST': return 1 + if t == 'OR': return sum(_gpr_est_cubes(c) for c in node[1]) + if t == 'AND': + p = 1 + for c in node[1]: + p *= _gpr_est_cubes(c) + if p > 1 << 62: return p + return p + + +_GPR_WARN = [] +def _gpr_factor_auto(node, budget=50000): + """Global factoring within budget; AND-split above it. Never splits an OR unless one single + OR-block alone exceeds budget (logged as a last resort -- raise the budget to avoid).""" + if node[0] == 'VAR': + return node + if _gpr_est_cubes(node) <= budget: + return _gpr_factor(_gpr_to_dnf(node)) + if node[0] == 'AND': + return ('AND', [_gpr_factor_auto(c, budget) for c in node[1]]) + _GPR_WARN.append("OR-block of ~%d cubes exceeds budget %d; split anyway." % (_gpr_est_cubes(node), budget)) + return ('OR', [_gpr_factor_auto(c, budget) for c in node[1]]) + + +def _gpr_to_string(n): + if n[0] == 'VAR': + return n[1] + if n[0] == 'CONST': + return '' # tautology -> no gene requirement + if n[0] == 'AND': + return ' and '.join(('(%s)' % _gpr_to_string(c)) if c[0] == 'OR' else _gpr_to_string(c) for c in n[1]) + return ' or '.join(('(%s)' % _gpr_to_string(c)) if c[0] == 'AND' else _gpr_to_string(c) for c in n[1]) + + +def simplify_gpr_string(rule, budget=50000): + """Return a leaf-minimized, boolean-equivalent monotone GPR string ('' passes through).""" + if not rule or not rule.strip(): + return rule + _GPR_VMAP.clear(); _GPR_VINV.clear(); _GPR_WARN.clear() + return _gpr_to_string(_gpr_factor_auto(_gpr_parse(rule), budget)) + + +def simplify_model_gprs(model, budget=50000): + """In place: replace each reaction's gene_reaction_rule with a leaf-minimized equivalent. + + Monotone AND/OR boolean-equivalence => flux/knockout semantics (and strain designs) unchanged; + only the GPR gadget built by extend_model_gpr shrinks. Any per-rule failure keeps the original. + """ + n = nchg = 0 + for r in model.reactions: + s = r.gene_reaction_rule + if not s: + continue + n += 1 + try: + new = simplify_gpr_string(s, budget) + if new and new != s: + r.gene_reaction_rule = new; nchg += 1 + except Exception as e: + logging.warning('gpr_simplify: kept original GPR for %s (%s)' % (r.id, type(e).__name__)) + logging.info(' GPR rule simplification: %d rules, %d rewritten.' % (n, nchg)) def compress_model(model, no_par_compress_reacs=set(), compression_backend='sparse_rref', propagate_gpr=False, @@ -1890,10 +2127,19 @@ def compress_model(model, no_par_compress_reacs=set(), compression_backend='spar with suppress_lp_context(model): cmp_mapReac = [] use_java = (compression_backend == 'efmtool_rref') + if use_java: + # The Python compressor re-expresses each lump in one member's units (see + # StoichMatrixCompressor._restore_group_scale); the legacy Java backend does not, so a + # lump can come out at an extreme scale. The returned map carries the factor, so + # expanding a design stays exact -- but a bound stated on a lumped reaction is read in + # the lump's units, which is how 'biomass >= 0.001' can end up below feasibility tolerance. + LOG.warning(' Compression backend "efmtool_rref" does not normalize lumped-reaction ' + 'scales; bounds and constraints on lumped reactions are expressed in the ' + 'lump\'s units. Use "sparse_rref" if you constrain lumped reactions.') LOG.info(' Removing blocked reactions.') remove_blocked_reactions(model) LOG.info(' Converting coefficients to rationals.') - stoichmat_coeff2rational(model) + stoichmat_coeff_to_fraction(model) coupled_changed = None # None = not yet computed run = 1 while True: @@ -1941,7 +2187,14 @@ def compress_model(model, no_par_compress_reacs=set(), compression_backend='spar run += 1 - # suppress_lp_context handles solver rebuild and objective restoration on exit + # suppress_lp_context handles solver rebuild, objective restoration and stale-group pruning + # on exit + if propagate_gpr: + # Leaf-minimize the propagated rules so any caller (incl. standalone compression, not just + # the SD pipeline) gets simplified GPRs. Monotone/boolean-equivalent -> designs unchanged; + # only the extend_model_gpr gadget shrinks. In the pipeline this runs again after reduce, but + # simplification is cheap and idempotent. + simplify_model_gprs(model) return cmp_mapReac @@ -1983,41 +2236,40 @@ def compress_model_coupled(model, compression_backend='sparse_rref', propagate_g Returns: dict: Mapping {compressed_id: {orig_id: factor, ...}} """ - # Save GPR AST bodies before either backend clears them - if propagate_gpr: - saved_gpr_bodies = {r.id: r.gpr.body for r in model.reactions} - - if compression_backend == 'efmtool_rref': - from .efmtool_cmp_interface import compress_model_java - reaction_map = compress_model_java(model, suppressed_reactions=suppressed_reactions) - # Java backend handles contradicting groups internally (CoupledContradicting). - # Clean up any remaining zero-flux reactions that the Java compressor created. - zero_flux = {r for r in model.reactions if r.lower_bound == 0 and r.upper_bound == 0} - for r in zero_flux: - reaction_map.pop(r.id, None) - if zero_flux: - model.remove_reactions(list(zero_flux), remove_orphans=True) - else: - # Clear gene rules to match Java behavior - for r in model.reactions: - r.gene_reaction_rule = '' - - result = compress_cobra_model(model, methods=CompressionMethod.standard(), in_place=True, - protected_reactions=protected_reactions) - reaction_map = result.reaction_map - # Python compressor handles contradicting groups internally via bounds - # intersection in _handle_compress (removes zero-flux groups and - # re-iterates to find new couplings). - - # Propagate GPR rules: AND-combine contributing reactions' GPR ASTs - if propagate_gpr: - for cmp_id, orig_map in reaction_map.items(): - try: - rxn = model.reactions.get_by_id(cmp_id) - except KeyError: - continue - gpr_bodies = [saved_gpr_bodies.get(orig_id) for orig_id in orig_map] - rxn.gene_reaction_rule = _combine_gpr_and(gpr_bodies) + # Compression is pure linear algebra; keep it off the optlang solver. + from straindesign.networktools import suppress_lp_context + with suppress_lp_context(model): + # Save GPR AST bodies before either backend clears them + if propagate_gpr: + saved_gpr_bodies = {r.id: r.gpr.body for r in model.reactions} + + if compression_backend == 'efmtool_rref': + from .efmtool_cmp_interface import compress_model_java + reaction_map = compress_model_java(model, suppressed_reactions=suppressed_reactions) + # Clean up any remaining zero-flux reactions that the Java compressor created. + zero_flux = {r for r in model.reactions if r.lower_bound == 0 and r.upper_bound == 0} + for r in zero_flux: + reaction_map.pop(r.id, None) + if zero_flux: + model.remove_reactions(list(zero_flux), remove_orphans=True) + else: + # Clear gene rules to match Java behavior + for r in model.reactions: + r.gene_reaction_rule = '' + + result = compress_cobra_model(model, methods=CompressionMethod.standard(), in_place=True, + protected_reactions=protected_reactions) + reaction_map = result.reaction_map + + # Propagate GPR rules: AND-combine contributing reactions' GPR ASTs + if propagate_gpr: + for cmp_id, orig_map in reaction_map.items(): + try: + rxn = model.reactions.get_by_id(cmp_id) + except KeyError: + continue + gpr_bodies = [saved_gpr_bodies.get(orig_id) for orig_id in orig_map] + rxn.gene_reaction_rule = _combine_gprs(gpr_bodies, 'and') return reaction_map @@ -2064,30 +2316,22 @@ def _parallel_key(i): cols, vals = stoichmat_T.rows[i], stoichmat_T.data[i] if not vals: return ((), fwd[i], rev[i], inh[i]) - f0 = float_to_rational(vals[0]) - stoich = tuple((int(c), float_to_rational(v) / f0) for c, v in zip(cols, vals)) + f0 = float_to_fraction(vals[0]) + stoich = tuple((int(c), float_to_fraction(v) / f0) for c, v in zip(cols, vals)) return (stoich, fwd[i], rev[i], inh[i]) # Find parallel reactions by exact key comparison (hash pre-filter, then full compare) - subset_list = [] - prev_found = set() protected = [r.id in protected_rxns for r in model.reactions] keys = [_parallel_key(i) for i in range(len(model.reactions))] - key_hashes = [hash(k) for k in keys] - for i in range(len(model.reactions)): - if i in prev_found: - continue - if protected[i]: - subset_list.append([i]) - continue - subset_i = [i] - for j in range(i + 1, len(model.reactions)): - if (not protected[j] and j not in prev_found - and key_hashes[i] == key_hashes[j] and keys[i] == keys[j]): - subset_i.append(j) - prev_found.add(j) - subset_list.append(subset_i) + # Group reactions that share an exact key in a single O(n) pass. dict preserves first-occurrence + # order, so each group's representative is its smallest index and subset_list stays ordered by + # ascending representative -- matching the surviving-reaction order after remove_reactions below. + # Protected reactions get a unique 2-tuple key (real keys are 4-tuples) so they never merge. + groups = {} + for i, key in enumerate(keys): + groups.setdefault(('\0protected', i) if protected[i] else key, []).append(i) + subset_list = list(groups.values()) # Lump parallel reactions del_rxns = [False] * len(model.reactions) @@ -2114,7 +2358,7 @@ def _parallel_key(i): for rxn_idx_group in subset_list: main_rxn = model.reactions[rxn_idx_group[0]] gpr_bodies = [old_gpr_bodies.get(old_reac_ids[j]) for j in rxn_idx_group] - group_gpr.append((main_rxn, _combine_gpr_or(gpr_bodies))) + group_gpr.append((main_rxn, _combine_gprs(gpr_bodies, 'or'))) remove_list = [model.reactions[i] for i in np.where(del_rxns)[0]] if remove_list: @@ -2148,14 +2392,12 @@ def _parallel_key(i): return rational_map -# ============================================================================= # Exports -# ============================================================================= __all__ = [ # Rational matrix and utilities 'RationalMatrix', - 'float_to_rational', + 'float_to_fraction', 'detect_max_precision', 'nullspace', 'basic_columns', @@ -2174,15 +2416,17 @@ def _parallel_key(i): 'compress_model_efmtool', # backward-compat alias 'compress_model_parallel', # GPR propagation helpers - '_gpr_ast_to_sympy', - '_sympy_to_gpr_string', - '_combine_gpr_and', - '_combine_gpr_or', + '_gpr_ast_to_expr', + '_expr_to_gpr_string', + '_combine_gprs', + # GPR rule simplification + 'simplify_model_gprs', + 'simplify_gpr_string', # Preprocessing 'remove_blocked_reactions', 'remove_ext_mets', 'remove_conservation_relations', 'remove_dummy_bounds', - 'stoichmat_coeff2rational', + 'stoichmat_coeff_to_fraction', 'stoichmat_coeff2float', ] diff --git a/straindesign/compute_strain_designs.py b/straindesign/compute_strain_designs.py index 66caf0e..7e53bc8 100644 --- a/straindesign/compute_strain_designs.py +++ b/straindesign/compute_strain_designs.py @@ -21,6 +21,7 @@ from typing import Dict, List, Tuple import numpy as np +import ast import logging import json import time @@ -30,9 +31,10 @@ from straindesign import SDModule, SDSolutions, select_solver, fva, DisableLogger, SDProblem, SDMILP from straindesign.names import * from straindesign.networktools import remove_ext_mets, bound_blocked_or_irrevers_fva, \ - reduce_gpr, extend_model_gpr, extend_model_regulatory, \ + extend_model_gpr, extend_model_regulatory, evaluate_gpr_ast, \ compress_model, compress_modules, compress_ki_ko_cost, expand_sd, filter_sd_maxcost, \ estimate_expansion_size, with_suppressed_lp, _silent_io +from straindesign.compression import simplify_model_gprs def _collect_no_par_compress_reacs(sd_modules): @@ -52,7 +54,218 @@ def _collect_no_par_compress_reacs(sd_modules): return reacs +# ── GPR reduction (pipeline-only: needs essential reactions + gene KO/KI costs) ── +# A module's flux range must exclude zero by more than this to count as essential. Ten times the +# backends' 1e-9 feasibility tolerance: below that a reported range is indistinguishable from one +# that touches zero. Erring high only leaves reactions knockable, which cannot lose a design. +_ESSENTIALITY_TOL = 1e-8 + + +def _essentials_from_limits(flux_limits): + """Reactions whose flux range inside a module excludes zero, i.e. essential to that module. + + ``flux_limits`` is an FVA result over the module's constrained polytope. Both bounds must share + a sign and stay clear of zero, so the reaction carries flux in every point of the module. + """ + return {reac_id for reac_id, limits in flux_limits.iterrows() + if np.min(abs(limits)) > _ESSENTIALITY_TOL and np.prod(np.sign(limits)) > 0} + + +def reduce_model_gprs(model, essential_reacs, gkis, gkos): + """Simplify GPR rules by removing non-targetable genes and reducing boolean expressions + + This function is used in preprocessing of computational strain design computations. Often, + certain reactions, for instance, reactions essential for microbial growth can/must not be + targeted by interventions. That can be exploited to reduce the set of genes in which + interventions need to be considered. + + Given a set of essential reactions that is to be maintained operational, some genes can be + removed from a metabolic model, either because they only affect only blocked reactions or + essential reactions, or because they are essential reactions and must not be removed. As a + consequence, the GPR rules of a model can be simplified using AST parsing for both DNF and non-DNF rules. + + + Example: + reduce_model_gprs(model, essential_reacs, gkis, gkos): + + Args: + model (cobra.Model): + A metabolic model that is an instance of the cobra.Model class containing GPR rules + + essential_reacs (list of str): + A list of identifiers of essential reactions. + + gkis, gkos (dict): + Dictionaries that contain the costs for gene knockouts and additions. E.g., + gkos={'adhE': 1.0, 'ldhA' : 1.0 ...} + + Returns: + (dict): + An updated dictionary of the knockout costs in which irrelevant genes are removed. + """ + + def ast_to_gene_reaction_rule(node): + """ + Convert an AST node back to gene reaction rule string format. + """ + if isinstance(node, ast.Name): + return node.id + elif isinstance(node, ast.BoolOp): + child_strings = [ast_to_gene_reaction_rule(child) for child in node.values] + if isinstance(node.op, ast.And): + return ' and '.join(f'({s})' if ' or ' in s else s for s in child_strings) + elif isinstance(node.op, ast.Or): + return ' or '.join(f'({s})' if ' and ' in s else s for s in child_strings) + else: + raise ValueError(f"Unsupported AST node type: {type(node)}") + + def simplify_gpr_ast(node, protected_genes_dict): + """ + Simplify GPR AST by setting protected genes to True and applying boolean simplification. + This is equivalent to the original string-based approach but operates purely on AST. + """ + return apply_gene_protection_to_ast(node, protected_genes_dict) + + def apply_gene_protection_to_ast(node, protected_genes_dict): + """ + Apply gene protection to AST by setting protected genes to True and simplifying boolean expressions. + Returns a simplified AST node with redundant terms removed and consistent gene ordering. + """ + if isinstance(node, ast.Name): + if node.id in protected_genes_dict: + return True + else: + return node + elif isinstance(node, ast.BoolOp): + # Recursively apply to children + new_children = [] + for child in node.values: + simplified_child = apply_gene_protection_to_ast(child, protected_genes_dict) + + if isinstance(node.op, ast.And): + if simplified_child is False: + return False + elif simplified_child is not True: + new_children.append(simplified_child) + elif isinstance(node.op, ast.Or): + if simplified_child is True: + return True + elif simplified_child is not False: + new_children.append(simplified_child) + + # Handle results + if not new_children: + return True if isinstance(node.op, ast.And) else False + elif len(new_children) == 1: + return new_children[0] + else: + # (b) De-dup: boolean simplification (absorption/dedup of OR terms) is delegated to + # simplify_model_gprs, which runs right after reduce_model_gprs on every path that + # runs reduce. Here we only apply the protected-gene substitution + True/False + # elimination and keep a stable child ordering. + sorted_children = sort_ast_nodes(new_children) + new_node = ast.BoolOp(op=node.op, values=sorted_children) + return new_node + else: + raise ValueError(f"Unsupported AST node type: {type(node)}") + + def sort_ast_nodes(nodes): + """Sort AST nodes for consistent ordering""" + + def node_sort_key(node): + if isinstance(node, ast.Name): + return (0, node.id) + elif isinstance(node, ast.BoolOp): + return (1, len(node.values), str(type(node.op))) + return (2, str(node)) + + return sorted(nodes, key=node_sort_key) + + def is_gene_essential_to_reaction_ast(reaction, gene_id): + """ + Determine if a gene is essential for a reaction using AST-based GPR analysis. + A gene is considered essential if removing it (setting it to False) makes + the entire GPR expression evaluate to False, rendering the reaction impossible. + """ + if not reaction.gene_reaction_rule: + return False + + # Skip reactions without gene associations + if not reaction.gpr or not reaction.gpr.body: + return False + + try: + # Test what happens if we knock out this gene using AST + gene_states = {gene_id: False} + result = evaluate_gpr_ast(reaction.gpr.body, gene_states) + return result is False + except Exception as e: + # Catch unsupported AST node types but don't fall back to string parsing + logging.warning(f'Unsupported AST node type in reaction {reaction.id} for gene {gene_id}: {e}') + return False + + # 1) Remove gpr rules from blocked reactions + blocked_reactions = [reac.id for reac in model.reactions if reac.bounds == (0, 0)] + for rid in blocked_reactions: + model.reactions.get_by_id(rid).gene_reaction_rule = '' + for g in model.genes[::-1]: # iterate in reverse order to avoid mixing up the order of the list when removing genes + if not g.reactions: + model.genes.remove(g) + + protected_genes = set() + + # 2. Protect genes that only occur in essential reactions + for g in model.genes: + if not g.reactions or {r.id for r in g.reactions}.issubset(essential_reacs): + protected_genes.add(g) + + # 3. Protect genes that are essential to essential reactions (AST-based analysis) + for r in [model.reactions.get_by_id(s) for s in essential_reacs]: + for g in r.genes: + if is_gene_essential_to_reaction_ast(r, g.id): + protected_genes.add(g) + + # 4. Remove essential genes, and knockouts without impact from gko_costs + [gkos.pop(pg.id) for pg in protected_genes if pg.id in gkos] + + # 5. Add all not-knockable genes to the protected list + [protected_genes.add(g) for g in model.genes if (g.id not in gkos) and (g.name not in gkos)] # support names or ids in gkos + + # 6. genes with kiCosts are kept (remove from protected list so they can be targeted) + gki_ids = [g.id for g in model.genes if (g.id in gkis) or (g.name in gkis)] # support names or ids in gkis + protected_genes = protected_genes.difference({model.genes.get_by_id(g) for g in gki_ids}) + protected_genes_dict = {pg.id: True for pg in protected_genes} + + # 7. Simplify GPR rules using AST-based boolean logic and remove non-targetable rules + for r in model.reactions: + if r.gene_reaction_rule and r.gpr and r.gpr.body: + try: + simplified = simplify_gpr_ast(r.gpr.body, protected_genes_dict) + + if simplified is True: + # Rule is always satisfied (cannot be knocked out) + model.reactions.get_by_id(r.id).gene_reaction_rule = '' + elif simplified is False: + # Rule is impossible - should not happen with proper protection + logging.error(f'Something went wrong during gpr rule simplification for {r.id}.') + elif isinstance(simplified, (ast.Name, ast.BoolOp)): + # Convert simplified AST back to string + new_rule = ast_to_gene_reaction_rule(simplified) + model.reactions.get_by_id(r.id).gene_reaction_rule = new_rule + # If simplified is the original node, keep original rule + except Exception as e: + logging.warning(f'Failed to simplify GPR rule for reaction {r.id}: {e}') + + # 8. Remove obsolete genes and protected genes + for g in model.genes[::-1]: + if not g.reactions or g in protected_genes: + model.genes.remove(g) + + return gkos + + @with_suppressed_lp + def compute_strain_designs(model: Model, **kwargs: dict) -> SDSolutions: """Computes strain designs for a user-defined strain design problem @@ -352,6 +565,24 @@ def compute_strain_designs(model: Model, **kwargs: dict) -> SDSolutions: # compression passes (keeps the coupled-exemption matching them by name) no_par_compress_reacs.update(no_coupled_compress_reacs) compression_backend = kwargs.get('compression_backend', 'sparse_rref') + # --- Reversibility pre-tightening (BEFORE compress #1) --- + # Sign-only FVA (cheaper than full FVA): fix lb/ub to 0 for directions carrying no flux in the + # base polytope. Design-neutral (a base-infeasible direction stays infeasible under any module + # constraint) -- the same tightening SD applies after compress #2, just moved up. Doing it here + # lets compress #1 fuse the now one-directional reactions and spares genuinely irreversible ones + # from the GPR fwd/rev split (which fires on lb<0). + from straindesign.speedy_fva import fast_reversibility + t0 = time.time() + _rev = fast_reversibility(cmp_model, solver=kwargs[SOLVER]) + _n_tight = 0 + for r in cmp_model.reactions: + can_fwd, can_rev = _rev[r.id] + if not can_fwd and float(r._upper_bound) > 0.0: + r._upper_bound = min(0.0, float(r._upper_bound)); _n_tight += 1 + if not can_rev and float(r._lower_bound) < 0.0: + r._lower_bound = max(0.0, float(r._lower_bound)); _n_tight += 1 + logging.info(' Reversibility pre-tightening fixed %d reaction directions (%.1fs).' + % (_n_tight, time.time() - t0)) logging.info('Compressing Network (' + str(len(cmp_model.reactions)) + ' reactions).') t0 = time.time() cmp_mapReac_1 = compress_model(cmp_model, no_par_compress_reacs, @@ -374,24 +605,32 @@ def compute_strain_designs(model: Model, **kwargs: dict) -> SDSolutions: if m[MODULE_TYPE] != SUPPRESS: # Essential reactions can only be determined from desired # or opt-/robustknock modules flux_limits = fva(cmp_model, solver=kwargs[SOLVER], constraints=m[CONSTRAINTS], compress=False) - for (reac_id, limits) in flux_limits.iterrows(): - if np.min(abs(limits)) > 1e-10 and np.prod(np.sign(limits)) > 0: # find essential - essential_reacs.add(reac_id) + essential_reacs.update(_essentials_from_limits(flux_limits)) # remove ko-costs (and thus knockability) of essential reactions [cmp_ko_cost.pop(er) for er in essential_reacs if er in cmp_ko_cost] # --- GPR extension on (possibly compressed) model --- if kwargs['gene_kos']: - if kwargs['compress'] is True or kwargs['compress'] is None: + # GPR reduction has two leaf-minimizing, boolean-equivalent (designs unchanged) steps: + # reduce_model_gprs (compress-only; also drops irrelevant/essential genes) and the monotone + # simplify_model_gprs. simplify_model_gprs stays separate rather than folded into + # reduce_model_gprs because it must ALSO run on the no-compress path (below). Running both here, + # before the count log, lets that log reflect the fully reduced gene/gpr counts and the + # combined elapsed time. + t_gpr = time.time() + compress_gpr = kwargs['compress'] is True or kwargs['compress'] is None + if compress_gpr: num_genes = len(cmp_model.genes) num_gpr = len([True for r in cmp_model.reactions if r.gene_reaction_rule]) logging.info('Preprocessing GPR rules (' + str(num_genes) + ' genes, ' + str(num_gpr) + ' gpr rules).') # removing irrelevant genes will also remove essential reactions from the list of knockable genes - uncmp_gko_cost = reduce_gpr(cmp_model, essential_reacs, uncmp_gki_cost, uncmp_gko_cost) - if len(cmp_model.genes) < num_genes or len([True for r in cmp_model.reactions if r.gene_reaction_rule]) < num_gpr: - num_genes = len(cmp_model.genes) - num_gpr = len([True for r in cmp_model.reactions if r.gene_reaction_rule]) - logging.info(' Simplified to ' + str(num_genes) + ' genes and ' + - str(num_gpr) + ' gpr rules.') + uncmp_gko_cost = reduce_model_gprs(cmp_model, essential_reacs, uncmp_gki_cost, uncmp_gko_cost) + simplify_model_gprs(cmp_model) + if compress_gpr and (len(cmp_model.genes) < num_genes or + len([True for r in cmp_model.reactions if r.gene_reaction_rule]) < num_gpr): + num_genes = len(cmp_model.genes) + num_gpr = len([True for r in cmp_model.reactions if r.gene_reaction_rule]) + logging.info(' Simplified to ' + str(num_genes) + ' genes and ' + + str(num_gpr) + ' gpr rules (%.1fs).' % (time.time() - t_gpr)) logging.info(' Extending metabolic network with gpr associations.') reac_map = extend_model_gpr(cmp_model, has_gene_names) for i, m in enumerate(sd_modules): @@ -455,28 +694,55 @@ def compute_strain_designs(model: Model, **kwargs: dict) -> SDSolutions: if not (float(r.lower_bound) == 0.0 and np.isinf(float(r.upper_bound)) and float(r.upper_bound) > 0)] - bound_blocked_or_irrevers_fva(cmp_model, solver=kwargs[SOLVER], compress=False, - reaction_list=_fva_scope) - logging.info(' FVA done (%.1fs).' % (time.time() - t0)) - - # FVA to identify essential reactions and size-1 MCS before building MILP - logging.info(' FVA(s) in compressed model to identify essential reactions.') essential_reacs = set() suppress_essential = set() cmp_size1_mcs = [] - # Scope FVA to knockable reactions only (essentiality of non-knockable reactions is irrelevant) knockable_ids = list(set(cmp_ko_cost.keys()) | set(cmp_ki_cost.keys())) - for m in sd_modules: - flux_limits = fva(cmp_model, solver=kwargs[SOLVER], constraints=m[CONSTRAINTS], - compress=False, reaction_list=knockable_ids) - essentials_in_module = set() - for (reac_id, limits) in flux_limits.iterrows(): - if np.min(abs(limits)) > 1e-10 and np.prod(np.sign(limits)) > 0: - essentials_in_module.add(reac_id) - if m[MODULE_TYPE] != SUPPRESS: - essential_reacs.update(essentials_in_module) - else: + + # With exactly one classical module, one FVA over the constrained module polytope can serve both + # model-bound tightening and module essentiality. This is only sound for a single module: applying + # one module's tighter ranges to the shared model could otherwise alter another module's polytope. + fold_module_fva = ( + len(sd_modules) == 1 + and sd_modules[0][MODULE_TYPE] in [SUPPRESS, PROTECT] + and sd_modules[0][INNER_OBJECTIVE] is None + ) + if fold_module_fva: + module = sd_modules[0] + fold_scope = sorted(set(_fva_scope) | set(knockable_ids)) + flux_limits = bound_blocked_or_irrevers_fva( + cmp_model, solver=kwargs[SOLVER], constraints=module[CONSTRAINTS], + compress=False, reaction_list=fold_scope) + module_limits = flux_limits.loc[ + [reac_id for reac_id in knockable_ids if reac_id in flux_limits.index]] + module['fva_bounds'] = module_limits + essentials_in_module = _essentials_from_limits(module_limits) + if module[MODULE_TYPE] == SUPPRESS: suppress_essential.update(essentials_in_module) + else: + essential_reacs.update(essentials_in_module) + logging.info(' Folded model/module FVA done (%.1fs).' % (time.time() - t0)) + else: + bound_blocked_or_irrevers_fva( + cmp_model, solver=kwargs[SOLVER], compress=False, reaction_list=_fva_scope) + logging.info(' FVA done (%.1fs).' % (time.time() - t0)) + + # FVA to identify essential reactions and size-1 MCS before building MILP + logging.info(' FVA(s) in compressed model to identify essential reactions.') + # FVA over each module's region, scoped to knockable reactions. The ranges serve two purposes: + # (1) essentiality for size-1 MCS detection, and (2) region-FVA subproblem tightening, read back + # in SDMILP, which is why SDProblem runs no region FVA of its own. flux_limits is stored on the + # module and flows to SDMILP via sd_modules. Scoping to knockable reactions keeps + # the LP count down (and only knockable reactions carry z-links to tighten anyway). + for module in sd_modules: + flux_limits = fva(cmp_model, solver=kwargs[SOLVER], constraints=module[CONSTRAINTS], + compress=False, reaction_list=knockable_ids) + module['fva_bounds'] = flux_limits + essentials_in_module = _essentials_from_limits(flux_limits) + if module[MODULE_TYPE] == SUPPRESS: + suppress_essential.update(essentials_in_module) + else: + essential_reacs.update(essentials_in_module) # Size-1 MCS detection: only for classical MCS problems (one SUPPRESS + any PROTECT) is_classical_mcs = (len([m for m in sd_modules if m[MODULE_TYPE] == SUPPRESS]) == 1 and @@ -540,6 +806,10 @@ def compute_strain_designs(model: Model, **kwargs: dict) -> SDSolutions: else: solution_approach = BEST + # SDMILP.enumerate_ksweep is an alternative POPULATE loop, disabled for now. It is complete only + # for integer-valued intervention costs, and was faster on CPLEX gene-MCS but slower on gurobi. + # enum_method = kwargs.pop('enum_method', 'populate') + dump_preprocessed = kwargs.pop('dump_preprocessed', None) if dump_preprocessed: diff --git a/straindesign/efmtool_cmp_interface.py b/straindesign/efmtool_cmp_interface.py index 29a03a7..190df4d 100644 --- a/straindesign/efmtool_cmp_interface.py +++ b/straindesign/efmtool_cmp_interface.py @@ -307,6 +307,13 @@ def jBigFraction2sympyRat(val): return jBigIntegerPair2sympyRat(val.getNumerator(), val.getDenominator()) +def jBigFraction2fraction(val): + """Convert Java BigFraction to fractions.Fraction.""" + from fractions import Fraction + r = jBigIntegerPair2sympyRat(val.getNumerator(), val.getDenominator()) + return Fraction(int(r.p), int(r.q)) + + def jBigIntegerPair2sympyRat(numer, denom): """Convert Java BigInteger pair to sympy Rational (requires sympy).""" import sympy @@ -378,13 +385,13 @@ def compress_model_java(model, suppressed_reactions=set()): dict: Reaction map from compressed to original reactions with scaling factors """ import jpype - from .networktools import stoichmat_coeff2rational + from .networktools import stoichmat_coeff_to_fraction # Initialize Java if not already done _init_java() # Convert to rational coefficients for Java - stoichmat_coeff2rational(model) + stoichmat_coeff_to_fraction(model) for r in model.reactions: r.gene_reaction_rule = '' @@ -442,7 +449,7 @@ def compress_model_java(model, suppressed_reactions=set()): model.reactions[r0_mi].subset_stoich = [] for ai in rxn_ai: mi = active_to_model[ai] - factor = jBigFraction2sympyRat(comprec.post.getBigFractionValueAt(ai, j)) + factor = jBigFraction2fraction(comprec.post.getBigFractionValueAt(ai, j)) model.reactions[mi] *= factor if model.reactions[mi].lower_bound not in (0, -float('inf')): model.reactions[mi].lower_bound /= abs(subset_matrix[ai, j]) @@ -472,7 +479,7 @@ def compress_model_java(model, suppressed_reactions=set()): merged_obj = 0.0 for ai in rxn_ai: mi = active_to_model[ai] - factor = jBigFraction2sympyRat(comprec.post.getBigFractionValueAt(ai, j)) + factor = jBigFraction2fraction(comprec.post.getBigFractionValueAt(ai, j)) merged_obj += _obj.pop(old_reac_ids[mi], 0.0) * float(factor) if merged_obj != 0: _obj[model.reactions[r0_mi].id] = merged_obj diff --git a/straindesign/lptools.py b/straindesign/lptools.py index bf18174..d70f79d 100644 --- a/straindesign/lptools.py +++ b/straindesign/lptools.py @@ -547,7 +547,7 @@ def fba(model, **kwargs) -> Solution: if min_cx <= 0 or isnan(min_cx): num_prob.add_eq_constraints(c, [-1.0]) else: - num_prob.add_eq_constraints(c, min_cx) + num_prob.add_eq_constraints(c, [-min_cx]) x, _, _ = num_prob.solve() elif status not in [OPTIMAL, UNBOUNDED]: status = INFEASIBLE diff --git a/straindesign/networktools.py b/straindesign/networktools.py index d8a65ee..120cdf7 100644 --- a/straindesign/networktools.py +++ b/straindesign/networktools.py @@ -104,6 +104,71 @@ def set_linear_coefficients(self, *a, **kw): _SOLVER_STUB = _SolverStub('__stub__') + +class _CarrierSolver: + """Backend-free stand-in solver attached to suppressed model copies. + + Reports the real optlang interface (so ``model.problem`` / ``select_solver`` resolve) and + exposes empty constraint/variable containers so cobra's ``add_metabolites`` / + ``Reaction.add_metabolites`` run without a live backend: constraint/variable lookups fall + through the permissive ``Container.__getitem__`` to ``_SOLVER_STUB`` (no-op coefficient/bound + setters) and ``add`` is a no-op. The copy is never solved -- FVA/FBA build their own MILP_LP + from the stoichiometry -- so no real backend is needed. Building an empty ``iface.Model()`` + instead (the previous behaviour) forced ``add_cons_vars`` to push every gadget metabolite into + a live Gurobi/CPLEX model during ``extend_model_gpr`` (~0.5s wasted per copy on iML1515). + """ + __slots__ = ('interface', 'constraints', 'variables') + + def __init__(self, interface): + from optlang.container import Container + self.interface = interface + self.constraints = Container() + self.variables = Container() + + def add(self, *a, **kw): + pass + + def remove(self, *a, **kw): + pass + + def update(self, *a, **kw): + pass + + # Picklable (dump_preprocessed pickles cmp_model): store the interface by module name and + # rebuild empty containers on load -- the carrier is never solved, so this suffices. + def __getstate__(self): + return self.interface.__name__ + + def __setstate__(self, name): + import importlib + from optlang.container import Container + self.interface = importlib.import_module(name) + self.constraints = Container() + self.variables = Container() + + +def _suppressed_copy(model): + """``Model.copy`` while LP updates are suppressed: no deep copy of the optlang backend. + + A plain copy deepcopies the live solver, which rebuilds the whole Gurobi/CPLEX model (~3s per + copy on iML1515). Under suppression nothing reads that solver -- FVA builds its own LP and + compression manipulates the stoichiometry directly -- so the solver is swapped for a stub while + copying (~0.3s) and the copy is given a backend-free ``_CarrierSolver`` of the same interface. + The carrier exposes ``.interface`` and empty constraint/variable containers, so the GPR + extension's ``add_metabolites`` calls run without building (or pushing constraints into) a live + solver. + """ + iface = model.solver.interface # captured before stubbing + saved = model._solver + orig_copy = next(o for cls, attr, o in _ORIG_COBRA if cls is Model and attr == 'copy') + try: + model._solver = _SOLVER_STUB + new = orig_copy(model) + finally: + model._solver = saved + new._solver = _CarrierSolver(iface) + return new + _ORIG_CONTAINER_GETITEM = None # saved Container.__getitem__ @@ -186,6 +251,25 @@ def _suppressed_remove_metabolites(self, metabolite_list, destructive=False): _remove_metabolites_direct(self, remove_set) +def _suppressed_add_metabolites(self, metabolite_list): + """Bypass solver constraint creation: direct DictList manipulation. + + Mirrors ``cobra.Model.add_metabolites`` dedup + ``_model`` wiring but skips the optlang + ``Constraint`` construction and ``add_cons_vars`` -- both need a live backend and, on the + suppressed copy, only build a throwaway solver the code never reads (FVA/FBA construct their + own MILP_LP; the original model's solver is rebuilt on suppress-exit via ``_populate_solver``). + Safe under ``extend_model_gpr`` because it de-dupes gadget metabolites before calling this. + """ + if not hasattr(metabolite_list, '__iter__'): + metabolite_list = [metabolite_list] + metabolite_list = [x for x in metabolite_list if x.id not in self.metabolites] + if not metabolite_list: + return + for x in metabolite_list: + x._model = self + self.metabolites += metabolite_list + + # -- Saved originals (None = not suppressed) ---------------------------------- _ORIG_SLC = None # (cls, method) for Constraint.set_linear_coefficients @@ -210,6 +294,7 @@ def _suppress_lp_updates(model): - Model._populate_solver → no-op (rebuild on context exit) - Model.remove_reactions → direct list manipulation - Model.remove_metabolites → direct list manipulation + - Model.add_metabolites → direct list manipulation (no backend constraints) - Container.__getitem__ → return stub for missing keys Safe to call when already suppressed (idempotent). The real methods @@ -267,6 +352,12 @@ def _suppress_lp_updates(model): if Model.remove_metabolites is not _suppressed_remove_metabolites: _ORIG_COBRA.append((Model, 'remove_metabolites', Model.remove_metabolites)) Model.remove_metabolites = _suppressed_remove_metabolites + if Model.add_metabolites is not _suppressed_add_metabolites: + _ORIG_COBRA.append((Model, 'add_metabolites', Model.add_metabolites)) + Model.add_metabolites = _suppressed_add_metabolites + if Model.copy is not _suppressed_copy: + _ORIG_COBRA.append((Model, 'copy', Model.copy)) + Model.copy = _suppressed_copy # Permissive solver container: return stub for missing keys global _ORIG_CONTAINER_GETITEM @@ -346,6 +437,13 @@ def suppress_lp_context(model): if hasattr(model, '_suppressed_obj'): del model._suppressed_obj if current_ids != _pre_ids: + if model.groups: + kept = {c.id for c in + list(model.reactions) + list(model.metabolites) + list(model.genes)} + for grp in model.groups: + stale = [m for m in grp.members if m.id not in kept] + if stale: + grp.remove_members(stale) try: solver_interface = model.solver.interface model._solver = solver_interface.Model() @@ -392,7 +490,7 @@ def _silent_io(): remove_ext_mets, remove_conservation_relations, remove_dummy_bounds, - stoichmat_coeff2rational, + stoichmat_coeff_to_fraction, stoichmat_coeff2float, ) @@ -401,7 +499,7 @@ def evaluate_gpr_ast(node, gene_states): """Evaluate a GPR AST node with given gene states. Supports arbitrary nesting of AND/OR operators (not limited to DNF/CNF). - Used by both gene_kos_to_constraints and reduce_gpr. + Used by both gene_kos_to_constraints and reduce_model_gprs. Args: node: An ast.Name or ast.BoolOp node from a parsed GPR rule @@ -660,288 +758,6 @@ def resolve_gene_constraints(model, constraints): return clean_constraints -def reduce_gpr(model, essential_reacs, gkis, gkos): - """Simplify GPR rules by removing non-targetable genes and reducing boolean expressions - - This function is used in preprocessing of computational strain design computations. Often, - certain reactions, for instance, reactions essential for microbial growth can/must not be - targeted by interventions. That can be exploited to reduce the set of genes in which - interventions need to be considered. - - Given a set of essential reactions that is to be maintained operational, some genes can be - removed from a metabolic model, either because they only affect only blocked reactions or - essential reactions, or because they are essential reactions and must not be removed. As a - consequence, the GPR rules of a model can be simplified using AST parsing for both DNF and non-DNF rules. - - - Example: - reduce_gpr(model, essential_reacs, gkis, gkos): - - Args: - model (cobra.Model): - A metabolic model that is an instance of the cobra.Model class containing GPR rules - - essential_reacs (list of str): - A list of identifiers of essential reactions. - - gkis, gkos (dict): - Dictionaries that contain the costs for gene knockouts and additions. E.g., - gkos={'adhE': 1.0, 'ldhA' : 1.0 ...} - - Returns: - (dict): - An updated dictionary of the knockout costs in which irrelevant genes are removed. - """ - - def ast_to_gene_reaction_rule(node): - """ - Convert an AST node back to gene reaction rule string format. - """ - if isinstance(node, ast.Name): - return node.id - elif isinstance(node, ast.BoolOp): - child_strings = [ast_to_gene_reaction_rule(child) for child in node.values] - if isinstance(node.op, ast.And): - return ' and '.join(f'({s})' if ' or ' in s else s for s in child_strings) - elif isinstance(node.op, ast.Or): - return ' or '.join(f'({s})' if ' and ' in s else s for s in child_strings) - else: - raise ValueError(f"Unsupported AST node type: {type(node)}") - - def simplify_gpr_ast(node, protected_genes_dict): - """ - Simplify GPR AST by setting protected genes to True and applying boolean simplification. - This is equivalent to the original string-based approach but operates purely on AST. - """ - return apply_gene_protection_to_ast(node, protected_genes_dict) - - def apply_gene_protection_to_ast(node, protected_genes_dict): - """ - Apply gene protection to AST by setting protected genes to True and simplifying boolean expressions. - Returns a simplified AST node with redundant terms removed and consistent gene ordering. - """ - if isinstance(node, ast.Name): - if node.id in protected_genes_dict: - return True - else: - return node - elif isinstance(node, ast.BoolOp): - # Recursively apply to children - new_children = [] - for child in node.values: - simplified_child = apply_gene_protection_to_ast(child, protected_genes_dict) - - if isinstance(node.op, ast.And): - if simplified_child is False: - return False - elif simplified_child is not True: - new_children.append(simplified_child) - elif isinstance(node.op, ast.Or): - if simplified_child is True: - return True - elif simplified_child is not False: - new_children.append(simplified_child) - - # Handle results - if not new_children: - return True if isinstance(node.op, ast.And) else False - elif len(new_children) == 1: - return new_children[0] - else: - # Apply additional simplifications for OR nodes - if isinstance(node.op, ast.Or): - new_children = remove_redundant_or_terms(new_children) - if len(new_children) == 1: - return new_children[0] - - # Sort children for consistent ordering (like string approach does) - sorted_children = sort_ast_nodes(new_children) - new_node = ast.BoolOp(op=node.op, values=sorted_children) - return new_node - else: - raise ValueError(f"Unsupported AST node type: {type(node)}") - - def remove_redundant_or_terms(children): - """ - Remove redundant terms from OR expressions using boolean logic simplification. - Example: (a and b and c) or (a and b) simplifies to (a and b) - since (a and b) is logically sufficient when both terms are present. - """ - # Convert AST nodes to comparable forms - simplified = [] - for child in children: - # Check if this child makes any other child redundant - is_redundant = False - for other in children: - if child is not other and is_subset_of(child, other): - # child is a subset of other, so other is redundant - is_redundant = False # Keep child, remove other later - elif child is not other and is_subset_of(other, child): - # other is a subset of child, so child is redundant - is_redundant = True - break - if not is_redundant: - simplified.append(child) - - # Remove duplicates - unique = [] - for child in simplified: - if not any(ast_nodes_equal(child, existing) for existing in unique): - unique.append(child) - - return unique if unique else children - - def is_subset_of(node1, node2): - """ - Check if node1 logically absorbs node2 in boolean algebra. - - In OR expressions: A or (A and B) = A - This means A absorbs (A and B) because A is simpler/more general. - - For absorption to work: node1 must be "simpler" than node2, - meaning node2 implies node1 (node2 is more restrictive). - - Examples: - - mobA absorbs (mobA and mobB) - - (a and b) absorbs (a and b and c) - """ - # Case 1: Single gene absorbs AND expression containing that gene - if isinstance(node1, ast.Name) and isinstance(node2, ast.BoolOp) and isinstance(node2.op, ast.And): - genes_in_and = get_genes_from_ast(node2) - return node1.id in genes_in_and - - # Case 2: Shorter AND expression absorbs longer AND expression with same genes - if (isinstance(node1, ast.BoolOp) and isinstance(node1.op, ast.And) and isinstance(node2, ast.BoolOp) and - isinstance(node2.op, ast.And)): - genes1 = get_genes_from_ast(node1) - genes2 = get_genes_from_ast(node2) - # node1 absorbs node2 if node1's genes are a proper subset of node2's genes - return genes1.issubset(genes2) and len(genes1) < len(genes2) - - return False - - def get_genes_from_ast(node): - """Extract set of genes from AST node""" - if isinstance(node, ast.Name): - return {node.id} - elif isinstance(node, ast.BoolOp): - genes = set() - for child in node.values: - genes.update(get_genes_from_ast(child)) - return genes - return set() - - def ast_nodes_equal(node1, node2): - """Check if two AST nodes are equivalent""" - if type(node1) != type(node2): - return False - if isinstance(node1, ast.Name): - return node1.id == node2.id - elif isinstance(node1, ast.BoolOp): - if type(node1.op) != type(node2.op): - return False - return (len(node1.values) == len(node2.values) and all(ast_nodes_equal(a, b) for a, b in zip(node1.values, node2.values))) - return False - - def sort_ast_nodes(nodes): - """Sort AST nodes for consistent ordering""" - - def node_sort_key(node): - if isinstance(node, ast.Name): - return (0, node.id) - elif isinstance(node, ast.BoolOp): - return (1, len(node.values), str(type(node.op))) - return (2, str(node)) - - return sorted(nodes, key=node_sort_key) - - def is_gene_essential_to_reaction_ast(reaction, gene_id): - """ - Determine if a gene is essential for a reaction using AST-based GPR analysis. - A gene is considered essential if removing it (setting it to False) makes - the entire GPR expression evaluate to False, rendering the reaction impossible. - """ - if not reaction.gene_reaction_rule: - return False - - # Skip reactions without gene associations - if not reaction.gpr or not reaction.gpr.body: - return False - - try: - # Test what happens if we knock out this gene using AST - gene_states = {gene_id: False} - result = evaluate_gpr_ast(reaction.gpr.body, gene_states) - return result is False - except Exception as e: - # Catch unsupported AST node types but don't fall back to string parsing - logging.warning(f'Unsupported AST node type in reaction {reaction.id} for gene {gene_id}: {e}') - return False - - # 1) Remove gpr rules from blocked reactions - blocked_reactions = [reac.id for reac in model.reactions if reac.bounds == (0, 0)] - for rid in blocked_reactions: - model.reactions.get_by_id(rid).gene_reaction_rule = '' - for g in model.genes[::-1]: # iterate in reverse order to avoid mixing up the order of the list when removing genes - if not g.reactions: - model.genes.remove(g) - - protected_genes = set() - - # 2. Protect genes that only occur in essential reactions - for g in model.genes: - if not g.reactions or {r.id for r in g.reactions}.issubset(essential_reacs): - protected_genes.add(g) - - # 3. Protect genes that are essential to essential reactions (AST-based analysis) - for r in [model.reactions.get_by_id(s) for s in essential_reacs]: - for g in r.genes: - if is_gene_essential_to_reaction_ast(r, g.id): - protected_genes.add(g) - - # 4. Remove essential genes, and knockouts without impact from gko_costs - [gkos.pop(pg.id) for pg in protected_genes if pg.id in gkos] - - # 5. Add all not-knockable genes to the protected list - [protected_genes.add(g) for g in model.genes if (g.id not in gkos) and (g.name not in gkos)] # support names or ids in gkos - - # 6. genes with kiCosts are kept (remove from protected list so they can be targeted) - gki_ids = [g.id for g in model.genes if (g.id in gkis) or (g.name in gkis)] # support names or ids in gkis - protected_genes = protected_genes.difference({model.genes.get_by_id(g) for g in gki_ids}) - protected_genes_dict = {pg.id: True for pg in protected_genes} - - # 7. Simplify GPR rules using AST-based boolean logic and remove non-targetable rules - for r in model.reactions: - if r.gene_reaction_rule and r.gpr and r.gpr.body: - try: - simplified = simplify_gpr_ast(r.gpr.body, protected_genes_dict) - - if simplified is True: - # Rule is always satisfied (cannot be knocked out) - model.reactions.get_by_id(r.id).gene_reaction_rule = '' - elif simplified is False: - # Rule is impossible - should not happen with proper protection - logging.error(f'Something went wrong during gpr rule simplification for {r.id}.') - elif isinstance(simplified, (ast.Name, ast.BoolOp)): - # Convert simplified AST back to string - new_rule = ast_to_gene_reaction_rule(simplified) - model.reactions.get_by_id(r.id).gene_reaction_rule = new_rule - # If simplified is the original node, keep original rule - except Exception as e: - logging.warning(f'Failed to simplify GPR rule for reaction {r.id}: {e}') - - # 8. Remove obsolete genes and protected genes - for g in model.genes[::-1]: - if not g.reactions or g in protected_genes: - model.genes.remove(g) - - return gkos - - -# backward-compat alias -remove_irrelevant_genes = reduce_gpr - - def extend_model_gpr(model, use_names=False): """Integrate GPR-rules into a metabolic model as pseudo metabolites and reactions using AST parsing @@ -1004,16 +820,14 @@ def warning_name_too_long(id, p=""): "\nOne of the generated reaction names is beyond or close to the limit of 255 "+\ "characters\npermitted by GLPK and Gurobi. The name of the newly generated "+\ "reaction or metabolite: \n "+id+",\ngenerated from reaction or metabolite:\n "+\ - p+"\n"+"was therefore trimmed to:\n "+id[0:MAX_NAME_LEN]+".\nThis trimming is "+\ - "usually safe, no guarantee is given. To avoid this message,\nuse the CPLEX "+\ - "solver or consider simplifying GPR rules or gene names in your model.") + p+"\n"+"was therefore trimmed to:\n "+truncate(id)+".\nThis trimming is "+\ + "usually safe, no guarantee is given. To avoid this message,\nconsider "+\ + "simplifying GPR rules or gene names in your model.") def truncate(id): h = hashlib.sha256(id.encode()).hexdigest()[:20] return id[0:MAX_NAME_LEN - 21] + "_" + h - solver = search('(' + '|'.join(avail_solvers) + ')', model.solver.interface.__name__)[0] - # Track created metabolites to avoid duplicates created_metabolites = set() @@ -1022,7 +836,7 @@ def create_gene_pseudoreaction(gene_id): gene_met_id = f'g_{gene_id}' # Check name length and truncate if necessary - if len(gene_met_id) > MAX_NAME_LEN and solver in {GUROBI, GLPK}: + if len(gene_met_id) > MAX_NAME_LEN: if truncate(gene_met_id) not in [m.id for m in model.metabolites]: warning_name_too_long(gene_met_id, gene_id) gene_met_id = truncate(gene_met_id) @@ -1039,7 +853,7 @@ def create_gene_pseudoreaction(gene_id): reaction_id = gene.id # Check name length and truncate if necessary - if len(reaction_id) > MAX_NAME_LEN and solver in {GUROBI, GLPK}: + if len(reaction_id) > MAX_NAME_LEN: warning_name_too_long(reaction_id, gene_id) reaction_id = truncate(reaction_id) @@ -1055,7 +869,7 @@ def create_and_metabolite(child_metabolites): and_met_id = "_and_".join(sorted(child_metabolites)) # Check name length and truncate if necessary - if len(and_met_id) > MAX_NAME_LEN and solver in {GUROBI, GLPK}: + if len(and_met_id) > MAX_NAME_LEN: if truncate(and_met_id) not in [m.id for m in model.metabolites]: warning_name_too_long(and_met_id, "AND combination") and_met_id = truncate(and_met_id) @@ -1068,7 +882,7 @@ def create_and_metabolite(child_metabolites): reaction_id = f"R_{and_met_id}" # Check name length and truncate if necessary - if len(reaction_id) > MAX_NAME_LEN and solver in {GUROBI, GLPK}: + if len(reaction_id) > MAX_NAME_LEN: warning_name_too_long(reaction_id, "AND combination") reaction_id = truncate(reaction_id) @@ -1084,7 +898,7 @@ def create_or_metabolite(child_metabolites): or_met_id = "_or_".join(sorted(child_metabolites)) # Check name length and truncate if necessary - if len(or_met_id) > MAX_NAME_LEN and solver in {GUROBI, GLPK}: + if len(or_met_id) > MAX_NAME_LEN: if truncate(or_met_id) not in [m.id for m in model.metabolites]: warning_name_too_long(or_met_id, "OR combination") or_met_id = truncate(or_met_id) @@ -1099,7 +913,7 @@ def create_or_metabolite(child_metabolites): reaction_id = f"R{i}_{or_met_id}" # Check name length and truncate if necessary - if len(reaction_id) > MAX_NAME_LEN and solver in {GUROBI, GLPK}: + if len(reaction_id) > MAX_NAME_LEN: warning_name_too_long(reaction_id, "OR combination") reaction_id = truncate(reaction_id) @@ -1140,7 +954,7 @@ def process_ast_node(node): r_rev.id = r.id + '_reverse_' + hex(hash(r))[8:] r_rev.lower_bound = np.max([0, r_rev.lower_bound]) reac_map[r.id].update({r_rev.id: -1.0}) - if len(r_rev.id) > MAX_NAME_LEN and solver in {GUROBI, GLPK}: + if len(r_rev.id) > MAX_NAME_LEN: warning_name_too_long(r_rev.id, r.id) r_rev.id = truncate(r_rev.id) rev_reac.add(r_rev) @@ -1332,7 +1146,7 @@ def compress_modules(sd_modules, cmp_mapReac): (list of SDModule): A list of strain design modules for the compressed network """ - sd_modules = modules_coeff2rational(sd_modules) + sd_modules = modules_coeff_to_fraction(sd_modules) for cmp in cmp_mapReac: reac_map_exp = cmp["reac_map_exp"] parallel = cmp["parallel"] @@ -1554,19 +1368,19 @@ def filter_sd_maxcost(sd, max_cost, kocost, kicost): return sd -def modules_coeff2rational(sd_modules): - """Convert coefficients to rational numbers using sympy.Rational""" - from .compression import float_to_rational +def modules_coeff_to_fraction(sd_modules): + """Convert SDModule coefficients to exact fractions.Fraction.""" + from .compression import float_to_fraction for i, module in enumerate(sd_modules): for param in [CONSTRAINTS, INNER_OBJECTIVE, OUTER_OBJECTIVE, PROD_ID]: if param in module and module[param] is not None: if param == CONSTRAINTS: for constr in module[CONSTRAINTS]: for reac in constr[0].keys(): - constr[0][reac] = float_to_rational(constr[0][reac]) + constr[0][reac] = float_to_fraction(constr[0][reac]) if param in [INNER_OBJECTIVE, OUTER_OBJECTIVE, PROD_ID]: for reac in module[param].keys(): - module[param][reac] = float_to_rational(module[param][reac]) + module[param][reac] = float_to_fraction(module[param][reac]) return sd_modules @@ -1586,7 +1400,7 @@ def modules_coeff2float(sd_modules): def bound_blocked_or_irrevers_fva(model, **kwargs): - """Use FVA to determine the flux ranges. Use this information to update the model bounds + """Use FVA to determine flux ranges, update model bounds, and return the ranges. If flux ranges for a reaction are narrower than its bounds in the mode, these bounds can be omitted, since other reactions must constrain the reaction flux. If (upper or lower) flux bounds are found to @@ -1610,6 +1424,7 @@ def bound_blocked_or_irrevers_fva(model, **kwargs): r._upper_bound = np.inf if limits.maximum <= -tol: r._upper_bound = min([0.0, r._upper_bound]) + return flux_limits # ── Portable, rational-safe model (de)serialisation ────────────────────────── diff --git a/straindesign/speedy_fva.py b/straindesign/speedy_fva.py index 3cfacfe..7c14dfb 100644 --- a/straindesign/speedy_fva.py +++ b/straindesign/speedy_fva.py @@ -37,11 +37,11 @@ from straindesign.solver_interface import MILP_LP from straindesign.pool import SDPool from straindesign.parse_constr import parse_constraints, lineqlist2mat -from straindesign.names import CONSTRAINTS, SOLVER, OPTIMAL, UNBOUNDED, GLPK, LP_METHOD_DUAL +from straindesign.names import CONSTRAINTS, SOLVER, OPTIMAL, UNBOUNDED, INFEASIBLE, GLPK, LP_METHOD_DUAL from straindesign.networktools import suppress_lp_context from straindesign.compression import ( compress_cobra_model, CompressionMethod, remove_conservation_relations, - stoichmat_coeff2rational, remove_blocked_reactions, + stoichmat_coeff_to_fraction, stoichmat_coeff2float, remove_blocked_reactions, ) @@ -69,17 +69,12 @@ def _compress_for_fva(model): """ cmp_maps = [] with suppress_lp_context(model): - # Fast copy: swap solver with empty stub so deepcopy(solver) is cheap - # (~0.3s vs ~3.3s on iML1515). Safe because speedy_fva builds its own + # Fast copy: the suppressed Model.copy skips the solver deepcopy (~0.3s vs ~3.3s on + # iML1515) and attaches a backend-free carrier. Safe because speedy_fva builds its own # MILP_LP and the compression pipeline is solver-independent. - saved_solver = model._solver - model._solver = model.problem.Model() - try: - cmp_model = model.copy() - finally: - model._solver = saved_solver + cmp_model = model.copy() remove_blocked_reactions(cmp_model) - stoichmat_coeff2rational(cmp_model) + stoichmat_coeff_to_fraction(cmp_model) n_before = len(cmp_model.reactions) # Single-pass coupled compression (NULLSPACE only, no RECURSIVE iteration) for r in cmp_model.reactions: @@ -680,10 +675,10 @@ def _rebuild_lp(): # Guard: LP optimum must not be worse than incumbent if direction == 1: val, inc = -obj_val, incumbent_max[j] - bad = np.isfinite(inc) and val < inc - 1e-6 * (1 + abs(inc)) + bad = np.isfinite(inc) and val < inc - _DEGEN_TOL * (1 + abs(inc)) else: val, inc = obj_val, incumbent_min[j] - bad = np.isfinite(inc) and val > inc + 1e-6 * (1 + abs(inc)) + bad = np.isfinite(inc) and val > inc + _DEGEN_TOL * (1 + abs(inc)) if bad: _rebuild_lp() C = [[j, float(sig)]] @@ -760,3 +755,186 @@ def _rebuild_lp(): fva_result = expanded return fva_result + + +# --------------------------------------------------------------------------- +# Fast exact reversibility (sign-only FVA) for pre-compression tightening +# --------------------------------------------------------------------------- + +_REV_SCAN_TOL = 1e-3 # co-option certifies only on flux comfortably above solver noise +_REV_REBUILD_EVERY = 200 +_ZERO_SNAP = 1e-11 # |flux| below this is solver noise, not a direction; 0 disables snapping +_DEGEN_TOL = 1e-6 # warm-start guard: fresh optimum must not fall below a known-achievable incumbent + + +def _rev_structural_sweep(model): + """Sound over-approximation of achievable directions via single-producer/consumer + + dead-end propagation (0 LP, sign-only). Returns (af, ar): af[id]=fwd not proven blocked, + ar[id]=rev not proven blocked. Every direction it kills is infeasible in ANY steady state, + hence also in the flux polytope, so tightening on it is lossless.""" + af = {r.id: r.upper_bound > 0 for r in model.reactions} + ar = {r.id: r.lower_bound < 0 for r in model.reactions} + met_rx = {mm.id: [(r.id, r.metabolites[mm]) for r in mm.reactions] for mm in model.metabolites} + changed = True + while changed: + changed = False + for e in met_rx.values(): + prod = [(i, 'f') for i, c in e if c > 0 and af[i]] + [(i, 'r') for i, c in e if c < 0 and ar[i]] + cons = [(i, 'f') for i, c in e if c < 0 and af[i]] + [(i, 'r') for i, c in e if c > 0 and ar[i]] + prx = {i for i, _ in prod}; crx = {i for i, _ in cons} + + def kill(i, d): + nonlocal changed + if d == 'f' and af[i]: + af[i] = False; changed = True + if d == 'r' and ar[i]: + ar[i] = False; changed = True + if not prod: + for i, d in cons: kill(i, d) + elif len(prx) == 1: + s = next(iter(prx)) + for i, d in cons: + if i == s: kill(i, d) + if not cons: + for i, d in prod: kill(i, d) + elif len(crx) == 1: + s = next(iter(crx)) + for i, d in prod: + if i == s: kill(i, d) + return af, ar + + +def fast_reversibility(model, solver=None, compress=True): + """Exact per-reaction reversibility on the ORIGINAL flux polytope, faster than full FVA. + + Returns {reaction_id: (can_fwd, can_rev)} for every reaction of ``model`` -- identical + directionality to FVA (a blocked reaction is (False, False)), used to fix lb/ub to 0 + BEFORE compression so genuinely one-directional reactions fuse (and avoid the GPR + fwd/rev split, which fires on lb<0). + + Method: (1) structural single-producer/consumer sweep tightens a copy's bounds (0 LP, + sound); (2) a single coupled compression of the tightened copy shrinks the LP (default + on -- the uncompressed matrix is per-LP pathological on some genome-scale models, e.g. + yeast-GEM: ~68 vs ~4 ms/LP compressed); (3) warm-started per-reaction max/min on the + compressed model (objective-only change) with a co-option scan that certifies other + reactions carrying flux; (4) map compressed min/max back to the original reactions. + Sign of the achieved min/max gives reversibility.""" + solver = select_solver(solver, model) + orig_rid = [r.id for r in model.reactions] + + # (1) structural sweep on the ORIGINAL model + tighten a copy's bounds (lossless) + af, ar = _rev_structural_sweep(model) + m = model.copy() + for r in m.reactions: + if not af[r.id]: + r.upper_bound = min(float(r.upper_bound), 0.0) + if not ar[r.id]: + r.lower_bound = max(float(r.lower_bound), 0.0) + + # (2) single coupled compress of the tightened model (default on) + if compress: + m, cmp_maps = _compress_for_fva(m) + stoichmat_coeff2float(m) + else: + cmp_maps = [] + + # (3) LP phase on the (compressed) model + cmp_rid = [r.id for r in m.reactions] + n = len(cmp_rid) + lb = np.array([float(r.lower_bound) for r in m.reactions]) + ub = np.array([float(r.upper_bound) for r in m.reactions]) + S = sparse.csr_matrix(create_stoichiometric_matrix(m)) + A_ineq = sparse.csr_matrix((0, n)); b_ineq = [] + A_eq = S; b_eq = [0.0] * S.shape[0] + + def build(): + return MILP_LP(A_ineq=A_ineq, b_ineq=b_ineq, A_eq=A_eq, b_eq=b_eq, + lb=lb.tolist(), ub=ub.tolist(), solver=solver) + lp = build() + + incumbent_max = np.full(n, -np.inf); incumbent_min = np.full(n, np.inf) + # Bound signs are exact model data, not solver output, so they decide on the exact sign: a + # direction counts as blocked only if the bound itself forbids it. A tolerance here would + # discard a direction whose achievable flux is merely small (a max of 1e-8 is still forward). + res_max = ub <= 0.0 # fwd already blocked by bounds (sweep/original) + res_min = lb >= 0.0 + incumbent_max[res_max] = ub[res_max] + incumbent_min[res_min] = lb[res_min] + fixed = ub == lb + res_max[fixed] = True; res_min[fixed] = True + incumbent_max[fixed] = ub[fixed]; incumbent_min[fixed] = lb[fixed] + + def scan(x): + nm = (~res_max) & (x > _REV_SCAN_TOL); res_max[nm] = True + np.maximum(incumbent_max, x, out=incumbent_max) + nn = (~res_min) & (x < -_REV_SCAN_TOL); res_min[nn] = True + np.minimum(incumbent_min, x, out=incumbent_min) + + n_lp = 0; prev_col = -1; seq = 0 + + def solve_dir(j, direction): + nonlocal prev_col, seq, n_lp + sig = -float(direction) + C = [[j, sig]] if (prev_col < 0 or prev_col == j) else [[j, sig], [prev_col, 0.0]] + if solver in ('cplex', 'gurobi'): + lp.backend.set_objective_idx(C) + else: + lp.set_objective_idx(C) + prev_col = j + r = lp.solve(); n_lp += 1; seq += 1 + return r + + # Feasibility preflight: one zero-objective solve proves the polytope is non-empty (so a later + # infeasible status can only come from the objective change, not the model) and its flux vector + # seeds every incumbent, so no subsequent warm-start optimum can silently contradict a flux + # already witnessed as achievable. + x_feas, _, status_feas = lp.solve(); n_lp += 1 + if status_feas == INFEASIBLE: + raise ValueError('fast_reversibility: the model has no steady-state flux distribution.') + if status_feas == OPTIMAL and x_feas: + scan(np.array(x_feas[:n], dtype=np.float64)) + + def unknown(j, direction): + """Record 'direction not determined': the incumbent goes to +/-inf so the direction is + reported as achievable. Tightening on an unproven bound could delete a feasible pathway; + reporting a spurious direction only forgoes tightening.""" + if direction == 1: res_max[j] = True; incumbent_max[j] = np.inf + else: res_min[j] = True; incumbent_min[j] = -np.inf + + for j in range(n): + for direction in (1, -1): + if (direction == 1 and res_max[j]) or (direction == -1 and res_min[j]): + continue + if seq > 0 and seq % _REV_REBUILD_EVERY == 0: + lp = build(); prev_col = -1 + x_list, obj_val, status = solve_dir(j, direction) + if status != OPTIMAL: + # UNBOUNDED is a proven infinite direction, every other nonoptimal status (time + # limit, numerical trouble) is simply unknown; both must not tighten. + unknown(j, direction) + continue + val = -obj_val if direction == 1 else obj_val + inc = incumbent_max[j] if direction == 1 else incumbent_min[j] + degen = (direction == 1 and np.isfinite(inc) and val < inc - _DEGEN_TOL * (1 + abs(inc))) or \ + (direction == -1 and np.isfinite(inc) and val > inc + _DEGEN_TOL * (1 + abs(inc))) + if degen: + lp = build(); prev_col = -1 + x_list, obj_val, status = solve_dir(j, direction) + if status != OPTIMAL: + unknown(j, direction) + continue + val = -obj_val if direction == 1 else obj_val + if direction == 1: res_max[j] = True; incumbent_max[j] = max(incumbent_max[j], val) + else: res_min[j] = True; incumbent_min[j] = min(incumbent_min[j], val) + scan(np.array(x_list[:n], dtype=np.float64)) + + # (4) expand compressed min/max back to original reactions + # Snapping only removes flux the solver cannot distinguish from zero; the direction decision + # itself is then the exact sign, so a small-but-real flux (1e-8) keeps its direction. + incumbent_max[np.abs(incumbent_max) < _ZERO_SNAP] = 0.0 + incumbent_min[np.abs(incumbent_min) < _ZERO_SNAP] = 0.0 + df = DataFrame({"minimum": incumbent_min, "maximum": incumbent_max}, index=cmp_rid) + if cmp_maps: + df = _expand_fva(df, cmp_maps, orig_rid) + return {r: (float(df.at[r, 'maximum']) > 0.0, float(df.at[r, 'minimum']) < 0.0) + for r in orig_rid} diff --git a/straindesign/strainDesignProblem.py b/straindesign/strainDesignProblem.py index 4f442ec..c7031d9 100644 --- a/straindesign/strainDesignProblem.py +++ b/straindesign/strainDesignProblem.py @@ -38,6 +38,10 @@ from straindesign.names import * import logging +# Margin a module flux range must clear on solvers whose reported ranges are only as tight as their +# own feasibility tolerance (1e-9): ten times that tolerance. +_MODULE_OVERRIDE_TOL = 1e-8 + class SDProblem: """Strain design MILP @@ -228,12 +232,64 @@ def __init__(self, model: Model, sd_modules: List[SDModule], *args, **kwargs): # np.savetxt("Ab_py.tsv", Ab.todense(), delimiter='\t') self.vtype = 'B' * self.num_z + 'C' * (self.z_map_vars.shape[1] - self.num_z) + def _module_bound_override(self, sd_module): + """Per-module bound override for a classical-MCS module (PROTECT or SUPPRESS). + + Returns a TARGETED SUBSET dict ``{rxn_id: (lo, hi)}`` of per-reaction bound overrides for this + module's block only (never written into the shared model). The bounds need NOT come from FVA -- + any source of a proven per-module bound works. Currently the source is the module's flux limits + (``sd_module['fva_bounds']``), computed once in compute_strain_designs' preprocessing. Only two + structural facts are carried (sign-only, never a magnitude): + - blocked in the module (min == max == 0) -> (0.0, 0.0) + - one-sided in the module (min >= 0) -> lo = 0.0 (never negative here) + - one-sided in the module (max <= 0) -> hi = 0.0 (never positive here) + Magnitudes are NOT touched (no non-binding-bound -> +/-inf relaxation -- that is the + fva_tighten behaviour a benchmark flagged as a regression). Values are returned as an override, + NOT written into the model, so this is scoped to the module's block only and cannot make a + reaction non-targetable for another module (the shared-z pitfall). If the limits are absent (a + bare SDProblem, not going through preprocessing), this returns no override -- it does NOT run a + fresh full-model FVA (that cost belongs in preprocessing, not the MILP constructor). + + Soundness: a reaction blocked within the module's block is already 0 across that whole block, so + fixing its bound to 0 (or fixing the sign of a one-sided reaction) does not remove any point of + it. For PROTECT it is the protected/desired set; for SUPPRESS it is the undesired set the primal + describes before farkas_dualize. In both cases the block is unchanged, so which knockout sets + keep it feasible (PROTECT) / make it infeasible (SUPPRESS) is unchanged -> the design set is + identical. + """ + limits = sd_module.get('fva_bounds') + if limits is None: + # Per-module flux limits are precomputed in compute_strain_designs' preprocessing and + # passed on the module. A bare SDProblem (a test, or a direct caller) supplies none and + # gets no override: that FVA belongs in preprocessing, not in the MILP constructor. The + # override only tightens bounds, so omitting it is design-neutral. + return {} + solver = getattr(self, SOLVER, None) + # SCIP and GLPK report flux ranges no tighter than their own feasibility tolerance (1e-9), so a + # reported zero there does not certify a zero. They therefore act only on a range that clears + # that tolerance with a safety factor; the exact solvers act on the exact sign. Either way the + # override only fixes a sign the module already forces, so a range too weak to act on merely + # forgoes tightening. + tol = _MODULE_OVERRIDE_TOL if select_solver(solver) in [SCIP, GLPK] else 0.0 + override = {} + for rid, lim in limits.iterrows(): + if lim.minimum > lim.maximum + tol: + # Numerically inconsistent range: neither sign is trustworthy, so act on neither. + logging.warning(' Module FVA range for %s is inconsistent (min %g > max %g), no bound ' + 'override applied.' % (rid, lim.minimum, lim.maximum)) + continue + lo = 0.0 if lim.minimum >= tol else None # convincingly non-negative in-module + hi = 0.0 if lim.maximum <= -tol else None # convincingly non-positive in-module + if lo is not None or hi is not None: # both sides fixed == blocked in-module + override[rid] = (lo, hi) + return override + def addModule(self, sd_module): """Generate module LP and z-linking-matrix for each module and add them to the strain design MILP - + Args: sd_module (straindesign.SDModule): - Modules to describe strain design problems like protected or suppressed flux states for + Modules to describe strain design problems like protected or suppressed flux states for MCS strain design or inner and outer objective functions for OptKnock. See description of SDModule for more information on how to set up modules. """ @@ -246,8 +302,15 @@ def addModule(self, sd_module): # 2. Construct LP for module if sd_module[MODULE_TYPE] in [PROTECT, SUPPRESS] and sd_module[INNER_OBJECTIVE] is None: # Classical MCS + # Tighten THIS block's bounds with region-FVA (blocked + reversibility). Sign-only, so it + # never over-tightens: it only fixes bounds the region already forces (a reaction + # blocked/one-sided in the region is already 0/one-sided there), leaving the region -- and + # hence the design set -- unchanged, while making the vacuous z-links droppable. Applied to + # both PROTECT and SUPPRESS: for SUPPRESS the undesired-region primal is bounded the same + # way before farkas_dualize, so the certificate is unchanged. + bound_override = self._module_bound_override(sd_module) A_ineq_p, b_ineq_p, A_eq_p, b_eq_p, lb_p, ub_p, c_p, z_map_constr_ineq_p, z_map_constr_eq_p, z_map_vars_p \ - = build_primal_from_cbm(self.model, V_ineq, v_ineq, V_eq, v_eq) + = build_primal_from_cbm(self.model, V_ineq, v_ineq, V_eq, v_eq, bound_override=bound_override) elif sd_module[MODULE_TYPE] in [PROTECT, SUPPRESS, OPTKNOCK, OPTCOUPLE]: c_in = linexprdict2mat(sd_module[INNER_OBJECTIVE], self.model.reactions.list_attr('id')) # by default, assume maximization of the inner objective @@ -691,11 +754,13 @@ def link_z(self): (1) Translate equality-KOs/KIs to two inequality-KOs/KIs (2) Translate variable-KOs/KIs to inequality-KIs/KOs - (3) Try to bound the problem with LPs - (4) Use LP-determined bounds to link z-variables, where such bounds were found + (3) Determine big-M values from variable bounds: zero/single-variable rows get a finite M + from the bounds; multi-variable rows are left unbounded (M=inf) + (4) Link z-variables via big-M for the rows that got a finite M (5) Translate remaining inequalities back to equalities when possible and link z via indicator constraints. If necessary, the solver interface will translate them to big-M constraints. (6) Remove redundant equalities from static problem + (7) Fix intervention binaries the linked MILP leaves free """ # 1. Split knockable equality constraints into foward and reverse direction @@ -707,11 +772,13 @@ def link_z(self): self.A_ineq = sparse.vstack((self.A_ineq, eq_constr_A)).tocsr() self.b_ineq += eq_constr_b self.z_map_constr_ineq = sparse.hstack((self.z_map_constr_ineq, z_eq)).tocsc() - # Remove knockable equalities from A_eq + # Remove knockable equalities from A_eq; the set keeps membership O(1) per row n_rows_eq = self.A_eq.shape[0] - self.A_eq = self.A_eq[[False if i in knockable_constr_eq else True for i in range(0, n_rows_eq)]] - self.b_eq = [self.b_eq[i] for i in range(0, len(self.b_eq)) if i not in knockable_constr_eq] - self.z_map_constr_eq = self.z_map_constr_eq[:, [False if i in knockable_constr_eq else True for i in range(0, n_rows_eq)]] + _kc_eq = set(int(i) for i in knockable_constr_eq) + keep_eq = [i not in _kc_eq for i in range(0, n_rows_eq)] + self.A_eq = self.A_eq[keep_eq] + self.b_eq = [self.b_eq[i] for i in range(0, len(self.b_eq)) if i not in _kc_eq] + self.z_map_constr_eq = self.z_map_constr_eq[:, keep_eq] # 2. Translate all variable knockouts to inequality knockouts numvars = self.A_ineq.shape[1] @@ -726,67 +793,70 @@ def link_z(self): lb_constr_b = [0 for _ in knockable_vars_leq0] bnd_constr_A = sparse.vstack((ub_constr_A, lb_constr_A)).tocsr() bnd_constr_b = ub_constr_b + lb_constr_b - var_kos = [knockable_vars[0][(knockable_vars[1] == i).nonzero()[0][0]] for i in knockable_vars_geq0 + knockable_vars_leq0] + # map each knockable var -> its first z in one pass, so the lookup below is O(1) + _vz = {} + for _z, _v in zip(knockable_vars[0].tolist(), knockable_vars[1].tolist()): + _vz.setdefault(_v, _z) + var_kos = [_vz[int(i)] for i in knockable_vars_geq0 + knockable_vars_leq0] z_lb_ub = -self.z_map_vars[:, knockable_vars_geq0 + knockable_vars_leq0] # add constraints to main problem self.A_ineq = sparse.vstack((self.A_ineq, bnd_constr_A)).tocsr() self.b_ineq += bnd_constr_b self.z_map_constr_ineq = sparse.hstack((self.z_map_constr_ineq, z_lb_ub)).tocsc() - # 3. Use LP to identify M-values for knockable constraints - # For this purpose, first construct a most relaxed LP-model (use all possible constraint-KOs, no possible var-KOs) - knockable_constr_ineq = np.sort(self.z_map_constr_ineq.nonzero()[1]) - - cont_vars = [False if i in self.idx_z else True for i in range(0, numvars)] - M_A_ineq = self.A_ineq[[False if i in knockable_constr_ineq else True for i in range(0, self.A_ineq.shape[0])], :][:, cont_vars] - M_b_ineq = [self.b_ineq[i] for i in range(0, self.A_ineq.shape[0]) if i not in knockable_constr_ineq] - M_A_eq = self.A_eq[:, cont_vars] - M_b_eq = self.b_eq.copy() + # 3. Big-M values for knockable constraints, read directly from the variable bounds. + # A knockable constraint a_ineq*x <= b relaxes via M = max(a_ineq*x) over the polytope; + # the z-coefficient carries the b offset so the knocked-out state relaxes to the tight + # bound a_ineq*x <= M (not b+M): + # sense > 0 (z=1 knocks out): z-coeff = (b - M) -> a_ineq*x + (b-M)*z <= b + # z=0: a_ineq*x <= b (active); z=1: a_ineq*x <= M (relaxed) + # sense < 0 (z=0 knocks out): z-coeff = (M - b), RHS := M -> a_ineq*x + (M-b)*z <= M + # z=1: a_ineq*x <= b (active); z=0: a_ineq*x <= M (relaxed) + # Zero/single-variable rows take a finite M from the bounds; multi-variable rows are + # unbounded on the polytope (M = +inf), which the linker realizes as an indicator + # constraint (gurobi/cplex) or the constant self.M (glpk/user-M). + knockable_constr_ineq = np.unique(self.z_map_constr_ineq.nonzero()[1]) + + _idxz = set(self.idx_z) # O(1) membership in the scan below + cont_vars = [i not in _idxz for i in range(0, numvars)] M_lb = [self.lb[i] for i in np.nonzero(cont_vars)[0]] M_ub = [self.ub[i] for i in np.nonzero(cont_vars)[0]] - # M_A contains a list of all knockable constraints. We need to maximize their value (M_A(i)*x) to get a good M - # Big-M knockout of a constraint a_ineq*x <= b, with M = max(a_ineq*x) over the relaxed - # polytope (b is the right-hand-side value). The z-coefficient carries the b offset so the - # knocked-out state relaxes to the TIGHT bound a_ineq*x <= M (not b+M): - # sense > 0 (z=1 knocks out): z-coeff = (b - M) -> a_ineq*x + (b-M)*z <= b - # z=0: a_ineq*x <= b (active); z=1: a_ineq*x <= M (relaxed) - # sense < 0 (z=0 knocks out): z-coeff = (M - b), RHS := M -> a_ineq*x + (M-b)*z <= M - # z=1: a_ineq*x <= b (active); z=0: a_ineq*x <= M (relaxed) - M_A = self.A_ineq[[True if i in knockable_constr_ineq else False for i in range(0, self.A_ineq.shape[0])], :][:, cont_vars] - M_A = list(M_A.toarray()) - M_b = [self.b_ineq[i] for i in range(0, self.A_ineq.shape[0]) if i in knockable_constr_ineq] - - processes = Configuration().processes - num_Ms = len(M_A) - processes = min(processes, num_Ms) + M_A = self.A_ineq[knockable_constr_ineq, :][:, cont_vars].tocsr() + num_Ms = M_A.shape[0] max_Ax = [np.nan] * num_Ms - # Dummy to check if optimization runs - # worker_init(M_A,M_A_ineq,M_b_ineq,M_A_eq,M_b_eq,M_lb,M_ub,list(solvers.keys())[0]) - # worker_compute(1) - - logging.info(' Bounding MILP.') - if processes > 1 and num_Ms > 1000: - with SDPool(processes, - initializer=worker_init, - initargs=(M_A, M_A_ineq, M_b_ineq, M_A_eq, M_b_eq, M_lb, M_ub, getattr(self, SOLVER), getattr(self, SEED))) as pool: - chunk_size = num_Ms // processes - for i, value in pool.imap_unordered(worker_compute, range(num_Ms), chunksize=chunk_size): - max_Ax[i] = value - else: - worker_init(M_A, M_A_ineq, M_b_ineq, M_A_eq, M_b_eq, M_lb, M_ub, getattr(self, SOLVER), getattr(self, SEED)) - for i in range(num_Ms): - _, max_Ax[i] = worker_compute(i) + # max(a*x) from bounds: zero rows -> 0; single-variable rows -> coeff*(ub if coeff>0 else lb), + # or +inf if that bound is infinite; multi-variable rows -> +inf (unbounded dual constraint). + n_zero = 0 + n_single = 0 + n_multi = 0 + for i in range(num_Ms): + row = M_A.getrow(i) + nnz = row.nnz + if nnz == 0: + max_Ax[i] = 0.0 + n_zero += 1 + elif nnz == 1: + col_idx = row.indices[0] + coeff = row.data[0] + if coeff > 0: + max_Ax[i] = coeff * M_ub[col_idx] if not isinf(M_ub[col_idx]) else np.inf + else: + max_Ax[i] = coeff * M_lb[col_idx] if not isinf(M_lb[col_idx]) else np.inf + n_single += 1 + else: + max_Ax[i] = np.inf + n_multi += 1 + logging.info(' Bounding MILP: %d constraints (%d zero, %d single-var, %d multi-var->indicator/M).' % + (num_Ms, n_zero, n_single, n_multi)) # round Ms up to 5 digits Ms = [np.ceil(M * 1e5) / 1e5 if not isinf(M) else self.M for M in max_Ax] # fill up M-vector also for notknockable reactions - Ms = [ - Ms[np.array([i == j - for j in knockable_constr_ineq]).nonzero()[0][0]] if i in knockable_constr_ineq else np.nan - for i in range(self.A_ineq.shape[0]) - ] + _Ms_full = np.full(self.A_ineq.shape[0], np.nan) + _Ms_full[knockable_constr_ineq] = np.array(Ms, dtype=float) + Ms = _Ms_full # 4. Link constraints to z-variables for available upper bounds self.z_map_constr_ineq = self.z_map_constr_ineq.tocsc() @@ -805,6 +875,63 @@ def link_z(self): self.b_ineq[row] = Ms[row] self.z_map_constr_ineq = self.z_map_constr_ineq.tocsc() + # 4b. (NOT ENABLED -- kept as a design note for the FVA-bounds work.) + # + # Consolidation counterpart to the ineq->eq lumping in step 5: an arity-1 row IS a bound, + # so carrying it as a row makes the MILP state the same restriction twice. Folding the + # tightest single-variable inequality per variable into lb/ub is safe HERE (and only here): + # - z-mapped rows are excluded, so no knockout link is folded away; + # - step 4 baked finite M's in as a z-coefficient, raising those rows to arity 2, so + # they are excluded automatically; + # - dualization is long done (prevent_boundary_knockouts runs pre-dualize inside + # build_primal_from_cbm), so an unconditional row and an unconditional bound are + # equivalent from here on. The same fold BEFORE dualization would NOT be safe: a + # positive lb on a knockable variable picks up a z-mapping when dualized, letting a + # KO relax it -- which is exactly what prevent_boundary_knockouts exists to prevent, + # and why reassign_lb_ub_from_ineq() guards on z_map_vars at its call site in + # addModule(). + # + # It is OFF because at this point A_ineq holds no arity-1 rows: step 4 lifts every + # single-variable knockable row to arity 2, and the rest are multi-variable, so the fold is + # a no-op. It becomes useful once non-knockable single-variable rows exist -- e.g. when + # per-module FVA ranges are folded in as bounds. + # + # If enabled, note two things about reusing reassign_lb_ub_from_ineq() here verbatim: + # (1) its arity scan is O(nnz^2) (`list(row_ineq).count(i)` per nonzero) and must be + # rewritten with np.bincount before it can run at genome scale -- it would dwarf + # the per-row LP this step already replaced; + # (2) it RAISES on lb > ub; at this stage a contradictory system is a legitimate + # INFEASIBLE and should be reported, not raised out of the MILP build. + # + # self.A_ineq = self.A_ineq.tocsr() + # self.A_ineq.eliminate_zeros() + # nnz_per_row = np.diff(self.A_ineq.indptr) + # z_rows = set(self.z_map_constr_ineq.nonzero()[1].tolist()) + # fold_rows = [i for i in np.nonzero(nnz_per_row == 1)[0].tolist() if i not in z_rows] + # if fold_rows: + # new_lb, new_ub = list(self.lb), list(self.ub) + # for i in fold_rows: + # j = int(self.A_ineq.indices[self.A_ineq.indptr[i]]) + # coef = float(self.A_ineq.data[self.A_ineq.indptr[i]]) + # val = float(self.b_ineq[i]) / coef + # if coef > 0: # coef*x <= b -> x <= b/coef (tightest ub = min) + # new_ub[j] = min(new_ub[j], val) + # else: # coef*x <= b -> x >= b/coef (tightest lb = max) + # new_lb[j] = max(new_lb[j], val) + # if any(l > u for l, u in zip(new_lb, new_ub)): + # logging.warning(' Bound folding produced lb > ub (infeasible static problem); ' + # 'keeping the rows and leaving it to the solver.') + # else: + # self.lb, self.ub = new_lb, new_ub + # keep = np.ones(self.A_ineq.shape[0], dtype=bool) + # keep[fold_rows] = False + # self.A_ineq = self.A_ineq[keep, :] + # self.b_ineq = [self.b_ineq[i] for i in range(len(keep)) if keep[i]] + # self.z_map_constr_ineq = self.z_map_constr_ineq.tocsc()[:, keep] + # Ms = [Ms[i] for i in range(len(keep)) if keep[i]] # Ms is indexed by A_ineq row + # NB knockable_constr_ineq holds pre-fold row indices but is dead after step 5's + # `tuple(...)` rebind, so the index shift above is harmless. + # 5. Translate back remaining inequalities to equations if applicable and link via indicator constraints knockable_constr_ineq = tuple(knockable_constr_ineq) knockable_constr_ineq_ic = [i for i in range(self.A_ineq.shape[0]) if isinf(Ms[i])] @@ -886,6 +1013,39 @@ def link_z(self): self.A_eq = self.A_eq[keep_eq, :] self.b_eq = [self.b_eq[i] for i in range(len(keep_eq)) if keep_eq[i]] + # 7. Fix intervention binaries the linked MILP leaves FREE. + # After ALL linking, a targetable KO z that gates no indicator and appears in no finite-M + # row controls nothing -- its only footprint is its own cost in the budget/objective rows + # (idx_row_maxcost/mincost/obj). Toggling it changes no constraint, so it is in no minimal + # cut; fix it to 0. Running this at the END of link_z (rather than up front on the z-maps) + # catches a strictly larger set: z's whose rows were lumped/removed by steps 5/b6 end up + # free here too (measured 8 vs 0 on e_coli, 6 vs 3 on iMLcore). Done explicitly rather than + # relying on solver presolve to spot the dominated column. + # NB indicators are exactly the rows whose box-bound big-M was infinite (arity >= 2). Step 7 + # does NOT inspect their feasible-region redundancy -- that is deferred to the region-FVA + # override and the upstream essentiality scans; here a fixable z simply carries no indicator, + # so ub=0 is the whole operation. Skip non-targetable (already fixed), inverted/KI z's, and + # lb>0 (essential KI) where ub=0 would give lb>ub. + Aic = self.A_ineq.tocsc() + Aec = self.A_eq.tocsc() if self.A_eq.shape[0] else None + budget_rows = {self.idx_row_maxcost, self.idx_row_mincost, self.idx_row_obj} + z_with_ind = set(int(b) for b in self.indic_constr.binv) if self.indic_constr is not None else set() + n_free = 0 + for z in range(self.num_z): + if self.z_non_targetable[z] or self.z_inverted[z] or self.lb[z] > 0 or self.ub[z] == 0: + continue + if z in z_with_ind: + continue + col = Aic.getcol(z).tocoo() + if any((r not in budget_rows) and v != 0 for r, v in zip(col.row.tolist(), col.data.tolist())): + continue # finite-M z-link -> controls a row + if Aec is not None and Aec.getcol(z).nnz: + continue # equality z-link + self.ub[z] = 0.0 + n_free += 1 + if n_free: + logging.info(' Fixed %d free intervention binaries to zero (no indicator or z-link).' % n_free) + class ContMILP: """Continuous representation of the strain design MILP. @@ -906,7 +1066,8 @@ def __init__(self, A_ineq, b_ineq, A_eq, b_eq, lb, ub, c, z_map_constr_ineq, z_m self.z_map_constr_eq = z_map_constr_eq self.z_map_vars = z_map_vars -def build_primal_from_cbm(model, V_ineq=None, v_ineq=None, V_eq=None, v_eq=None, c=None) -> \ +def build_primal_from_cbm(model, V_ineq=None, v_ineq=None, V_eq=None, v_eq=None, c=None, + bound_override=None) -> \ Tuple[sparse.csr_matrix, Tuple, sparse.csr_matrix, Tuple, Tuple, Tuple, sparse.csr_matrix, sparse.csr_matrix, sparse.csr_matrix]: """Builds primal LP from constraint-based model and (optionally) additional constraints. @@ -947,7 +1108,7 @@ def build_primal_from_cbm(model, V_ineq=None, v_ineq=None, V_eq=None, v_eq=None, V_eq = sparse.csr_matrix((0, numr)) v_eq = [] if c is None: - c = [i.objective_coefficient for i in model.reactions] + c = [0.0] * numr S = sparse.csr_matrix(create_stoichiometric_matrix(model)) # fill matrices A_eq = sparse.vstack((S, V_eq)) @@ -956,6 +1117,17 @@ def build_primal_from_cbm(model, V_ineq=None, v_ineq=None, V_eq=None, v_eq=None, b_ineq = v_ineq.copy() lb = [float(v.lower_bound) for v in model.reactions] ub = [float(v.upper_bound) for v in model.reactions] + if bound_override: + for i, r in enumerate(model.reactions): + ov = bound_override.get(r.id) + if ov is not None: + lo, hi = ov + if lo is not None: + lb[i] = max(lb[i], float(lo)) + if hi is not None: + ub[i] = min(ub[i], float(hi)) + if lb[i] > ub[i]: # numeric guard: never emit an inconsistent block + lb[i], ub[i] = float(lo), float(hi) z_map_vars = sparse.identity(numr, 'd', format="csc") z_map_constr_eq = sparse.csc_matrix((numr, A_eq.shape[0])) z_map_constr_ineq = sparse.csc_matrix((numr, A_ineq.shape[0])) @@ -1191,50 +1363,70 @@ def reassign_lb_ub_from_ineq(A_ineq, b_ineq, A_eq, b_eq, lb, ub, if z_map_vars is None: z_map_vars = sparse.csc_matrix((numz, numr)) + # Rows carrying exactly one entry ARE bounds. A bincount over the nonzero row indices finds + # them and the single (column, value) is read straight off the COO, which keeps the scan + # linear in nnz -- necessary for this to be usable at genome scale. + def _single_entry_rows(A, z_map_constr): + """Ascending indices of rows with exactly one nonzero, excluding knockable rows, + plus row->column and row->value lookups for those rows.""" + rows, cols = A.nonzero() # scipy .nonzero() already drops explicit zeros + counts = np.bincount(rows, minlength=A.shape[0]) + single = np.nonzero(counts == 1)[0] + knockable = set(z_map_constr.nonzero()[1].tolist()) + keep = np.array([i for i in single.tolist() if i not in knockable], dtype=int) + row2col = np.full(A.shape[0], -1, dtype=int) + row2val = np.zeros(A.shape[0], dtype=float) + if len(single): + sel = np.isin(rows, single) + row2col[rows[sel]] = cols[sel] + data = np.asarray(A.tocsr()[rows[sel], cols[sel]]).ravel() + row2val[rows[sel]] = data + return keep, row2col, row2val + # translate entries to lb or ub - # find all entries in A_ineq - row_ineq = A_ineq.nonzero()[0] - # filter for rows with only one entry - var_bound_constraint_ineq = [i for i in row_ineq if list(row_ineq).count(i) == 1] - # exclude knockable constraints - var_bound_constraint_ineq = [i for i in var_bound_constraint_ineq if i not in z_map_constr_ineq.nonzero()[1]] + var_bound_constraint_ineq, ineq_row2col, ineq_row2val = _single_entry_rows(A_ineq, z_map_constr_ineq) # retrieve all bounds from inequality constraints for i in var_bound_constraint_ineq: - idx_r = A_ineq[i, :].nonzero()[1][0] # get reaction from constraint (column of entry) - if A_ineq[i, idx_r] > 0: # upper bound constraint - ub[idx_r] += [b_ineq[i] / A_ineq[i, idx_r]] + idx_r = int(ineq_row2col[i]) # get reaction from constraint (column of entry) + coef = ineq_row2val[i] + if coef > 0: # upper bound constraint + ub[idx_r] += [b_ineq[i] / coef] else: # lower bound constraint - lb[idx_r] += [b_ineq[i] / A_ineq[i, idx_r]] - - # find all entries in A_eq - row_eq = A_eq.nonzero()[0] - # filter for rows with only one entry - var_bound_constraint_eq = [i for i in row_eq if list(row_eq).count(i) == 1] - # exclude knockable constraints - var_bound_constraint_eq = [i for i in var_bound_constraint_eq if i not in z_map_constr_eq.nonzero()[1]] + lb[idx_r] += [b_ineq[i] / coef] + + var_bound_constraint_eq, eq_row2col, eq_row2val = _single_entry_rows(A_eq, z_map_constr_eq) + # knockability is a property of the VARIABLE here, so precompute it per column once instead + # of slicing z_map_vars inside the loop + col_has_z = np.asarray((z_map_vars != 0).sum(axis=0)).ravel() > 0 # retrieve all bounds from equality constraints # and partly set lb or ub derived from equality constraints, for instance: # If x = 5, set ub = 5 and keep the inequality constraint -x <= -5. # If x = -5, set lb =-5 and keep the inequality constraint x <= -5. A_ineq_new = sparse.csr_matrix((0, numr)) b_ineq_new = [] + eq_rows_to_keep_as_ineq = [] for i in var_bound_constraint_eq: - idx_r = A_eq[i, :].nonzero()[1][0] # get reaction from constraint (column of entry) - if any(z_map_vars[:, idx_r]): # if reaction is knockable - if A_eq[i, idx_r] * b_eq[i] > 0: # upper bound constraint - ub[idx_r] += [b_eq[i] / A_eq[i, idx_r]] - A_ineq_new = sparse.vstack((A_ineq_new, -A_eq[i, :])) - b_ineq_new += [-b_eq[i]] - elif A_eq[i, idx_r] * b_eq[i] < 0: # lower bound constraint - lb[idx_r] += [b_eq[i] / A_eq[i, idx_r]] - A_ineq_new = sparse.vstack((A_ineq_new, A_eq[i, :])) - b_ineq_new += [b_eq[i]] + idx_r = int(eq_row2col[i]) # get reaction from constraint (column of entry) + coef = eq_row2val[i] + if col_has_z[idx_r]: # if reaction is knockable + if coef * b_eq[i] > 0: # upper bound constraint + ub[idx_r] += [b_eq[i] / coef] + eq_rows_to_keep_as_ineq += [(i, -1.0)] + elif coef * b_eq[i] < 0: # lower bound constraint + lb[idx_r] += [b_eq[i] / coef] + eq_rows_to_keep_as_ineq += [(i, 1.0)] else: ub[idx_r] += [0.0] lb[idx_r] += [0.0] else: - lb[idx_r] += [b_eq[i] / A_eq[i, idx_r]] - ub[idx_r] += [b_eq[i] / A_eq[i, idx_r]] + lb[idx_r] += [b_eq[i] / coef] + ub[idx_r] += [b_eq[i] / coef] + # build the retained direction(s) in ONE vstack instead of growing the matrix per row + if eq_rows_to_keep_as_ineq: + idx = [i for i, _ in eq_rows_to_keep_as_ineq] + sgn = np.array([s for _, s in eq_rows_to_keep_as_ineq]) + A_ineq_new = sparse.diags(sgn) @ A_eq.tocsr()[idx, :] + b_ineq_new = [s * b_eq[i] for i, s in eq_rows_to_keep_as_ineq] # set tightest bounds (avoid inf) lb = [max([i for i in l if not isinf(i)] + [np.nan]) for l in lb] ub = [min([i for i in u if not isinf(i)] + [np.nan]) for u in ub] @@ -1246,19 +1438,23 @@ def reassign_lb_ub_from_ineq(A_ineq, b_ineq, A_eq, b_eq, lb, ub, if any(np.greater(lb, ub)): raise Exception("There is a lower bound that is greater than its upper bound counterpart.") - # remove constraints that became redundant + # remove constraints that became redundant (boolean masks keep this linear in the row count) numineq = A_ineq.shape[0] - A_ineq = A_ineq[[False if i in var_bound_constraint_ineq else True for i in range(0, numineq)]] - b_ineq = [b_ineq[i] for i in range(0, len(b_ineq)) if i not in var_bound_constraint_ineq] - z_map_constr_ineq = z_map_constr_ineq[:, [False if i in var_bound_constraint_ineq else True for i in range(0, numineq)]] + keep_ineq = np.ones(numineq, dtype=bool) + keep_ineq[var_bound_constraint_ineq] = False + A_ineq = A_ineq[keep_ineq] + b_ineq = [b_ineq[i] for i in range(0, len(b_ineq)) if keep_ineq[i]] + z_map_constr_ineq = z_map_constr_ineq[:, keep_ineq] numeq = A_eq.shape[0] - A_eq = A_eq[[False if i in var_bound_constraint_eq else True for i in range(0, numeq)]] - b_eq = [b_eq[i] for i in range(0, len(b_eq)) if i not in var_bound_constraint_eq] + keep_eq = np.ones(numeq, dtype=bool) + keep_eq[var_bound_constraint_eq] = False + A_eq = A_eq[keep_eq] + b_eq = [b_eq[i] for i in range(0, len(b_eq)) if keep_eq[i]] # add equality constraints that transformed to inequality constraints A_ineq = sparse.vstack((A_ineq, A_ineq_new)) b_ineq += b_ineq_new if numz: - z_map_constr_eq = z_map_constr_eq[:, [False if i in var_bound_constraint_eq else True for i in range(0, numeq)]] + z_map_constr_eq = z_map_constr_eq[:, keep_eq] z_map_constr_ineq = sparse.hstack((z_map_constr_ineq, sparse.csc_matrix((numz, A_ineq_new.shape[0])))) return A_ineq, b_ineq, A_eq, b_eq, lb, ub, z_map_constr_ineq, z_map_constr_eq else: @@ -1322,47 +1518,3 @@ def prevent_boundary_knockouts(A_ineq, b_ineq, lb, ub, z_map_constr_ineq, z_map_ z_map_constr_ineq = sparse.hstack([z_map_constr_ineq, sparse.csc_matrix((numz, new_z_cols))]) return A_ineq, b_ineq, lb, ub, z_map_constr_ineq - - -def _worker_cleanup(): - """Dispose the global LP and solver environment on worker exit.""" - global lp_glob - try: - if lp_glob is not None and hasattr(lp_glob, 'solver'): - from io import StringIO - from contextlib import redirect_stdout, redirect_stderr - with redirect_stdout(StringIO()), redirect_stderr(StringIO()): - if lp_glob.solver == 'gurobi': - lp_glob.backend.dispose() - import gurobipy as gp - gp.disposeDefaultEnv() - elif lp_glob.solver == 'cplex': - lp_glob.backend.end() - lp_glob = None - except Exception: - pass - - -def worker_init(A, A_ineq, b_ineq, A_eq, b_eq, lb, ub, solver, seed): - """Helper function for determining bounds on linear expressions""" - global lp_glob - lp_glob = MILP_LP(A_ineq=A_ineq, b_ineq=b_ineq, A_eq=A_eq, b_eq=b_eq, lb=lb, ub=ub, solver=solver, seed=seed) - if lp_glob == CPLEX: - lp_glob.backend.parameters.lpmethod.set(1) - lp_glob.backend.parameters.threads.set(1) - elif solver == 'gurobi': - lp_glob.backend.params.Threads = 1 - lp_glob.solver = solver - lp_glob.A = A - if solver in ('gurobi', 'cplex'): - import atexit - atexit.register(_worker_cleanup) - - -def worker_compute(i) -> Tuple[int, float]: - """Helper function for determining bounds on linear expressions""" - global lp_glob - # maximize by minimizing negative objective and negating result - lp_glob.set_objective(-lp_glob.A[i]) - min_cx = -lp_glob.slim_solve() - return i, min_cx diff --git a/tests/conftest.py b/tests/conftest.py index 7cccb7d..f1dbaa7 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -13,6 +13,7 @@ def pytest_addoption(parser): for name, help_text in [ ("--medium", "Run iMLcore genome-scale benchmarks (~4 min total)."), ("--large", "Run iML1515 large-model benchmarks (several min/solver)."), + ("--java", "Run JPype/JVM tests on Linux/macOS too (flaky, see jpype#934)."), ]: try: parser.addoption(name, action="store_true", default=False, help=help_text) @@ -49,9 +50,12 @@ def pytest_collection_modifyitems(config, items): # CI runners due to a GC finalization race (jpype#934). Windows is unaffected. # Tested jpype1==1.5.0 pinning — no improvement (still segfaults, plus no # Python 3.13 wheel causing build failures on macOS ARM64). - if platform.system() != 'Windows': + # --java forces them on anyway, which is how a Java-backend change gets verified without + # round-tripping through the Windows CI leg. + if platform.system() != 'Windows' and not config.getoption("--java", default=False): skip_java = pytest.mark.skip( - reason="JPype JNI crashes non-deterministically on Linux/macOS (jpype#934)") + reason="JPype JNI crashes non-deterministically on Linux/macOS (jpype#934); " + "pass --java to run anyway") for item in items: if "java" in item.keywords: item.add_marker(skip_java) diff --git a/tests/test_04_preprocessing.py b/tests/test_04_preprocessing.py index b9fee61..32893d3 100644 --- a/tests/test_04_preprocessing.py +++ b/tests/test_04_preprocessing.py @@ -10,15 +10,15 @@ compress_model_coupled, compress_model_parallel, remove_blocked_reactions, - stoichmat_coeff2rational, + stoichmat_coeff_to_fraction, remove_conservation_relations, stoichmat_coeff2float, - _combine_gpr_and, - _combine_gpr_or, - _gpr_ast_to_sympy, - _sympy_to_gpr_string, + _combine_gprs, + _gpr_ast_to_expr, + _expr_to_gpr_string, ) -from sympy import simplify_logic, And as SA, Or as SO, Symbol as SS +from cobra.core.gene import GPR +from sympy import simplify_logic # ── GPR extension + compression ────────────────────────────────────── @@ -61,138 +61,100 @@ def test_gpr_extension_compression2(model_gpr): # ── GPR propagation helper unit tests ──────────────────────────────── -class TestGprAstToSympy: +class TestGprAstToExpr: + # None is special-cased (identity check); the rest share one assertion shape. def test_none_returns_none(self): - assert _gpr_ast_to_sympy(None) is None - - def test_single_gene(self): - node = ast.Name(id='g1') - result = _gpr_ast_to_sympy(node) - assert result == SS('g1') - - def test_and_expression(self): - node = ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]) - result = _gpr_ast_to_sympy(node) - assert result == SA(SS('g1'), SS('g2')) - - def test_or_expression(self): - node = ast.BoolOp(op=ast.Or(), values=[ast.Name(id='g1'), ast.Name(id='g2')]) - result = _gpr_ast_to_sympy(node) - assert result == SO(SS('g1'), SS('g2')) - - def test_nested(self): + assert _gpr_ast_to_expr(None) is None + + @pytest.mark.parametrize("node,expected", [ + (ast.Name(id='g1'), 'g1'), + (ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]), + ('and', ('g1', 'g2'))), + (ast.BoolOp(op=ast.Or(), values=[ast.Name(id='g1'), ast.Name(id='g2')]), + ('or', ('g1', 'g2'))), # (g1 and g2) or g3 - node = ast.BoolOp(op=ast.Or(), values=[ + (ast.BoolOp(op=ast.Or(), values=[ ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]), - ast.Name(id='g3') - ]) - result = _gpr_ast_to_sympy(node) - expected = SO(SA(SS('g1'), SS('g2')), SS('g3')) - assert result == expected - - -class TestSympyToGprString: - def test_none_returns_empty(self): - assert _sympy_to_gpr_string(None) == '' - - def test_single_symbol(self): - assert _sympy_to_gpr_string(SS('g1')) == 'g1' - - def test_and(self): - result = _sympy_to_gpr_string(SA(SS('g1'), SS('g2'))) - assert result == 'g1 and g2' - - def test_or(self): - result = _sympy_to_gpr_string(SO(SS('g1'), SS('g2'))) - assert result == 'g1 or g2' - - def test_nested_and_in_or(self): - # g3 or (g1 and g2) - expr = SO(SA(SS('g1'), SS('g2')), SS('g3')) - result = _sympy_to_gpr_string(expr) - assert result == '(g1 and g2) or g3' - - def test_nested_or_in_and(self): - # g3 and (g1 or g2) - expr = SA(SO(SS('g1'), SS('g2')), SS('g3')) - result = _sympy_to_gpr_string(expr) - assert result == '(g1 or g2) and g3' + ast.Name(id='g3')]), + ('or', (('and', ('g1', 'g2')), 'g3'))), + # g1 and (g2 and g3) -> flattened + (ast.BoolOp(op=ast.And(), values=[ + ast.Name(id='g1'), + ast.BoolOp(op=ast.And(), values=[ast.Name(id='g2'), ast.Name(id='g3')])]), + ('and', ('g1', 'g2', 'g3'))), + ], ids=['single_gene', 'and', 'or', 'nested', 'nested_same_op_flattened']) + def test_gpr_ast_to_expr(self, node, expected): + assert _gpr_ast_to_expr(node) == expected + + +class TestExprToGprString: + @pytest.mark.parametrize("expr,expected", [ + (None, ''), + ('g1', 'g1'), + (('and', ['g1', 'g2']), 'g1 and g2'), + (('or', ['g1', 'g2']), 'g1 or g2'), + (('or', [('and', ['g1', 'g2']), 'g3']), '(g1 and g2) or g3'), + (('and', [('or', ['g1', 'g2']), 'g3']), '(g1 or g2) and g3'), + ], ids=['none', 'single_gene', 'and', 'or', 'nested_and_in_or', 'nested_or_in_and']) + def test_expr_to_gpr_string(self, expr, expected): + assert _expr_to_gpr_string(expr) == expected def test_deterministic_sorting(self): - # Should always produce same order - result1 = _sympy_to_gpr_string(SA(SS('g2'), SS('g1'), SS('g3'))) - result2 = _sympy_to_gpr_string(SA(SS('g3'), SS('g1'), SS('g2'))) + result1 = _expr_to_gpr_string(('and', ['g2', 'g1', 'g3'])) + result2 = _expr_to_gpr_string(('and', ['g3', 'g1', 'g2'])) assert result1 == result2 == 'g1 and g2 and g3' + def test_roundtrips_through_cobra(self): + # whatever we render must parse back to an equivalent rule in cobra + rule = _expr_to_gpr_string(('or', [('and', ['g1', 'g2']), 'g3'])) + assert GPR.from_string(rule).as_symbolic() == \ + GPR.from_string('(g1 and g2) or g3').as_symbolic() -class TestCombineGprAnd: - def test_all_empty(self): - assert _combine_gpr_and([None, None]) == '' - def test_single_non_empty(self): - node = ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]) - result = _combine_gpr_and([node]) - assert result == 'g1 and g2' - - def test_skip_empty(self): - """Empty GPR (None) should be skipped in AND combination.""" - node = ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]) - result = _combine_gpr_and([node, None, None]) - assert result == 'g1 and g2' - - def test_two_non_empty(self): - node1 = ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]) - node2 = ast.Name(id='g3') - result = _combine_gpr_and([node1, node2]) - assert result == 'g1 and g2 and g3' - - def test_simplification(self): - """AND of overlapping expressions should simplify.""" - # (g1 and g2) AND (g1 and g3) -> g1 and g2 and g3 - node1 = ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]) - node2 = ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g3')]) - result = _combine_gpr_and([node1, node2]) - assert result == 'g1 and g2 and g3' - - def test_empty_list(self): - assert _combine_gpr_and([]) == '' +class TestCombineGprAnd: + @pytest.mark.parametrize("nodes,expected", [ + ([None, None], ''), + ([ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')])], + 'g1 and g2'), + # Empty GPR (None) should be skipped in AND combination. + ([ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]), None, None], + 'g1 and g2'), + ([ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]), + ast.Name(id='g3')], + 'g1 and g2 and g3'), + # AND of overlapping expressions should simplify: (g1 and g2) AND (g1 and g3). + ([ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]), + ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g3')])], + 'g1 and g2 and g3'), + ([], ''), + ], ids=['all_empty', 'single_non_empty', 'skip_empty', 'two_non_empty', + 'simplification', 'empty_list']) + def test_combine_gprs_and(self, nodes, expected): + assert _combine_gprs(nodes, 'and') == expected class TestCombineGprOr: - def test_any_empty_returns_empty(self): - """If any reaction has empty GPR (always active), result is empty.""" - node = ast.Name(id='g1') - result = _combine_gpr_or([node, None]) - assert result == '' - - def test_all_empty(self): - assert _combine_gpr_or([None, None]) == '' - - def test_two_non_empty(self): - node1 = ast.Name(id='g1') - node2 = ast.Name(id='g2') - result = _combine_gpr_or([node1, node2]) - assert result == 'g1 or g2' - - def test_deduplication(self): - """OR with duplicate terms should deduplicate (sympy constructor).""" - # g1 OR g1 -> g1 - node1 = ast.Name(id='g1') - node2 = ast.Name(id='g1') - result = _combine_gpr_or([node1, node2]) - assert result == 'g1' + @pytest.mark.parametrize("nodes,expected", [ + # If any reaction has empty GPR (always active), result is empty. + ([ast.Name(id='g1'), None], ''), + ([None, None], ''), + ([ast.Name(id='g1'), ast.Name(id='g2')], 'g1 or g2'), + # OR with duplicate terms should deduplicate: g1 OR g1 -> g1. + ([ast.Name(id='g1'), ast.Name(id='g1')], 'g1'), + ([], ''), + ], ids=['any_empty_returns_empty', 'all_empty', 'two_non_empty', + 'deduplication', 'empty_list']) + def test_combine_gprs_or(self, nodes, expected): + assert _combine_gprs(nodes, 'or') == expected def test_no_absorption(self): - """OR does raw merge — absorption is deferred to reduce_gpr.""" + """OR does raw merge — absorption is deferred to simplify_model_gprs.""" # (g1 and g2) OR g1 -> kept as-is (not simplified to g1) node1 = ast.BoolOp(op=ast.And(), values=[ast.Name(id='g1'), ast.Name(id='g2')]) node2 = ast.Name(id='g1') - result = _combine_gpr_or([node1, node2]) + result = _combine_gprs([node1, node2], 'or') assert 'g1 and g2' in result and 'or' in result - def test_empty_list(self): - assert _combine_gpr_or([]) == '' - # ── GPR propagation integration tests (model_gpr.xml) ──────────────── @@ -205,7 +167,7 @@ class TestModelGprCompression: def test_coupled_compression_propagates_gpr(self, gpr_model): """Coupled compression should AND-combine GPR rules, skipping empty ones.""" remove_blocked_reactions(gpr_model) - stoichmat_coeff2rational(gpr_model) + stoichmat_coeff_to_fraction(gpr_model) remove_conservation_relations(gpr_model) orig_gprs = {r.id: r.gene_reaction_rule for r in gpr_model.reactions} @@ -256,7 +218,7 @@ def test_coupled_group_r4_r5_r6_rdex(self, gpr_model): DNF: (g1 & g4 & g7 & g8) | (g1 & g4 & g5 & g8 & g9) """ remove_blocked_reactions(gpr_model) - stoichmat_coeff2rational(gpr_model) + stoichmat_coeff_to_fraction(gpr_model) remove_conservation_relations(gpr_model) reac_map = compress_model_coupled(gpr_model, propagate_gpr=True) @@ -271,11 +233,9 @@ def test_coupled_group_r4_r5_r6_rdex(self, gpr_model): f"Expected r4, r5, r6 to be merged. Reaction map: {reac_map}" from cobra.core.gene import GPR - parsed = GPR.from_string(target_rxn.gene_reaction_rule) - result_sympy = _gpr_ast_to_sympy(parsed.body) - - g1, g4, g5, g7, g8, g9 = [SS(f'g{i}') for i in [1, 4, 5, 7, 8, 9]] - expected = SO(SA(g1, g4, g7, g8), SA(g1, g4, g5, g8, g9)) + result_sympy = GPR.from_string(target_rxn.gene_reaction_rule).as_symbolic() + expected = GPR.from_string( + '(g1 and g4 and g7 and g8) or (g1 and g4 and g5 and g8 and g9)').as_symbolic() assert simplify_logic(result_sympy ^ expected) == False, \ f"GPR mismatch. Got: {result_sympy}, expected: {expected}" @@ -289,7 +249,7 @@ def test_coupled_group_r3_rpex(self, gpr_model): AND combine (skip empty): just r3's GPR = g8 or (g3 and g6) """ remove_blocked_reactions(gpr_model) - stoichmat_coeff2rational(gpr_model) + stoichmat_coeff_to_fraction(gpr_model) remove_conservation_relations(gpr_model) reac_map = compress_model_coupled(gpr_model, propagate_gpr=True) @@ -302,11 +262,8 @@ def test_coupled_group_r3_rpex(self, gpr_model): assert target_rxn is not None from cobra.core.gene import GPR - parsed = GPR.from_string(target_rxn.gene_reaction_rule) - result_sympy = _gpr_ast_to_sympy(parsed.body) - - g3, g6, g8 = SS('g3'), SS('g6'), SS('g8') - expected = SO(g8, SA(g3, g6)) + result_sympy = GPR.from_string(target_rxn.gene_reaction_rule).as_symbolic() + expected = GPR.from_string('g8 or (g3 and g6)').as_symbolic() assert simplify_logic(result_sympy ^ expected) == False, \ f"GPR mismatch. Got: {result_sympy}, expected: {expected}" @@ -333,13 +290,13 @@ def test_efmtool_coupled_gpr_matches_sparse_rref(self, gpr_model, java_available # Sparse RREF path remove_blocked_reactions(gpr_model) - stoichmat_coeff2rational(gpr_model) + stoichmat_coeff_to_fraction(gpr_model) remove_conservation_relations(gpr_model) rref_map = compress_model_coupled(gpr_model, compression_backend='sparse_rref', propagate_gpr=True) # Efmtool path remove_blocked_reactions(model_java) - stoichmat_coeff2rational(model_java) + stoichmat_coeff_to_fraction(model_java) remove_conservation_relations(model_java) java_map = compress_model_coupled(model_java, compression_backend='efmtool_rref', propagate_gpr=True) @@ -365,8 +322,8 @@ def gpr_by_group(model, reac_map): continue from cobra.core.gene import GPR - sym_rref = _gpr_ast_to_sympy(GPR.from_string(gpr_rref).body) - sym_java = _gpr_ast_to_sympy(GPR.from_string(gpr_java).body) + sym_rref = GPR.from_string(gpr_rref).as_symbolic() + sym_java = GPR.from_string(gpr_java).as_symbolic() assert simplify_logic(sym_rref ^ sym_java) == False, \ f"GPR mismatch for group {sorted(group_key)}: rref='{gpr_rref}', java='{gpr_java}'" diff --git a/tests/test_07_compression.py b/tests/test_07_compression.py index 6840dcf..2397847 100644 --- a/tests/test_07_compression.py +++ b/tests/test_07_compression.py @@ -69,7 +69,7 @@ def test_python_compression_basic(model_gpr): def test_python_compression_coupled_function(model_small_example): """compress_model_coupled with compression_backend='sparse_rref' returns a dict.""" - nt.stoichmat_coeff2rational(model_small_example) + nt.stoichmat_coeff_to_fraction(model_small_example) nt.remove_conservation_relations(model_small_example) reac_map = nt.compress_model_coupled(model_small_example, compression_backend='sparse_rref') assert isinstance(reac_map, dict) @@ -77,7 +77,7 @@ def test_python_compression_coupled_function(model_small_example): def test_compression_coefficient_type(model_small_example): """Compression coefficients are exact rational number types.""" - nt.stoichmat_coeff2rational(model_small_example) + nt.stoichmat_coeff_to_fraction(model_small_example) nt.remove_conservation_relations(model_small_example) reac_map = nt.compress_model_coupled(model_small_example, compression_backend='sparse_rref') for new_reac, old_reacs in reac_map.items(): @@ -85,9 +85,9 @@ def test_compression_coefficient_type(model_small_example): assert is_rational_type(coeff), (f"Coefficient for {old_reac} in {new_reac}: expected rational, got {type(coeff)}") -def test_stoichmat_coeff2rational_uses_rational_type(model_small_example): - """stoichmat_coeff2rational converts all coefficients to rational types.""" - nt.stoichmat_coeff2rational(model_small_example) +def test_stoichmat_coeff_to_fraction_uses_rational_type(model_small_example): + """stoichmat_coeff_to_fraction converts all coefficients to rational types.""" + nt.stoichmat_coeff_to_fraction(model_small_example) for reaction in model_small_example.reactions: for metabolite, coeff in reaction._metabolites.items(): assert is_rational_type(coeff), (f"Coefficient for {metabolite.id} in {reaction.id}: expected rational, got {type(coeff)}") @@ -161,21 +161,51 @@ def test_compression_parity_reaction_count(jpype_available): model_java.reactions), (f"Reaction count mismatch: sparse_rref={len(model_py.reactions)}, efmtool_rref={len(model_java.reactions)}") +def _trace_lump(cmp_maps, orig_id): + """Follow an original reaction through the compression rounds. + + Returns (compressed_id, factor) with orig_flux == factor * compressed_flux. + """ + cur, factor = orig_id, 1.0 + for rnd in cmp_maps: + for new_id, members in rnd["reac_map_exp"].items(): + if cur in members: + factor *= float(members[cur]) + cur = new_id + break + return cur, factor + + @pytest.mark.java def test_fba_equivalence(jpype_available): - """Both compression backends produce compressed models with the same optimal FBA value (straindesign FBA).""" - model_py = load_model("e_coli_core") - nt.compress_model(model_py, compression_backend='sparse_rref') - model_java = load_model("e_coli_core") - nt.compress_model(model_java, compression_backend='efmtool_rref') + """Both backends preserve the uncompressed optimum once the lump factor is applied. - biomass_py = next((r.id for r in model_py.reactions if 'biomass' in r.id.lower()), None) - biomass_java = next((r.id for r in model_java.reactions if 'biomass' in r.id.lower()), None) - assert biomass_py and biomass_java, "Could not find biomass reaction" - - val_py = sd.fba(model_py, obj={biomass_py: 1}, obj_sense='maximize').objective_value - val_java = sd.fba(model_java, obj={biomass_java: 1}, obj_sense='maximize').objective_value - assert abs(val_py - val_java) < 1e-6, (f"FBA objective mismatch: sparse_rref={val_py}, efmtool_rref={val_java}") + A lump's overall scale is free: only its ratios are fixed, so the raw objective value of a + lumped reaction is backend-specific and not a meaningful thing to compare. sparse_rref + re-expresses each lump in one member's units, efmtool_rref does not, so their biomass columns + differ by a constant factor. What must agree -- and what a caller actually relies on -- is the + flux recovered through the compression map. + """ + base = load_model("e_coli_core") + biomass = next((r.id for r in base.reactions if 'biomass' in r.id.lower()), None) + assert biomass, "Could not find biomass reaction" + ref = sd.fba(base, obj={biomass: 1}, obj_sense='maximize').objective_value + + recovered = {} + for backend in ('sparse_rref', 'efmtool_rref'): + model = load_model("e_coli_core") + cmp_maps = nt.compress_model(model, compression_backend=backend) + cmp_id, factor = _trace_lump(cmp_maps, biomass) + assert cmp_id in [r.id for r in model.reactions], ( + f"{backend}: compression map names {cmp_id}, which is not in the compressed model") + val = sd.fba(model, obj={cmp_id: 1}, obj_sense='maximize').objective_value + recovered[backend] = factor * val + + for backend, val in recovered.items(): + assert abs(val - ref) < 1e-6, ( + f"{backend}: recovered optimum {val} != uncompressed {ref}") + assert abs(recovered['sparse_rref'] - recovered['efmtool_rref']) < 1e-6, ( + f"Backend mismatch after mapping back: {recovered}") def test_cobra_optimize_after_compression():