From 498d1d59be71d50aa9ef461f06f1d936c15c85ab Mon Sep 17 00:00:00 2001 From: Eduard Kerkhoven Date: Sun, 19 Jul 2026 00:12:04 +0200 Subject: [PATCH] =?UTF-8?q?confidence:=20cross-model=20validation=20of=20t?= =?UTF-8?q?he=20facet=20bands=20(=C2=A710.2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run measure_confidence_facets.py over three further models beyond yeast-GEM (Human-GEM, iYali, panAsp/pAo) and record the result: - Two of the three bands §6 could not exercise DO fire on other real models -- formula-generic on all three, formula-unparseable on both drafts -- so they are calibrated, not yeast artefacts. Only charge-unknown stays fixture-only across all four (a mass-balanced reaction with a charge-less metabolite is genuinely rare). - gpr+literature fires only where reactions carry a pubmed annotation. - The gene-rubric-vs-recorded-Confidence-Level check needs a per-reaction-curated model: iYali and pAo record a blanket value, so they are not graded ground truth the way yeast-GEM is. Also generalise the script to load SBML or cobra YAML/JSON by extension (Human-GEM ships as YAML). Docs: §6 caveat resolved with a pointer to §10.2; §10.2 written up and marked done. -W docs build clean. --- docs/studies/confidence_tracking.md | 62 ++++++++++++++++------------ scripts/measure_confidence_facets.py | 12 +++++- 2 files changed, 46 insertions(+), 28 deletions(-) diff --git a/docs/studies/confidence_tracking.md b/docs/studies/confidence_tracking.md index 5c2b434..d1814b3 100644 --- a/docs/studies/confidence_tracking.md +++ b/docs/studies/confidence_tracking.md @@ -179,10 +179,11 @@ disagreements are informative rather than embarrassing: 43 gene-less reactions c ### What yeast-GEM cannot test -Three bands never fire on it, and are therefore exercised only by synthetic fixtures: `formula-unparseable` -(no parenthesised formulas), `formula-generic` (no residual lands in an R group), and `charge-unknown` (all -2748 metabolites carry a charge). This is stated rather than hidden — a distribution measured on one model is -not a guarantee about another. +Three bands never fire on it: `formula-unparseable` (no parenthesised formulas), `formula-generic` (no +residual lands in an R group), and `charge-unknown` (all 2748 metabolites carry a charge). This is stated +rather than hidden — a distribution measured on one model is not a guarantee about another. The cross-model +sweep in [§10.2](#102-validate-beyond-one-model) resolves it: `formula-generic` and `formula-unparseable` +both fire on other real models; only `charge-unknown` stays exercised solely by synthetic fixtures. ## 7. Integration with the existing tools @@ -203,10 +204,8 @@ not a guarantee about another. **Shipped:** `ConfidenceEntry`, `ReactionConfidence`, `get_confidence` / `set_confidence` / `clear_confidence`, `read_confidence`, `mark_curated`, `equation_exempt`, `gene_association_exempt`, `score_localization_confidence`, `score_equation_confidence`, `score_gene_association_confidence`, -`confidence_report`, `facet_summary`. Storage lives in `reaction.notes["raven_confidence"]`; there is no -separate save step — the record serialises with the model. - -**Planned:** an umbrella `annotate_confidence(model, facets=[...])` — see [§10](#10-what-is-left). +`annotate_confidence`, `confidence_report`, `facet_summary`. Storage lives in +`reaction.notes["raven_confidence"]`; there is no separate save step — the record serialises with the model. ## 9. Standards alignment (for the paper) @@ -238,30 +237,39 @@ Two cautions govern the mapping: ## 10. What is left -facet is planned. The wiring (§10.1) and the standards mapping (§10.3) are shipped; what remains is -validating beyond one model (§10.2) and the paper write-up. +The **facet set is closed**: `localization`, `equation` and `gene_association` are shipped, and no further +facet is planned. The wiring (§10.1), the standards mapping (§10.3) and the cross-model validation (§10.2) +are done; what remains is the paper write-up. -### 10.1 Wire the facets together — shipped +### 10.1 Wire the facets together — done -- **`annotate_confidence(model, proposal=..., scores=...)`** runs every applicable scorer in one call and - returns `{facet: count}`, so a caller need not know the three scorer names and their argument shapes. - `localization` runs only when a proposal and score table are given; a facet whose inputs are absent is - skipped rather than failed — the same abstain-rather-than-guess rule the scores themselves follow. -- **`curation_priority` reads the record.** A reaction a curator has settled with `mark_curated` is dropped - from the review queue (`include_curated=False`), closing the loop between the two tools: score → review → - curate → *stop being asked about it*. -- **The SBO precondition names its remedy.** The no-SBO-terms warning points at - `raven_toolbox.annotation.add_sbo_terms(model)`. +- **`annotate_confidence(model, *, proposal=None, scores=None, facets=None)`** runs every applicable scorer + in one call and skips a facet whose inputs are absent (`localization` needs a proposal and a score table; + the other two need only the model) rather than failing — the same abstain-rather-than-guess rule the + scores themselves follow. +- **`curation_priority` now skips curated placements.** A reaction whose `localization` facet is + `mark_curated` is dropped from the review queue (pass `include_curated=True` to keep it), closing the + score → review → curate loop so a settled reaction stops resurfacing. +- **The SBO-precondition warning now names its remedy** — `raven_toolbox.annotation.add_sbo_terms(model)`. ### 10.2 Validate beyond one model -Every number in §6 comes from yeast-GEM, and §6 already flags that three bands (`formula-unparseable`, -`formula-generic`, `charge-unknown`) never fire there — they are covered only by synthetic fixtures. A -distribution measured on one model is not a guarantee about another. Running -`scripts/measure_confidence_facets.py` over Human-GEM and a non-curated draft would show whether the bands -are calibrated or merely yeast-shaped, and would exercise the branches yeast cannot reach. The -gene-rubric-vs-`Confidence Level` check only replicates on a model that records that note, so its absence -elsewhere is itself worth reporting. +Done. `scripts/measure_confidence_facets.py` (now reads SBML or YAML/JSON) was run on three more models — +**Human-GEM** (12 877 rxns), **iYali** (1924) and **panAsp/pAo** (1917) — vs yeast-GEM's §6 baseline: + +| equation band | yeast-GEM | Human-GEM | iYali | pAo | +|---|---|---|---|---| +| `formula-generic` (0.3) | 0 | 66 | 40 | 26 | +| `formula-unparseable` (0.3) | 0 | 0 | 8 | 67 | +| `charge-unknown` (0.6) | 0 | 0 | 0 | 0 | + +Two of the three bands yeast-GEM could not exercise fire on other real models, so they are calibrated, not +yeast artefacts. Only `charge-unknown` stays fixture-only — a mass-balanced reaction with a charge-less +metabolite is rare (curated models charge everything; drafts fail mass balance first). + +`gpr+literature` (0.9) fires only where reactions carry a `pubmed` annotation (iYali 749; the others 0). The +gene-rubric-vs-`Confidence Level` check needs a per-reaction-curated model: iYali and pAo record a blanket +value (all 2, all 0), so only yeast-GEM's varied notes are usable ground truth. ### 10.3 Standards alignment for the paper — mapping shipped diff --git a/scripts/measure_confidence_facets.py b/scripts/measure_confidence_facets.py index 8bdc323..442352d 100644 --- a/scripts/measure_confidence_facets.py +++ b/scripts/measure_confidence_facets.py @@ -40,6 +40,16 @@ def _recorded(rxn) -> int | None: return None +def _load_model(path: Path): + """Load a model by extension: SBML (.xml/.sbml), cobra YAML (.yml/.yaml) or cobra JSON (.json).""" + suffix = path.suffix.lower() + if suffix in (".yml", ".yaml"): + return cobra.io.load_yaml_model(str(path)) + if suffix == ".json": + return cobra.io.load_json_model(str(path)) + return cobra.io.read_sbml_model(str(path)) + + def main(argv=None): ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) @@ -48,7 +58,7 @@ def main(argv=None): ap.add_argument("--head", type=int, default=12, help="rows of the review queue to print") args = ap.parse_args(argv) - model = cobra.io.read_sbml_model(str(args.model)) + model = _load_model(args.model) total = len(model.reactions) print(f"=== {model.id or args.model.name}: {total} reactions, {len(model.metabolites)} metabolites ===")