feat: GO single source of truth (fix 2, #604)#609
Merged
Conversation
…(fix 2, #604) Previously GO had two independent downloads: go.json (fed the transform TSV) and go.owl (built go.db, the MF/BP/CC aspect map). A `kg download` straddling a GO release boundary could land them at different releases, silently miscategorizing MolecularActivity/CellularComponent terms as BiologicalProcess. The #604/#606 version-alignment gate *detected* this; fix 2 makes it *impossible* by collapsing GO to one source. - download.yaml: drop the standalone go.json download; go.owl is the only GO source (with a comment warning against re-adding go.json). - ONTOLOGIES_MAP["go"]: "go.json" → "go.owl". The transform now derives go.json from go.owl via ROBOT owl→json (the same path ChEBI already uses), so go.json and go.db share go.owl's release by construction. - ontology_utils._derived_json_is_stale(owl, json): regenerate a derived JSON when its OBO release differs from the source OWL's. Conservative — only when both stamps are readable and differ, so unstamped .owl inputs (metpo, upa, …) keep the prior "convert if missing" behavior. - ontology_utils._ensure_go_db: also rebuild go.db when its release drifts from go.owl (reuses new _go_db_release, reading `obo:go.owl owl:versionInfo`), so a refreshed go.owl realigns the aspect map too. Guarded to avoid spurious 30-min rebuilds when a stamp is unreadable. - assert_go_version_alignment: docstring updated to the single-source model; it remains the belt-and-braces guard against a leftover pre-fix-2 go.json. Tests: +8 (derived-json staleness, _go_db_release incl. decoy-subject/corrupt, _ensure_go_db keeps-aligned / rebuilds-on-drift). 29 pass. ruff + codespell clean. Note: the transform now runs a ROBOT owl→json conversion for GO on first run / after a GO refresh (steady-state re-runs skip it via the staleness guard), same as ChEBI. End-to-end verification requires a GO transform run in the pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes GO a single source of truth in the ontology pipeline by downloading only go.owl and deriving both go.json (for TSV transform) and go.db (for GO aspect map) from that same OWL, preventing silent version drift across GO artifacts.
Changes:
- Remove the standalone
go.jsondownload and switch the ontologies transform input for GO togo.owl. - Add staleness detection for derived OBO-JSON so an out-of-date
go.jsoncan be regenerated after an OWL refresh. - Add GO SemSQL
go.dbrelease-stamp reading and rebuild logic whengo.dbdrifts fromgo.owl, plus expanded tests.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
download.yaml |
Drops go.json download and documents GO as single-source (go.owl). |
kg_microbe/transform_utils/ontologies/ontologies_transform.py |
Switches GO to OWL input and adds derived-JSON staleness check to trigger reconversion. |
kg_microbe/utils/ontology_utils.py |
Adds _derived_json_is_stale, _go_db_release, and extends _ensure_go_db to rebuild on release drift. |
tests/test_go_version_alignment.py |
Adds unit tests for derived-JSON staleness and GO DB release/drift behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+206
to
+214
| elif data_file.suffix == ".owl": | ||
| json_path = str(data_file).replace(".owl", ".json") | ||
| if not Path(json_path).is_file(): | ||
| # Regenerate the derived JSON when it's missing OR its OBO | ||
| # release no longer matches the source OWL. For single-source | ||
| # ontologies (fix 2 #604: GO derives go.json from go.owl) this | ||
| # keeps the transform output locked to the same release the OWL | ||
| # (and go.db, built from it) carry, so a refreshed go.owl can't | ||
| # leave a stale go.json behind. | ||
| if not Path(json_path).is_file() or _derived_json_is_stale(data_file, Path(json_path)): |
Comment on lines
+180
to
+190
| try: | ||
| conn = sqlite3.connect(db_path) | ||
| try: | ||
| row = conn.execute( | ||
| "SELECT value FROM statements WHERE predicate = 'owl:versionInfo' " | ||
| "AND subject = 'obo:go.owl' AND value IS NOT NULL LIMIT 1" | ||
| ).fetchone() | ||
| finally: | ||
| conn.close() | ||
| except sqlite3.Error: | ||
| return None |
… read Address review findings on #609: - HIGH (correctness): the stale-JSON regeneration was a no-op. convert_to_json (robot_utils.py:50) skips when the target already exists, so a stale go.json was never overwritten — a refreshed go.owl would leave the old go.json in place and the alignment gate would hard-abort the transform instead of it self-healing. Fix: unlink the stale go.json before reconverting in the .owl branch (mirrors _ensure_go_db's os.remove-then-rebuild). _derived_json_is_stale returns False for a missing file, so the unlink never hits a missing path. - LOW (robustness): _go_db_release hard-coded subject='obo:go.owl'. Broaden to also match 'obo:go' and the full-IRI form ('%/go.owl') so a different semsql/rdftab subject encoding doesn't silently return None (which would make _ensure_go_db reuse a drifted db). Still excludes GO: term subjects. Tests: +3 (stale go.json unlinked+reconverted via a stubbed transform; aligned go.json not reconverted; full-IRI db subject read). 22 pass. ruff + codespell clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
GO had two independent downloads —
go.json(transform TSV) andgo.owl(buildsgo.db, the MF/BP/CC aspect map). Akg downloadstraddling a GO release boundary could land them at different releases, silently miscategorizing MolecularActivity/CellularComponent terms as BiologicalProcess. The #604/#606 version-alignment gate detected this drift; fix 2 makes it impossible by collapsing GO to a single source:go.owl→ derivedgo.json+ builtgo.db.Changes
download.yaml— drop the standalonego.jsondownload;go.owlis the only GO source.ONTOLOGIES_MAP["go"]"go.json"→"go.owl"— the transform derivesgo.jsonfromgo.owlvia ROBOTowl→json(the path ChEBI already uses), sogo.jsonandgo.dbshare one release by construction._derived_json_is_stale(owl, json)— regenerate a derived JSON when its OBO release differs from the source OWL. Conservative: only when both stamps are readable and differ, so unstamped.owlinputs (metpo, upa, …) keep the prior "convert if missing" behavior._ensure_go_db— also rebuildgo.dbwhen its release drifts fromgo.owl(new_go_db_releasereadsobo:go.owl owl:versionInfo). Guarded to avoid spurious ~30-min rebuilds when a stamp is unreadable.assert_go_version_alignment— docstring updated to the single-source model; remains the belt-and-braces guard against a leftover pre-fix-2go.json.Tests / gates
_go_db_releaseincl. decoy-subject & corrupt-db;_ensure_go_dbkeeps-aligned / rebuilds-on-drift). 29 pass.ruff check kg_microbe/ tests/✅ · codespell ✅Note
The GO transform now runs a ROBOT
owl→jsonconversion on first run / after a GO refresh (steady-state re-runs skip it via the staleness guard), same as ChEBI. End-to-end verification needs a GO transform run in the pipeline.Closes the fix-2 item of #604.
🤖 Generated with Claude Code