Skip to content

Emit kg-phenio release metadata (kozahub-metadata-schema)#195

Merged
kevinschaper merged 2 commits into
masterfrom
release-metadata
May 7, 2026
Merged

Emit kg-phenio release metadata (kozahub-metadata-schema)#195
kevinschaper merged 2 commits into
masterfrom
release-metadata

Conversation

@kevinschaper

Copy link
Copy Markdown
Collaborator

Summary

kg-phenio now writes output/release-metadata.yaml describing the upstream sources consumed (phenio + the two upheno SSSOM mappings) and the artifacts produced (merged-kg.tar.gz). The receipt nests phenio's own per-ontology versions — read from phenio's `phenio-upstream-versions.tsv` release asset — as a `Release` inside the phenio entry, which itself sits under kg-phenio.

The receipt shape (excerpt):

```yaml
source: kg-phenio
source_version: v2026-04-14 # phenio's tag, used as the primary upstream version
build_version: kg-phenio_v2026-04-14__
sources:

  • id: phenio
    name: PHENIO
    version: v2026-04-14
    version_method: github_release_api
    sources:
    • {id: infores:bfo, version: 2019-08-26, version_method: owl_version_iri, urls: [...]}
    • {id: infores:chebi, version: 2026-04-20, version_method: owl_version_iri, urls: [...]}
    • {id: infores:mondo, version: ..., ...}

    ~20 ontologies from phenio-upstream-versions.tsv

  • {id: infores:upheno-cross-species, ...}
  • {id: infores:upheno-species-independent, ...}
    artifacts:
  • {path: merged-kg.tar.gz, sha256: ...}
    ```

When monarch-ingest's aggregator picks this up (follow-up PR), HP/MONDO/CHEBI/etc. versions in monarch-kg's receipt trace back through `kg-phenio → phenio → infores:`, full provenance preserved.

Files

  • `versions.py` (top-level, outside the `kg_phenio/` package so the metadata script doesn't pull in kgx via package init): `get_source_versions()` builds the nested phenio `Release` plus the two upheno peer leaves.
  • `scripts/write_metadata.py`: standard kozahub-metadata-schema wrapper.
  • `pyproject.toml`: `kozahub-metadata-schema` git dep.
  • `Jenkinsfile`: new `Metadata` stage between Merge and Publish; the file is included in the s3 sync alongside the KG tarball.

Test plan

  • Local dry run with phenio's manifest from an unreleased checkout: `PHENIO_MANIFEST_PATH=../phenio/src/ontology/phenio-upstream-versions.tsv python scripts/write_metadata.py` produces a valid receipt with all 6 ontologies (bfo/bspo/chebi/monochrom/pato/ro) showing real versions, plus the upheno mappings with `http_last_modified` versions.
  • First production Jenkins build of kg-phenio after merge — confirms the Metadata stage runs and the file lands in s3.

Notes

  • The next phenio release will publish `phenio-upstream-versions.tsv` (`RELEASE_ASSETS` in `phenio.Makefile` already references it). Until then the production fetch will 404 and the `Metadata` stage will fail — by design (we'd rather notice loudly than silently emit an empty manifest).
  • Companion PR coming on monarch-ingest to support a non-default metadata URL so kg-phenio's receipt (which lives at `https://kg-hub.berkeleybop.io/kg-phenio/current/release-metadata.yaml\`, not a GitHub release) can be fetched by `ingest download-release-metadata`.

kevinschaper added a commit to monarch-initiative/monarch-ingest that referenced this pull request May 5, 2026
kg-phenio publishes its release-metadata.yaml to KG-Hub S3, not to a
GitHub release, so the default `releases/latest/download/...` URL pattern
doesn't reach it. Add an optional `metadata_url:` field on a `repo:`
entry that overrides where the receipt is fetched from. Also support
metadata-only entries (no `files:`) — they contribute to the build
receipt but aren't transform targets.

- ingests.yaml: new `kg_phenio` entry with metadata_url pointing at
  kg-hub.berkeleybop.io and no files.
- ingest_utils.py: `get_release_metadata_sources()` returns (repo, url)
  tuples honoring the override; `is_metadata_only()` helper for
  filtering.
- cli_utils.py: `download_release_metadata()` takes the (repo, url)
  list. `transform_one` raises a clear error on metadata-only entries;
  `transform_all` filters them out of its iteration.
- Tests updated; live `ingest download-release-metadata` fetches
  23/24 (kg-phenio's first release pending PR Knowledge-Graph-Hub/kg-phenio#195).
@kevinschaper
kevinschaper marked this pull request as draft May 6, 2026 23:01
kg-phenio now writes output/release-metadata.yaml describing the upstream
sources consumed (phenio + the two upheno SSSOM mappings) and the
artifacts produced (merged-kg.tar.gz). The receipt nests phenio's own
per-ontology versions — read from phenio's phenio-upstream-versions.tsv
release asset — as a Release under the phenio entry, which itself sits
under kg-phenio.

When phenio's manifest is published, an HP/MONDO/CHEBI/etc. version
appearing in monarch-kg's aggregated build receipt traces back through
kg-phenio → phenio → infores:<ontology> for full provenance.

- versions.py (top-level, outside kg_phenio/ to avoid pulling kgx into
  the metadata script): get_source_versions() builds the nested phenio
  Release plus the two upheno peer leaves.
- scripts/write_metadata.py: standard kozahub-metadata-schema wrapper.
- pyproject.toml: kozahub-metadata-schema git dep.
- Jenkinsfile: new Metadata stage between Merge and Publish; the
  release-metadata.yaml is included in the s3 sync alongside the
  KG tarball.

For local development against an unreleased phenio checkout, set
PHENIO_MANIFEST_PATH=../phenio/src/ontology/phenio-upstream-versions.tsv
- New `metadata` click command that wraps scripts/write_metadata.py so the
  release-metadata.yaml gets emitted as part of a local `python run.py`
  workflow, not just under Jenkins. Matches request to make everything
  reproducible without Jenkins.
- Bump kozahub-metadata-schema dep from `branch = "main"` to
  `branch = "collapse-to-single-release-class"` so the writer emits the
  new single-Release-class shape (id/version at top level instead of
  source/source_version). Will revert to main once that PR merges.

Verified locally end-to-end: with PHENIO_MANIFEST_PATH pointing at a
locally-built phenio-upstream-versions.tsv, the writer emits the new
shape with phenio nested as a Release whose own sources are the 19
upstream ontology Releases.
@kevinschaper
kevinschaper marked this pull request as ready for review May 7, 2026 15:12
@kevinschaper
kevinschaper merged commit 976c781 into master May 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant