Epic: sample onboarding, multi-source metadata, and publication (MVP) #266
Copy link
Copy link
Open
Description
Plan: docs/plans/onboarding-and-publication.md (PR #273).
MVP. About 470 samples from 4 studies go through the whole path:
- registered
- run under
cmgd_mpa4.2 2.3.0 - staged to the lake as Parquet
- published with cMD and Olm metadata to
cmgd-public, Zenodo and a Hugging Face Dataset repo - navigable by a reviewer agent from
llms.txtin 4 fetches or fewer
Cohort:
| Study | Samples | Source |
|---|---|---|
| NishiwakiH_2024 | 167 | #188 |
| JoS_2022 | 156 | #186 |
| PernigoniN_2021 | 74 | Olm × cMD overlap |
| HoylesL_2018 | 73 | Olm × cMD overlap |
Estimated compute: about 11.3k core-h at the 2.3.x rehearsal rate of 24 core-h per sample.
Decisions to confirm before Phase 1 (plan section "Decisions to confirm"):
- D1: which catalog holds the metadata (
cmgd_lakeor the shared lake'scmgd_metaschema) - D2: add-study write-through to v2
- D3: Olm already contains cMD (
external.cmd) - D4: per-table licence (Olm data is CC-BY-4.0)
Related: #193 #195 #196 #57 #55 #185–#190
Phases
- Onboarding MVP Phase 1: metadata store and Olm ingest #267 Phase 1: metadata store and Olm ingest
- Onboarding MVP Phase 2: cMD source, xref, harmonized view, cMD vs Olm comparison #268 Phase 2: cMD source, xref, harmonized view, cMD vs Olm comparison
- Onboarding MVP Phase 3: add-study path on v2 with cost estimates; run the 470-sample cohort #269 Phase 3: add-study path on v2 with cost estimates; run the 470-sample cohort
- Onboarding MVP Phase 4: stage and release with metadata and provenance #270 Phase 4: stage and release with metadata and provenance
- Onboarding MVP Phase 5: Zenodo + Hugging Face mirrors and reviewer-agent acceptance #271 Phase 5: Zenodo + HF mirrors and reviewer-agent acceptance
Later, outside the MVP: #272
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
seandavi commentedon Oct 16, 2026
Decisions (2026-10-16, autonomous run while user is offline):
cmgd_meta(lake.cmgd_meta.*), so a single snapshot covers both results and metadata. This keeps the user's choice of per-source tables in the cmgd lake, just without the retiredcmgd_lakecatalog.external.cmdrows when treating Olm as an independent source.evidence_quote.cmgd_mpa4.2samples already completed, plus metadata. The 470-sample cohort runs on Alpine and Anvil and lands in the next release. Bridges-2 stays off (max_concurrent_runs: 0).Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
seandavi commentedon Oct 16, 2026
Discussion issues opened (no commitment yet): #290 AWS Open Data, #291 R / Bioconductor access, #292 Zenodo deposit strategy at corpus scale.