Skip to content

Epic: sample onboarding, multi-source metadata, and publication (MVP) #266

Open

Description

@seandavi

Plan: docs/plans/onboarding-and-publication.md (PR #273).

MVP. About 470 samples from 4 studies go through the whole path:

  1. registered
  2. run under cmgd_mpa4.2 2.3.0
  3. staged to the lake as Parquet
  4. published with cMD and Olm metadata to cmgd-public, Zenodo and a Hugging Face Dataset repo
  5. navigable by a reviewer agent from llms.txt in 4 fetches or fewer

Cohort:

Study Samples Source
NishiwakiH_2024 167 #188
JoS_2022 156 #186
PernigoniN_2021 74 Olm × cMD overlap
HoylesL_2018 73 Olm × cMD overlap

Estimated compute: about 11.3k core-h at the 2.3.x rehearsal rate of 24 core-h per sample.

Decisions to confirm before Phase 1 (plan section "Decisions to confirm"):

  • D1: which catalog holds the metadata (cmgd_lake or the shared lake's cmgd_meta schema)
  • D2: add-study write-through to v2
  • D3: Olm already contains cMD (external.cmd)
  • D4: per-table licence (Olm data is CC-BY-4.0)

Related: #193 #195 #196 #57 #55 #185–#190

Phases

Later, outside the MVP: #272

Activity

  1. seandavi commented on Oct 16, 2026

    @seandavi
    OwnerAuthor

    Decisions (2026-10-16, autonomous run while user is offline):

    • D1: metadata goes in the shared lake under schema cmgd_meta (lake.cmgd_meta.*), so a single snapshot covers both results and metadata. This keeps the user's choice of per-source tables in the cmgd lake, just without the retired cmgd_lake catalog.
    • D2: fix the add-study trigger and add write-through to v2 as the first step of Phase 3.
    • D3: ingest Olm's long determinations table and exclude external.cmd rows when treating Olm as an independent source.
    • D4: license Olm-derived tables CC-BY-4.0 per table and don't redistribute evidence_quote.
    • First release: built from the 125 cmgd_mpa4.2 samples already completed, plus metadata. The 470-sample cohort runs on Alpine and Anvil and lands in the next release. Bridges-2 stays off (max_concurrent_runs: 0).
    • Zenodo and Hugging Face: no public DOI is minted without the user's sign-off. The mirror code is tested against Zenodo sandbox and a private HF repo if credentials exist.

    Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

  2. seandavi commented on Oct 16, 2026

    @seandavi
    OwnerAuthor

    Discussion issues opened (no commitment yet): #290 AWS Open Data, #291 R / Bioconductor access, #292 Zenodo deposit strategy at corpus scale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions