Repository navigation
R access to new cmgd releases via cmgdr (ExperimentHub is being retired) #291
Description
Give R users a path to the new releases. The curatedMetagenomicData Bioconductor package has always reached its users through ExperimentHub, which is being retired, so cmgdr reading our releases directly is the way forward.
What exists: seandavi/cmgdr
cmgdr is a low-level R client that reads the published layout directly: the JSON indexes, manifests, Parquet, per-study files and the frozen DuckLake catalog.
- It imports only
curl,digestandjsonlite. duckdbandTreeSummarizedExperimentare optional.- Its download cache is verified by checksum.
That is the "read our releases directly" option, already built.
Checked against the first MVP release (cmgd_mpa4.2-2.3.0 / 2026-10-16), using pkgload::load_all on cmgdr 409ebad:
cmgd_release(),cmgd_tables()(all 12 tables, with each table's licence: Olm-derived tables show CC-BY-4.0) andcmgd_studies()all work.cmgd_species_tse(rel, "BaiX_2021")returns 1245 species × 24 samples.cmgd_connect()attaches the catalog. SQL onsample_metadata_harmonizedworks (health_condition 125, country 125, sex 124, age_years 89, antibiotics 70).
Gaps for the new metadata
- Sample metadata in
colData.cmgd_species_tse()usesqc.tsvascolData. The metadata (samples,sample_metadata_harmonized/_cmd/_olm) is reachable only through SQL. Add acmgd_sample_metadata(rel, source = c("harmonized","cmd","olm"))that returns one row per sample, and an option to join it into the TSE'scolData. - Licence and attribution. When Olm-derived columns are returned, show the CC-BY-4.0 attribution, for example as a message or in
metadata(tse). - Provenance. Expose
provenance.json(pipeline, databases, metadata source releases) on the release object. - Search by run accession. The samples table has
run_accessionsas text joined with;. A helper to find a sample from an SRR would help. - Add a fixture release with metadata tables to cmgdr's tests (the fixtures already come from nextflow_telemetry).
Distribution
ExperimentHub is going away, so it is not an option. What remains:
- Where cmgdr lives. It could go on CRAN, on Bioconductor as a software package (check Bioconductor's policy on packages that read from outside web resources), or stay on GitHub with r-universe.
- Relationship to the existing curatedMetagenomicData package. The existing package depends on ExperimentHub, so it needs a new backend either way. Should it be rebuilt on cmgdr, keeping
returnSamples()andsampleMetadatafor existing users? Who maintains it? (@lwaldron?) MetaPhlAn 4.2 / vJan26 also uses a different taxonomy from the current package data. - Citable versions for R users. Without ExperimentHub's snapshots, cite a milestone release by its DOI (Zenodo deposit strategy at corpus scale (milestones, what to include) #292), and have cmgdr pin a release by date.
Suggested first step
Add cmgd_sample_metadata() and a colData join to cmgdr, using the release published now. That is small, and it makes the new metadata usable from R immediately.
🤖 Generated with Claude Code
Activity
[-]R / Bioconductor access to new cmgd releases (ExperimentHub or direct)[/-][+]R access to new cmgd releases: extend cmgdr for metadata; decide on ExperimentHub[/+][-]R access to new cmgd releases: extend cmgdr for metadata; decide on ExperimentHub[/-][+]R access to new cmgd releases via cmgdr (ExperimentHub is being retired)[/+]seandavi commentedon Oct 16, 2026
Position (tentative, not yet discussed with the team):
What that means for the checklist above:
cmgd_sample_metadata()still fits cmgdr. It returns the published metadata tables (harmonized, cmd or olm) as plain data frames, with the licence and attribution attached.colDatajoin, the API compatibility questions (returnSamples(),sampleMetadata) and the handling of the taxonomy change belong to the user-facing package. cmgdr's existingcmgd_species_tse()stays as a convenience.To raise with the cMD team (@lwaldron): who builds and owns the user-facing package, and what it needs from cmgdr.
seandavi commentedon Oct 16, 2026
cmgd_sample_metadata()landed in cmgdr (seandavi/cmgdr#3, closes seandavi/cmgdr#2). All 9 acceptance tests pass, and CI is green on macOS, Ubuntu and Windows. Verified live:cmgd_sample_metadata(rel, study = "BaiX_2021", wide = TRUE)returns 24 × 10, with licence CC-BY-4.0 and the Olm DOI in the attribution message. One thing it shows: numbers come out as "44.0", and cMD has 53/53.0 duplicates, which wide output;-joins. That is a metadata-store normalization issue, tracked in #288.lwaldron commentedon Oct 16, 2026
Makes sense to me. I'll see how observing cmgdr affects the cMD refactor.
lwaldron commentedon Oct 16, 2026
Oh and I see the earlier question now, yes I would aim to preserve the cMD API, especially returnSamples() and sampleMetadata(). I have wrote permission to cMD on Bioconductor and can act as maintainer for now.
seandavi commentedon Oct 16, 2026
Scope settled (cmgdr stays a low-level client; a user-facing package builds on it) and
cmgd_sample_metadata()shipped in seandavi/cmgdr#3. The remaining work (fixtures, provenance, run-accession lookup, distribution, ownership of the user-facing package) is tracked in seandavi/cmgdr#4. #292 and #288 stay here.