You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Apply to AWS Open Data Sponsorship Program for cmgd releases #290
Apply to the AWS Open Data Sponsorship Program to host cmgd releases on S3, as a mirror of the canonical R2 copy.
Why
If accepted, AWS pays S3 storage and data transfer for 2 years, renewable. Compute isn't covered.
The data is listed in the Registry of Open Data on AWS, where people discover it.
People can query it in place from AWS compute.
R2 already has free egress, so the gain is discoverability and long-term hosting off our bill, not cost.
Size estimate, from the first release: about 0.6 MB per sample. That's roughly 40 GB at 60k samples and roughly 330 GB at Olm-corpus scale (about 535k samples).
Eligibility
Licences. The data must be usable by anyone for any purpose. CC0 (results, cMD metadata) and CC-BY-4.0 (Olm-derived tables) both qualify; licences that forbid commercial use do not.
Format. Open, cloud-optimized formats. Parquet and the frozen DuckLake catalog already meet this.
Documentation and help for users. The release site, llms.txt, the dataset card and the DuckDB examples cover most of it. We'd add a short tutorial.
Demonstrated interest. This is the weak point today, with 125 samples and no usage numbers yet.
Plan (not started)
Wait for the release that includes the 470-sample MVP cohort, and collect basic access numbers from R2 analytics.
If accepted: copy each release folder to S3 unchanged (aws s3 sync) and add it as a mirror target next to Zenodo and HF in nf-etl mirror. The catalog uses relative data paths, so it attaches from S3 unchanged.
Write the Registry of Open Data YAML entry.
Alternatives considered
Source Cooperative. S3-compatible and free for now, backed by Radiant Earth with AWS in-kind support. It has little track record with bioscience datasets. Keep it as the fallback if AWS declines.
Activity