Skip to content

Apply to AWS Open Data Sponsorship Program for cmgd releases #290

Open

Description

@seandavi

Apply to the AWS Open Data Sponsorship Program to host cmgd releases on S3, as a mirror of the canonical R2 copy.

Why

  • If accepted, AWS pays S3 storage and data transfer for 2 years, renewable. Compute isn't covered.
  • The data is listed in the Registry of Open Data on AWS, where people discover it.
  • People can query it in place from AWS compute.
  • R2 already has free egress, so the gain is discoverability and long-term hosting off our bill, not cost.
  • Size estimate, from the first release: about 0.6 MB per sample. That's roughly 40 GB at 60k samples and roughly 330 GB at Olm-corpus scale (about 535k samples).

Eligibility

  • Licences. The data must be usable by anyone for any purpose. CC0 (results, cMD metadata) and CC-BY-4.0 (Olm-derived tables) both qualify; licences that forbid commercial use do not.
  • Format. Open, cloud-optimized formats. Parquet and the frozen DuckLake catalog already meet this.
  • Documentation and help for users. The release site, llms.txt, the dataset card and the DuckDB examples cover most of it. We'd add a short tutorial.
  • Demonstrated interest. This is the weak point today, with 125 samples and no usage numbers yet.

Plan (not started)

  • Wait for the release that includes the 470-sample MVP cohort, and collect basic access numbers from R2 analytics.
  • Apply at https://application.opendata.aws. Review may run quarterly, so check the current cycle.
  • If accepted: copy each release folder to S3 unchanged (aws s3 sync) and add it as a mirror target next to Zenodo and HF in nf-etl mirror. The catalog uses relative data paths, so it attaches from S3 unchanged.
  • Write the Registry of Open Data YAML entry.

Alternatives considered

  • Source Cooperative. S3-compatible and free for now, backed by Radiant Earth with AWS in-kind support. It has little track record with bioscience datasets. Keep it as the fallback if AWS declines.

Refs: https://aws.amazon.com/opendata/open-data-sponsorship-program, https://aws.amazon.com/opendata/faqs/

Part of epic #266.

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    discussionOpen question; no decision or commitment yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions