Move datasets to Zenodo, downloaded on demand and cached with BiocFileCache by lwaldron · Pull Request #3 · lima1/curatedBladderData

lwaldron · 2026-10-06T15:55:45Z

Hi Markus, as offered in #2. The 12 ExpressionSets now live on Zenodo and are downloaded on first use instead of shipping inside the package. The source tarball goes from about 75 MB to 1.4 MB.

Zenodo record: https://zenodo.org/records/23192472 (deposited by me, as you suggested). Tell me if you want the creator list or license changed and I'll edit the record.

What users see

curatedBladderData()                      # list the 12 datasets
eset <- curatedBladderData("GSE89_eset")  # downloads once, then cached
esets <- curatedBladderData(c("GSE89_eset", "GSE37317_eset"))   # named list

data(GSE89_eset) still works. Each data/*.rda is replaced by a small data/*.R stub that fetches through the same cache and prints a deprecation note once per session, so existing scripts and the two reverse dependencies keep running. The download is deferred until the object is actually used, not at data() time.

The cache is package-specific, at tools::R_user_dir("curatedBladderData", "cache"), and every download is checked against an md5 recorded in inst/extdata/zenodo-manifest.csv. That CSV is the single place URLs live, so a future Zenodo version means editing one file.

Why it checks without network access

R CMD check loads every file in data/, which would otherwise mean downloading everything on every check. The stubs detect the package's own check (via _R_CHECK_PACKAGE_NAME_) and load small offline subsets bundled in inst/extdata/testdata/ instead, 1.4 MB for all 12. Reverse-dependency checks, where that variable holds a different package name, still get the real data.

Two details that are easy to undo by accident, both commented in place:

data/datalist must stay. Without it, R CMD INSTALL runs data() on every file to build its index, which would download everything at install time.
BuildResaveData: no in DESCRIPTION. Without it, R CMD build executes the stubs and re-saves them as .rda, putting the data straight back into the tarball.

Verification

All 12 objects on Zenodo are all.equal to the .rda files they replace, compared against the pre-refactor commit (inst/scripts/data-integrity-report.Rmd does this and asserts it).
Manifest md5s and sizes match what the Zenodo API reports for all 12 files.
R CMD check passes. New testthat tests cover the manifest, the getter, the stubs and an opt-in live download.
The vignette builds on the offline subsets, so it needs no network.

Incidental fixes, happy to drop any of these

These were pre-existing and unrelated to Zenodo. I fixed them because they were in files I was already touching, but say the word and I'll pull them out:

biocViews listed OvarianCancerData. Removed, and GEO added. There is no BladderCancerData term in the vocabulary.
man/curatedBladderData-package.Rd described ovarian cancer, carried the curatedOvarianData author list, and gave a maintainer that disagreed with DESCRIPTION. Rewritten to match DESCRIPTION, so it now credits you alone. Worth your eye, since it changes an author list.
Depends: affy was unused. affy appeared only in one eval=FALSE vignette chunk, where exprs() and pData() actually come from Biobase. Swapped to Biobase.
createEsetList.R errored when the filters excluded every dataset, because which() was called on an empty list. Same bug existed in curatedCRCData and curatedOvarianData.

Version set to 1.49.1. Note your GitHub copy said 1.19.2 while Bioconductor devel is at 1.49.0: the release process bumps versions on the Bioconductor git without touching GitHub. I confirmed the two are otherwise byte-identical before starting.

The same pattern is now in curatedCRCData and curatedOvarianData, if you want to compare: waldronlab/curatedCRCData#5 and waldronlab/curatedOvarianData#10.

🤖 Generated with Claude Code

Replaces the 75 MB of data/*.rda with on-demand download from Zenodo, cached per-file in a package-specific BiocFileCache and verified against md5 checksums in inst/extdata/zenodo-manifest.csv. URLs are placeholders until the Zenodo record is published. - New exported getter curatedBladderData(): no args lists datasets, one name returns an ExpressionSet, several return a named list; test=TRUE loads small offline subsets from inst/extdata/testdata/. - data(X) keeps working via data/*.R delayedAssign stubs that delegate to the getter with a once-per-session deprecation message. During this package's own R CMD check the stubs resolve to the offline subsets, so checking needs no network. data/datalist is retained so installation does not execute the stubs, and BuildResaveData: no stops R CMD build from re-saving them as .rda. - createEsetList.R loads through the getter, with a new test.mode option in the patientselection config. - Vignette documents the caching and runs on the offline subsets. - Adds testthat tests and inst/scripts/ for regenerating everything. Also fixes several pre-existing issues noticed along the way: biocViews claimed OvarianCancerData, the package help page described ovarian cancer with the wrong author list and maintainer, affy was depended on but unused, and createEsetList.R errored when filters excluded every dataset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

All 12 uploaded files verified: the manifest md5s and sizes match what the Zenodo API reports, every object is all.equal to the .rda it replaces (inst/scripts/data-integrity-report.Rmd), and the full test suite passes including a live download and a cache hit. R CMD check is clean. The source tarball drops from about 75 MB to 1.4 MB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

Manifest sizes are operationally significant but are not verified by the integrity report, and two documentation inaccuracies remain.

Review effort: Balanced
Findings: 1 · 2

Open (3)

Validate manifest size_bytes to prevent repeated cache eviction · New
Correct study count: 12 datasets represent 10 studies · New
Fix nonexistent curation template provenance path · New

What changed in this PR

Moves 12 bladder cancer datasets to Zenodo with verified, on-demand caching while preserving legacy data() compatibility.

Changes:

Adds the curatedBladderData() retrieval API and delayed compatibility stubs.
Adds offline fixtures, tests, manifests, and integrity tooling.
Updates package metadata, documentation, and vignette workflows.

File	Description
`R/getData.R`	Implements retrieval, caching, and validation.
`DESCRIPTION`	Updates dependencies and package metadata.
`NAMESPACE`	Exports the getter and imports dependencies.
`NEWS.md`	Documents the release changes.
`external_data_store.txt`	Removes obsolete external-data configuration.
`tests/testthat.R`	Enables testthat.
`tests/testthat/test-getData.R`	Tests manifests, retrieval, and stubs.
`vignettes/curatedBladderData_vignette.rnw`	Documents the new data workflow.
`man/curatedBladderData.Rd`	Documents the getter API.
`man/curatedBladderData-package.Rd`	Corrects package-level documentation.
`inst/extdata/zenodo-manifest.csv`	Records dataset URLs and checksums.
`inst/extdata/patientselection_all.config`	Adds offline test mode.
`inst/extdata/createEsetList.R`	Loads datasets through the getter.
`inst/scripts/make-data.R`	Generates uploads, manifests, and stubs.
`inst/scripts/make-test-data.R`	Generates offline fixtures.
`inst/scripts/data-integrity-report.Rmd`	Verifies migrated data.
`data/GSE13507_eset.R`	Adds a delayed legacy-data stub.
`data/GSE1827_eset.R`	Adds a delayed legacy-data stub.
`data/GSE19915.GPL3883_eset.R`	Adds a delayed legacy-data stub.
`data/GSE19915.GPL5186_eset.R`	Adds a delayed legacy-data stub.
`data/GSE31189_eset.R`	Adds a delayed legacy-data stub.
`data/GSE31684_eset.R`	Adds a delayed legacy-data stub.
`data/GSE32894_eset.R`	Adds a delayed legacy-data stub.
`data/GSE37317_eset.R`	Adds a delayed legacy-data stub.
`data/GSE5287_eset.R`	Adds a delayed legacy-data stub.
`data/GSE89_eset.R`	Adds a delayed legacy-data stub.
`data/PMID17099711.GPL8300_eset.R`	Adds a delayed legacy-data stub.
`data/PMID17099711.GPL91_eset.R`	Adds a delayed legacy-data stub.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+local_md5 <- tools::md5sum(file.path(params$upload_dir, manifest$filename))
+stopifnot(identical(unname(local_md5), manifest$md5))
+
+record <- unique(sub(".*records/([0-9A-Za-z_]+)/.*", "\\1", manifest$url))
+stopifnot(length(record) == 1L)
+api <- jsonlite::fromJSON(paste0("https://zenodo.org/api/records/", record))
+zen <- data.frame(filename = api$files$key,
+                  md5 = sub("^md5:", "", api$files$checksum))
+merged <- merge(manifest[, c("filename", "md5")], zen, by = "filename",
+                suffixes = c(".manifest", ".zenodo"))
+stopifnot(nrow(merged) == nrow(manifest),
+          identical(merged$md5.manifest, merged$md5.zenodo))


-Description: The curatedBladderData package provides relevant functions and data for gene expression analysis in patients with bladder cancer.
-Depends: R (>= 2.10.0), affy
-Suggests: BiocStyle, survival, xtable, sva, genefilter, logging
+Description: Manually curated bladder cancer ExpressionSet objects from 12


+## The datasets themselves were produced by the curatedBladderData curation
+## pipeline (see the package vignette and inst/extdata/the curation template);
+## this script re-serializes the .rda files that shipped inside the package


lwaldron · 2026-10-06T16:06:53Z

One more thing I noticed but deliberately did not touch, since I don't know the right answer: inst/CITATION points at the curatedOvarianData paper. It tells anyone citing curatedBladderData to cite "curatedOvarianData: Clinically Annotated Data for the Ovarian Cancer Transcriptome" (Database 2013, doi:10.1093/database/bat013), with that paper's author list and abstract.

It looks like the same copy-paste as the biocViews and help-page issues, but a citation felt like the wrong thing for me to guess at, so I left it alone. Worth pointing at whatever the correct bladder reference is, or at the package itself if there isn't a paper.

Same caveat applies to the author list on the help page, which is why that change is in the droppable section: I reduced it to match DESCRIPTION, but if the original list reflects real contributors then DESCRIPTION is the thing that's incomplete, and that's your call rather than mine.

🤖 Drafted with Claude Code

- The integrity report verified md5 but never size_bytes, although R/getData.R compares size_bytes against the cached file on every cache hit. A wrong size in the manifest would therefore evict and re-download a perfectly good file on every call, silently and forever. The report now asserts size against both the local upload and the Zenodo API. - DESCRIPTION said "12 studies". It is 12 objects from 10 studies: GSE19915 and PMID17099711 each contributed data on two platforms. - make-data.R pointed at "inst/extdata/the curation template", a path that does not exist, left over from adapting the script from curatedCRCData. Removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

lwaldron · 2026-10-06T16:10:28Z

Pushed 8d84dc7 addressing all three Copilot comments. All three were correct:

The integrity report checked md5 but never size_bytes, even though R/getData.R compares size_bytes against the cached file on every cache hit. A wrong size in the manifest would have silently evicted and re-downloaded a perfectly good file on every single call. The report now asserts size against both the local upload and the Zenodo API. This was the valuable one.
DESCRIPTION said "12 studies". It is 12 objects from 10 studies, since GSE19915 and PMID17099711 each contributed data on two platforms. Corrected.
make-data.R referenced a path that does not exist, left over from adapting the script from curatedCRCData. Removed.

🤖 Drafted with Claude Code

R/getData.R compares size_bytes against the cached file on every cache hit, so a wrong size in the manifest would silently evict and re-download a valid file on every call, with no error. The integrity report checked md5 only, and so was blind to the one field that drives that behaviour. It now asserts size against both the local upload and the Zenodo API. The published manifest is correct: the strengthened report passes against Zenodo record 23125734 unchanged. This closes the gap for future regenerations. Found by Copilot review on lima1/curatedBladderData#3, which carries the same report. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

R/getData.R compares size_bytes against the cached file on every cache hit, so a wrong size in the manifest would silently evict and re-download a valid file on every call, with no error. The integrity report checked md5 only, and so was blind to the one field that drives that behaviour. It now asserts size against both the local upload and the Zenodo API. The published manifest is correct: the strengthened report passes against Zenodo record 23125873 unchanged. This closes the gap for future regenerations. Found by Copilot review on lima1/curatedBladderData#3, which carries the same report. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

lwaldron and others added 2 commits October 6, 2026 11:29

Copilot AI balanced review requested due to automatic review settings October 6, 2026 15:55

lwaldron mentioned this pull request Oct 6, 2026

Offer: move data to Zenodo with BiocFileCache caching, as just done for curatedCRCData and curatedOvarianData #2

Open

Copilot started reviewing on behalf of lwaldron October 6, 2026 15:56 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

lima1 merged commit d6d092c into lima1:master Oct 6, 2026

Navigation Menu

Repository navigation

Move datasets to Zenodo, downloaded on demand and cached with BiocFileCache - #3

Move datasets to Zenodo, downloaded on demand and cached with BiocFileCache#3
lima1 merged 3 commits into
lima1:masterfrom
lwaldron:zenodo-refactor

lwaldron commented Oct 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

lwaldron commented Oct 6, 2026 •

edited

Loading

Uh oh!

lwaldron commented Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

lwaldron commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What users see

Why it checks without network access

Verification

Incidental fixes, happy to drop any of these

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

lwaldron commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lwaldron commented Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lwaldron commented Oct 6, 2026 •

edited

Loading

lwaldron commented Oct 6, 2026 •

edited

Loading