Repository navigation
Move datasets to Zenodo with BiocFileCache caching - #5
Merged
Merged
Conversation
Replace the ~400 MB of data/*.rda with on-demand download from Zenodo, cached in a package-specific BiocFileCache and verified against md5 checksums in inst/extdata/zenodo-manifest.csv (placeholder URLs until the Zenodo record is published): - New exported getter curatedCRCData(): no args lists datasets, one name returns an ExpressionSet, several return a named list; test=TRUE loads small offline subsets from inst/extdata/testdata/. - data(X) stays working via data/*.R delayedAssign stubs that resolve through the getter (deprecation message once per session); during this package's own R CMD check they load the offline subsets, so checking needs no network. datalist retained so installation does not execute the stubs; BuildResaveData: no so R CMD build does not. - createEsetList.R loads through the getter; new test.mode option in the patientselection configs; fixed which() on an empty eset list. - RUnit scaffolding replaced with testthat; vignette updated to run on the offline subsets and document caching; CI restored to R CMD check + BiocCheck + pkgdown (reverted to pkgdown-only in 414e753 only because checking 400 MB exhausted runner memory). - inst/scripts/: make-data.R, make-test-data.R (provenance; Zenodo upload preparation) and data-integrity-report.Rmd (verification). Closes #4 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All 34 uploaded files verified: manifest md5s and sizes match the Zenodo API, every object is all.equal to its pre-refactor data/*.rda (inst/scripts/data-integrity-report.Rmd, which also fixes premature worktree removal from a top-level on.exit in a knitr chunk), and the full test suite passes including a live download. make-data.R now enumerates from the upload directory when re-run to finalize URLs after data/ holds only stubs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Cached-file verification, compatibility tests, cache assertions, and the pkgdown dependency need correction.
Review effort: Balanced
Findings: 2
Open (3)
What changed in this PR
Moves large datasets to Zenodo with cached, checksum-aware retrieval while retaining legacy data() compatibility.
Changes:
- Adds
curatedCRCData()and BiocFileCache-backed downloads. - Adds offline fixtures, testthat coverage, migration scripts, and documentation.
- Restores reusable CI and pkgdown configuration.
| File | Description |
|---|---|
R/getData.R |
Implements retrieval and caching. |
DESCRIPTION |
Updates dependencies and metadata. |
NAMESPACE |
Exports the getter. |
NEWS.md |
Documents migration. |
vignettes/curatedCRCData_vignette.Rmd |
Documents cached access. |
man/curatedCRCData.Rd |
Documents the getter. |
man/curatedCRCData-package.Rd |
Updates package documentation. |
tests/testthat.R |
Adds testthat runner. |
tests/testthat/test-getData.R |
Tests retrieval and manifests. |
inst/unitTests/test_counts.R |
Removes legacy RUnit test. |
inst/unitTests/test_checkCurated.R |
Removes legacy validation test. |
inst/unitTests/test_checkCurated.Rout |
Removes obsolete output. |
inst/scripts/make-data.R |
Generates uploads and stubs. |
inst/scripts/make-test-data.R |
Generates offline fixtures. |
inst/scripts/data-integrity-report.Rmd |
Adds integrity verification. |
inst/extdata/zenodo-manifest.csv |
Registers Zenodo datasets. |
inst/extdata/patientselection.config |
Adds offline test mode. |
inst/extdata/patientselection_all.config |
Adds offline test mode. |
inst/extdata/createEsetList.R |
Loads through the getter. |
external_data_store.txt |
Removes obsolete storage config. |
data/datalist |
Updates dataset ordering. |
data/GSE11237_eset.R |
Adds delayed data stub. |
data/GSE12225.GPL3676_eset.R |
Adds delayed data stub. |
data/GSE12945_eset.R |
Adds delayed data stub. |
data/GSE13067_eset.R |
Adds delayed data stub. |
data/GSE13294_eset.R |
Adds delayed data stub. |
data/GSE14095_eset.R |
Adds delayed data stub. |
data/GSE14333_eset.R |
Adds delayed data stub. |
data/GSE16125.GPL5175_eset.R |
Adds delayed data stub. |
data/GSE17536_eset.R |
Adds delayed data stub. |
data/GSE17537_eset.R |
Adds delayed data stub. |
data/GSE17538.GPL570_eset.R |
Adds delayed data stub. |
data/GSE18105_eset.R |
Adds delayed data stub. |
data/GSE2109_eset.R |
Adds delayed data stub. |
data/GSE21510_eset.R |
Adds delayed data stub. |
data/GSE21815_eset.R |
Adds delayed data stub. |
data/GSE24549.GPL5175_eset.R |
Adds delayed data stub. |
data/GSE24550.GPL5175_eset.R |
Adds delayed data stub. |
data/GSE2630_eset.R |
Adds delayed data stub. |
data/GSE26682.GPL570_eset.R |
Adds delayed data stub. |
data/GSE26682.GPL96_eset.R |
Adds delayed data stub. |
data/GSE26906_eset.R |
Adds delayed data stub. |
data/GSE27544_eset.R |
Adds delayed data stub. |
data/GSE28702_eset.R |
Adds delayed data stub. |
data/GSE3294_eset.R |
Adds delayed data stub. |
data/GSE33113_eset.R |
Adds delayed data stub. |
data/GSE39582_eset.R |
Adds delayed data stub. |
data/GSE3964_eset.R |
Adds delayed data stub. |
data/GSE4045_eset.R |
Adds delayed data stub. |
data/GSE4526_eset.R |
Adds delayed data stub. |
data/GSE45270_eset.R |
Adds delayed data stub. |
data/TCGA.COAD_eset.R |
Adds delayed data stub. |
data/TCGA.READ_eset.R |
Adds delayed data stub. |
data/TCGA.RNASeqV2_eset.R |
Adds delayed data stub. |
data/TCGA.RNASeqV2.READ_eset.R |
Adds delayed data stub. |
.Rbuildignore |
Excludes site metadata. |
.github/workflows/ci.yml |
Adopts reusable CI workflow. |
_pkgdown.yml |
Configures package site. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| Suggests: | ||
| survival, | ||
| RUnit, | ||
| testthat (>= 3.0.0), |
Comment on lines
+40
to
+55
| if (fresh) { | ||
| md5 <- unname(tools::md5sum(path)) | ||
| if (!identical(md5, meta$md5[i])) { | ||
| res <- BiocFileCache::bfcquery(bfc, meta$url[i], field = "fpath", | ||
| exact = TRUE) | ||
| if (nrow(res)) | ||
| BiocFileCache::bfcremove(bfc, res$rid) | ||
| if (retry) { | ||
| message("Checksum mismatch; retrying the download once...") | ||
| return(.getDataFile(name, retry = FALSE)) | ||
| } | ||
| stop("md5 checksum mismatch for '", name, "' downloaded from ", | ||
| meta$url[i], | ||
| "; the corrupt copy was removed from the cache.") | ||
| } | ||
| } |
Comment on lines
+62
to
+63
| data("GSE3294_eset", package = "curatedCRCData", envir = e) | ||
| expect_true(exists("GSE3294_eset", envir = e)) |
lwaldron
added a commit
that referenced
this pull request
Oct 3, 2026
- Verify cached files by size on every load (cheap truncation check); evict and re-download on mismatch. Full md5 still runs after each fresh download. - Stub test now forces the delayed binding under the check guard and asserts the class, instead of exists(), which never forces. - Drop suppressMessages() inside expect_silent() so a regression that re-downloads on a cache hit fails the test. - Integrity report smoke test uses a throwaway R_USER_CACHE_DIR instead of deleting the user's real package cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
Copilot's review addressed in 3a019f3 (pushed to devel as 2.45.4): cached files are now size-checked on every load with evict-and-redownload on mismatch, the stub test forces the delayed binding under the check guard instead of using exists(), suppressMessages() no longer defeats expect_silent(), and the integrity report uses a throwaway cache instead of deleting the user's real one. The missing-pkgdown point is real but belongs in the shared workflow — fixed in waldronlab/.github#4 rather than adding pkgdown to Suggests. 🤖 Drafted with Claude Code |
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the ~400 MB of
data/*.rdawith on-demand download from Zenodo record 23125734, cached per-file in a package-specific BiocFileCache and verified against md5 checksums ininst/extdata/zenodo-manifest.csv.curatedCRCData(): no args lists datasets, one name returns an ExpressionSet, several return a named list;test=TRUEloads small offline subsets bundled ininst/extdata/testdata/.data(X)keeps working viadata/*.RdelayedAssign stubs that delegate to the getter with a once-per-session deprecation message; during this package's own check the stubs resolve to the offline subsets, so no network is needed.datalistis retained so installation doesn't execute the stubs, andBuildResaveData: nokeepsR CMD buildfrom doing so.createEsetList.Rloads through the getter (newtest.modeconfig option); vignette runs end-to-end on the offline subsets and documents caching.inst/scripts/data-integrity-report.Rmdconfirms all 34 Zenodo objects areall.equalto the pre-refactordata/*.rdawith md5s matching the Zenodo API.Closes #4
🤖 Generated with Claude Code