Repository navigation
Move datasets to Zenodo with BiocFileCache caching - #10
Merged
Merged
Conversation
Replace the ~380 MB of data/*.rda with on-demand download from Zenodo, cached in a package-specific BiocFileCache and verified against md5 checksums in inst/extdata/zenodo-manifest.csv (placeholder URLs until the Zenodo record is published): - New exported getter curatedOvarianData(): no args lists datasets, one name returns an ExpressionSet, several return a named list; test=TRUE loads small offline subsets from inst/extdata/testdata/. - data(X) stays working via data/*.R delayedAssign stubs that resolve through the getter (deprecation message once per session); during this package's own R CMD check they load the offline subsets, so checking needs no network. datalist retained so installation does not execute the stubs; BuildResaveData: no so R CMD build does not. - createEsetList.R loads through the getter; new test.mode option in the patientselection configs; fixed which() on an empty eset list. - RUnit scaffolding replaced with testthat; vignette updated to run on the offline subsets and document caching; replaced the dead DFCI URL with the GitHub URL and added BugReports; removed the committed build/ PDF. - inst/scripts/: make-data.R, make-test-data.R (provenance; Zenodo upload preparation) and data-integrity-report.Rmd (verification). Closes #9 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same fixes as curatedCRCData: make-data.R enumerates from the upload directory when data/ holds only stubs, and the integrity report no longer removes its comparison worktree prematurely (top-level on.exit in a knitr chunk fires at the end of the expression, not the chunk). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All 30 uploaded files verified: manifest md5s and sizes match the Zenodo API, every object is all.equal to its pre-refactor data/*.rda (inst/scripts/data-integrity-report.Rmd), and the full test suite passes including a live download. R CMD check Status: OK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The external-data migration and reverse-dependency compatibility need human verification, alongside resolution of the retained issues.
Review effort: Balanced
Findings: 1
Open (2)
What changed in this PR
Moves datasets to Zenodo to reduce package size for R-universe while preserving legacy data() access.
Changes:
- Adds on-demand downloads with checksums and BiocFileCache caching.
- Supports offline subsets for examples, tests, and vignettes.
- Updates documentation, dataset selection, and migration tooling.
| File | Description |
|---|---|
vignettes/curatedOvarianData.Rmd |
Documents caching and uses offline examples. |
tests/testthat/test-getData.R |
Tests manifests, getters, and caching. |
tests/testthat.R |
Adds testthat entry point. |
tests/runTests.R |
Removes RUnit entry point. |
R/getData.R |
Implements downloads, caching, and legacy loading. |
NEWS.md |
Records migration changes. |
NAMESPACE |
Exports getter and imports dependencies. |
man/curatedOvarianData.Rd |
Documents the getter. |
man/curatedOvarianData-package.Rd |
Updates package documentation. |
inst/unitTests/test_counts.R |
Removes old dataset-count checks. |
inst/unitTests/test_checkCurated.R |
Removes old metadata checks. |
inst/scripts/make-test-data.R |
Generates offline fixtures. |
inst/scripts/make-data.R |
Prepares uploads, manifests, and stubs. |
inst/scripts/data-integrity-report.Rmd |
Adds migration integrity checks. |
inst/extdata/zenodo-manifest.csv |
Records dataset URLs and checksums. |
inst/extdata/patientselection.config |
Adds offline test-mode option. |
inst/extdata/patientselection_all.config |
Adds offline test-mode option. |
inst/extdata/createEsetList.R |
Loads datasets through getters. |
external_data_store.txt |
Removes external-data directory listing. |
DESCRIPTION |
Updates dependencies and build settings. |
data/TCGA.RNASeqV2_eset.R |
Adds legacy loading stub. |
data/TCGA.mirna.8x15kv2_eset.R |
Adds legacy loading stub. |
data/TCGA_eset.R |
Adds legacy loading stub. |
data/PMID19318476_eset.R |
Adds legacy loading stub. |
data/PMID17290060_eset.R |
Adds legacy loading stub. |
data/PMID15897565_eset.R |
Adds legacy loading stub. |
data/GSE9891_eset.R |
Adds legacy loading stub. |
data/GSE8842_eset.R |
Adds legacy loading stub. |
data/GSE6822_eset.R |
Adds legacy loading stub. |
data/GSE6008_eset.R |
Adds legacy loading stub. |
data/GSE51088_eset.R |
Adds legacy loading stub. |
data/GSE49997_eset.R |
Adds legacy loading stub. |
data/GSE44104_eset.R |
Adds legacy loading stub. |
data/GSE32063_eset.R |
Adds legacy loading stub. |
data/GSE32062.GPL6480_eset.R |
Adds legacy loading stub. |
data/GSE30161_eset.R |
Adds legacy loading stub. |
data/GSE30009_eset.R |
Adds legacy loading stub. |
data/GSE26712_eset.R |
Adds legacy loading stub. |
data/GSE26193_eset.R |
Adds legacy loading stub. |
data/GSE2109_eset.R |
Adds legacy loading stub. |
data/GSE20565_eset.R |
Adds legacy loading stub. |
data/GSE19829.GPL8300_eset.R |
Adds legacy loading stub. |
data/GSE19829.GPL570_eset.R |
Adds legacy loading stub. |
data/GSE18520_eset.R |
Adds legacy loading stub. |
data/GSE17260_eset.R |
Adds legacy loading stub. |
data/GSE14764_eset.R |
Adds legacy loading stub. |
data/GSE13876_eset.R |
Adds legacy loading stub. |
data/GSE12470_eset.R |
Adds legacy loading stub. |
data/GSE12418_eset.R |
Adds legacy loading stub. |
data/E.MTAB.386_eset.R |
Adds legacy loading stub. |
data/datalist |
Reorders dataset index. |
.Rbuildignore |
Excludes website and citation metadata. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+81
to
+86
| unlink(tools::R_user_dir(pkg, "cache"), recursive = TRUE) | ||
| eset <- getter(smallest) # downloads | ||
| stopifnot(is(eset, "ExpressionSet")) | ||
| eset2 <- getter(smallest) # must hit the cache silently | ||
| stopifnot(identical(dim(eset), dim(eset2))) | ||
| dim(eset) |
Comment on lines
+73
to
+74
| expect_silent(suppressMessages( | ||
| eset2 <- curatedOvarianData("GSE30009_eset"))) |
lwaldron
added a commit
that referenced
this pull request
Oct 3, 2026
- Verify cached files by size on every load (cheap truncation check); evict and re-download on mismatch. Full md5 still runs after each fresh download. - Stub test now forces the delayed binding under the check guard and asserts the class, instead of exists(), which never forces. - Drop suppressMessages() inside expect_silent() so a regression that re-downloads on a cache hit fails the test. - Integrity report smoke test uses a throwaway R_USER_CACHE_DIR instead of deleting the user's real package cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
Copilot's review addressed in 9530aca (pushed to devel as 1.51.3): the integrity report now uses a throwaway R_USER_CACHE_DIR instead of deleting the user's real package cache, and suppressMessages() no longer defeats expect_silent() in the cache-hit test. Also applied the same fixes as waldronlab/curatedCRCData#5 (size check on cached files, stub test forces the binding), since the two packages deliberately share this code. 🤖 Drafted with Claude Code |
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the ~380 MB of
data/*.rdawith on-demand download from Zenodo record 23125873, cached per-file in a package-specific BiocFileCache and verified against md5 checksums ininst/extdata/zenodo-manifest.csv. Same design as waldronlab/curatedCRCData#5:curatedOvarianData(): no args lists datasets, one name returns an ExpressionSet, several return a named list;test=TRUEloads small offline subsets bundled ininst/extdata/testdata/.data(X)keeps working viadata/*.RdelayedAssign stubs that delegate to the getter with a once-per-session deprecation message; during this package's own check the stubs resolve to the offline subsets, so no network is needed.datalistis retained so installation doesn't execute the stubs, andBuildResaveData: nokeepsR CMD buildfrom doing so.createEsetList.Rloads through the getter (newtest.modeconfig option); vignette runs end-to-end on the offline subsets, including the CXCL12 survival meta-analysis and ComBat examples, and documents caching.build/PDF removed.inst/scripts/data-integrity-report.Rmdconfirms all 30 Zenodo objects areall.equalto the pre-refactordata/*.rdawith md5s matching the Zenodo API.The two Suggests-level dependents (doppelgangR, TOP) keep working through the
data()stubs.Closes #9
🤖 Generated with Claude Code