From 15e4413b2dddfeefe14dccbc2682bba56d8919e8 Mon Sep 17 00:00:00 2001 From: chross22 <52218551+chross22@users.noreply.github.com> Date: Thu, 6 Aug 2026 17:56:53 -0400 Subject: [PATCH 1/2] Declare the R version the package needs R CMD build inferred a dependency on R >= 4.1.0 on every build and warned about it, because the package uses the |> pipe introduced there (R/accessEnvDat.R and R/matchData.R) without saying so in DESCRIPTION. 4.1.0 is the floor the syntax actually implies: the pipe and the \(x) function shorthand both arrive in that release, and the package uses the former in two files and the latter nowhere. %||% is defined in R/variables.R rather than taken from base, which is what keeps the floor below the 4.4.0 that base's version would require. Co-Authored-By: Claude Opus 5 --- DESCRIPTION | 2 ++ 1 file changed, 2 insertions(+) diff --git a/DESCRIPTION b/DESCRIPTION index d925749..bada8a2 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -18,6 +18,8 @@ Description: Downloads environmental data from the Copernicus Marine Service License: MIT + file LICENSE Encoding: UTF-8 Roxygen: list(markdown = TRUE) +Depends: + R (>= 4.1.0) Imports: utils, graphics, From 04cba8a4162b1e310bf3e3c9e1b193791a147b05 Mon Sep 17 00:00:00 2001 From: chross22 <52218551+chross22@users.noreply.github.com> Date: Thu, 6 Aug 2026 17:57:05 -0400 Subject: [PATCH 2/2] Make the parallel downloading findable in the README The section existed but did not answer the question a reader arrives with, which is whether there is an option and what it is called. Three things were in the way. Neither this section nor the daily one was in the table of contents, so a reader scanning the contents for it concluded there was nothing to find. The section described the behaviour - downloads "go out four at a time" - without naming n_workers or saying it defaults to 4. Someone looking for the argument had to read the whole paragraph to learn it exists. Both examples passed frequency = "daily", which reads as parallelism being a daily-only feature. It is not: any fetch spanning more than one time step is more than one download, and a fifteen-year monthly record is 180 of them. The examples are monthly now, and say so. Adds what it deliberately does not do, since both are easy to assume the other way: cached files are read from disk rather than re-fetched, and each accessEnvDat() call is one dataset, so SST and CHL remain two calls run one after the other. Co-Authored-By: Claude Opus 5 --- README.Rmd | 31 ++++++++++++++++++++++++------- README.md | 31 ++++++++++++++++++++++++------- 2 files changed, 48 insertions(+), 14 deletions(-) diff --git a/README.Rmd b/README.Rmd index bb91ef1..b58ffb6 100644 --- a/README.Rmd +++ b/README.Rmd @@ -30,6 +30,8 @@ The goal of datamatch is to pull environmental data from Copernicus Marine Servi - [Set up](#set-up) — the Copernicus client, sign-in, and where downloads are cached - [Quick start](#quick-start) - [One call per product](#one-call-per-product) — why `SST` and `CHL` are two fetches + - [Monthly or daily](#monthly-or-daily) — monthly by default, and how to fetch particular days + - [Downloads run in parallel](#downloads-run-in-parallel) — `n_workers`, and what is already cached **Choosing what to fetch** @@ -209,17 +211,32 @@ would come from, and `NA` where there is none. ### Downloads run in parallel -Days already in the cache are read straight from disk. Only the missing ones are -downloaded, and those go out four at a time, since a Copernicus subset request -spends nearly all of its time waiting on the API rather than on your machine. -Fetching a month of daily SST and SSS takes about 40 seconds this way against -about 160 serially. +**`n_workers` controls this, and it defaults to 4** — parallel downloading is on +without being asked for. A Copernicus subset request spends nearly all of its +time waiting on the API rather than on your machine, so four requests in flight +take barely longer than one. Fetching a month of daily SST and SSS takes about +40 seconds this way against about 160 serially. ```{r quickstart-workers, eval = FALSE} -accessEnvDat(vars = "SST", frequency = "daily", n_workers = 8, ...) # more -accessEnvDat(vars = "SST", frequency = "daily", n_workers = 1, ...) # serial +# The default. 15 years x 12 months is 180 downloads, four at a time +accessEnvDat(vars = "SST", years = 2003:2017, months = 1:12, bounding_box = bb) + +accessEnvDat(vars = "SST", years = 2003:2017, months = 1:12, bounding_box = bb, + n_workers = 8) # more at once +accessEnvDat(vars = "SST", years = 2003:2017, months = 1:12, bounding_box = bb, + n_workers = 1) # one at a time ``` +**This is not a daily-only feature.** Any fetch spanning more than one time step +is more than one download, so a long monthly record benefits as much as a daily +one — the 180 downloads above are monthly. + +Two things it deliberately does not do. It does not re-download what you already +have: files in the cache are read straight from disk, and a fully cached call +starts no workers at all. And it does not parallelise across calls — each +`accessEnvDat()` call is one dataset, so fetching SST and CHL is two calls, run +one after the other with each parallel inside itself. + The limit is the service, not your cores, so raising `n_workers` far past 8 mostly earns rate limiting. A day that fails does not abandon the others: every day is attempted, the ones that succeeded stay cached, and the error names each diff --git a/README.md b/README.md index bd300ba..bbc8dec 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,8 @@ The goal of datamatch is to pull environmental data from Copernicus Marine Servi - [Set up](#set-up) — the Copernicus client, sign-in, and where downloads are cached - [Quick start](#quick-start) - [One call per product](#one-call-per-product) — why `SST` and `CHL` are two fetches + - [Monthly or daily](#monthly-or-daily) — monthly by default, and how to fetch particular days + - [Downloads run in parallel](#downloads-run-in-parallel) — `n_workers`, and what is already cached **Choosing what to fetch** @@ -207,18 +209,33 @@ would come from, and `NA` where there is none. ### Downloads run in parallel -Days already in the cache are read straight from disk. Only the missing ones are -downloaded, and those go out four at a time, since a Copernicus subset request -spends nearly all of its time waiting on the API rather than on your machine. -Fetching a month of daily SST and SSS takes about 40 seconds this way against -about 160 serially. +**`n_workers` controls this, and it defaults to 4** — parallel downloading is on +without being asked for. A Copernicus subset request spends nearly all of its +time waiting on the API rather than on your machine, so four requests in flight +take barely longer than one. Fetching a month of daily SST and SSS takes about +40 seconds this way against about 160 serially. ``` r -accessEnvDat(vars = "SST", frequency = "daily", n_workers = 8, ...) # more -accessEnvDat(vars = "SST", frequency = "daily", n_workers = 1, ...) # serial +# The default. 15 years x 12 months is 180 downloads, four at a time +accessEnvDat(vars = "SST", years = 2003:2017, months = 1:12, bounding_box = bb) + +accessEnvDat(vars = "SST", years = 2003:2017, months = 1:12, bounding_box = bb, + n_workers = 8) # more at once +accessEnvDat(vars = "SST", years = 2003:2017, months = 1:12, bounding_box = bb, + n_workers = 1) # one at a time ``` +**This is not a daily-only feature.** Any fetch spanning more than one time step +is more than one download, so a long monthly record benefits as much as a daily +one — the 180 downloads above are monthly. + +Two things it deliberately does not do. It does not re-download what you already +have: files in the cache are read straight from disk, and a fully cached call +starts no workers at all. And it does not parallelise across calls — each +`accessEnvDat()` call is one dataset, so fetching SST and CHL is two calls, run +one after the other with each parallel inside itself. + The limit is the service, not your cores, so raising `n_workers` far past 8 mostly earns rate limiting. A day that fails does not abandon the others: every day is attempted, the ones that succeeded stay cached, and the error names each