Skip to content

Commit e01f157

Browse files
chross22claude
andcommitted
Expire the index cache on each provider's publishing cadence
The AMOC cache I added had no expiry, so it would have gone stale silently as RAPID published new releases. Fixing that properly meant giving every index the same treatment rather than special-casing one. Each entry now records how its source updates: monthly for the NOAA indices, annual for RAPID, none for LCR, which was published with a paper and ends at 2014. That cadence sets how long a cached copy may be reused - a week, a month, or forever - so a living index re-downloads on its own and a finished one is never re-fetched pointlessly. Three things back it up. The staleness check is on the data rather than the cache, because a fresh download of a stale file is still stale. If a series ends further behind the present than its provider's own lag, that is reported with the command to force a refresh, and the note that if refreshing changes nothing the provider has not published either. A failed download with a cached copy on disk returns the cached copy and warns, rather than erroring. A provider being briefly down should not become an outage here. Saying so is what keeps it from being mistaken for current. climate_index_status() reports what is cached and what is due without downloading anything, so it is safe offline. refresh_climate_index() forces a re-fetch and skips the indices that cannot change. Downloading also moves out of the NetCDF reader into cached_index_file(), which every format now shares. The text indices were previously re-downloaded on every call and are now cached too. That made the mocking in the AMOC tests unnecessary: read_rapid_netcdf() takes a path, so those tests pass a fixture directly instead of intercepting the cache. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 556486c commit e01f157

11 files changed

Lines changed: 619 additions & 84 deletions

NAMESPACE

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,13 @@
22

33
S3method(print,datamatch_dictionary)
44
S3method(print,datamatch_index_dictionary)
5+
S3method(print,datamatch_index_status)
56
export(accessEnvDat)
67
export(as_markdown)
78
export(attach_bathymetry)
89
export(attach_climate_index)
910
export(bathymetry_variables)
11+
export(climate_index_status)
1012
export(climate_indices)
1113
export(copernicus_variables)
1214
export(covariate_columns)
@@ -24,6 +26,7 @@ export(plot_env)
2426
export(plot_matched)
2527
export(plot_series)
2628
export(product_url)
29+
export(refresh_climate_index)
2730
export(upscale_grid)
2831
export(upscale_time)
2932
export(variable_dataset)

R/climate_indices.R

Lines changed: 289 additions & 50 deletions
Large diffs are not rendered by default.

man/cached_index_file.Rd

Lines changed: 29 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

man/climate_index_status.Rd

Lines changed: 29 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

man/fetch_climate_index.Rd

Lines changed: 31 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

man/index_cache_age.Rd

Lines changed: 18 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

man/index_max_age.Rd

Lines changed: 20 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 8 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

man/refresh_climate_index.Rd

Lines changed: 32 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

man/warn_if_stale.Rd

Lines changed: 25 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)