Commit 81d96bf
Add 8 snow-related variables for hydrology departure (v0.2.0) (#55)
* Initialize PWF baseline for #48
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Snow backfill script with hourly accumulation handling
Phase 1 of #48. New scripts/backfill_edh_snow.py imports safeguards
from scripts/_lib.py (preflight_single_instance, with_retry,
write_geotiff, log, get_token) and produces both layers of snow
variables from a single hourly EDH fetch per year:
Monthly natives (12-band/year COG):
snow_depth mm SWE monthly mean of daily sde * rsn
snowfall mm monthly sum of daily sf (accum-handled)
snowmelt mm monthly sum of daily smlt (accum-handled)
snow_cover % monthly mean of daily snowc
Annual derived (1-band/year COG):
swe_max mm annual max of daily sde * rsn
snowfall_fraction % 100 * annual_sum_sf / annual_sum_tp
snowmelt_doy_50 day DOY when cumsum(smlt) >= 50% of annual
snowmelt_rate_peak mm/wk annual max of 7-day rolling sum daily smlt
New hourly_accum_to_daily() helper implements the 00:00 UTC reset
trick: sf and smlt at valid_time = t (00:00 UTC) represent the
accumulation from t-24h to t, so daily total for day D = value at
D+1 00:00 UTC. Pattern was documented in the #36 EDH-migration
archive but never implemented in code; this is the first call site.
Smoke-tested on year 2020 (leap year): all 8 outputs written in
228s. Median DOY-50 = May 8 (freshet), median peak SWE = 276 mm,
median snowfall_fraction = 27% — values span plausible BC ranges,
alpine pixels reach plausible extremes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add 8 snow vars to cd_variables registry, pct_point_diff anomaly type
Phase 4 of #48 - consumer-side registry, anomaly branch, tests.
cd_variables() now ships 15 vars (7 existing + 8 snow):
Monthly natives (pct_normal except snow_cover):
swe, snowfall, snowmelt pct_normal %
snow_cover pct_point_diff %
Annual derived (mostly absolute):
swe_max absolute mm
snowfall_fraction pct_point_diff %
snowmelt_doy_50 absolute day
snowmelt_rate_peak absolute mm/wk
Renamed snow_depth -> swe in script and registry: the value
sde * rsn evaluates to kg/m^2 = mm of water (SWE), not vertical
snow depth. Original name was misleading; caught during registry
design. Running backfill uses the pre-rename script via its
already-loaded code; outputs will be mv'd snow_depth_*.tif -> swe_*.tif
after backfill completes (one-line shell op, no re-fetch).
cd_anomaly() adds the pct_point_diff branch. Formula is identical to
absolute (value - baseline_mean), combined via %in% - the distinct
type is for downstream display semantics. cap_pct does NOT apply to
pct_point_diff (covered by new test); only pct_normal is clamped.
Tests: 166 PASS, 0 FAIL. New cases in test-cd_anomaly.R for
pct_point_diff arithmetic and cap non-application; bumped
test-cd_variables.R count 7 -> 15 plus two new membership tests;
fixed the duplicate pct_normal assertion in test-cd_fetch.R.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Stage 3 aggregation + catalog parsing for snow vars
Phase 2b of #48 - extend the Stage 3 R pipeline to aggregate the
4 monthly snow natives (swe, snowfall, snowmelt, snow_cover) into
multi-year COGs alongside the existing 7 vars, and stack the 4
annual-derived snow scalars (swe_max, snowfall_fraction,
snowmelt_doy_50, snowmelt_rate_peak) into 1-band-per-year multi-year
COGs in a new Step 1b. agg_methods extended for the new monthly
vars (sum for snowfall/snowmelt, mean for swe/snow_cover).
Caught a substring-match bug in cd_stac_item() while dry-running the
new pipeline: the parser used grepl(v, name_parts) which matches
substrings, so swe_max_annual.tif was being mis-routed under the
`swe` variable (because "swe" is a substring of "swe_max"). The bug
was latent before because no existing variable name was a substring
of another. Fixed with strict {var}_{period} exact-match against
both registries (cd_variables() and cd_periods()).
Catalog now has 59 items (35 existing + 24 new):
4 monthly snow x 5 periods (annual + winter/spring/summer/fall)
4 annual snow x 1 period (annual)
Verified via dry-run: cd_catalog() correctly returns 5 periods for
each monthly snow var and 1 period (annual) for each annual-derived
scalar. Existing tmean catalog entries unchanged.
Tests: 166 PASS, 0 FAIL (no regression from the parser change).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Phase 2 + 2b: snow vars live on S3 (catalog.json now 59 items)
Full 76-year backfill landed (~2h52min main run + 3min retry for year
2022 ClientPayloadError). Renamed 75 snow_depth_*.tif outputs to
swe_*.tif since sde * rsn evaluates to mm SWE, not vertical snow depth.
Stage 3 R aggregation + S3 push completed. 24 new COGs uploaded (~114 MB):
4 monthly snow x 5 periods (annual + winter/spring/summer/fall)
4 annual derived x 1 period (annual)
plus an updated catalog.json with 35+24 = 59 STAC items.
cd_catalog() against the default S3 URL now returns the full set;
verified all 8 new vars at expected periods.
Updates planning files only - the script changes that produced this
result already landed in 37a7510 (Stage 3 extension + cd_stac_item
exact-match parser fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Phase 5: Snowpack vignette section + bibliography wiring
Phase 5 of #48. New "Snowpack" section in vignettes/peace-fwcp.Rmd
between "Daytime Highs" and "Recent vs Pre-warming". Wires up the
bibliography YAML field for the first time and consumes the
citation map produced by #53 (snow methodology lit review).
vignettes/references.bib (new, 22 KB, 11 entries):
Generated by rbbt::bbt_write_bib() from BBT keys for the 11
papers in the NewGraphEnvironment/hydrology Zotero collection
added during #53. Static bib committed so vignette renders
on CI without needing Zotero/BBT.
Vignette YAML:
Adds bibliography: references.bib + link-citations: true.
Snowpack section structure:
- Intro: snowpack as hinge of BC hydrology, salmon-migration
framing, broad context (Mote 2018, Pederson 2011, Najafi 2017
BC attribution, Kang 2016 Fraser freshet)
- Methodology footnote: ERA5-Land bias caveat (Kouki 2023),
raw MK is correct per Yue and Wang 2002
- Headline numbers paragraph: SWE annual -10 percent, summer
SWE -75 percent, spring snowmelt +37 percent (freshet
earlier), annual snowfall flat (-6 percent) so the SWE
decline is warmth-driven not less-snow-driven
- Seasonal-curve table: pivoted from monthly faceted plot to
seasonal table because monthly aggregations aren't on S3
(COG schema is annual + 4 seasons). Seasonal level still
tells the "when does snow accumulate / melt" story cleanly.
- 4 annual time-series plots with Theil-Sen lines: swe_max,
snowmelt_doy_50, snowmelt_rate_peak, snowfall_fraction
- 3-finding interpretation: snow leaving earlier (not falling
less), freshet shifting into spring, summers becoming
snow-free. Each finding tied to a specific cited paper.
Recent vs Pre-warming table extended:
Added no_pct_vars list (snow_cover, snowfall_fraction,
snowmelt_doy_50) to NA-out the Delta-percent column for vars
where it's not meaningful. All 8 new vars appear in the table.
data-raw/peace_fwcp_vignette_data.R:
Extracted pct_normal_vars list to include swe, snowfall,
snowmelt alongside prcp, soil_moisture for the regional and
per-ecoregion cd_compare(method = "pct_change") calls. Re-ran
precompute: peace_fwcp.rds is now 270 KB (was 160 KB) with
all 15 vars.
Render time 8.7 s. 166 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Phase 3: ASWS QA cross-check + vignette bias caveat update
Phase 3 of #48. New data-raw/qa_snow_validation.R cross-checks
ERA5-Land swe_max against BC ASWS automated snow-pillow daily
SWE at 4 active stations in the FWCP Peace AOI. Outputs land in
planning/active/ for the eventual archive.
Site selection (via bcsnowdata::snow_auto_location() spatially
intersected with the FWCP Peace AOI):
Pine Pass 1400 m 37 yr (1989-2025)
Mount Sheba 1490 m 7 yr (2019-2025)
Ware Upper 1565 m 10 yr (2016-2025)
Aiken Lake 1050 m 41 yr (1985-2025)
Germansen Landing was in the original list but lacks usable SWE
record at this location.
Findings (95 paired site-years, pooled r = 0.51):
1. Bias is direction-variable, NOT uniformly high:
- Pine Pass: ERA5 underestimates by 61% (1140 vs 439 mm)
- Mount Sheba: ERA5 underestimates by 32% (936 vs 633 mm)
- Ware Upper: ERA5 close match (+1%, 250 vs 254 mm)
- Aiken Lake: ERA5 overestimates by 54% (265 vs 407 mm)
This contradicts a naive read of Kouki et al. 2023 (which
reported 150-200% NH-wide overestimate). At BC interior sites
ERA5-Land overestimates as Kouki found; at high-snowpack
Coast-Mountain-spillover sites it underestimates badly.
2. Bias is approximately stable over time at all 4 sites:
regression of (ERA5 - ASWS) on year is non-significant
everywhere (p > 0.2). This supports the vignette claim that
"trends are still defensible" even if absolute values are
biased.
Vignette updated: replaced the earlier quote of Kouki's
NH-wide 150-200% figure with our specific BC findings in the
methodology footnote of the Snowpack section. Re-rendered: 9.2 s.
DESCRIPTION: bcsnowdata added to Suggests.
Manual-survey secondary cross-check skipped — the ASWS bias
stability is clean enough at all 4 sites that the secondary
check isn't needed to support the vignette claim. Documented
as deferred in findings.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Phase 6: monthly GHA pipeline supports snow vars; README updated
Phase 6 of #48. The monthly GHA at .github/workflows/climate-update.yml
runs scripts/pipeline_update_edh.R, so that's the file that needs to
know about the snow vars — the workflow YAML stays unchanged.
scripts/pipeline_update_edh.R changes:
* agg_methods extended with the 4 monthly snow natives
(swe / snowfall / snowmelt / snow_cover); same shape as the
7 existing 12-band-per-year vars, flow through cd_aggregate
identically.
* annual_vars list added for the 4 annual derived snow scalars
(swe_max / snowfall_fraction / snowmelt_doy_50 /
snowmelt_rate_peak); these are 1-band per year, stored in
data/backfill/annual/, bypass cd_aggregate.
* Step 3 now calls both backfill_edh_all.py AND backfill_edh_snow.py
for each candidate year, and verifies all 15 outputs wrote
(7 core monthly + 4 snow monthly + 4 snow annual). The Python
scripts skip incomplete years individually, so a partial year
on EDH cleanly defers all 15 to the next run.
* Step 4 split into a monthly path (cd_aggregate from 12-band)
and an annual path (1-band straight onto the existing COG).
Refactored the per-(var, period) append logic into a small
append_to_cog() helper to avoid duplicating the grid-alignment
check between the two paths.
* Header docstring updated to describe the new flow.
README.md: variable inventory now lists all 15 vars grouped as
core climate / snow monthly natives / snow annual derived, with a
periods note clarifying that annual-derived vars only have an
"annual" period (not the seasonal+annual schema of the monthly
natives).
parse() clean on the R script. 166 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Vignette polish: simpler bias methodology + per-ecoregion snow + interp
Round of vignette refinements after the first user review of the
Snowpack section:
* Methodology paragraph: rewritten in plain language. Two-paragraph
structure now distinguishes the two stacked error sources at any
point comparison: (1) scale mismatch between a 80 km^2 cell average
and a single point measurement, and (2) the Northern-Hemisphere-
scale cell-mean bias documented by Kouki 2023. Walks the reader
through what our QA at Pine Pass and Aiken Lake actually shows.
Dropped the file pointer to the QA results.
* Acronyms expanded inline. DJF/MAM/JJA/SON now reads "winter
(December-February)" etc.
* New "Snow per ecoregion" subsection inside Per-Ecoregion Variation.
Two facet plots: peak SWE anomaly per ecoregion + snowmelt DOY-50
anomaly per ecoregion. Brief intro paragraph above the plots
pointing to the WSG x ecoregion table for watershed-group mapping;
deeper interpretation lives in the Interpretation section.
* Interpretation section gains a snowpack finding: snow leaving the
region earlier (not falling less); freshet timing shifting
uniformly across all 5 ecoregions (~1 day/decade earlier, p < 0.01
in every ecoregion); peak SWE detectable only in the regional
aggregate because year-to-year variability dominates at the
ecoregion scale. Earlier draft asserted an elevation gradient that
is NOT present in the data; corrected.
* Salmon-migration framing removed - the FWCP Peace is non-anadromous
(Bennett Dam blocks salmon from the Williston watershed). Replaced
with the FWCP-accurate resident-salmonid framing (bull trout,
Arctic grayling, mountain whitefish, rainbow trout, kokanee) and
freshet relevance for channel morphology / spawning gravel
mobilization / off-channel rearing habitat.
* "## References" heading added at the end so pandoc places the
auto-generated bibliography under an explicit section.
Render time 9.4 s, 11 citations resolve to References, 166 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 38d8bf8 commit 81d96bf
21 files changed
Lines changed: 1868 additions & 92 deletions
File tree
- R
- data-raw
- inst/vignette-data
- planning/active
- scripts
- tests/testthat
- vignettes
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
6 | | - | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
7 | 10 | | |
8 | 11 | | |
9 | 12 | | |
10 | 13 | | |
11 | 14 | | |
12 | 15 | | |
13 | | - | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
14 | 19 | | |
15 | 20 | | |
16 | 21 | | |
| |||
43 | 48 | | |
44 | 49 | | |
45 | 50 | | |
46 | | - | |
| 51 | + | |
47 | 52 | | |
48 | 53 | | |
49 | 54 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
78 | | - | |
79 | | - | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
80 | 84 | | |
81 | 85 | | |
82 | 86 | | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
88 | 99 | | |
89 | 100 | | |
90 | 101 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
4 | | - | |
5 | | - | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
6 | 8 | | |
7 | 9 | | |
8 | 10 | | |
9 | 11 | | |
10 | 12 | | |
11 | | - | |
12 | | - | |
13 | | - | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
14 | 21 | | |
15 | 22 | | |
16 | 23 | | |
| |||
22 | 29 | | |
23 | 30 | | |
24 | 31 | | |
25 | | - | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
26 | 37 | | |
27 | 38 | | |
28 | 39 | | |
29 | | - | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
30 | 49 | | |
31 | | - | |
32 | 50 | | |
33 | 51 | | |
34 | 52 | | |
35 | 53 | | |
36 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
37 | 57 | | |
38 | 58 | | |
39 | 59 | | |
40 | 60 | | |
41 | 61 | | |
42 | | - | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
43 | 65 | | |
44 | 66 | | |
45 | 67 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | | - | |
56 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
57 | 60 | | |
58 | 61 | | |
59 | 62 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
48 | 52 | | |
49 | | - | |
| 53 | + | |
50 | 54 | | |
51 | 55 | | |
52 | 56 | | |
| |||
58 | 62 | | |
59 | 63 | | |
60 | 64 | | |
61 | | - | |
| 65 | + | |
62 | 66 | | |
63 | 67 | | |
64 | 68 | | |
| |||
0 commit comments