Performance enhancement: NEX-GDDP fetch pipeline (review + next steps)

[Sammyjoseph999]

Performance enhancement: NEX-GDDP fetch pipeline

@peetmate — assigning to you to review the NEX-GDDP performance work and assess further enhancement opportunities.

Background

The NEX-GDDP ensemble/analysis paths were slow because they fetched data one year at a time and ran models serially. A series of changes introduced two optimizations across the ensemble modules:

1. Span-fetch — fetch the full date range once per (model, scenario) in a single GEE call, then slice per year in memory (was one GEE call per year).
2. Parallelism — run the per-model/per-projection work concurrently via ThreadPoolExecutor, auto-scaled by default (--workers, capped at 10 to match the GEE connection pool). The season-analysis ensemble stays serial (its global monkeypatch/redirect_stdout is not thread-safe) but still benefits from span-fetch.

Measured impact

• Long-term climatology (16 models x 2 scenarios x 30 years): >15 min -> ~2 min.
• Ensemble hazards (640 projections): GEE calls 640 -> 32, fetch wall-clock ~3 min.
• Season analysis: ~21x fewer GEE fetches per model (one span vs one per year).

Associated commits / PRs (flagged for review)

• perf: parallelize ensemble fetch and fetch full span once per projection #111 — parallelize ensemble fetch + span-fetch (061d5c0, 2100efb)
• fix+perf: ensemble print KeyError (#79) and parallel span-fetch #112 — ensemble print fix + parallel span-fetch
• perf: parallelize NEX-GDDP climatology + span-fetch (#81) #113 — parallelize NEX-GDDP climatology + span-fetch (e3e4e8b, 9ec82c0)
• fix: force matplotlib Agg backend to avoid Tk thread crash in climatology #116 — force matplotlib Agg backend (climatology thread crash) (389b912)
• perf: parallelize remaining ensembles + --workers flags #126 — parallelize remaining ensembles + --workers flags (c9c9c2f)
• perf+fix: span-fetch season analysis, revert unsafe season parallelism, cap workers to pool #128 — span-fetch season analysis, revert unsafe season parallelism, cap workers to GEE pool (c96ab94)

Affected modules: calculate_hazards/ensemble_hazards.py, climatology/long_term_climatology.py, climate_statistics/ensemble_statistics.py, compare_periods/ensemble_periods.py, season_analysis/ensemble.py, season_analysis/seasons.py.

For Pete to assess

• Whether the worker cap (10) is optimal vs GEE quota/rate limits, and whether the season-analysis ensemble can be made safely parallel (remove the global monkeypatch in analyze_one_model).
• Caching repeated (model, scenario, location) fetches across runs.
• Any remaining per-year or per-call hot spots in the GEE path (fetch_data/source_data/sources/gee.py, nex_gddp.py).

[Sammyjoseph999]

@peetmate — heads-up on a NEX-GDDP correctness fix that just landed on main (#132, via #133), relevant to this performance review.

nex_gddp.py previously filtered by a single scenario across the whole date range. Because NASA/GDDP-CMIP6 stores ≤2014 only under historical and 2015+ under the SSP, any request spanning the 2014/2015 boundary silently dropped the pre-2015 years (e.g. a 1990-2016 ssp245 request returned only 2015-2016). Fixed by splitting into a historical segment (≤2014) + SSP segment (≥2015).

Why it matters for performance work: this fix is in nex_gddp.py itself, so it affects every NEX-GDDP consumer over historical periods — climatology, statistics, compare_periods, compare_datasets — not just the module where it was reported. Two things to weigh during your review:

• Any benchmarks/caching keyed on NEX-GDDP fetches should account for the split now issuing two scenario segments (hence up to 2x the chunked calls) for boundary-spanning ranges.
• The span-fetch optimizations already in place still apply per segment.

Surfacing here so it's on your radar alongside the parallelization work tracked in this issue.