Plan unified visualization and reporting layer for climate-toolkit outputs

[peetmate]

Plan unified visualization and reporting layer for climate-toolkit outputs

Summary

Current toolkit mostly prints terminal tables and saves JSON/CSV, with limited plotting only in climatology. We need deliberate visualization/reporting layer spanning:

• historical climate summaries
• season analysis
• period comparisons
• NEX-GDDP ensemble comparisons
• hazards outputs
• weather-station candidate review and station-vs-grid diagnostics

This should be treated as major enhancement project, not ad hoc plot additions.

Why this matters

• terminal outputs getting too dense for users to monitor or interpret
• many modules now return rich structured outputs that are difficult to inspect quickly
• ensemble workflows need uncertainty shown visually, not only printed means
• station-vs-grid workflow needs diagnostic plots/maps to support source selection
• current plotting logic is isolated in climate_tookit/climatology/long_term_climatology.py and not reusable across package

Current state in repo

• climate_tookit/climatology/long_term_climatology.py already writes annual time-series and monthly climatology PNGs via matplotlib
• weather-station workflow now emits HTML candidate maps/tables, but layout and reporting are still narrow-purpose
• most other modules emit terminal tables + JSON/CSV only
• no shared plotting API, no shared style/config, no common report bundle convention

Research notes: recommended stack and patterns

Core stack

1. xarray + matplotlib for baseline static figures

   • xarray plotting is built around labeled arrays and uses CF-style metadata (long_name, standard_name, units) for axis labels.
   • Good default for deterministic CLI-friendly PNG export.
   • Source: xarray plotting guide
     https://docs.xarray.dev/en/stable/user-guide/plotting.html

2. Cartopy for publication-quality climate maps

   • Strong choice for projected climate maps, anomaly maps, gridded fields, multi-panel map layouts.
   • Better than plain scatter/leaflet for static atlas-style outputs.
   • Source: Cartopy docs
     https://cartopy.readthedocs.io/stable/

3. hvPlot / HoloViews / Panel for optional interactive HTML dashboards

   • Good for time-series brushing, fan charts, linked filtering, small multiples, crossfilter dashboards.
   • Panel especially relevant if we later want self-contained local dashboards/reports.
   • Sources:
     https://hvplot.holoviz.org/en/docs/latest/user_guide/Gridded_Data.html
     https://panel.holoviz.org/

4. Folium for lightweight standalone HTML maps

   • Best for simple site/station maps with popups, lines, tiles, overlays, no heavy app server.
   • Likely right default for weather-station review maps and possibly multi-site summaries.
   • Source:
     https://python-visualization.github.io/folium/latest/

5. geemap as optional Earth Engine exploration layer, not hard dependency

   • Useful for GEE-backed exploratory maps, inspectors, split maps, timeseries interaction.
   • Should stay optional because package must not force GEE-bound visualization path for all users.
   • Source:
     https://geemap.org/

Ensemble / uncertainty best practice

• Do not show ensemble mean alone.
• Prefer percentile bands / fan charts / spread summaries alongside central estimate.
• xclim.ensembles already provides create_ensemble, ensemble_mean_std_max_min, and ensemble_percentiles.
• This aligns with our existing need to show inter-model spread clearly.
• Source:
  https://xclim.readthedocs.io/en/stable/apidoc/xclim.ensembles.html
  https://xclim.readthedocs.io/en/stable/notebooks/ensembles.html

Verification / station-vs-grid diagnostics

• Daily precipitation correlation alone is weak and often misleading.
• Need multi-timescale diagnostics:
  • daily event skill
  • monthly totals
  • seasonal totals
  • annual descriptive totals
  • bias / MAE / RMSE / correlation
  • wet-day contingency metrics
• xskillscore is strong fit for standardized verification metrics if we move more calculations onto xarray-aligned structures.
• Source:
  https://xskillscore.readthedocs.io/en/stable/

Design principles for toolkit visualization layer

1. Machine-readable first, plot second

   • plotting should consume standardized output payloads, not terminal text
   • each module should expose stable report-ready structures

2. Static first, interactive second

   • first target: robust PNG/SVG/HTML exports from CLI
   • second target: richer interactive dashboards where justified

3. Uncertainty-aware by default

   • ensemble plots should show spread, percentiles, counts of contributing models, and missingness

4. CF/unit aware labeling

   • variable labels and units should come from harmonized metadata layer
   • avoid hard-coded inconsistent labels across modules

5. Separate map outputs from time-series/stat-table outputs

   • site/station geography problem differs from seasonal or climatology diagnostics

6. Report bundle convention

   • one run should be able to emit organized bundle, e.g.:
     • report.json
     • tables/*.csv
     • figures/*.png
     • maps/*.html
     • index.html

7. Optional dependency groups

   • keep installable base package lean
   • likely extras such as viz, maps, dashboard

Recommended scope by module

1. climate_statistics

• annual rainfall seasonality panel
• monthly climatology bars/lines
• detected/fixed season window timeline
• seasonal summary cards/tables
• humid/perhumid guard explanation box

2. season_analysis

• rainfall + ET0 + water balance time series with detected onset/cessation markers
• season-within-window timeline
• intra-season dry-spell visualization
• multi-year onset/cessation scatter / box plots

3. compare_periods.periods

• baseline vs focal/future seasonal bars
• anomaly plots
• percent-change panels
• “review required” annotation panels when season comparability weak

4. compare_periods.ensemble_periods

• ensemble fan charts for temperature / precipitation / water balance metrics
• baseline vs future ridge/small-multiple model spreads
• season-specific uncertainty bars
• model contribution / missingness panel

5. calculate_hazards

• hazard scorecards
• threshold exceedance frequency plots
• seasonal hazard comparison panels
• later: map-ready hazard layers if/when spatial workflows mature

6. weather_station

• nearby-station candidate map with basemap, distance lines, completeness scaling
• station coverage calendar heatmaps
• station-vs-grid monthly/seasonal comparison panels
• variable-by-variable source ranking summary
• pooled multi-station diagnostics plots

Proposed implementation phases

Phase 0: output contracts and metadata

• define shared report/figure schema
• normalize variable labels/units for plot use
• identify canonical “tidy” table per module
• decide where xarray structures are worth preserving

Phase 1: shared plotting foundation

• create climate_tookit.visualization package
• add plot config, palettes, label helpers, file naming helpers
• add shared table-to-figure utilities
• add optional dependency extras in packaging

Phase 2: static CLI exports

• add reusable PNG/SVG exports for:
  • climatology
  • season analysis
  • compare_periods
  • ensemble_periods
  • weather_station compare

Phase 3: map/report outputs

• formalize HTML map generation for station candidate review
• add report bundle writer and simple index.html
• add multi-site map and summary outputs where relevant

Phase 4: interactive dashboards

• prototype Panel/hvPlot report for:
  • ensemble compare
  • weather station compare
  • multi-site exploration
• keep optional; do not block CLI use

Phase 5: docs and examples

• user guide for figure/report outputs
• example commands and screenshots
• notes on when to prefer static vs interactive outputs

Suggested package structure

climate_tookit/visualization/
  __init__.py
  config.py
  labels.py
  io.py
  styles.py
  timeseries.py
  climatology.py
  seasons.py
  compare_periods.py
  ensembles.py
  hazards.py
  weather_station.py
  maps.py
  report_bundle.py

Initial heavy-lift decisions to make before coding too much

1. Should canonical plot inputs be pandas tables, xarray objects, or dual-format adapters?
2. Which outputs must be supported in pure CLI batch mode?
3. Which dependencies stay optional?
4. Should index.html be plain Jinja/Folium output first, with Panel later?
5. How much of weather-station compare should move onto xarray/xskillscore structures first?

Concrete first tranche after this issue

1. Inventory each module’s structured outputs and define canonical plot-ready tables.
2. Design minimal shared visualization API and output bundle layout.
3. Implement two exemplars first:
   • ensemble_periods uncertainty plot set
   • weather_station.compare diagnostics + map bundle
4. Only then propagate to other modules.

References

• xarray plotting guide: https://docs.xarray.dev/en/stable/user-guide/plotting.html
• Cartopy docs: https://cartopy.readthedocs.io/stable/
• hvPlot docs: https://hvplot.holoviz.org/en/docs/latest/user_guide/Gridded_Data.html
• Panel docs: https://panel.holoviz.org/
• Folium docs: https://python-visualization.github.io/folium/latest/
• geemap docs: https://geemap.org/
• xclim ensembles docs: https://xclim.readthedocs.io/en/stable/apidoc/xclim.ensembles.html
• xclim ensemble notebook: https://xclim.readthedocs.io/en/stable/notebooks/ensembles.html
• xskillscore docs: https://xskillscore.readthedocs.io/en/stable/

[Sammyjoseph999]

Starting on Phase 0 (output contracts) — an inventory of what each module actually returns today, since every later phase depends on standardizing these. Grounded in the current code.

Current state (verified)

• Only matplotlib is a runtime dependency; plotting lives only in climatology/long_term_climatology.py and compare_datasets.py.
• No climate_tookit/visualization package exists yet.
• Most modules return nested, heterogeneous dicts (metadata + indicators interleaved) plus a JSON/CSV dump — not tidy, plot-ready tables.

Per-module output inventory

Module	Entry point	Current output shape	Saved artifacts	Proposed canonical plot-ready table(s)
climate_statistics	analyze_climate_statistics()	nested dict: per-season rainfall/temp extremes, water-balance (NDWS/NDWL0), heat indices, SPEI	JSON	seasonal_summary (season × metric), monthly_climatology (month × var)
season_analysis	fetch_and_analyze_years[_fixed]()	per-year onset/cessation, dry-spell, livestock THI block (mean_thi, days_severe, …)	JSON	onset_cessation (year × onset/cessation), daily_water_balance (date × precip/ET0/balance), thi_daily
compare_periods.periods	compare()	baseline vs focal dicts + anomalies, "review required" flags	JSON	period_delta (metric × baseline/focal/anomaly)
compare_periods.ensemble_periods	ensemble driver	per-model stats + aggregated means	JSON	ensemble_long (model × metric × value) → drives fan charts/percentiles
calculate_hazards	calculate_hazards()	nested dict: indicator defs, analysis_window (onset/cessation), soil/crop params, thresholds, warnings	JSON	hazard_scorecard (indicator × value/threshold/status)
weather_station.compare	compare_station_to_grids()	station metadata + per-variable grid-vs-station frames	JSON + HTML	station_meta, grid_vs_station_long (var × timescale × metric), station_points (geo)

Cross-cutting observations that should drive the schema

1. Payloads mix metadata and metrics. Plotting wants a flat "long" metric table (entity, metric, value, unit) separated from a metadata/provenance block. Recommend every module expose a to_report_payload() returning {metadata, tables: {name: tidy_df}, provenance}.
2. No shared units/label layer. Labels are hard-coded per module. Phase 0 should define a single variable→(long_name, units) map (CF-style) so figures label consistently.
3. DataFrames are discarded after JSON serialization in several paths. For time-series/water-balance/ensemble plots we need the daily/per-model frames preserved (or reconstructable) — worth deciding per module which frames to keep.

Recommendation on the 5 open design questions

1. Canonical plot input: tidy pandas long tables as the contract, with optional xarray adapters only where gridded/CF metadata adds value (ensembles, maps). Avoids forcing xarray on every module.
2. Pure-CLI batch: all static exports (PNG/SVG + the report bundle) must work headless; interactive dashboards stay opt-in.
3. Optional deps: base = matplotlib only; extras viz (nothing new), maps (folium/cartopy), dashboard (panel/hvplot), verify (xskillscore).
4. index.html: start with plain Jinja + Folium (deterministic, no server); Panel later.
5. Weather-station onto xarray/xskillscore: defer — wrap existing frames in the tidy grid_vs_station_long contract first; move to xskillscore only if the metric set grows.

Suggested next tranche (matches the issue's own plan)

Build the two exemplars against this contract before touching other modules:

1. ensemble_periods uncertainty plot set (fan chart + model-contribution/missingness panel) from ensemble_long.
2. weather_station.compare diagnostics + candidate map from grid_vs_station_long + station_points.

Flagging for team direction rather than self-assigning — if the tidy-payload contract and the pandas-first / static-first calls above look right, I can take the Phase-0 schema definitions + first exemplar.