Skip to content

Add new ghcn data access - #74

Open
crvernon wants to merge 8 commits into
mainfrom
bug/data-limit
Open

Add new ghcn data access#74
crvernon wants to merge 8 commits into
mainfrom
bug/data-limit

Conversation

@crvernon

Copy link
Copy Markdown
Contributor

This pull request adds support for the next-generation NCEI GHCN-hourly (GHCNh) weather data source to the diyepw package, ensuring continued generation of AMY EPW files beyond NOAA's discontinuation of the ISD-Lite product in August 2025. The update includes new data normalization utilities, expanded APIs, and comprehensive documentation on the new data source and station identifier mapping.

Major new features and improvements:

Support for GHCNh data and ISD-Lite discontinuation:

  • Added GHCNh (Global Historical Climatology Network hourly) as a new data source for observed weather, with automatic fallback when ISD-Lite is unavailable or incomplete due to NOAA discontinuing ISD-Lite after August 2025. The documentation now explains the transition, fallback strategy, and details about the new data source.
  • Introduced a WMO-to-GHCN station identifier mapping, with a packaged CSV and helper function wmo_to_ghcn_id(), to resolve differences between ISD-Lite and GHCNh station IDs. Documentation describes how the mapping is built and used.

API and CLI enhancements:

  • Updated the API and CLI to support the new data source: create_amy_epw_file() and create_amy_epw_files_for_years_and_wmos() now accept a source argument ("auto", "isd_lite", or "ghcnh") and an optional station_id for direct GHCN access. CLI documentation explains usage and fallback behavior. [1] [2]
  • Added new top-level exports: analyze_amy_file, get_ghcnh_file, and wmo_to_ghcn_id in diyepw/__init__.py for unified access to both ISD-Lite and GHCNh workflows.

Data normalization and analysis:

  • Implemented _ghcnh_normalize.py, a new module for reading and converting GHCNh PSV files into the canonical AMY DataFrame schema, including logic for collapsing sub-hourly observations to hourly and normalizing ISD-Lite data for consistency.
  • Added analyze_amy_file.py, a new analysis module that works with both ISD-Lite and GHCNh formats, superseding the original ISD-Lite-only analysis.

Infrastructure improvements:

  • Added _http.py with HTTP helpers enforcing bounded timeouts, improving reliability when downloading from NOAA/NCEI endpoints.

These changes ensure that diyepw remains functional and robust as NOAA transitions away from ISD-Lite, while providing users with clear documentation and flexible APIs for working with both legacy and next-generation weather datasets.

@ericmartinpe

Copy link
Copy Markdown
Collaborator

While reviewing this PR I found and fixed a few issues, and ran additional testing:

  • Wind direction sentinel bug: GHCNh's 999 missing-value sentinel for wind_direction wasn't being converted to NaN, so it could flow into the canonical AMY data as a bogus 999° entry. Fixed by sanitizing Wind_Direction by range ([0, 360]) rather than relying on exact-value matching, since other columns (e.g. Sea_Level_Pressure) can legitimately read ~999. Added regression tests in tests/test_ghcnh_normalize.py.
  • Leap-year hardcoding: analyze_amy_file.py hardcoded 8760 hours/year in its missing-row calculations, so leap years were always reported as short by 24 hours. Now infers 8760 vs 8784 from the file's own timestamps (or an explicit year argument). Added tests/test_analyze_amy_file.py.
  • Confusing error on double fallback failure: in source="auto", if ISD-Lite failed and the WMO had no GHCNh station mapping, the raised error was a bare, misleading LookupError naming only the GHCNh side. Now raises one combined error describing both failures. Added a regression test.
  • .gitignore: added diyepw/data/ghcnh_files/*, matching the existing noaa_isd_lite_files entry, so downloaded GHCNh PSVs don't show up as untracked files.

Testing

  • Live source="auto" runs across multiple WMO/year combinations, including years straddling the ISD-Lite discontinuation.
  • CLI test of create_amy_epw_files_for_years_and_wmos with --source isd_lite and --source ghcnh --station-id.
  • Batch/mixed-mode run across years straddling the cutoff, confirming per-combination errors are isolated correctly without crashing the batch.
  • Full test suite passing (37 tests).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants