noaa-weather

A standalone Facetwork example package providing FFL workflows and handlers for working with NOAA climate data:

• GHCN catalog — discover and ingest Global Historical Climatology Network station data
• NDBC buoys — fetch and summarise National Data Buoy Center marine observations
• ISD-Lite — hourly station observations with offline mocks for tests
• Climate analysis — yearly state aggregates, multi-decade trends (temperature, precipitation, snowfall), linear regressions
• Reverse geocoding — Nominatim-backed station-to-place lookup with on-disk cache
• Reporting — per-station HTML reports, choropleth warming maps, batch summaries
• Extreme events — detect heat waves, cold snaps, wet/dry spells and heavy rain/snow days per station or region, with per-decade trends and dependency-free SVG/HTML charts
• Quality control — surface how much of a station's GHCN record NOAA flagged as failing QC (overall %, per element/year/check), roll it up to an observation-weighted region rate with a worst-stations ranking, and render it as a dependency-free SVG/HTML chart — so a reader can judge data credibility before trusting a trend

Discovered by the Facetwork runner via the facetwork.domains entry point declared in pyproject.toml. After pip install -e ., Facetwork's fw runner start --domain noaa-weather and fw ffl seed pick this package up automatically.

FFL at a glance

The domain is driven from FFL, Facetwork's workflow language. A step is name = Facet(args); andThen foreach fans the per-station work out across the fleet, and a step placed after the block fans it back in:

namespace my.weather {

    use weather.Catalog
    use weather.QC

    /** Per-station QC in parallel, then one region-level rollup. */
    workflow RegionQCRollup(country: String = "US", state: String = "NY", max_stations: Int = 5,
        start_year: Int = 1950, end_year: Int = 2026) => (status: String, narrative: String) andThen {

        discovery = weather.Catalog.DiscoverStations(
            country = $.country, state = $.state, max_stations = $.max_stations) andThen foreach station in $.stations {

            qc = weather.QC.SummarizeQualityFlags(
                station_id = $.station.station_id,
                state = $$.state,
                start_year = $$.start_year,
                end_year = $$.end_year)

            yield RegionQCRollup(status = "station_done", narrative = "")
        }

        region = weather.QC.AggregateRegionQC(
            country = $.country, state = $.state,
            start_year = $.start_year, end_year = $.end_year,
            station_count = discovery.station_count)

        yield RegionQCRollup(status = "completed", narrative = region.narrative)
    }
}

fw ffl run --primary my.ffl --library src/noaa_weather/ffl/weather.ffl \
  --workflow my.weather.RegionQCRollup --inputs '{"state": "NY"}'

📖 docs/ffl-examples.md — the full example gallery: fan-out/fan-in, Json loop variables, tuning the extreme-event thresholds, chaining charts onto aggregates, catch per station, when guards, and overriding this domain's own RetryPolicy/RateLimit mixins. Every snippet there is compile-checked.

Feature specifications

Per-feature specs live under docs/ — one document per feature, each grounded in the FFL docstrings, handler code, and shared tool libraries. Start with the flagship, Climate Analysis & Regional Trends.

Feature	Spec
Station catalog discovery (weather.Catalog)	docs/catalog-discovery.md
Station CSV ingest (weather.Ingest)	docs/ingest.md
Climate analysis & regional trends (weather.Analysis)	docs/trends.md
Quality-control surfacing (weather.QC)	docs/quality-control.md
Extreme-event detection (weather.Extremes)	docs/extremes.md
Marine buoys / NDBC (weather.Marine)	docs/marine.md
Reverse geocoding (weather.Geocode)	docs/geocode.md
Element vocabulary (weather.Vocab)	docs/vocab.md
Regional climate report bundle (weather.Report)	docs/climate-report.md
Workflows & fan-out composition (weather.workflows, weather.Cache)	docs/workflows.md
Storage, cache & sidecars (cross-cutting)	docs/storage-and-cache.md

See docs/README.md for the full grouped index.

Install

git clone https://github.com/rlemke/fwh_noaa_weather.git ~/fw_handlers/fwh_noaa_weather
cd ~/fw_handlers/fwh_noaa_weather
pip install -e .

This registers the package under the facetwork.domains entry-point group, making it discoverable by any Facetwork installation in the same environment.

Run from a Facetwork checkout

All commands below assume your shell is in the Facetwork checkout and the noaa-weather package is installed in the same Python environment that runs Facetwork (pip install -e ~/fw_handlers/fwh_noaa_weather).

Cold start: dashboard + runner together

fw ffl seed --include noaa-weather           # one-time, seeds FFL
fw runner start --domain noaa-weather -- --log-format text

This brings up the dashboard on :8080 and a runner that polls for noaa-weather tasks.

Add a runner to an already-running stack

If the Facetwork dashboard is already up and you just want another runner attached to it (after pulling new noaa-weather code, or to scale out):

fw runner start --domain noaa-weather --no-dashboard -- --log-format text

Run standalone

PYTHONPATH=src python agent.py

Layout

fwh_noaa_weather/
├── pyproject.toml                  # facetwork.domains entry point
├── README.md
├── CLAUDE.md                       # guidance for Claude Code in this repo
├── USER_GUIDE.md                   # human-facing walkthrough
├── agent-spec/                     # tools-pattern, cache-layout specs
├── agent.py                        # standalone AgentPoller variant
├── conftest.py                     # pytest fixtures
├── tests/                          # repo-level integration tests
├── scripts/                        # operational scripts (seed-climate-data, …)
└── src/noaa_weather/
    ├── __init__.py                 # exports `domain: DomainPackage`
    ├── handlers/                   # event-facet subpackages (one per domain)
    │   ├── analysis/
    │   ├── catalog/
    │   ├── extremes/               # extreme-event detection + SVG/HTML charts
    │   ├── geocode/
    │   ├── ingest/
    │   ├── marine/
    │   ├── qc/                     # quality-control surfacing (Q-flag rejection rates)
    │   ├── report/
    │   └── shared/                 # ghcn_utils, weather_utils — shims into tools/_noaa_tools
    ├── ffl/                        # weather.ffl + compiled JSON
    └── tools/                      # CLI utilities (CLI .py + .sh wrappers + _noaa_tools/)
        ├── _noaa_tools/                   # download / parse / analysis / mocks / charts
        ├── *.py                    # discover-stations, fetch-station-csv, …
        └── *.sh                    # shell wrappers

The tools/ dir gives every operation a CLI (e.g. download-ghcn-catalog.sh, fetch-station-csv.sh, compute-region-trend.sh); the FFL handlers call into the same tools/_noaa_tools/ modules via the handlers/shared/<domain>_utils.py shim, so the two surfaces share one cache and one implementation.

Required infrastructure

Service	Purpose
MongoDB	Facetwork registry + workflow state, plus weather_reports / climate_trends / extreme-event rollup collections
MinIO / S3 (optional)	Shared durable cache + outputs under FW_STORAGE=s3 — lets a multi-server fleet share artifacts with no shared disk

The cache + outputs backend is chosen by FW_STORAGE (local | hdfs | s3) rooted at FW_DATA_ROOT. On s3, downloads and durable outputs land in shared MinIO/S3 while readers get a real local file via a localize() read-through cache; scratch/staging always stay on local disk (FW_LOCAL_SCRATCH).

The package falls back to deterministic hash-based mocks when requests isn't installed or NOAA endpoints are unreachable, so unit tests run fully offline. See USER_GUIDE.md for the end-to-end walkthrough.

License

Apache 2.0 — see LICENSE.