Test framework for `site_forecast`

Revision note (2026-07-15). Rewritten from the original proposal into a status tracker: the framework is roughly half implemented (commits f4e84ae, 0c703a1) and the suite passes — 62 tests, offline, ~7 s. Scope clarification: these tests are a safety net for the migration in DESIGN.md, written against the current API (okay, save_data, .df). They will be revised alongside the migration and do not need to anticipate the future Result/Op API; a more complete test suite gets built out after the migration. NOMADS tests are deferred (see below).

Context

Two constraints from the user:

No network on every test run. Fixtures replay cached query results so the default pytest is offline and fast.
Detect schema changes infrequently. A separate, opt-in run hits the real services and validates response structure.

Standing decisions:

Lean two-layer fixtures. VCR cassette for open_meteo (the only query with non-trivial custom parsing of a binary FlatBuffers wire format); parquet/netcdf/rowset snapshots for the rest. For monitor, snapshot the raw rowset so parse_rows/rowset_to_dataframe get exercised too.
pandera schemas as the single source of truth for query DataFrame contracts — reused by live schema tests and as documentation.
pytest markers (schema, slow) gate live/long tests; default pytest is offline.
Thin assertion layer on query tests. The fixtures and pandera schemas are the durable assets — they validate the data, which survives the DESIGN.md refactor unchanged. The assertions on the wrapper (okay, save_data) are deliberately thin and against the current API; they get rewritten to Result status-gate assertions during DESIGN migration steps 2–6. Do not pre-build tests against the future API.

Status

Done

Remaining, in priority order

None — the offline-suite milestone is complete. The next build-out (fuller plotter/orchestrator tests, nomads, the slow end-to-end lane) happens alongside/after the DESIGN.md migration.

Deferred

nomads_cutout — the module was written some time ago but is not yet run in operations or wired to any consumer. It will eventually be fully implemented (see DESIGN.md Section 10); defer its offline test, fixture, pandera schema, and live schema test until it is brought into service after the migration. Do not build fixtures for it speculatively.
slow end-to-end lane (instantiating the orchestrator with mocked queries and asserting outputs) — defer to the post-migration build-out, where it targets Forecast(profile, ctx) directly.

Interaction with the DESIGN.md migration

Purpose. This suite exists so that DESIGN migration steps 2–10 can ship incrementally with a regression net under them. Coverage of the current behavior is the goal; elegance of the assertions is not.
What survives the migration unchanged: replay fixtures (cassettes, parquet/netcdf snapshots, pickled rowsets), the pandera schemas, the pure-function unit tests, and the live schema lane.
What gets rewritten during the migration: the thin wrapper assertions in tests/query/ (assert q.okay → status-gate assertions on q.result, then on operation Results; save_data() checks → generic serializer checks). This happens naturally as each DESIGN step lands; it is expected churn, not rework to avoid.
After the migration, build the fuller suite: plotters unit-tested with hand-built dict[str, Result], validate_profile tests, the slow end-to-end lane, and nomads once implemented.

Fixture strategy per query

open_meteo (cassette, done): pytest-recording records via @pytest.mark.vcr. The module-level CACHE_SESSION = requests_cache.CachedSession(".cache", expire_after=3600) short-circuits HTTP at the requests-cache layer, so VCR (at urllib3) never sees the call — the bypass_openmeteo_cache conftest fixture swaps in a plain requests.Session for the duration of each test. Note this also bypasses the retry adapter; the retry configuration is covered separately by unit/test_open_meteo_session.py. Cassettes store the FlatBuffers binary as base64 in YAML — opaque but functional.

ndfd (parquet, done): stores the parsed DataFrame. The text parser is dead in offline tests — its schema-change detection is the responsibility of the live schema test.

herbie_maps (netcdf, done): the refresh script produces a tiny subset — ~50×50 grid centered on the VLA (all sub-hourly steps kept), 524 KB. get_tcolw is mocked to return the cached Dataset; the real reduction pipeline runs offline. The quantile/mean-at-radius reductions are checked as known-answer unit tests on synthetic grids (exact values) and as invariants on the fixture-driven HerbieQuery. (The exploratory full-size netcdfs — test_tcolw.netcdf ~403 MB and test_tcolw_subset.netcdf — now live under data/old_data/, still untracked; the committed fixture is produced fresh by the refresh script, never copied from working data.)

monitor (pickled rowset, done): psycopg2.connect monkeypatched to a mock whose cursor fetchall() returns pre-pickled raw rows by call order, so parse_rows and rowset_to_dataframe — the real schema-bearing parsers — execute against representative data.

Schema validation (pandera)

tests/schemas/queries.py declares one DataFrameSchema per in-service query class, plus an assert_herbie_dataset structural check for the xarray-based Herbie query (pandera targets DataFrames). Two consumers:

Offline test loads the fixture, asserts it satisfies the schema — catches accidental fixture mutation and serves as a documented contract.
Live test (-m schema) hits the real service, asserts the result satisfies the schema — primary mechanism for detecting upstream structural change.

Schemas declare columns, dtypes, nullability, and index/MultiIndex names. Value ranges are deliberately not locked (those drift), only structural shape. These schemas are also the foothold for the schema-validated Result evolution sketched in DESIGN.md Section 10.

Run modes

pytest — offline, fast, default (and the CI lane)
pytest -m schema — live, run before releases or on a weekly cron
pytest -m slow — end-to-end (post-migration)
pytest -m "" — everything

Known issues

The suite surfaces ~21 warnings in two families, both worth fixing before upstream deprecations become errors:

Pandas4Warning: 'd' is deprecated from Station.sun_rise_and_sets and from the tests themselves (pd.Timedelta("1d") → "1D"), plus one concat default-sorting deprecation.
DeprecationWarning: get_cmap ... removed in 3.11 from the Matplotlib colormap helpers.

Verification (for each remaining slice)

uv sync --all-extras installs cleanly
ruff check . && ruff format --check . passes
pytest — all offline tests pass in well under a minute with no network traffic
pytest -m schema — run manually once after adding a live test
pytest --cov=site_forecast — coverage report generated
CI green once .github/workflows/test.yml lands

FilesExpand file tree

test_plan.md

Latest commit

History