Cloud-native geospatial data products built on Canadian wildfire research — daily fire spread, fire weather, suppression effectiveness, and long-term fire regime change.
A monorepo of six products sharing one data layer. The engineering goal is that everything here runs on a laptop and serves from static hosting; the scientific goal is that every number is reproducible and every uncertainty is reported.
📖 Project foundations — data inventory, research threads, roadmap, standards · 📓 Work journal
| Product | What it is | Status | |
|---|---|---|---|
| ① | FireSpread Canada | Daily fire-growth data cube (CFSDB) + spread model + web explorer | 🔨 in progress |
| ② | pyfiredanger-grid | Validated FWI System over gridded weather, on PyPI | 📋 scaffold |
| ③ | Time-to-Contain | Survival analysis of Alberta suppression effectiveness | 📋 planned |
| ④ | Day-of-Burn | Satellite hotspots → daily fire progression, with uncertainty | 📋 planned |
| ⑤ | Fire Regime Atlas | 1930→2023 trend & seasonality, as a scrollytelling site | 📋 planned |
| ⑥ | Ignition Risk | Human vs lightning ignition surface, spatially cross-validated | 📋 planned |
The Canadian Fire Spread Database point-level deliverable is 13.2 GB uncompressed across 20 zipped annual CSVs, developed on a machine with ~16 GB of free disk. Unzipping is not an option.
Every conversion in pipelines/ therefore streams:
zipfile member handle → Arrow CSV RecordBatchReader → ParquetWriter
one record batch at a time. Peak memory is a single batch, the decompressed CSV never touches disk, and the result is hive-partitioned Parquet that DuckDB can query with predicate pushdown — including from the browser via DuckDB-WASM, so the front ends need no API server.
Requires uv, and the raw data archive for anything but the tests.
uv syncCheck which inputs are visible:
uv run wfp pathsConvert the CFSDB fire-day table to GeoParquet:
uv run wfp cfsdb groupsStream the point archives to partitioned Parquet (all 20 years, or name specific ones):
uv run wfp cfsdb points 2021 2023Compare cross-validation strategies on the fire-day table, and ask what the model uses:
uv run firespread cvuv run firespread diagnoseThen query the converted data directly, no database required:
duckdb -c "SELECT year, count(*) rows, avg(sprdistm) mean_spread_m FROM 'data/processed/cfsdb_points/**/*.parquet' GROUP BY year ORDER BY year"uv run pytest -m "not slow"The suite runs entirely on synthetic CFSDB-shaped fixtures, so CI needs no access to the archive.
Tests marked slow / needs_archive validate the data contract against the real files and are
skipped when it is not mounted:
uv run pytest -m slowLint, format, and type-check:
uv run ruff check . && uv run ruff format --check . && uv run mypy pipelines/src packages/pyfiredanger-grid/src products/firespread-canada/srcdocs/ foundations + journal
packages/
pyfiredanger-grid/ Product ② — installable library
pipelines/ ingest & conversion (wfp CLI)
src/wildfire_pipelines/
config.py paths, CRS constants
schema.py CFSDB data contract
cfsdb.py streaming conversions
products/
firespread-canada/ Product ① — CV harness, spread model, diagnostics
notebooks/ exploration only — nothing ships from here
data/ gitignored; raw is re-fetchable, derived is rebuildable
Area and distance are computed only in EPSG:3978 (NAD83 / Canada Atlas Lambert). The Alberta
perimeter shapefile ships in EPSG:3400 (Alberta 10-TM Forest, a Transverse Mercator projection)
which is not valid for area computation. Every pipeline stage declares its CRS explicitly; see
config.py.
This repository contains original analysis code. The underlying datasets and several reference
implementations are the work of others and are credited in full in
FOUNDATIONS.md §6 — in particular the Canadian
Fire Spread Database (Q. Barber et al., NRCan), the Canadian National Fire Database (CFS),
Alberta Wildfire, and the firedanger package (D. Steinfeld, Uni Bern).
Code is MIT. Data retains its original licence — see the attribution section before redistributing any derived artifact.