A runnable demonstrator that detects persistent (climate-warming) contrails along flight trajectories and proposes small flight-level changes to avoid them, weighing the avoided contrail climate forcing against the extra fuel burn.
This is the data-science / data-engineering half of a EUROCONTROL-style Contrail Avoidance project: a Polars/Pandas pipeline plus a Databricks-style notebook that decide which flights form persistent contrails and optimise an avoidance manoeuvre. Data is synthetic but physically plausible (see scope note). It runs end-to-end on a laptop (macOS, Python 3.11, no GPU).
flowchart LR
G["generate.py<br/>seeded synthetic data"] --> W["weather_grid.parquet<br/>lat × lon × FL × time<br/>temp_K, RHi%"]
G --> F["flights.parquet<br/>trajectories of waypoints"]
W --> P["pipeline.py<br/>nearest-cell join"]
F --> P
P --> PH["physics.py<br/>SAC ∩ ISSR<br/>persistent-contrail flag"]
PH --> DET["per-flight segments<br/>length km, duration h"]
DET --> AV["avoidance.py<br/>FL-change optimiser<br/>±2000/±4000 ft"]
AV --> M["metrics.py<br/>fuel→CO2,<br/>contrail EF→CO2e"]
M --> OUT["fleet climate impact<br/>t CO2e avoidable vs t fuel CO2"]
OUT --> D["dashboard/app.py<br/>Streamlit + Plotly"]
OUT --> NB["notebooks/<br/>Databricks notebook"]
make install # create .venv (Python 3.11) + install deps + editable package
make data # generate the synthetic weather grid + flights into data/
make pipeline # detect persistent-contrail segments per flight
make avoidance # run the FL-change optimiser + print fleet climate impact
make test # pytest (physics + pipeline + avoidance)
make notebook # run the Databricks-style notebook as a script, end-to-end
make dashboard # streamlit run dashboard/app.pymake install requires python3.11 on the PATH. Everything else uses .venv.
A contrail is the line of ice cloud behind an aircraft. Most evaporate in seconds and don't matter for climate. The ones that matter are persistent contrails that spread into thin "contrail cirrus" and trap heat — collectively a climate forcing comparable to aviation's CO₂.
A persistent contrail needs two conditions at the same point in the sky:
- Schmidt-Appleman Criterion (SAC) — the air must be cold enough for the hot, moist engine exhaust to reach water saturation as it mixes and cools, so droplets form and freeze. The critical temperature
T_critdepends on pressure (altitude) and humidity. (Schmidt 1941; Appleman 1953; Schumann 1996.) - Ice-supersaturation (ISSR) — the ambient air must already be supersaturated with respect to ice (relative humidity over ice
RHi > 100 %). Only then does the fresh ice contrail persist and spread instead of sublimating immediately. (Kärcher 2018.)
So in the code:
persistent_contrail = SAC_met(T, p, RH) AND (RHi > 100 %)
src/contrail_avoidance/physics.py implements sac_threshold(...), is_persistent_contrail(...) and the saturation-vapour-pressure helpers, each documented with references. The SAC is the Schumann (1996) closed-form threshold from the exhaust mixing-line slope, with documented turbofan constants. It is a simplified, directional approximation suitable for a demo — not CoCiP/APCEMM.
Avoidance. A small fraction of flights cause most contrail forcing, and ISSR layers are thin. So for each contrail-forming flight we test ±2000/±4000 ft cruise-level changes, re-fly the trajectory against the weather grid, and pick the change that exits the ISSR while minimising extra fuel — but only if it is net-beneficial:
net_benefit = avoided_contrail_CO2e - extra_fuel_CO2
Contrail forcing → CO₂e uses the Energy-Forcing approach (Teoh et al. 2020); fuel → CO₂ uses the ICAO 3.16 kg-CO₂/kg-fuel factor. All constants are named in src/contrail_avoidance/metrics.py.
With the committed seeds (make data && make pipeline && make avoidance), the 30-flight / 96,000-cell synthetic scenario produces roughly:
| metric | value |
|---|---|
| weather-grid ISSR cells | ~3.8% of the grid (coherent patches) |
| flights forming persistent contrails | 4 / 30 |
| total persistent-contrail length | ~536 km |
| recommended diversions | 4 (all net-beneficial) |
| avoidable contrail forcing | ~1.57 t CO₂e |
| extra-fuel penalty | ~0.12 t CO₂ (~0.04 t fuel) |
| net climate benefit | ~1.45 t CO₂e |
Read the direction (a small fuel cost buys a much larger climate benefit), not the absolute numbers.
The demo is built so the only thing that changes between "laptop" and "production on Databricks" is the data source and the storage layer — the physics and the pipeline are unchanged.
Medallion layout (Delta Lake + Unity Catalog):
| layer | table | content |
|---|---|---|
| bronze | coav.bronze.weather_grid |
raw 4D weather reanalysis (ERA5) |
| bronze | coav.bronze.flights |
raw 4D flight tracks (OpenSky) |
| silver | coav.silver.waypoints_flagged |
waypoints joined to weather, SAC/ISSR flags |
| gold | coav.gold.flight_summary |
per-flight contrail length/duration |
| gold | coav.gold.avoidance_decisions |
FL-change recommendations + CO₂e tradeoff |
Locally these are regenerable Parquet files in data/ (git-ignored, rebuilt by make data). On Databricks they are managed Delta tables in Unity Catalog, with lineage and access control. The notebook (notebooks/contrail_avoidance.py, in the # Databricks notebook source format) narrates exactly this flow and is importable as a Databricks notebook and runnable as a local script.
Orchestration. A Databricks Workflow job chains the tasks ingest → pipeline → avoidance → publish, scheduled per weather-update cycle. The compute is plain Polars/NumPy, so it runs on a single node; at full ERA5 resolution the nearest-cell join becomes a Spark spatial / H3 join while the pure-NumPy physics functions are reused unchanged.
- Weather grid → ERA5 reanalysis (ECMWF Copernicus): temperature and specific humidity on pressure levels; derive
RHifrom specific humidity + temperature. Swapgenerate.py's grid for an ERA5 loader; the schema (lat, lon, flight_level, time_idx, temp_k, rhi_percent, pressure_pa) stays the same. - Flights → OpenSky Network ADS-B state vectors: real 4D tracks (lat/lon/altitude/time). Swap
generate_flightsfor an OpenSky ingest; the waypoint schema is unchanged.
- Python + Pandas / Polars / NumPy — a real columnar Polars pipeline (joins, group-bys, segment aggregation) with documented Pandas interop and vectorised NumPy physics.
- Databricks / Data-Lakehouse — explicit bronze/silver/gold Delta + Unity Catalog framing, a Databricks-source notebook, and a Workflow orchestration design.
- Technical documentation — referenced physics docstrings, ADRs (
docs/DECISIONS.md), and this README aimed at both experts and non-experts. - ATM / aviation-climate domain understanding — correct, referenced treatment of the Schmidt-Appleman Criterion, ice-supersaturation, the contrail-vs-CO₂ climate tradeoff, and operationally realistic ±2000/±4000 ft avoidance manoeuvres.
This repo uses synthetic, deterministic weather and flights. The ISSR patches and trajectories are constructed to be physically plausible and coherent, but they are not real meteorology or real traffic. The SAC/ISSR detection and the contrail→CO₂e conversion are simplified, first-order models chosen for transparency and directional correctness — they are not operational and deliberately avoid the complexity of CoCiP/APCEMM and a full GWP* treatment. The value of the demo is the end-to-end engineering and the correct physical structure, with clean seams (physics.py, metrics.py, data loaders) where higher-fidelity models and real ERA5/OpenSky data would drop in.
src/contrail_avoidance/
generate.py synthetic 4D weather grid + flights -> Parquet
physics.py SAC threshold, ISSR, persistent-contrail detection (referenced)
pipeline.py Polars nearest-cell join + per-flight segment aggregation
avoidance.py FL-change optimiser + climate-vs-fuel tradeoff
metrics.py fuel->CO2 and contrail-EF->CO2e conversions (documented constants)
notebooks/ Databricks-source notebook (.py) + .ipynb
dashboard/app.py Streamlit + Plotly demo dashboard
tests/ pytest: physics boundary cases + pipeline/avoidance integration
docs/DECISIONS.md ADRs
- Schumann, U. (1996). On conditions for contrail formation from aircraft engines. Meteorol. Z. 5, 4–23.
- Kärcher, B. (2018). Formation and radiative forcing of contrail cirrus. Nat. Commun. 9, 1824.
- Teoh, R. et al. (2020). Mitigating the climate forcing of aircraft contrails by small-scale diversions and technology adoption. Environ. Sci. Technol. 54, 2941–2950.
- Lee, D. S. et al. (2021). The contribution of global aviation to anthropogenic climate forcing for 2000 to 2018. Atmos. Environ. 244, 117834.