Skip to content

Add AWI and UTIG/KOPRI helicopter data, fix handling of swath data, shared line-km rule - #1

Merged
thomasteisberg merged 10 commits into
mainfrom
add-direct-sources
Sep 22, 2026
Merged

Add AWI and UTIG/KOPRI helicopter data, fix handling of swath data, shared line-km rule#1
thomasteisberg merged 10 commits into
mainfrom
add-direct-sources

Conversation

@thomasteisberg

@thomasteisberg thomasteisberg commented Sep 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Brings three sources the BedMap + xOPR pipeline was missing into every figure, and makes the bar charts count line-km with the same gap rule as the density maps.

New sources (extra_sources.py, opr_tracks.py)

  • AWI radar sounders (EMR, UWB, UWBM) from the AWI map server WFS layer, dated per profile, 1994/95 to 2025/26. Replaces the AWI BedMap files, which held 185k km against 462k km here for 2001-2025 (it appears some AWI data was missing from BedMap between 2008-2014). Product duplicates (60/600 ns pulses, standard/qlook) are removed; residual repeat lines are within 16% of the geometric union per season.
  • KOPRI/UTIG helicopter radar over the Amundsen Sea (QIceRadar layers ASE2-ASE6, ~14.8k km), absent from BedMap and xOPR. Seasons 2017, 2019, 2021, 2023, 2025 (alternating Thwaites cruises); ASE3/ASE4 confirmed from TDR file names. Counted as no raw access (Level 1B/2 products only, no raw release).
  • xOPR real flight tracks from CReSIS per-segment CSVs for collections in OPR_SUBSTITUTIONS: the 2009 Twin Otter season (replacing the BedMap MUSIC swath file that cannot be chained) and the xOPR-only 2022-2024 seasons. opr_tracks.py --verify shows CSV tracks within 0-5% of STAC and within 10% of BedMap except the expected 2009 case.

Shared rules (bedmap_common.py)

  • BM2 files superseded by renamed BM3 files (8 RNRF + UTIG_2008_ICECAP, ~55k km double counted) are dropped.
  • One gap rule, MAX_POINT_SPACING_M = 1.5 km, for the density grids and the per-campaign line-km. Raised from 1 km so airborne surveys sampled at 1 km (SOAR-LVS-WLK, CASERTZ, MAMOG) are kept; traverses and 25 km-spaced GANOVEX files still drop out. Per-file km are cached in bedmap_campaign_km.csv (rebuild: uv run python bedmap_common.py).
  • TEMPORAL_OVERRIDES spreads the SOAR 1996-2001 file over its real years instead of the 1999-2000 placeholder.
  • campaign_years(): dated sources go to their season; BedMap keeps the calendar spread.

Figures

  • Bar charts extended to 2025. Combined chart reconciles xOPR only against the CReSIS/NASA/UTIG pool so other providers add on top (the old max(bedmap, open) clipped AWI 2022+ to zero). AWI is "commitment to release" throughout.
  • Country chart: y axis in thousands of km, larger fonts, USA at the bottom, opt-in --reference-line KM --reference-label TEXT instead of the fixed HAPS line.
  • Density and coastal scripts include the extra tracks (--no-extra to disable).
  • Pages site gains the by-country chart.

Antarctic combined totals (2001-2025 vs 2001-2023 before): open 558k (557k), commitment 710k (415k), no access 693k (709k).

Notes

  • CI now fetches ~500 MB of source data on demand into radar_cache/ (AWI GeoJSON, QIceRadar gpkg, CReSIS CSVs); caching that directory between runs would help.
  • PRIC/CHINARE lines after 2018/19 are deliberately out of scope (season labels unverified).
  • Analysis notes in claude_notes/ (gitignored) and the plan in claude_plans/20260914-add-awi-and-kopri-sources.md.

Test plan

  • uv run python opr_tracks.py --verify
  • Both bar charts, Antarctic density map, ASE zoom, coastal gaps regenerated from cache
  • CI run on this branch (workflow only triggers on main; use workflow_dispatch)

🤖 Generated with Claude, edited by a human

https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5

thomasteisberg and others added 8 commits September 14, 2026 14:39
@thomasteisberg
Unify deduplication across scripts
0b7cf90
@thomasteisberg @claude
Add AWI, KOPRI helicopter and xOPR-track sources to all figures
d3e49be 
- extra_sources.py: AWI ice-sounder tracks from the AWI WFS layer (dated per
  profile, product duplicates removed) replace the AWI BedMap files; KOPRI/UTIG
  helicopter surveys (QIceRadar ASE2-ASE6) with confirmed seasons.
- opr_tracks.py: real flight geometry from CReSIS per-segment CSVs for
  collections in bedmap_common.OPR_SUBSTITUTIONS (2009 Twin Otter swath case,
  xOPR-only 2022-2024 seasons); --verify compares CSV/STAC/BedMap lengths.
- bedmap_common: drop BM2 files superseded by renamed BM3 files, substituted
  files and AWI files; campaign_years() puts dated sources in their season.
- Bar charts extended to 2025; combined chart reconciles xOPR only against the
  CReSIS/NASA/UTIG pool so other providers add on top.
- Density and coastal scripts include the extra tracks (--no-extra to disable).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5
@thomasteisberg @claude
Bar charts use gap-filtered point line-km; shared 1.5 km gap rule
a65f728 
- bedmap_common: load_bedmap_points/point_segments hold the one gap rule
  (MAX_POINT_SPACING_M, raised 1 km -> 1.5 km to keep near-1 km airborne
  surveys while still dropping traverses and 25 km-spaced GANOVEX files);
  campaign_point_km() caches per-file km in bedmap_campaign_km.csv and
  load_bedmap_catalog fills line_km from it instead of catalog geometry.
- plot_survey_density.load_bedmap uses the shared functions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5
@thomasteisberg @claude
Country chart: --no-haps variant, y axis in thousands of km, larger f…
a79ab55 
…onts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5
@thomasteisberg @claude
Country chart: stack USA at the bottom
4c9957b 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5
@thomasteisberg @claude
Spread the SOAR 1996-2001 BedMap file over its real years (TEMPORAL_O…
1aa4510 
…VERRIDES)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5
@thomasteisberg @claude
Pages: add the by-country chart (no HAPS line)
ff5d449 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5
@thomasteisberg @claude
Country chart: opt-in --reference-line KM / --reference-label instead…
bc21cda 
… of --no-haps

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5
@thomasteisberg thomasteisberg changed the title Add AWI, KOPRI helicopter and xOPR-track sources; shared line-km rule Add AWI and UTIG/KOPRI helicopter data, fix handling of swath data, shared line-km rule Sep 16, 2026
thomasteisberg and others added 2 commits September 16, 2026 14:24
@thomasteisberg @claude
Fetch AWI tracks in filtered pages with retries (CI download was trun…
6e4a8f9 
…cated)

The single 245 MB GetFeature response was cut off on the CI runner. Use a
server-side radar_system filter (8.8k of 40.7k features) and 2000-feature
pages, and retry truncated downloads for all sources.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5
@thomasteisberg @claude
Cut density-map memory: per-file BedMap segments (float32) and chunke…
1bba79c 
…d gridding

The CI runner was killed on the density step (~20 GB peak after the shared
gap rule loaded all 81M points as one frame). bedmap_segments() now reads
one parquet file at a time and bin_line_km() grids in 5M-segment chunks;
peak is ~4.8 GB for the full run. Output and the km table are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019UvrC4evHHkT7qWWL96sD5
@thomasteisberg

thomasteisberg commented Sep 16, 2026

Copy link
Copy Markdown
Member Author

Deployment failure expected (can only deploy from main). Build passes, which is the important part

@thomasteisberg
thomasteisberg merged commit 0ea435f into main Sep 22, 2026
1 of 2 checks passed

This branch had an error being deployed

1 failed deployment
github-pages — 1bba79c8 Deployed Sep 16, 2026 by thomasteisberg via deploy #15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@thomasteisberg