Radiometric calibration checks: image-combine seams and surface saturation

[thomasteisberg]

Human-written context: I found a couple of types of calibration issues from other work. This PR adds metrics that allow downstream data users to filter out these types of calibration issues. It does not remove any data from the dataset -- decisions about how to filter are left to downstream users.

The two calibration issues targeted are:

1. Evidence of surface saturation (which artificially reduces the power different between bed and surface)
2. Power jumps at the image combine seams resulting from mis-calibration of multiple channels or multiple waveforms when the radar data was processed

Flagging the image combine issues requires that this pipeline downloads all of the individual images as well as the combined version. This increases the data downloads by almost 4x (most frames have 3 images plus the combined), but everything still seems to run fine and the total dataset build for both ice sheets still takes less than 2 days on a single machine.

Claude description:

Adds two radiometric calibration checks to the pipeline and exposes their results as per-trace/per-frame values for downstream filtering (values and statuses, not verdicts).

What's included

• Image-combine seam check (per frame, on by default): recovers each image's effective combine scaling empirically (w_i = median(combined − img_i) over its sourced section — works regardless of img_comb_weights_mode, including auto seasons with >100 dB scalings), then measures the residual seam step in the overlap of weight-corrected images. New per-trace img_comb_offset_dB / img_comb_pair, per-frame status/mean attrs, full status vocabulary; failures degrade to statuses and never skip a frame.
• Surface saturation second pass (store-level, no downloads): fits per-season clip ceilings via a binned upper-quantile flat-envelope rule (Theil–Sen), separately per surface-source-image population, with sparse-adjacent-bin merging to keep thin-tail range support. New per-trace surface_source_image_index (provenance flag) and surface_ceiling_margin_dB (NaN where no credible ceiling), season dict with levels/CI/pileup/support and measured img2 bias (cross_cap_step_db).
• Pipeline integration: config-normalized default-on, typed frame attrs, auto-migrating appends (removes a silent variable skip), calibration-only backfill CLI (parallel, resumable), science-data fingerprint for second-pass staleness; viewer int8 handling (fixes a latent float-reinterpret bug); docs incl. docs/dataset_changelog.md with suggested QC filters.
• Param support back to the 2012-era layout (param_combine_wf_chan, old 2-element img_comb, wfs-as-list-of-dicts) and a mid-ice-record-tail noise-floor guard.

Validation

• 76 unit tests; regression check re-processed 5 production frames across both stores with all 25 pre-existing variables byte-identical.
• Full backfill of ase (815), greenland (5,698), antarctica (6,147 frames); per-store sha256 verification: science arrays untouched, only calibration fields added.
• ASE store updated on S3 (rehearsal); greenland/antarctica pushed alongside this PR.

Key findings shipped in the data: systematic seam errors in 2016_Greenland_P3 (+7.4 dB), 2017/2013_Antarctica_Basler; saturation ceilings for 2012_Antarctica_DC8 (−30 dB), 2014_Greenland_P3 (−31.6 dB, 16.2 dB img2 bias step), 2013_Greenland_P3 (−50.6 dB).

🤖 Generated with Claude Code

https://claude.ai/code/session_01YPMWF1uxKRL2pmtgdyJuM8