🤖 from Claude
Design/scoping issue capturing an investigation into how to carry ATL03 photon signal confidence through zagg's t-digest aggregation, so that a signal-weighted photon-flux-vs-height distribution can be produced per grid cell. Refs #265 (HHDC / photon flux — this is a sub-concern of that product), #281 (declarative t-digest), #217 (chunk-scoped digest streaming). No code yet; this is a decision-space writeup with current leans for review.
All numbers below were measured on real v007 granules (Greenland RGT 1336 across seasons, and 61 NEON passes) with h5py; ATBD/data-dict citations are from the v007 NSIDC docs.
1. Background findings (established)
signal_conf_ph is a per-surface categorical, not a probability. Shape [N_photons, 5], int8; the 5 columns are independent signal-finding passes per surface type (order: land, ocean, sea_ice, land_ice, inland_water). Domain {-2 possible_tep, -1 not_considered, 0 noise, 1 buffer, 2 low, 3 med, 4 high}.
- The ordinal is misleading — there is a hard break between 1 and 2. The ATBD's own signal predicate is
signal_conf_ph > 1 (so {0,1} = background, {2,3,4} = signal).
1 = buffer is not a weak-signal tier: it is the vertical padding collar added so the signal set spans ≥20 m (ATBD §5.1.1) — background photons hugging the surface.-1 = not_considered means the per-surface mask didn't run there.- v007 also ships
weight_ph (uint16 0–65535, DDA/RBF photon density = a genuinely continuous per-photon score) and experimental signal_class_ph (categorical, thresholded from weight_ph).
Confidence is not reconstructable from an aggregated digest. It is a Poisson-outlier density classification in along-track × vertical histograms at ~20 m rate, with SNR tiers SNRbin = Hz/μ_bg where μ_bg comes from the external atmospheric-histogram stream, plus DEM gating, per-surface tuning, and slant histogramming (ATBD §5.1.4.2). Aggregation destroys the along-track axis, the background rate, and slope — so the signal/background decision must happen at ingest; it cannot be recomputed post-aggregation.
Signal fraction is driven by solar background, not surface or latitude — it ranges ~11–91%. Same Greenland ice track across seasons:
| regime |
solar elev |
background |
signal % (of evaluated) |
| night (Dec) |
−33° to −40° |
0.01 MHz |
~91% |
| day (Jun) |
+30° to +47° |
7–10 MHz |
~20% (up to 85% noise) |
NEON mixed passes: 40% signal / 60% background in aggregate, bimodally spread 3%→99.6% per pass. Two earlier assumptions are therefore retracted: (a) "background is small, store just a count" — false in daylight, where background is the majority; (b) "signal is ~99% high, so tiny low/med lanes" — the high-fraction of signal drops to ~43% in bright daylight, with low+med climbing to ~50%.
No clean single-surface label exists in the product. geolocation/surf_type [n_seg, 5] is per-segment and multi-hot ("which types are possible", 0/1) — ~49% of segments flag 3 types at once. There is no per-photon surface type and no NASA-provided "primary" surface. So any per-cell surface choice is ours to define (a priority rule keyed on the multi-hot mask), broadcast segment→photon via the existing ph_index_beg/segment_ph_cnt link.
The multi-column split confounds — quantified. Using "any column > 1" vs "the land_ice column > 1", the two disagree on 2% (day) to 10% (night) of photons. ~80–86% of conflicts are soft (buffer in one surface, signal ≥2 in another — the same near-surface photon under different pad windows); ~18% of night conflicts are hard (high in one column, noise in another) ≈ 1.8% of all photons. The confound only exists because the mask flags multiple surfaces (96% concordance across land/land_ice/inland_water).
DDA vs categorical (cross-tab, both regimes). signal_class_ph ≥ 1 vs categorical > 1: agreement 91–97%, but phi (Matthews) only 0.36–0.76 — best in daylight (0.69–0.76), i.e. where signal-weighting matters. The DDA is systematically ~7% more inclusive (picks up the buffer-collar photons), one-directional. weight_ph tracks categorical confidence well in daylight (Pearson r = 0.886, monotonic noise→high) but weakly at night (r = 0.585). The DDA is surface-agnostic (one call per photon → no confound) but experimental.
2. Decision space (with current leans)
(A) How to select signal photons at ingest.
- (A1) Union:
any active column > 1 — grab all photons any surface calls signal. ← current lean. It is an idempotent OR (order-independent), decision-free (no primary-surface rule), and reversible (a union can always be narrowed downstream; a committed single-column subset cannot be widened). Cost: includes the ~1.8% hard-contradiction + cross-surface buffer-collar photons (defensible — a real return some tuning found).
- (A2) Commit to one surface column (the product's surface). Cleaner/tighter per NASA convention, but requires the self-defined primary-surface rule and is not reversible. Recoverable as a special case of (A1) if the presence field is kept.
- (A3) DDA
signal_class_ph ≥ 1 or a weight_ph threshold. Confound-free single call, good daytime correlation, but experimental and ~7% looser than categorical.
(B) Digest structure. Lean: two strata per cell — a signal digest and a noise/background digest (partition at >1). On-demand merge reconstructs the total ("photon flux") distribution; each digest's total weight is the exact stratum photon count, so the signal/noise count split is free and needs no separate field. Merging two strata is a single deterministic 2-way t-digest merge (the non-associativity only bites the spatial fan-out, not this recombination). Open sub-choice: full noise digest vs. a scalar noise count when only its magnitude is wanted.
(C) Companion 64-bit field. Lean: per-surface (and/or per-level low/med/high) composition, stored as quantized fractions, merged as digest-weighted means. Because the digest already holds the exact total, the field only needs composition, not magnitude — so fractions (bounded [0,1]) never saturate regardless of cell size, and f_merged = (N_A·f_A + N_B·f_B)/(N_A+N_B) is an order-independent monoid using the digest weight. Counts-on-demand = fraction × N_signal; presence is the degenerate fraction > 0. ~5 surfaces × 8 bits = 40 bits, fits with room.
- Note the per-surface counts overlap (multi-hot) — they are marginals,
Σ > N_signal, and cannot split the digest into per-surface sub-distributions. They also carry little information away from surface boundaries (96% concordance). Alternatives considered: OR-presence semilattice only (cheapest, provenance without counts); 3-bit-per-column ordinal bounding interval; full per-(surface,level) presence bitset (5×7 = 35 bits, lossless single-obs).
(D) What stays out of scope. Reconstructing a per-surface height distribution requires per-surface strata (K digests), not a packable field — deferred unless a product needs land_ice-only-vs-land distributions distinctly.
3. Open questions for review
- Is the aggregation weighting between cells (weight each cell by its signal content) or within cell (reshape the height PDF)? Between-cell is fully served by the companion field + digest weights; within-cell tunable weighting needs the two strata in (B).
- Confirm the selection lean (A1) union over (A2) commit / (A3) DDA. If a specific product wants tighter surface fidelity, (A2)/(A3) can layer on.
- Is per-surface composition (C) worth 40 bits given 96% concordance, or is OR-presence provenance enough for v1?
- Threshold: keep
> 1 (ATBD-standard) as the signal cut, or expose it (e.g. == 4 high-only) as a config knob?
References
🤖 from Claude
Design/scoping issue capturing an investigation into how to carry ATL03 photon signal confidence through zagg's t-digest aggregation, so that a signal-weighted photon-flux-vs-height distribution can be produced per grid cell. Refs #265 (HHDC / photon flux — this is a sub-concern of that product), #281 (declarative t-digest), #217 (chunk-scoped digest streaming). No code yet; this is a decision-space writeup with current leans for review.
All numbers below were measured on real v007 granules (Greenland RGT 1336 across seasons, and 61 NEON passes) with
h5py; ATBD/data-dict citations are from the v007 NSIDC docs.1. Background findings (established)
signal_conf_phis a per-surface categorical, not a probability. Shape[N_photons, 5], int8; the 5 columns are independent signal-finding passes per surface type (order: land, ocean, sea_ice, land_ice, inland_water). Domain{-2 possible_tep, -1 not_considered, 0 noise, 1 buffer, 2 low, 3 med, 4 high}.signal_conf_ph > 1(so{0,1}= background,{2,3,4}= signal).1 = bufferis not a weak-signal tier: it is the vertical padding collar added so the signal set spans ≥20 m (ATBD §5.1.1) — background photons hugging the surface.-1 = not_consideredmeans the per-surface mask didn't run there.weight_ph(uint16 0–65535, DDA/RBF photon density = a genuinely continuous per-photon score) and experimentalsignal_class_ph(categorical, thresholded fromweight_ph).Confidence is not reconstructable from an aggregated digest. It is a Poisson-outlier density classification in along-track × vertical histograms at ~20 m rate, with SNR tiers
SNRbin = Hz/μ_bgwhereμ_bgcomes from the external atmospheric-histogram stream, plus DEM gating, per-surface tuning, and slant histogramming (ATBD §5.1.4.2). Aggregation destroys the along-track axis, the background rate, and slope — so the signal/background decision must happen at ingest; it cannot be recomputed post-aggregation.Signal fraction is driven by solar background, not surface or latitude — it ranges ~11–91%. Same Greenland ice track across seasons:
NEON mixed passes: 40% signal / 60% background in aggregate, bimodally spread 3%→99.6% per pass. Two earlier assumptions are therefore retracted: (a) "background is small, store just a count" — false in daylight, where background is the majority; (b) "signal is ~99% high, so tiny low/med lanes" — the high-fraction of signal drops to ~43% in bright daylight, with low+med climbing to ~50%.
No clean single-surface label exists in the product.
geolocation/surf_type[n_seg, 5]is per-segment and multi-hot ("which types are possible", 0/1) — ~49% of segments flag 3 types at once. There is no per-photon surface type and no NASA-provided "primary" surface. So any per-cell surface choice is ours to define (a priority rule keyed on the multi-hot mask), broadcast segment→photon via the existingph_index_beg/segment_ph_cntlink.The multi-column split confounds — quantified. Using "any column > 1" vs "the land_ice column > 1", the two disagree on 2% (day) to 10% (night) of photons. ~80–86% of conflicts are soft (buffer in one surface, signal
≥2in another — the same near-surface photon under different pad windows); ~18% of night conflicts are hard (highin one column,noisein another) ≈ 1.8% of all photons. The confound only exists because the mask flags multiple surfaces (96% concordance across land/land_ice/inland_water).DDA vs categorical (cross-tab, both regimes).
signal_class_ph ≥ 1vs categorical> 1: agreement 91–97%, but phi (Matthews) only 0.36–0.76 — best in daylight (0.69–0.76), i.e. where signal-weighting matters. The DDA is systematically ~7% more inclusive (picks up the buffer-collar photons), one-directional.weight_phtracks categorical confidence well in daylight (Pearson r = 0.886, monotonic noise→high) but weakly at night (r = 0.585). The DDA is surface-agnostic (one call per photon → no confound) but experimental.2. Decision space (with current leans)
(A) How to select signal photons at ingest.
any active column > 1— grab all photons any surface calls signal. ← current lean. It is an idempotent OR (order-independent), decision-free (no primary-surface rule), and reversible (a union can always be narrowed downstream; a committed single-column subset cannot be widened). Cost: includes the ~1.8% hard-contradiction + cross-surface buffer-collar photons (defensible — a real return some tuning found).signal_class_ph ≥ 1or aweight_phthreshold. Confound-free single call, good daytime correlation, but experimental and ~7% looser than categorical.(B) Digest structure. Lean: two strata per cell — a signal digest and a noise/background digest (partition at
>1). On-demand merge reconstructs the total ("photon flux") distribution; each digest's total weight is the exact stratum photon count, so the signal/noise count split is free and needs no separate field. Merging two strata is a single deterministic 2-way t-digest merge (the non-associativity only bites the spatial fan-out, not this recombination). Open sub-choice: full noise digest vs. a scalar noise count when only its magnitude is wanted.(C) Companion 64-bit field. Lean: per-surface (and/or per-level low/med/high) composition, stored as quantized fractions, merged as digest-weighted means. Because the digest already holds the exact total, the field only needs composition, not magnitude — so fractions (bounded
[0,1]) never saturate regardless of cell size, andf_merged = (N_A·f_A + N_B·f_B)/(N_A+N_B)is an order-independent monoid using the digest weight. Counts-on-demand =fraction × N_signal; presence is the degeneratefraction > 0. ~5 surfaces × 8 bits = 40 bits, fits with room.Σ > N_signal, and cannot split the digest into per-surface sub-distributions. They also carry little information away from surface boundaries (96% concordance). Alternatives considered: OR-presence semilattice only (cheapest, provenance without counts); 3-bit-per-column ordinal bounding interval; full per-(surface,level) presence bitset (5×7 = 35 bits, lossless single-obs).(D) What stays out of scope. Reconstructing a per-surface height distribution requires per-surface strata (K digests), not a packable field — deferred unless a product needs land_ice-only-vs-land distributions distinctly.
3. Open questions for review
> 1(ATBD-standard) as the signal cut, or expose it (e.g.== 4high-only) as a config knob?References
surf_type/signal_conf_ph/weight_ph/signal_class_phdefinitions.