Aggregation (consolidation) arm: measure the tier lever for the sub-Tier-2 products

[emmanuelmathot]

Motivation

For three of the benchmarked products (SWOT Raster100m, SWOT LakeSP, S2 Let-it-snow) the measured verdict is the same: the cloud-native output is sub-Tier-2 per delivered unit, so the object-grouping lever is not tiling, sharding or row-group size, it is aggregation across units. The CNES report states that recommendation (D2 ch. 6.4, 6.5, 7.3) but supports it with a first-order estimate, not a measurement: for example "order 60 to 80 MiB per UTM zone per cycle" is the per-tile COG size times a tile count, not a consolidated object that was written and read.

The harness converts one source component into one output object, so the aggregation lever is currently the one recommendation it cannot verify. This is the missing arm that turns the tier advice from arithmetic into evidence.

Ask

An aggregation (consolidation) arm: convert N source products into one output object per format, then measure it with the existing metric set.

• Raster (COG / GeoZarr). Mosaic the granules of a group into one object: for SWOT Raster100m the natural group is one UTM zone, one cycle; for LIS a tile across dates (which for GeoZarr means a time dimension, the time-cube shape already identified for S2).
• Vector (GeoParquet / FlatGeobuf). Concatenate the passes of a group into one file, with the row groups sized so a bbox query still prunes.
• Point cloud (COPC). Merge adjacent tiles into one octree, which is the CO3D question as much as the SWOT one.
• Config surface. A grouping key on the arm (something like aggregate.by: utm_zone+cycle or a regex capture over the product name) plus the group size, so a run can sweep "how many units per object" and show where the group crosses the Tier 2 and Tier 3 floors.
• What it must report. The aggregated object's size and tier verdict, the write cost of consolidating, and above all a subset read on the aggregate: the whole point is that a bigger object stays range-addressable, so a random bbox read on the consolidated object must be compared against the same read on the per-unit objects. If aggregation clears the tier but ruins partial access, that is the finding.

Acceptance

• An aggregation mode selectable per arm, unit-tested on synthetic multi-unit fixtures for at least the raster and vector paths.
• Committed runs for SWOT Raster100m (per UTM zone per cycle) and SWOT LakeSP (per cycle), reporting object count and size before and after, tier verdicts, write cost and subset-read latency both ways.
• The D2 estimate replaced by a measured number.

Touch points

src/cng_benchmark/runner.py (product grouping), the format adapters (cog.py, geozarr.py, geoparquet.py, copc.py), configs/benchmarks/, docs/configuration.md.

Related

• CNES study: D2 ch. 6.4, 6.5, 7.3; consolidated-metrics.md section 4 ("the other size regime: small products need aggregation").
• Feeds the LIS and CO3D issues in this batch, where the delivered unit is also sub-tier.

[emmanuelmathot]

Scope note: this issue also has to carry the Sentinel time cube, and one metric that does not exist yet

Re-reading what the Sentinel arms actually wrote, the aggregation question is larger than the sub-Tier-2 products this issue was opened for.

Every GeoZarr number in the study is a per-scene, per-band store. formats/geozarr.py says it outright ("Time-stacking the source scenes into a 3D cube is a separate, deferred concern … nothing here stacks"), and the committed S1 layouts confirm it: chunk_shape [1098, 1098], shard_shape [10980, 10980], one store per polarisation of one acquisition. So the benchmark compares a Zarr shaped exactly like a COG against that COG. That is a fair container comparison, and it is worth keeping as the control, but it is also the configuration in which Zarr has the least to offer: no time axis to amortise, and a display and read penalty against GDAL.

The consequence for the deliverable is sharp. The CNES report recommends COG as primary for Sentinel-1 and Sentinel-2 and names GeoZarr as the alternative for the time-cube / analysis mix. The measured evidence neither supports nor refutes that alternative, because the cube was never built. Meanwhile the time cube is the thing CNES explicitly asked to see demonstrated (Guillaume, 2026-05-13: one Zarr store per tile indexed by time, sharded over time to clear the tier, STAC items pointing into it with a slice parameter).

What to add here

• A stacked GeoZarr arm on Sentinel-1 RTC: one MGRS tile, N acquisitions, VV and VH, into one store chunked and sharded over (t, y, x). The adapter already accepts a 3D [t, y, x] chunk/shard shape and uses the trailing two dims, so the parameter surface exists; what is missing is a reader that presents N products as one stacked component.
• The same for Sentinel-2 L2A (path B in the report), where the time-stack is the stated analysis path.
• A time-series read metric. This is the decisive one and it does not exist today: metrics/read.py measures a random spatial window on a single object. The cube's whole claim is that a pixel or small window across all dates is one contiguous read instead of N file opens, so the comparison that matters is stacked-Zarr against the equivalent traverse over N per-scene COGs. Without that metric the stacked arm would only report size and tier, which is the least interesting half.

Feasibility: data is not the obstacle

A probe against sentinel1-s2l1c-sprid (2026-08-04) found 112 acquisition dates for T31TCJ in 2026 alone, across 8 months, at the expected S1 cadence:

T31TCJ/2026/01/01/, 01/02/, 01/06/, 01/07/, 01/08/, 01/12/, 01/13/, 01/14/, 01/18/, …

So a 12-date or 24-date stack over one tile is a run away, on data the runner already reads. Suggested first shape: one tile, 12 dates, VV and VH, chunk [1, 1098, 1098], shard over time so the object clears the tier, against the same 12 dates as per-scene COGs.

Why it is worth doing even under time pressure

It is the only arm in the remaining set that can change a recommendation rather than sharpen a number. Today the report says COG primary partly because the measured Zarr is slower; a cube that turns a 12-date time series into one contiguous read is where that ordering could reverse for the analysis mix. Either result is publishable: if the cube wins on time-series reads and loses on display, that is exactly the two-path recommendation the report already frames, but with evidence under it.

[emmanuelmathot]

Build the stacked S1 RTC store on EOPF-Explorer/data-model, and mind one hard conflict

Direction for the stacked arm: do not invent a private layout. EOPF-Explorer/data-model (the eopf-geozarr package) is the reference converter behind the EOPF Explorer, and the CNES report's whole positioning argument is that CNES should track EOPF rather than diverge. A benchmark that measures a bespoke store undercuts that argument, whatever the numbers say.

What to take from it

• The pyramid layout: overview levels as child groups 0/, 1/, 2/ under a resolution group, COG-style /2 downsampling with a configurable minimum dimension (default 256). This already matches what feat(geozarr): build a multiscale pyramid by default, described by the Zarr conventions #78 implemented, so the geometry agrees.
• Native CRS preservation: no reprojection to a web-mercator tile matrix set; the tile matrix set is expressed in the product's own CRS. Correct for S1 RTC on MGRS.
• The chunk-alignment helper (calculate_aligned_chunk_size), which exists precisely to stop chunk/encoding mismatches when writing through dask. Directly relevant to a (t, y, x) store.
• The entry points, so we call rather than reimplement: create_geozarr_dataset() and setup_datatree_metadata_geozarr_spec_compliant(), plus the CLI eopf-geozarr convert | info | validate.
• eopf-geozarr validate as an acceptance gate on the arm. This is the cheapest credibility we can buy: an external tool, not ours, asserting the store we benchmark is a faithful GeoZarr. The report leans on the claim that "a store without multiscales is not a faithful GeoZarr"; having the reference implementation validate our output makes that claim checkable rather than rhetorical.

The conflict, stated plainly

eopf-geozarr targets GeoZarr spec 0.4 and requires, on every array: _ARRAY_DIMENSIONS, CF standard_name, a grid_mapping attribute pointing at a CF grid-mapping variable, and GeoTransform on that variable.

Our multiscales module deliberately writes none of those. From its own docstring:

There is no CF spatial_ref grid-mapping variable duplicating the CRS and transform, no grid_mapping attribute pointing at one, and no _ARRAY_DIMENSIONS — Zarr v3 names an array's axes in its own dimension_names field.

The reasoning in #78 was sound (0.4 is deprecated, v1 has not landed, so write the published conventions v1 will assemble). But the consequence is concrete: the stores we benchmark today would fail eopf-geozarr validate, and CNES would be shown numbers for a store the ESA-side reference converter rejects.

Suggested resolution, cheapest first: emit both. The two metadata sets are additive, not contradictory — _ARRAY_DIMENSIONS, standard_name, grid_mapping and GeoTransform can sit alongside the zarr_conventions documents, costing a few hundred bytes per array and no restructuring. Then the store validates against the reference implementation and declares the conventions v1 will assemble, and the report can say exactly that. If that turns out to be unacceptable for some reason, the fallback is to state the divergence explicitly in the report rather than let it pass silently.

Where data-model stops, and why that is interesting

Its specification-contribution doc defers time handling to future work, and the package has no Sentinel-1-specific content. So for the two things this arm needs most, the time axis and the S1 RTC shape, there is nothing upstream to copy.

That is an opportunity rather than a blocker. The stacked-cube convention (how t is named, chunked and sharded, how per-date STAC items point into one store with a slice parameter, which is exactly Guillaume's 2026-05-13 ask) is a gap the EOPF work has flagged and invited contributions on. Doing it here and proposing it upstream turns a CNES requirement into a community contribution, which is squarely D1's remit and strengthens the D2 positioning chapter: CNES is not waiting on the convention, it is helping write the part it needs.

[emmanuelmathot]

Correction, and the part that actually matters: the S1 structure to stack against

My previous comment said EOPF-Explorer/data-model has no Sentinel-1-specific content. That was wrong, taken from a README summary rather than the tree. It has a full typed model of the Sentinel-1 product, src/eopf_geozarr/data_api/s1.py, and that, not the spec-version question, is what the stacked arm should follow.

The hierarchy (Sentinel1Root)

Root                                   attrs: stac_discovery, other_metadata
├── S01SIWGRD_<timestamp>_..._VH/      one group per polarisation
│   ├── conditions/                    antenna_pattern, attitude, orbit, gcp,
│   │                                  terrain_height, azimuth_fm_rate,
│   │                                  doppler_centroid, coordinate_conversion
│   ├── measurements/                  imagery + azimuth_time, ground_range, line, pixel
│   └── quality/                       calibration, noise (azimuth / range)
└── S01SIWGRD_<timestamp>_..._VV/      same, per polarisation

Two structural facts worth noting before designing anything:

• The EOPF product sits in radar geometry; geocoding is a separate step driven by the GCPs in conditions/gcp (conversion/sentinel1_reprojection.py). CNES S1Tiling RTC output is already gridded on MGRS, so we skip that step and inherit only the structure.
• The whole hierarchy is declared with pydantic-zarr GroupSpec / ArraySpec over closed=True TypedDicts (src/eopf_geozarr/pyz/). It is a machine-checkable contract, not a document. Our stacked store should declare its own spec the same way, so conformance is a test rather than a claim.

What it dictates for the stacked arm

1. Polarisation stays a group; time goes inside measurements. EOPF splits VV and VH at the top precisely because each carries its own conditions and quality. So the cube is (t, y, x) per polarisation group, not a 4-D array with a polarisation axis. That also keeps the per-polarisation shard sizing we already measured comparable.
2. Keep the measurements / conditions / quality split, even though an RTC tile needs far less of conditions than a GRD product does. It is what makes the store recognisably an S1 store to anyone who knows EOPF, which is the whole point of aligning.
3. Per-acquisition metadata becomes time-indexed. Orbit, attitude, incidence angle, calibration and noise LUTs are per-scene upstream. Stacked, each grows a leading time axis or moves under a time-indexed conditions/. This is the real design work in this arm.
4. stac_discovery at the root is the hook for the CNES ask. The store already carries its own STAC metadata; stacked, that becomes one entry per acquisition, which is exactly the "one STAC item per scene, asset pointing into the same store with a slice parameter" pattern Guillaume asked for on 2026-05-13. The field exists upstream; the stacked semantics do not.

Net

Points 1, 2 and 4 are reuse. Point 3 is new work, because time is the one axis the upstream model has not addressed (its spec-contribution doc defers time handling explicitly). So the stacked S1 RTC convention is worth writing up and sending back upstream rather than keeping local, which is D1 work and strengthens the D2 positioning argument at the same time.

[emmanuelmathot]

Out of scope for the current CNES contract (decided 2026-08-05). Both halves of this issue, consolidating sub-Tier-2 products and the Sentinel time-stacked cube, are carried to a possible phase 2 rather than the final campaign.

Nothing here is blocked by data or tooling: the S1 archive holds 112 acquisition dates for T31TCJ in 2026, the adapter already accepts a 3D [t, y, x] chunk/shard shape, and the structure to follow is settled (the EOPF Sentinel-1 hierarchy, see the comment above). It is a scope decision, so the design notes in this issue stay as the starting point if a second phase happens.

The study report now states both as out of scope and marks the affected recommendations as reasoned rather than measured: the aggregation lever for the sub-Tier-2 products is presented as sizing arithmetic, and the analysis-side GeoZarr option for S1 / S2 rests on the format's properties, since every GeoZarr number in the report is the per-scene form.

Keeping this issue open as the phase-2 entry point rather than closing it.