GeoZarr's multiscale pyramid fragments the tier verdict: S1 drops Tier3->Tier2, S2 drops to no tier, despite matched total bytes

[emmanuelmathot]

Distinct from #102: this is within one component, bundling won't touch it

#102/786fe51 fixes duplication across components sharing a grid (coordinate arrays and pyramid metadata written once per component instead of once per store). This issue is about fragmentation within a single component's own multiscale pyramid, which bundle_components does not address, because it is not a duplication problem, it is a structural consequence of Zarr's multiscale convention: each pyramid level is its own array with its own shape, so it necessarily gets its own shard file(s), its own zarr.json, and (today) its own x/y coordinate arrays.

What was measured

Final CNES campaign, wave 4, harness 0.7.1 (pre-#102), one component per store (today's default even after #102 lands, unless bundle_components is set):

S1 RTC, VV band, one scene, chunk_shape: [1098,1098], shard_shape: [10980,10980], multiscale_levels: 5: 48 physical objects for one array. Level 0 (native res) is one 96.9 MB shard; levels 1-5 are 1-3 shards each, shrinking fast (24.1 MB, then 4.1/1.1/0.77 MB split across 3 shards, then 370 KB, then 94 KB); every level also carries its own data/zarr.json, x/zarr.json, y/zarr.json (3 metadata files) and its own x/y coordinate chunk (2 more files), so 5 sidecar files per level x 6 levels = 30 of the 48 objects are metadata or coordinates, mostly under 20 KB.

Aggregate consequence, whole runs:

	S1 RTC (2 bands x 20 scenes)	S2 MAJA (28 bands x 16 scenes)
Harness's own object count (object_profile.count, shard-data only)	448	4305
Mean	36.7 MiB	6.05 MiB
Median	1.58 MiB	0.41 MiB
Tier verdict	tier2-disk (COG: tier3-tape)	none at all (COG: tier2-disk)
Raw physical object count (everything under the store, coordinates + metadata + data)	1948	19361

The tier verdict drops a full tier for both missions, despite total store bytes being within 0.3-2% of COG's (the pyramid-completeness and matched-codec corrections earlier in this campaign made the byte totals converge; this is what's left after that). And this is the optimistic number: object_profile.count already excludes metadata and coordinate files (448/4305), which are themselves real S3 objects a Datalake lifecycle policy tiers individually. Counted honestly (everything under the store), the picture is 4.3-4.5x worse in object count, dragged even further toward tiny objects.

Why bundling doesn't fix this

bundle_components shares the coordinate arrays and group metadata across VV/VH or the 28 MAJA bands, which helps: for S1 it would cut the ~30 metadata/coordinate objects per component down to one shared set (a real, worthwhile win, since 2 components today pay for coordinates twice). But every component still needs its own 6-level pyramid, and each level of that pyramid is still its own array, still needs its own shard file(s) regardless of bundling. Bundling reduces the metadata/coordinate multiplier; it does not touch the multiplier from pyramid depth itself, which is the larger contributor by count (30 of 48 objects were metadata/coordinates in the S1 sample, but the bytes problem, the reason median collapses to 1.58 MiB, comes from the levels themselves being genuinely small: level 5 was 94 KB of actual data).

Why COG doesn't have this problem

A COG's overview pyramid lives as embedded IFDs inside one physical file. Adding overviews costs bytes (confirmed already: overview_bytes on the GeoZarr layouts) but never costs objects, because GDAL writes them into the same TIFF. GeoZarr's multiscale convention, as implemented (one array per level, multiscales per the Zarr conventions), makes overview depth pay in object count as well as bytes, in a way that has no COG analogue and, as far as I can tell, no way around within the current Zarr v3 sharding model, since a shard is scoped to one array and each level is a different array.

Ask

Not asking for a full fix in this issue, since I'm not certain there is a clean one, but three things worth doing:

1. Confirm the diagnosis is right — is there a Zarr v3 mechanism I'm missing that could pack multiple pyramid levels' shards into one physical object? (Consolidated metadata is a read-side convention, not a write-side object-count reducer, as far as I understand it; flag if I'm wrong.)
2. If there is no fix, make the tradeoff explicit and default-visible. Right now a GeoZarr arm's summary.md reports total bytes and overview_bytes but nothing about the tier consequence of pyramid depth. A pyramid_object_overhead or similar on GeoZarrLayout, or a warning in the tier section when multiscale_levels > 0 measurably drops the verdict versus a flat store, would keep this from being rediscovered the way Bundle multi-component conversions into one store (GeoZarr) or offer multi-band (COG) instead of one object per component #102 was.
3. Report honestly on whatever ships. If #102's honest enumerate_store_objects (including coordinates) becomes the default reported number, the CNES report should quote whichever count Datalake would actually see, not the shard-only one; happy to align on which once you've decided.

Where this lands in the study

This is now the dominant open question for chapters 6.1/6.2. The campaign already closed "codec explains the gap" (measured: 0.3-2.0% effect, not ~25%) and "no pyramid explains the gap" (fixed, sizes converge to within 2%). What's left, and what this issue is about, is that the pyramid GeoZarr needed to converge on size is exactly what fragments its tier verdict. Correcting for cross-component duplication (#102) will help but, on the evidence above, will not close this by itself.

Related

#102 (cross-component duplication, the fix already on main, unreleased). CNES study: D2 ch. 6.1/6.2, testbed/ops/final-campaign.md wave 4.