docs(design): add the zarr-specs#399 many-file cases and a per-chunk codec sketch

[d-v-b]

🤖 AI text below 🤖

This adds an executable demo, design/experiments/inhomogeneous/, of the five many-file cases in zarr-developers/zarr-specs#399, and a sketch of the issue's design 2.1 (per-chunk codec selection) that expresses each case as one Zarr array. VirtualiZarr parses every file in both paths. The demo shows what a single Zarr array can and can't hold today, and how per-chunk context in refkv reference values could fill the gap.

case	one Zarr array today (concatenated virtual datasets)	switch sketch
1. codec eras (gzip, shuffle + gzip, uncompressed)	not expressible: one codec chain per array	bit for bit
2. monthly files (31, 28, 31 days)	not expressible: needs variable-length chunks (ZEP003)	bit for bit
3. ragged inner chunks (4, 4, 2)	not expressible: partial chunks inside a regular grid	bit for bit
4. per-file CF scale_factor / add_offset	one set of attributes for all files (see below)	bit for bit
5. varying dtype (int16, float32)	not expressible: one dtype per array	bit for bit
6. concatenating two combined arrays	n/a	bit for bit: case tables merge by union, every chunk header copied as is

In cases 1, 2, 3 and 5, VirtualiZarr refuses to concatenate and says what is missing. That is the safe behavior, and the gap it reports is the issue's subject.

Case 4 isn't specific to any tool. CF scaling is array-level attributes, so concatenating has to pick one set:

• "override" (the default in xr.concat and open_virtual_mfdataset) keeps the first file's, and the other files decode off by up to 19.6.
• "no_conflicts" refuses.
• "drop_conflicts" leaves the raw integers.

The sketch (its own section of demo.py) works like this:

• Each per-file difference is its own codec. These are the file's own compression codecs, zarr-extensions cast_value and scale_offset, and a new crop for HDF5's whole-stored edge chunks.
• A generic switch codec picks, per chunk, which chain applies.
  • Cases are templates. A case may leave numbers open ({"parameter": k}).
  • Cases are keyed by content. A case's key is the first 8 bytes of the SHA-256 of its canonical JSON.
  • The header fills them in. A chunk's header is its case's key followed by the case's parameters.
  • zarr.json holds the kinds of encoding; per-chunk numbers travel with each chunk. Case 4's three files share one case.
• refkv reference values carry that index. Each chunk is Concat(Range(data=<varint>), Range(source=<file>, offset, length)), so no store injects anything at read time and no codec does I/O.
• Month-length and ragged files use the zarr-extensions rectilinear chunk grid.

Scaling (uv run demo.py scaling): every chunk is packed with its own CF scale and offset, as GRIB packs each message.

chunks	one case per chunk: zarr.json / open	one case, two parameters: zarr.json / open
1,000	402 kB / 0.01 s	0.8 kB / 0.00 s
10,000	4.0 MB / 0.25 s	0.8 kB / 0.02 s
100,000	40.1 MB / 2.96 s	0.8 kB / 0.27 s

Both read 200 sampled chunks bit for bit, and reading one chunk takes about 2 ms either way. With parameters, what still grows is the archive's own key index, which any archive of 100,000 chunks has.

The README explains why the chooser stays format-agnostic, how it composes (outer codecs, nested switch, sharding, the conditional precedent, manifest-level concatenation), and its limits.

How results are checked:

• Reference: each file read on its own by xarray and h5netcdf, which apply HDF5's filters, CF scaling and _FillValue.
• Comparison: bit for bit (np.array_equal, NaN equal to NaN); otherwise the largest difference is reported.
• Data: one smooth field continues across the files, so a misdecoded file shows as a break at its boundary in figures/.

Notes

• To run it: cd design/experiments/inhomogeneous && uv run demo.py (and uv run demo.py scaling). The dependencies are declared inline (PEP 723): refkv from PyPI, VirtualiZarr, cast-value-rs (which zarr-python's cast_value needs) and matplotlib. uv builds an environment for this script alone. The agent checked the run by copying demo.py alone into an empty directory.
• cast_value fill-value bug: while building this, the agent found that zarr-python accepts cast_value metadata whose fill value can't round-trip, which the cast_value spec says MUST be an error. The fix is the draft zarr-developers/zarr-python#4510. The demo maps NaN to each file's CF _FillValue with scalar_map.
• Status of the new codecs: switch and crop are defined only in this demo; they are not registered zarr extensions.
• Scope: the PR only adds files under design/experiments/inhomogeneous/, with no dependency on other PRs.

🤖 Generated with Claude Code