docs(design): add the zarr-specs#399 many-file cases and a per-chunk codec sketch

design/experiments/inhomogeneous/output.txt

@@ -0,0 +1,61 @@
+
+==============================================================================
+1. codec eras: gzip, gzip, shuffle + gzip, uncompressed (GOES-16)
+------------------------------------------------------------------------------
+  era_0.nc: shape=(8, 6) chunks=(4, 6) float32 gzip
+  era_1.nc: shape=(8, 6) chunks=(4, 6) float32 gzip
+  era_2.nc: shape=(8, 6) chunks=(4, 6) float32 gzip+shuffle
+  era_3.nc: shape=(8, 6) chunks=(4, 6) float32 uncompressed
+  one Zarr array today: not one array: NotImplementedError: The ManifestArray class cannot concatenate arrays which were stored using different codecs, But found codecs (BytesCodec(endian='little'), Zlib(codec_name='numc
+  switch sketch:        one array, 3 switch cases: identical to the files (bit for bit)
+                        the archive is 4,147 bytes; the chunks stay in the 4 files (44,339 bytes), read in place
+
+==============================================================================
+2. rectilinear: monthly files of 31, 28 and 31 days, one chunk each
+------------------------------------------------------------------------------
+  month_0.nc: shape=(31, 6) chunks=(31, 6) float32 gzip
+  month_1.nc: shape=(28, 6) chunks=(28, 6) float32 gzip
+  month_2.nc: shape=(31, 6) chunks=(31, 6) float32 gzip
+  one Zarr array today: not one array: ValueError: Cannot concatenate arrays with inconsistent chunk shapes: (28, 6) vs (31, 6) .Requires ZEP003 (Variable-length Chunks).
+  switch sketch:        one array, 2 switch cases: identical to the files (bit for bit)
+                        the archive is 3,698 bytes; the chunks stay in the 3 files (34,740 bytes), read in place
+
+==============================================================================
+3. ragged inner chunks: 10 steps per file chunked by 4 (4, 4, 2)
+------------------------------------------------------------------------------
+  ragged_0.nc: shape=(10, 6) chunks=(4, 6) float32 gzip
+  ragged_1.nc: shape=(10, 6) chunks=(4, 6) float32 gzip
+  ragged_2.nc: shape=(10, 6) chunks=(4, 6) float32 gzip
+  one Zarr array today: not one array: ValueError: Cannot concatenate arrays with partial chunks because only regular chunk grids are currently supported. Concat input 0 has array length 10 along the concatenati
+  switch sketch:        one array, 1 switch case: identical to the files (bit for bit)
+                        the archive is 3,811 bytes; the chunks stay in the 3 files (33,433 bytes), read in place
+
+==============================================================================
+4. per-file scale_factor / add_offset on int16 (BOM BARPA)
+------------------------------------------------------------------------------
+  scaled_0.nc: shape=(10, 6) chunks=(5, 6) int16 gzip ×0.01 +280
+  scaled_1.nc: shape=(10, 6) chunks=(5, 6) int16 gzip ×0.02 +270
+  scaled_2.nc: shape=(10, 6) chunks=(5, 6) int16 gzip ×0.005 +290
+  one Zarr array today: combined, with the first file's attributes for every file (combine_attrs='override'; 'no_conflicts' refuses instead): max |difference| 19.6; 120 of 180 values off by more than 0.001
+  switch sketch:        one array, 1 switch case: identical to the files (bit for bit)
+                        the archive is 3,849 bytes; the chunks stay in the 3 files (33,190 bytes), read in place
+
+==============================================================================
+5. varying dtype: int16, int16, float32
+------------------------------------------------------------------------------
+  dtype_0.nc: shape=(10, 6) chunks=(5, 6) int16 gzip
+  dtype_1.nc: shape=(10, 6) chunks=(5, 6) int16 gzip
+  dtype_2.nc: shape=(10, 6) chunks=(5, 6) float32 gzip
+  one Zarr array today: not one array: ValueError: Cannot combine arrays with inconsistent dtypes, but got 2 distinct dtypes: {dtype('<f4'), dtype('<i2')}
+  switch sketch:        one array, 2 switch cases: identical to the files (bit for bit)
+                        the archive is 3,759 bytes; the chunks stay in the 3 files (33,173 bytes), read in place
+
+==============================================================================
+6. concatenating two combined arrays (gzip, shuffle + gzip | gzip, uncompressed)
+------------------------------------------------------------------------------
+  array A: 2 cases ['7508e002dae26618', 'b86f3102cff388cc']
+  array B: 2 cases ['7508e002dae26618', 'e2879ab0551143ab']
+  A + B:   3 cases (the shared gzip case once); every chunk header copied unchanged: True; codecs equal to combining all four files at once: True
+  concatenated, saved and reopened: identical to the files (bit for bit)
+
+figures: <repo>/design/experiments/inhomogeneous/figures

design/experiments/inhomogeneous/scaling_output.txt

@@ -0,0 +1,7 @@
+  chunks  design                       zarr.json    archive     open   read 1  checked
+   1,000  one case per chunk             401.7kB     0.54MB    0.02s    2.6ms  200 chunks bit for bit
+   1,000  one case, two parameters         0.8kB     0.17MB    0.00s    1.4ms  200 chunks bit for bit
+  10,000  one case per chunk           4,014.2kB     5.41MB    0.25s    1.8ms  200 chunks bit for bit
+  10,000  one case, two parameters         0.8kB     1.72MB    0.07s    1.8ms  200 chunks bit for bit
+ 100,000  one case per chunk          40,139.3kB    54.43MB    2.98s    1.8ms  200 chunks bit for bit
+ 100,000  one case, two parameters         0.8kB    17.50MB    0.25s    1.7ms  200 chunks bit for bit