[rhoadesScholar]

Registers the offset chunk key encoding: the default encoding applied to the chunk grid
coordinate plus a fixed per-dimension integer offset, so that adding k whole chunks before
array index 0 is two metadata edits (shape[a] += k * chunk_shape[a], offset[a] -= k) and
no chunk changes its key.

Array indexing stays zero-based; shape, the regular chunk grid and the codecs keep their
meaning. With an all-zero offset the keys are byte-for-byte those of default. An
implementation that does not know the name refuses the array, which is why this is a new
encoding rather than a field on the regular grid (current readers would ignore such a field
and address every chunk from grid coordinate 0).

This is the chunk-level extension named as the alternative when zarr-specs#122 / #144 declined
a change to array indexing (rabernat: "an optional extension that would operate at the chunk
level, i.e. creating chunks like -1.0.0"; d-v-b: "explicitly tracking the name of the first
chunk in some metadata"), and the direction jbms pointed at when closing zarr-specs#9. It
keeps to the one field it needs; sub-chunk growth and a user-facing non-zero array origin are
out of scope.

What it costs today versus with the encoding. Growing a default-encoded array by two
chunk rows in front, on one machine (NVMe, zarr-python 3.4.1), wall seconds:

chunks (64³ uint8)	rename every chunk file	copy to a new array	offset: one zarr.json write
512	0.03	0.32	0.004
4 096	0.13	3.1	0.006
32 768 (8 GiB)	1.0	26.5	0.016

The rename column is the best case and only exists on a filesystem; on an object store the
rename is a copy + delete per chunk (44 s at 32 768 chunks through the zarr store API). The
offset column does not grow with the array. Script and raw rows:
https://github.com/rhoadesScholar/zarr-chunk-offset/tree/main/bench

Implementations

• zarr-python plugin, the reference: https://github.com/rhoadesScholar/zarr-chunk-offset
  (zarr.chunk_key_encoding entry point, zarr-python >= 3.1.3), with adopt and
  prepend_chunks and tests, including a metadata-only prepend on a copy of a real
  multiscale level.
• zarr-python core: OffsetChunkKeyEncoding plus Array.prepend_chunks / Array.prepend,
  branch https://github.com/rhoadesScholar/zarr-python/tree/offset-chunk-key-encoding
  (Add Array.prepend and the regular_bounded chunk grid zarr-python#4516).
• tensorstore zarr3 driver: reads and writes it, including key-range deletion on shrink with
  negative key components,
  https://github.com/rhoadesScholar/tensorstore/tree/zarr3-offset-chunk-key-encoding
  (zarr3: support the regular_bounded chunk grid google/tensorstore#296).
• neuroglancer: reads it,
  https://github.com/rhoadesScholar/neuroglancer/tree/zarr-offset-chunk-key-encoding
  (zarr: read the regular_bounded chunk grid google/neuroglancer#1133).

Motivation: volumes assembled from parts that arrive over time in no fixed order (our case:
serial-section pages registered as they are imaged, where a page can land below pages already
rendered). Licensed under CC BY 3.0 as the registry requires.

Issues

• Closes Negative chunk indexes, offset chunk origin zarr-specs#9 (negative chunk indexes, offset chunk origin): this is that
  proposal as a registered extension.
• Addresses the chunk-grid-origin half of declaring the origin of an array #30, and does so with a
  new name rather than an origin key on the regular grid because of the concern raised there:
  a key that current readers would ignore cannot be made safe. The array-index-space origin and
  underfilling the first chunk are left to that discussion.
• Refs Support for non-zero origin zarr-specs#122 (non-zero array origin; indexing is unchanged here) and
  Proposal: open ended zarrs zarr-specs#39 (open-ended arrays that grow at both ends).

[rhoadesScholar]

Ready for review. Implementations are open as zarr-developers/zarr-python#4516 (core + Array.prepend), google/tensorstore#296 (read/write) and google/neuroglancer#1133 (read). cc @d-v-b @jbms @clbarnes

[jbms]

Defining this as a chunk_key_encoding is indeed a clever, minimal way to specify it.

However, there are some drawbacks:

• Tied to the single default chunk_key_encoding, which is awkward. One of the problems with the default chunk key encoding is that the lexicographical order of chunks does not match the lexicographical order of keys. Adding negative numbers into the default encoding makes the problem worse, though.
• Shifts the logical coordinate space of the array rather than allowing negative coordinates. Sometimes that is what you want, but in other cases you may want to retain the existing meaning of coordinates. More generally it does not allow shifting the logical coordinate space of the array by an amount that does not lie on a chunk boundary.
• zarr allows the last chunk to be partial, meaning if shape does not end on a chunk boundary the last chunk is logically "clipped". This proposal does not allow the same thing for the first chunk, though.

What I'd rather see is an origin parameter added to the regular and rectilinear chunk grids that specifies the logical array coordinate corresponding to the start of chunk 0 in each dimension. Whether negative coordinates are supported or not would be up to the chunk_key_encoding --- default and v2 would support it as in your proposal, even though it makes the ordering worse. Then separately there would be a way to specify lower and upper bounds on the logical coordinate space of the array, as an alternative to shape (which would imply a lower bound of 0).

[d-v-b]

I agree with the points raised by @jbms. Since this very important feature requires coordinated changes across multiple metadata fields, it might make sense to start with a rough plan. Hence this issue: #79

[rhoadesScholar]

Thanks both, agreed on all three points. I missed #79 before opening this, sorry for the overlap. I've added notes from the prototype there (#79 (comment)), and I'll treat this PR and the three implementation PRs as a prototype to rework once the metadata shape settles.

[rhoadesScholar]

Superseded by #81, which registers a regular_bounded chunk grid with origin and lower_bound, following the design discussed here and in #79. The three implementation PRs have been reworked onto it. Closing in favour of #81.