Epic: fleshing out a low-level library

[kylebarron]

The below was written by Claude, and then I'll split it out into individual issues.

---

zarrista low-level API gap analysis

Date: 2026-07-09
Scope: What functionality is missing from zarrista to be a complete
low-level Zarr API, measured against the zarrs v0.23 Rust crate it wraps.

zarrista already covers the hard parts: the codec pipeline, sharding/subchunks,
async I/O via obstore + icechunk, DLPack/Arrow/numpy export, array creation via
a typed builder, and whole-chunk read+write. The gaps below are what remain,
roughly in priority order.

---

1. Writing arbitrary array subsets (highest impact)

zarrista can read any subset (retrieve_array_subset, __getitem__) but can
only write whole chunks (store_chunk). zarrs exposes, but zarrista does
not:

• store_array_subset / store_array_subset_elements — write an arbitrary
  region spanning partial chunks
• store_chunk_subset — write part of a single chunk
• __setitem__ — the natural numpy-style counterpart to __getitem__
• partial_encoder — the experimental_partial_encoding flag is already in
  CodecOptions, but no method consumes it

Without subset writing, the write API is only usable when write regions are
chunk-aligned. This is asymmetric with the read side and is the first thing
most users will hit.

2. Indexing / selection is very limited

src/array/selection.rs supports only int, step-1 slice, and Ellipsis (and is
self-described as "vibe coded"). Missing:

• Strided slices (step != 1)
• None / newaxis
• Boolean masks
• Integer/fancy array indexing — including orthogonal (oindex) and vectorized
  (vindex) modes that zarr-python offers

3. Group & hierarchy creation

• No group creation. src/group/sync.rs only has open() +
  store_metadata(); there is no builder or create to make a new group from
  scratch, and no create_group / create_array convenience on Group.
• No consolidate_metadata() writer. You can read consolidated_metadata,
  but zarrs' Node::consolidate_metadata (which builds it) isn't exposed — so
  you can't produce the consolidated metadata that makes cloud reads fast.
• Node isn't a pyclass (src/node/sync.rs:27 TODO); no hierarchy_tree /
  tree-printing.

4. Array manipulation

• No resize / set_shape — zarrs supports it; there is no way to
  grow/shrink an existing array.
• No metadata version conversion (to_v) exposed.
• Builder stubs unfinished: codec_options (src/array/builder.rs:73) and
  storage_transformers (src/array/builder.rs:154) are commented out.

5. Zarr V2 reading

V2 metadata types exist internally (PyArrayMetadataV2) and group opens force
MetadataConvertVersion::V3, so V2→V3 conversion partly works — but there is no
first-class V2 read path or way to inspect the original V2 metadata. Given how
much real-world Zarr data is still V2, this matters.

6. Codecs not exposed

Currently wired: transpose, bitround, blosc, gzip, zstd, crc32c.
zarrs supports many more:

• array→bytes: no explicit bytes (endianness) constructor, no sharding
  constructor function (only via builder), and missing vlen-utf8,
  vlen-bytes, packbits, zfp, pcodec
• bytes→bytes: missing bz2, zlib, adler32, fletcher32, shuffle,
  gdeflate
• array→array: missing fixedscaleoffset, reshape, squeeze

Many are behind zarrs feature flags (zfp, pcodec, bz2, …) not currently
enabled in Cargo.toml.

7. Data types not exposed

Current: bool, int/uint 8–64, float16/32/64, complex64/128, string, bytes, raw
bits. Missing the practically important ones:

• numpy.datetime64 / numpy.timedelta64 (need zarrs chrono/jiff
  features)
• bfloat16, and the float8_* / complex_bfloat16 family (ML use cases)
• fixed_length_utf32, and the niche int2/int4/uint2/uint4

8. Stores

Currently: sync FilesystemStore / MemoryStore, plus async via obstore
ObjectStore and icechunk Session (covers S3/GCS/HTTP well). Gaps:

• ZIP store (zarrs_zip) — reading Zarr-in-a-zip is common
• Plain HTTP read-only store (zarrs_http) — obstore covers this, but no
  lightweight direct option
• No sync remote store — all remote I/O forces the async path
• Diagnostic adapters (UsageLogStorageAdapter,
  PerformanceMetricsStorageAdapter) not exposed

9. Bulk & cached operations

• Multi-chunk ops: retrieve_chunks / store_chunks (operate on a region
  of chunks at once) aren't exposed — only single-chunk methods.
• Chunk cache: zarrs' LRU chunk caches aren't user-controllable; shard-cache
  management is flagged TODO in src/array/sync.rs.

10. Global configuration

zarrs has a global_config() (validate_checksums, store_empty_chunks,
concurrency targets, consolidated-metadata usage, etc.). zarrista only exposes
per-call CodecOptions — there is no process-wide config module.