Skip to content

feat(zarr-metadata)!: read what zarr-python writes in a ZARR_PYTHON_CONTEXT scope - #426

Closed
d-v-b wants to merge 14 commits into
mainfrom
claude/zarr-python-scope-a042b7
Closed

feat(zarr-metadata)!: read what zarr-python writes in a ZARR_PYTHON_CONTEXT scope#426
d-v-b wants to merge 14 commits into
mainfrom
claude/zarr-python-scope-a042b7

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 29, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Add the ZARR_PYTHON_CONTEXT scope, which reads what zarr-python writes: CORE_AND_EXTENSIONS_CONTEXT plus zarr-python's own data types null_terminated_bytes (NumPy's S) and variable_length_bytes, which no specification registers. CORE_AND_EXTENSIONS_CONTEXT now also reads the fixed_length_utf32 data type (NumPy's U) and the vlen-utf8 and vlen-bytes codecs that zarr-extensions registers, so the strings zarr-python writes are checked instead of read as unclaimed.

Rename the v3 scopes CORE and CORE_AND_EXTENSIONS to CORE_CONTEXT and CORE_AND_EXTENSIONS_CONTEXT, so that a scope's name says it is a ZarrV3Context.

Fixes #423

What the agent checked

The agent generated v3 metadata with zarr-python 3.4.2 for <U3, S4, str and VariableLengthBytes() arrays. All four now read with no problem and no unclaimed field in ZARR_PYTHON_CONTEXT. Before this change, CORE_AND_EXTENSIONS left the string array's vlen-utf8 codec unclaimed as well as the data types.

Written by zarr-python Registered in zarr-extensions Scope
fixed_length_utf32 yes CORE_AND_EXTENSIONS_CONTEXT
vlen-utf8, vlen-bytes codecs yes CORE_AND_EXTENSIONS_CONTEXT
null_terminated_bytes no (zarr-python warns the spelling may change) ZARR_PYTHON_CONTEXT
variable_length_bytes no (zarr-python warns the spelling may change) ZARR_PYTHON_CONTEXT

Rules the new definitions check:

  • length_bytes is a positive multiple of 4 for fixed_length_utf32, and at least 1 for null_terminated_bytes.
  • A fill value fits the type, and is valid base64 where zarr-python writes base64. The canonical null_terminated_bytes fill value drops trailing null bytes.
  • fixed_length_utf32 is multi-byte, so the bytes codec requires an endian.
  • The vlen-* codecs reject a data type of fixed size. This uses the data type's declared storage, so it does not tell string from bytes: a name check would refuse zarr-python's own variable_length_bytes + vlen-bytes pairing.

Not covered

  • zarr-python's raw_bytes (NumPy V): its type names would collide with the RawBytes* names that r* already exports at the top level.
  • The legacy structured alias that earlier zarr-python 3.x releases wrote.
  • numcodecs.* codecs.
  • The v2 scope CORE_V2 keeps its name.

just check passes on Python 3.11 and the newest supported version, including the strict docs build.

🤖 Generated with Claude Code

d-v-b added 2 commits October 29, 2026 20:02
@d-v-b
feat(zarr-metadata)!: read what zarr-python writes in a ZARR_PYTHON_C…
927ca74 
…ONTEXT scope

Add definitions for the extensions zarr-python writes. zarr-extensions
registers `fixed_length_utf32` and the `vlen-utf8` and `vlen-bytes`
codecs, so they join CORE_AND_EXTENSIONS_CONTEXT. zarr-python's own
`null_terminated_bytes` and `variable_length_bytes`, which no
specification registers, are only in the new ZARR_PYTHON_CONTEXT, so the
default scope stays spec-pure.

Rename CORE to CORE_CONTEXT and CORE_AND_EXTENSIONS to
CORE_AND_EXTENSIONS_CONTEXT, so that a scope's name says it is a context.

Fixes #423

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
docs(zarr-metadata): number the changelog fragments for PR 426
9688909 
Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b d-v-b mentioned this pull request Oct 29, 2026
Draft
d-v-b added 6 commits October 31, 2026 17:08
@d-v-b
chore(zarr-metadata): merge main into the ZARR_PYTHON_CONTEXT branch
8740e09 
Resolve the conflict in test_pipelines.py: keep the unclaimed `acme.vlen`
codec, at the codec-level location zarr-developers#4574 gives a codec's chunk problems.
Rename CORE to CORE_CONTEXT in main's new test_rule_locations.py, and move
the expectations in test_zarr_python_scope.py to zarr-developers#4574's locations: a
codec's chunk problem at the codec, a missing key at the object that
lacks it.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
docs(zarr-metadata): number the changelog fragments for PR 4583
339aad4 
Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
fix(zarr-metadata): keep the old scope names and tighten the string r…
beb0697 
…ules

Keep `CORE` and `CORE_AND_EXTENSIONS` importable from `zarr_metadata.v3`
and `zarr_metadata.v3.definition` as the same scopes with a
DeprecationWarning, so a consumer such as pydantic-zarr keeps working
across the rename.

Check that `vlen-utf8` encodes `string` and `vlen-bytes` encodes `bytes`
(or zarr-python's `variable_length_bytes`), as zarr-extensions says,
instead of accepting any data type whose values vary in size.

Do not count trailing null bytes against the size of a
`null_terminated_bytes` fill value, since they are padding. Spell a
`fixed_length_utf32` fill value without its trailing U+0000 padding, so
`"a"` and `"a\u0000"` have one canonical spelling.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
feat(zarr-metadata): read zarr-python's raw_bytes and rename CORE_V2
443df3e 
Add zarr-python's `raw_bytes` data type (NumPy's `V`) to
ZARR_PYTHON_CONTEXT, so the scope reads every data type zarr-python
writes. Its names start with `ZarrPython`, since `RawBytes` already names
the spec's raw bits. A fill value longer than `length_bytes` is a problem,
because zarr-python cuts it short, and the canonical spelling pads it with
null bytes, as zarr-python writes it.

Rename CORE_V2 to CORE_V2_CONTEXT, so every scope's name ends in
`_CONTEXT`. The old name keeps working with a DeprecationWarning, through
the same `renamed_scope` helper as the v3 names, which moves to
`zarr_metadata._renamed`.

Share the base64 decoding and spelling of fill values in `_byte.py`, and
say that the vlen codecs' `configuration` key is optional, since
zarr-python writes an empty object.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
refactor(zarr-metadata): declare the v2 scope's old name once
42f9fad 
Move CORE_V2_CONTEXT and its table of old names into `v2/_core.py`, as
`v3/_core.py` holds the v3 scopes, so `zarr_metadata.v2` and
`zarr_metadata.v2.definition` share one table.

Say why a `fixed_length_utf32` fill value counts its trailing U+0000 code
points, as the spec requires, while `null_terminated_bytes` follows
zarr-python and does not, and test a value that only its padding makes too
long.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b
Merge branch 'main' into claude/zarr-python-scope-a042b7
ed7dd65
d-v-b and others added 6 commits November 1, 2026 09:44
@d-v-b
Merge branch 'main' into claude/zarr-python-scope-a042b7
fff2ca3
@d-v-b
fix(zarr-metadata): give main's strategies the renamed scopes and new…
bf93387 
… definitions

The Hypothesis strategies merged from main (zarr-developers#4575) imported `CORE`,
`CORE_AND_EXTENSIONS` and `CORE_V2` from modules that now hold only the
`_CONTEXT` names, and their coverage test requires a strategy for every
definition in `CORE_AND_EXTENSIONS_CONTEXT`. Rename the imports, and add
strategies for `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes`; each
vlen codec draws nothing for a data type it does not read.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b @claude
fix(zarr-metadata): draw zarr-python's own data types in the testing …
2f19193 
…strategies

`DEFAULT_STRATEGIES` holds a strategy for each definition the package
defines, but `null_terminated_bytes`, `variable_length_bytes` and
`raw_bytes` had none, so `fields()` raised `LookupError` for them. Draw
each, and check the strategies against `ZARR_PYTHON_CONTEXT`, which
holds every definition. Say that the documents in
test_zarr_python_scope.py leave out the compressors zarr-python adds.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b @claude
test(zarr-metadata): pin the registered extensions to CORE_AND_EXTENS…
7b51c8d 
…IONS_CONTEXT

The strategies are now checked against `ZARR_PYTHON_CONTEXT`, which no
longer proves that `fixed_length_utf32`, `vlen-utf8` and `vlen-bytes`
are in `CORE_AND_EXTENSIONS_CONTEXT`. Assert it directly.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b @claude
refactor(zarr-metadata): let the vlen strategies ask their codec's rules
f57e2a8 
Each vlen codec strategy named its compatible data types a second time;
draw nothing for exactly the chunks the codec's own chunk rules refuse,
so the two cannot drift. Draw `fixed_length_utf32` fill values with
trailing U+0000 padding too, which the canonical spelling removes.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b
chore(zarr-metadata): merge main into the ZARR_PYTHON_CONTEXT branch
c989cdb 
Main already holds this branch as zarr-developers#4583 (bc177b3, whose tree is this
branch's tip), followed by later work such as the declaration rename in
zarr-developers#4592. Every conflict therefore resolves to main's side, and the merged
tree is main's.

Assisted-by: ClaudeCode:claude-opus-5-5
@d-v-b

d-v-b commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI text below 🤖

This PR landed as zarr-developers#4583 (bc177b3 on main), whose tree matched this branch's tip. After the agent merged main into the branch, the diff against main is empty, so there is nothing left to merge here.

@d-v-b d-v-b closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

needs release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

zarr-metadata: ship definitions for zarr-python's registered extension data types

1 participant

@d-v-b