Skip to content

feat(context): scope ArrayIO's extension resolution with a Context - #420

Draft
d-v-b wants to merge 3 commits into
claude/arrayio-lazy-metadata-48d8e5from
claude/arrayio-context
Draft

feat(context): scope ArrayIO's extension resolution with a Context#420
d-v-b wants to merge 3 commits into
claude/arrayio-lazy-metadata-48d8e5from
claude/arrayio-context

Conversation

@d-v-b

@d-v-b d-v-b commented Oct 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Merge posture: this PR is the deprecation path for the global mutable state that #419 deliberately still honors. Without a context every binding behaves exactly as before (default registries, globally configured pipeline); a context is the explicit, per-binding replacement, and its pipeline choice is captured into the same slot the global configuration fills without one, so precedence lives in one place.

Added a context to zarr.ArrayIO and zarr.AsyncArrayIO: a value defining the data types, codecs and codec pipeline in scope for one array. Extension names in a stored document or in build's keyword arguments resolve through the context instead of the default registries; a kind the context leaves unset falls back to the defaults, while a kind it sets is a closed world, and a name outside it fails to resolve with an error naming the scope.

The type is the existing zarr.core.context.Context — the extensions-in-scope value the structured data type resolution already threads through nested reads — extended with codecs: Mapping[str, type[Codec]] and codec_pipeline: type[CodecPipeline], and exported from zarr.experimental (the ArrayIO classes themselves are public in the zarr namespace). An optional context= is threaded through the resolution functions: parse_dtype and the data-type getters, codec normalization (_resolve_codec, _parse_*_codec, parse_codecs), from_dict on both metadata classes, parse_array_metadata, create_codec_pipeline, and build_array_metadata. With no context, every path is byte-identical to before.

The layering and the metadata-model binding land in #419; this PR only adds the context. with_context completes the with_* family: one binding is rebound under another scope, so a codec pipeline is chosen per binding instead of through the global configuration — the pipeline-parity, sharded-indexing-rejection and fork-pool tests now run that way, with no zarr.config.set in any of them. With the document-first binding the context is consumed only where the document resolves into classes, so opening an array whose codec names fall outside the context still works for document operations, and resolution fails when the data path is built.

Scope notes:

  • Zarr format 2 filters and compressors are numcodecs objects, resolved by numcodecs; no context governs them (documented on from_dict and build_array_metadata).
  • Codec instances pass through any context: it scopes name resolution, not values.
  • Chunk key encodings and chunk grids are not yet context kinds; they can be added when their registries become extension points.
Stack

Stacked on #419 (claude/arrayio-lazy-metadata-48d8e5); merge that first. This PR is the claude/arrayio-context branch.

🤖 Generated with Claude Code

d-v-b added a commit that referenced this pull request Oct 29, 2026
@d-v-b
docs: changelog fragment for #420
28b4221 
Assisted-by: ClaudeCode:claude-fable-5
@d-v-b d-v-b mentioned this pull request Oct 29, 2026
Draft
@d-v-b
d-v-b force-pushed the claude/arrayio-context branch 3 times, most recently from 89fa080 to be7fcdc Compare October 29, 2026 12:28
@d-v-b d-v-b changed the title feat(experimental): scope ArrayIO's extension resolution with a Context feat(context): scope ArrayIO's extension resolution with a Context Oct 29, 2026
@d-v-b
d-v-b force-pushed the claude/arrayio-context branch 9 times, most recently from e2bcdc9 to 45896a5 Compare October 31, 2026 16:56
@d-v-b d-v-b mentioned this pull request Oct 31, 2026
Draft
d-v-b and others added 3 commits November 1, 2026 13:27
@d-v-b
feat(context): scope ArrayIO's extension resolution with a Context
205b83a 
Extends zarr.core.context.Context — the extensions-in-scope value the
structured data type resolution already uses — with codecs and
codec_pipeline, and threads an optional context through the resolution
functions: parse_dtype and the data-type getters, parse_codecs,
metadata from_dict on both formats, parse_array_metadata, and the
per-format builders. A kind the context leaves None resolves from the
default registries exactly as before; a kind it sets is a closed world,
and a name outside it fails to resolve, naming the scope — including
parse_dtype's variable-length string shortcut. The codecs mapping is
copied read-only at construction.

ArrayIO, AsyncArrayIO and AsyncArray take the context; with_context
rebinds one array under another scope, derived bindings keep their
context, and a context's pipeline choice is captured into the same slot
the global configuration fills without one. Documented limits: the
scope governs top-level names (codecs nested inside a sharding codec
resolve through the default registry), and a binding built from a
resolved metadata object only picks up the pipeline. The
pipeline-parity, sharded-indexing-rejection and fork-pool tests select
their codec pipeline per binding instead of mutating the global
configuration — the context is the explicit replacement for that
global state, and the path to deprecating it.

Addresses the roborev review of this branch (job 1360).

Refs zarr-developers#4217

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b @claude
fix(context): resolve default codecs and legacy sources under the scope
9680d25 
A closed codec scope now checks the default v3 codec chain, so a binding
never stores a document its own context cannot read back. Legacy resolved
metadata is re-resolved under a context in one place, `AsyncArrayIO.__init__`
(via `_legacy_resolved_in_context`, which carries the repair mark), so a
binding constructed from a resolved object and one rebound by `with_context`
follow the same rule. Tests cover the default-chain rejection, a data round
trip through a rebound binding, the repair mark surviving `with_context`, and
the variable-length-UTF8 shortcut resolving through a context's registry.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b @claude
fix(context): resolve once in the builders and check with_metadata's …
2e180d6 
…scope

build_v2/build_v3 resolve their metadata under the context, then passed it to
the public constructor, which re-resolved it from to_dict(): every scoped
build resolved twice. They now bind through a private _from_resolved, which
still reads the object as parse_array_metadata does (v2 metadata with a chunk
size of 0 as repaired), and the shared field setup lives in _bind.
with_metadata reads a legacy object as the constructor does, under the
binding's context when there is one, so a closed scope rejects names outside
it there too. The variable-length-UTF8 test registers a subclass, so it fails
if the shortcut ignores the context.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-v-b
d-v-b force-pushed the claude/arrayio-context branch from de992b7 to 2e180d6 Compare November 1, 2026 12:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@d-v-b