From ffb4656761b62c811e8668840c0a8a2479b3db56 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 26 Oct 2026 13:28:20 +0100 Subject: [PATCH 1/3] chore(zarr-metadata): build 0.6.0 changelog Run `towncrier build --version 0.6.0` in `packages/zarr-metadata/`, which moves the 23 pending fragments into `CHANGELOG.md`. Assisted-by: ClaudeCode:claude-opus-5-5 --- packages/zarr-metadata/CHANGELOG.md | 273 ++++++++++++++++++ packages/zarr-metadata/changes/4311.doc.md | 2 - packages/zarr-metadata/changes/4365.bugfix.md | 1 - packages/zarr-metadata/changes/4378.bugfix.md | 24 -- packages/zarr-metadata/changes/4378.misc.1.md | 5 - packages/zarr-metadata/changes/4378.misc.md | 4 - packages/zarr-metadata/changes/4420.bugfix.md | 29 -- packages/zarr-metadata/changes/4421.bugfix.md | 4 - packages/zarr-metadata/changes/4422.bugfix.md | 3 - .../zarr-metadata/changes/4422.feature.md | 7 - packages/zarr-metadata/changes/4423.bugfix.md | 7 - packages/zarr-metadata/changes/4426.bugfix.md | 5 - packages/zarr-metadata/changes/4428.bugfix.md | 4 - packages/zarr-metadata/changes/4430.misc.md | 4 - .../zarr-metadata/changes/4432.feature.md | 14 - .../zarr-metadata/changes/4434.feature.1.md | 10 - .../zarr-metadata/changes/4434.feature.md | 50 ---- .../zarr-metadata/changes/4436.feature.md | 12 - packages/zarr-metadata/changes/4443.bugfix.md | 3 - .../zarr-metadata/changes/4443.feature.2.md | 10 - .../zarr-metadata/changes/4443.feature.3.md | 6 - .../zarr-metadata/changes/4443.feature.4.md | 20 -- .../zarr-metadata/changes/4443.feature.md | 13 - .../zarr-metadata/changes/4490.feature.md | 17 -- 24 files changed, 273 insertions(+), 254 deletions(-) delete mode 100644 packages/zarr-metadata/changes/4311.doc.md delete mode 100644 packages/zarr-metadata/changes/4365.bugfix.md delete mode 100644 packages/zarr-metadata/changes/4378.bugfix.md delete mode 100644 packages/zarr-metadata/changes/4378.misc.1.md delete mode 100644 packages/zarr-metadata/changes/4378.misc.md delete mode 100644 packages/zarr-metadata/changes/4420.bugfix.md delete mode 100644 packages/zarr-metadata/changes/4421.bugfix.md delete mode 100644 packages/zarr-metadata/changes/4422.bugfix.md delete mode 100644 packages/zarr-metadata/changes/4422.feature.md delete mode 100644 packages/zarr-metadata/changes/4423.bugfix.md delete mode 100644 packages/zarr-metadata/changes/4426.bugfix.md delete mode 100644 packages/zarr-metadata/changes/4428.bugfix.md delete mode 100644 packages/zarr-metadata/changes/4430.misc.md delete mode 100644 packages/zarr-metadata/changes/4432.feature.md delete mode 100644 packages/zarr-metadata/changes/4434.feature.1.md delete mode 100644 packages/zarr-metadata/changes/4434.feature.md delete mode 100644 packages/zarr-metadata/changes/4436.feature.md delete mode 100644 packages/zarr-metadata/changes/4443.bugfix.md delete mode 100644 packages/zarr-metadata/changes/4443.feature.2.md delete mode 100644 packages/zarr-metadata/changes/4443.feature.3.md delete mode 100644 packages/zarr-metadata/changes/4443.feature.4.md delete mode 100644 packages/zarr-metadata/changes/4443.feature.md delete mode 100644 packages/zarr-metadata/changes/4490.feature.md diff --git a/packages/zarr-metadata/CHANGELOG.md b/packages/zarr-metadata/CHANGELOG.md index c1e9f81a61..657e56373c 100644 --- a/packages/zarr-metadata/CHANGELOG.md +++ b/packages/zarr-metadata/CHANGELOG.md @@ -2,6 +2,279 @@ +## 0.6.0 (2026-10-26) + +### Features + +- **Breaking:** `ValidationProblem` and `MetadataValidationError` check at run + time what their annotations declare, and raise `TypeError` otherwise. A + problem's `loc` is a tuple of strings and integers, so a list, which was + accepted before, is now refused, as is a `bool`; its `message` is a string + and its `kind` one of the kinds; an error holds problems. A caller that is + not type-checked fails where it builds the value, not later, where a + location is read. ([#4422](https://github.com/zarr-developers/zarr-python/pull/4422)) +- `zarr_metadata.typed_json.check(value, SomeTypedDict)` type-checks a JSON + value against any of the package's TypedDicts -- `ZarrV3ArrayMetadataJSON`, + `GzipCodecConfiguration`, or one of your own -- and gives back a value of + it, or None, with every problem located. A TypedDict reads as the typing + spec defines it, whatever the runtime's attributes say: a key is required + by its own `Required` or `NotRequired`, or else by the `total` of the class + that declared it, read from the evaluated annotations, so a module written + with `from __future__ import annotations` reads like any other; `closed`, + `extra_items` and openness come from the class, or from its bases when it + says neither; and each annotation is evaluated in the module of the class + that wrote it. A key a closed TypedDict does not declare is reported, as + the new problem kind `unknown_key`, and left out of the value, which still + comes back; a union of TypedDicts tagged by a `Literal` key is read by the + branch the tag names. ([#4432](https://github.com/zarr-developers/zarr-python/pull/4432)) +- **Breaking:** `must_understand: false` is refused at every extension point + of an array document, codecs and storage transformers included, not only + at `data_type`, `chunk_grid` and `chunk_key_encoding`: ignoring a codec + gives wrong bytes as surely as ignoring a data type gives wrong values, + and the spec naming only three points is read as an oversight. A document + that declares a codec or a storage transformer ignorable now has a problem + at `codecs.N.must_understand` or `storage_transformers.N.must_understand`, + so the readers refuse it, and no model writes one. The JSON schema `zarr_metadata.pydantic` generates for an array + document says the same, and so does `resolve`, which reads one field on + its own. ([#4434](https://github.com/zarr-developers/zarr-python/pull/4434)) +- Each extension's metadata is a definition: a value holding the name the + metadata carries, the TypedDict its configuration is -- the one + declaration of that JSON -- and plain functions over that TypedDict for + the rules. `zarr_metadata.definition` and `zarr_metadata.v3.definition` are the + doors, and their module docstrings the guide. Every extension the package models is one: the + codecs `blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`, + `sharding_indexed`, `transpose` and `zstd`; every data type, raw bits + included (`r16` is read by the `r*` definition with `{"bits": 16}`, the size + its name carries); the `regular` and `rectilinear` chunk grids; and the `default` + and `v2` chunk key encodings. Each is a constant in its own module -- + `GZIP_CODEC`, `STRUCT_DATA_TYPE` -- and `CORE` holds what the + specification defines, `CORE_AND_EXTENSIONS` what `zarr-extensions` + registers besides. A third party writes a TypedDict, a rule function and + a definition, and adds it to a scope; a definition checks itself + when built, and nothing happens at class creation. + + Reading a field is three steps, each usable on its own by a caller + holding nothing but JSON. `check(value, SomeTypedDict)`, from + `zarr_metadata.typed_json`, type-checks JSON against a TypedDict and needs + nothing else; `definition.read_configuration(configuration)` is the check, each nested + field's envelope judged, and then the rules; `resolve(field, + CodecDefinition, scope)` reads a whole field in a scope -- its envelope + judged, its name related to a definition, its configuration judged, each + nested field read the same way -- and returns what the scope made of it, + with every problem located. A name nothing in scope claims is left + unjudged. A shard's pipelines, a cast's target + type and a struct's field types are nested fields, annotated `ZarrV3CodecField` + and `ZarrV3DataTypeField`, and read in the scope their field is read in. + A codec definition also says whether the size + of what it gives out is `static` or `dynamic`, and a member annotated + `ZarrV3StaticCodecField` takes codecs of static size only: a shard's + `index_codecs`, where the spec forbids a compressor, so a `gzip` there is a + problem at its place. + + The TypedDict reads as the typing spec defines it -- `total`, `Required`, + `NotRequired`, `closed` and `extra_items` mean what they mean to a type + checker, whether or not its module postpones annotations. A key a closed + TypedDict does not declare is reported, as `unknown_key`, and left out of + what the rules are handed. A definition refuses a TypedDict that says + nothing of the keys it does not declare, and a member typed + `ZarrV3MetadataFieldJSON`, which would check as plain JSON. + + `canonical_of(field, problems)` gives a field without problems in its + simplest equivalent spelling: each nested field in its own simplest + spelling, then its definition's own `canonical`, and the envelope in the + fewest words every reader takes, with no `must_understand`. A field with + any problem, an unknown key included, has + none, since a simpler spelling would erase what its author wrote. What + `canonical` gives is judged again, and one that does not hold is a + `ValueError`, a fault in the definition. ([#4434](https://github.com/zarr-developers/zarr-python/pull/4434)) +- **Breaking:** the v3 readers read each extension point through its + definition: `read_array_metadata_v3` judges the data type, the chunk + grid, the chunk key encoding, each codec and each storage transformer + against the definition that claims its name in a scope, + `CORE_AND_EXTENSIONS` unless a `context` is passed, so a gzip `level` of + 99, or a key a codec does not declare, is a problem located in the + configuration, and a document holding one, accepted before, is refused. + So do the v3 group readers, and the v3 model classes' `from_json` and + `from_key_value` take the same `context`. + A name nothing in scope claims is left unjudged. A problem in a field + that a configuration holds, such as a codec inside a shard, is reported + at that field's own location, and the field that holds it is still read. ([#4436](https://github.com/zarr-developers/zarr-python/pull/4436)) +- **Breaking:** a v3 array's `fill_value` is judged against its `data_type`. + Each data type's definition declares the JSON shape of its fill value, + `fill_value`, and the rules for one of that shape, `fill_value_rules`: an + `int8` fill value of 300, a `float32` hex string of another width, and a + struct fill value missing a field are each a problem at `fill_value`, + where the package accepted them before. `fill_value_problems(data_type, + value)` judges a fill value against a data type field a scope read, and + a field that is read keeps the fields it read inside as + `nested`, a `Nested` mapping by location. A data type nothing + in scope claims leaves its fill value unjudged. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) +- **Breaking:** a v3 array's sharding codec is judged against the shard it + is handed, and its two pipelines are read as the array's is. Its + `chunk_shape` has an inner chunk length for each axis of the shard, + dividing each length the shards take along it -- the shard being the + chunk the codec is handed, so a transposed shard is divided along its + transposed axes. Its inner codecs are handed the inner chunks, of the + shard's data type, and its index codecs the shard index, of `uint64`, + with an axis more than the shard. A shard that does not divide, or a + pipeline that does not fit what it is handed -- an index `bytes` codec + without an `endian` -- has a problem where it sits, where the package + accepted it before. A codec definition + says what the pipelines it holds are handed, `pipelines`, and each + codec's `Stage` keeps the stages of those pipelines as `inner`. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) +- **Breaking:** a v3 array's `chunk_grid` is judged against its `shape`, by + the grid's definition. A regular grid whose `chunk_shape` does not have + one length per dimension of the shape, and a rectilinear grid whose + `chunk_shapes` does not have one entry per dimension, or whose chunk + lengths fall short of their dimension, each have a problem in + `chunk_grid.configuration`, where the package accepted them before. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) +- **Breaking:** a v3 array's `codecs` are read as a pipeline: in order -- + array -> array codecs, then one array -> bytes codec, then bytes -> bytes + codecs -- and each codec judged against the chunk it is handed, which is + the grid's chunks of the array's data type, as the array -> array codecs + before it hand them on. A codec out of order, a second array -> bytes + codec or none at all; a `bytes` codec without an `endian` handed numbers + of several bytes, or handed values that vary in size; a `transpose` whose + `order` has another number of axes than its chunk; a `cast_value` to or + from a data type that models no real numbers, wrapping to one that is + not integral, or mapping a scalar that is not a fill value of the data + type on its side; a `scale_offset` handed values that are no numbers, or + with an `offset` or `scale` that is not a value of its data type; and a + struct field whose values vary in size are each a problem where they + sit, where the package accepted them before. A reading records each + codec with the chunk it is handed, as a `Stage`, and the lengths a + grid's chunks take; `storage_of(data_type)` says how a data type's + values are stored. A + definition's `rules` are handed the fields its configuration holds as + the scope read them, and the kinds gain `chunk_lengths` (grids), + `storage` (data types), and `chunk_rules` and `transition` (codecs). ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) +- **Breaking:** metadata is now read against the definitions in a scope, a + `ZarrV3Context` or `ZarrV2Context` that a reader builds from the codecs, + data types and other fields it supports, and a model is the document + together with the scope it was read in. The `read_*` readers report every + problem at its location, `unclaimed_fields` lists each field the scope + does not know, so a reader can tell whether it can open a node, repair + readers undo known zarr-python writer bugs, and JSON Schemas are exported. + Code written for 0.5.0 must change: models are no longer dataclasses, + their field members hold `AcceptedField` or `UnclaimedField` values, + `update` takes JSON and takes members the spec does not define through + `extra_fields`, the `validate_*`, `is_*` and `parse_*` functions for + documents and the `...Partial` TypedDicts are removed, some names moved to + another module, the v2 constants and models follow the v3 names and + shapes, and a v2 time dtype without a unit is refused. `to_json` now + writes each field as it was read, rather than shortening a codec to a bare + name that zarr-python could not open, and zarr-metadata now depends on + `annotated-types`. ([#4490](https://github.com/zarr-developers/zarr-python/pull/4490)) + +### Bugfixes + +- The v2 array validator follows the spec on two points it was stricter than: members outside the `.zarray` definition are accepted when they are JSON values with string keys, and are kept and written back by `to_json` and `to_key_value`; any other member is a problem, as in a v3 document (the spec says other keys ["SHOULD NOT be present ... and SHOULD be ignored"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L91-L92), unlike `.zgroup`'s ["MUST NOT"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L313)), and `filters: []` is accepted (["a list of JSON objects providing codec configurations, or null"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L76-L79) sets no minimum). The JSON Schema that `zarr_metadata.pydantic` generates follows: it accepts other members, and the filter pipeline has no minimum length. ([#4365](https://github.com/zarr-developers/zarr-python/pull/4365)) +- Concrete v3 entity types are now assignable to the fields they describe. + Previously, none of the package's canonical codec / chunk-grid / + chunk-key-encoding / data-type types (e.g. `BloscCodecMetadata`, + `RegularChunkGridMetadata`) satisfied `ZarrV3MetadataFieldJSON`, so a + type checker rejected putting them into the very fields they document + (`codecs`, `chunk_grid`, `data_type`, ...). Three changes fix this: + + - `ZarrV3NamedConfigJSON.name` and `.configuration` are now `ReadOnly` + (PEP 705), making them covariant so concrete `name: Literal[...]` and + required-`configuration` shapes are accepted. + - `ZarrV3NamedConfigJSON` is now `closed` (PEP 728): the spec's + named-configuration envelope has exactly `name` / `configuration` / + `must_understand`, and closing the type also makes it usable as a + `JSONValue` (needed for e.g. the `sharding_indexed` inner `codecs`). + - Every concrete `*Object` / `*Configuration` TypedDict is now `closed`, + and object forms declare `must_understand: NotRequired[bool]` (any v3 + metadata field may carry the extension member). + + **Soft-breaking** for type-checking consumers: dicts with keys beyond the + declared shape no longer satisfy the closed types, and `name` / + `configuration` can no longer be mutated through `ZarrV3NamedConfigJSON`. + Both were previously accepted by type checkers but produced documents + outside the spec's shapes. Generating a pydantic JSON Schema for + the `zarr_metadata.pydantic` field types no longer emits warnings. + + ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) +- Attributes are user data, and a non-finite number in them is read, + validated and written back. zarr-python writes attributes with the + defaults of Python's `json` module, so an xarray `_FillValue` or a CF + `missing_value` of NaN is stored as a bare `NaN`, which RFC 8259 lacks; + the model refused such a document and could not read the store at all. + A node's `attributes`, a v2 `.zattrs`, the `.zattrs` entries of a v2 + `.zmetadata`, and the attributes of the nodes a v3 group's inline + `consolidated_metadata` holds may now hold `NaN`, `Infinity` and + `-Infinity`, in every reader, such as `read_array_metadata_v3`, and in + each model's `from_json`, `from_key_value` and `to_key_value`. Wherever + the spec interprets a value it spells those numbers as strings, so + anywhere else a non-finite number is still refused, and the store reader + now says where one is instead of failing to decode the document. + + A model whose document the reader refuses raises + `MetadataValidationError` with every problem, instead of being written: + + - a non-finite number outside attributes, which was refused before too; + - a non-string attribute key, which was written as a string; + - a value that is not JSON, which raised `TypeError`; + - anything else the reader refuses, such as `dimension_names` longer than + `shape`, which was written and then refused on read. + + Any sequence is written as an array. A number too large for a float, such + as `1e400`, is read as infinity, so in attributes it is now accepted and + written back as `Infinity`. + + Through `zarr_metadata.pydantic`, pydantic writes a non-finite attribute + as `null` unless the enclosing model sets `ser_json_inf_nan="constants"`. + + ([#4420](https://github.com/zarr-developers/zarr-python/pull/4420)) +- `from_key_value` takes a mapping keyed by `str` or by any narrower key + type, so reading back what `to_key_value` wrote type-checks. `to_key_value` + keys its result by the store-key literals, and since a mapping's key type is + invariant, a `Mapping[str, bytes]` parameter refused it. ([#4421](https://github.com/zarr-developers/zarr-python/pull/4421)) +- `MetadataValidationError` pickles and copies with its message. A copy was + built from the error's message, and its message came out as the + original's characters, one per line. ([#4422](https://github.com/zarr-developers/zarr-python/pull/4422)) +- A `bytes` or `bytearray` is not a JSON array. Where a document expects an + array, an empty one passed as an array with no items: the v3 parser + returned `storage_transformers=b""`, and the v2 readers took `dtype=b""`, + then failed when the model was written. The readers and models now refuse + one at `storage_transformers` and at a v2 `dtype` and its field records. At + `codecs`, `dimension_names` and v2 `filters`, which already refused one, + the problem now says the value has the wrong type. ([#4423](https://github.com/zarr-developers/zarr-python/pull/4423)) +- A model read by `from_json` or `from_key_value` shares no mutable state + with the document it was read from, as a document from `to_json` already + shares none with its model. A frozen model held the caller's own + dictionaries, so editing the input after reading it changed the model, + and could make `to_key_value` fail. ([#4426](https://github.com/zarr-developers/zarr-python/pull/4426)) +- `from_key_value` checks that each store value it reads is `bytes`, as its + signature says: a `str` was decoded as a JSON document, and `None` or a + `memoryview` raised `TypeError`. Each is now a `MetadataValidationError` + with an `invalid_type` problem at the store key. ([#4428](https://github.com/zarr-developers/zarr-python/pull/4428)) +- `ZarrV3ArrayMetadata.create_default(shape=...)` derives a chunk length of 1 + for a dimension of length 0, where it wrote 0: the regular grid asks for + chunk lengths greater than zero, and `zarr` does not open a grid with one. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + +### Improved Documentation + +- The docs now render the release notes as a page instead of linking to + `CHANGELOG.md` on GitHub, so each docs version shows its own changelog. ([#4311](https://github.com/zarr-developers/zarr-python/pull/4311)) + +### Misc + +- `ZarrV2ConsolidatedMetadataJSON.zarr_consolidated_format` is typed + `Literal[1]` rather than `int`. Format 1 is the only defined `.zmetadata` + format and the runtime validator already rejected anything else, so the + type now carries the constraint the validator enforces instead of + contradicting it. ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) +- **Breaking:** `MetadataValidationError.problems` is now a tuple instead of + a list. Iteration and indexing are unchanged; code that changes a report + must first copy it with `list(problems)`. `MetadataValidationError` still + accepts any problem sequence. ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) +- `ZarrV2ZGroupJSON` and `ZarrV2GroupMetadataJSON` are closed + (PEP 728), as the spec's "Other keys MUST NOT be present" and the v2 group + validator already said, and as its pydantic schema already was. A type + checker now refuses a v2 group document with a key it does not declare. ([#4430](https://github.com/zarr-developers/zarr-python/pull/4430)) + + ## 0.5.0 (2026-08-14) ### Bugfixes diff --git a/packages/zarr-metadata/changes/4311.doc.md b/packages/zarr-metadata/changes/4311.doc.md deleted file mode 100644 index 674389780c..0000000000 --- a/packages/zarr-metadata/changes/4311.doc.md +++ /dev/null @@ -1,2 +0,0 @@ -The docs now render the release notes as a page instead of linking to -`CHANGELOG.md` on GitHub, so each docs version shows its own changelog. diff --git a/packages/zarr-metadata/changes/4365.bugfix.md b/packages/zarr-metadata/changes/4365.bugfix.md deleted file mode 100644 index 09a76e3b71..0000000000 --- a/packages/zarr-metadata/changes/4365.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -The v2 array validator follows the spec on two points it was stricter than: members outside the `.zarray` definition are accepted when they are JSON values with string keys, and are kept and written back by `to_json` and `to_key_value`; any other member is a problem, as in a v3 document (the spec says other keys ["SHOULD NOT be present ... and SHOULD be ignored"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L91-L92), unlike `.zgroup`'s ["MUST NOT"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L313)), and `filters: []` is accepted (["a list of JSON objects providing codec configurations, or null"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L76-L79) sets no minimum). The JSON Schema that `zarr_metadata.pydantic` generates follows: it accepts other members, and the filter pipeline has no minimum length. diff --git a/packages/zarr-metadata/changes/4378.bugfix.md b/packages/zarr-metadata/changes/4378.bugfix.md deleted file mode 100644 index 9142a6fe6e..0000000000 --- a/packages/zarr-metadata/changes/4378.bugfix.md +++ /dev/null @@ -1,24 +0,0 @@ -Concrete v3 entity types are now assignable to the fields they describe. -Previously, none of the package's canonical codec / chunk-grid / -chunk-key-encoding / data-type types (e.g. `BloscCodecMetadata`, -`RegularChunkGridMetadata`) satisfied `ZarrV3MetadataFieldJSON`, so a -type checker rejected putting them into the very fields they document -(`codecs`, `chunk_grid`, `data_type`, ...). Three changes fix this: - -- `ZarrV3NamedConfigJSON.name` and `.configuration` are now `ReadOnly` - (PEP 705), making them covariant so concrete `name: Literal[...]` and - required-`configuration` shapes are accepted. -- `ZarrV3NamedConfigJSON` is now `closed` (PEP 728): the spec's - named-configuration envelope has exactly `name` / `configuration` / - `must_understand`, and closing the type also makes it usable as a - `JSONValue` (needed for e.g. the `sharding_indexed` inner `codecs`). -- Every concrete `*Object` / `*Configuration` TypedDict is now `closed`, - and object forms declare `must_understand: NotRequired[bool]` (any v3 - metadata field may carry the extension member). - -**Soft-breaking** for type-checking consumers: dicts with keys beyond the -declared shape no longer satisfy the closed types, and `name` / -`configuration` can no longer be mutated through `ZarrV3NamedConfigJSON`. -Both were previously accepted by type checkers but produced documents -outside the spec's shapes. Generating a pydantic JSON Schema for -the `zarr_metadata.pydantic` field types no longer emits warnings. diff --git a/packages/zarr-metadata/changes/4378.misc.1.md b/packages/zarr-metadata/changes/4378.misc.1.md deleted file mode 100644 index 5f5eb1f238..0000000000 --- a/packages/zarr-metadata/changes/4378.misc.1.md +++ /dev/null @@ -1,5 +0,0 @@ -`ZarrV2ConsolidatedMetadataJSON.zarr_consolidated_format` is typed -`Literal[1]` rather than `int`. Format 1 is the only defined `.zmetadata` -format and the runtime validator already rejected anything else, so the -type now carries the constraint the validator enforces instead of -contradicting it. diff --git a/packages/zarr-metadata/changes/4378.misc.md b/packages/zarr-metadata/changes/4378.misc.md deleted file mode 100644 index 83d55866d4..0000000000 --- a/packages/zarr-metadata/changes/4378.misc.md +++ /dev/null @@ -1,4 +0,0 @@ -**Breaking:** `MetadataValidationError.problems` is now a tuple instead of -a list. Iteration and indexing are unchanged; code that changes a report -must first copy it with `list(problems)`. `MetadataValidationError` still -accepts any problem sequence. diff --git a/packages/zarr-metadata/changes/4420.bugfix.md b/packages/zarr-metadata/changes/4420.bugfix.md deleted file mode 100644 index 138dcd1a31..0000000000 --- a/packages/zarr-metadata/changes/4420.bugfix.md +++ /dev/null @@ -1,29 +0,0 @@ -Attributes are user data, and a non-finite number in them is read, -validated and written back. zarr-python writes attributes with the -defaults of Python's `json` module, so an xarray `_FillValue` or a CF -`missing_value` of NaN is stored as a bare `NaN`, which RFC 8259 lacks; -the model refused such a document and could not read the store at all. -A node's `attributes`, a v2 `.zattrs`, the `.zattrs` entries of a v2 -`.zmetadata`, and the attributes of the nodes a v3 group's inline -`consolidated_metadata` holds may now hold `NaN`, `Infinity` and -`-Infinity`, in every reader, such as `read_array_metadata_v3`, and in -each model's `from_json`, `from_key_value` and `to_key_value`. Wherever -the spec interprets a value it spells those numbers as strings, so -anywhere else a non-finite number is still refused, and the store reader -now says where one is instead of failing to decode the document. - -A model whose document the reader refuses raises -`MetadataValidationError` with every problem, instead of being written: - -- a non-finite number outside attributes, which was refused before too; -- a non-string attribute key, which was written as a string; -- a value that is not JSON, which raised `TypeError`; -- anything else the reader refuses, such as `dimension_names` longer than - `shape`, which was written and then refused on read. - -Any sequence is written as an array. A number too large for a float, such -as `1e400`, is read as infinity, so in attributes it is now accepted and -written back as `Infinity`. - -Through `zarr_metadata.pydantic`, pydantic writes a non-finite attribute -as `null` unless the enclosing model sets `ser_json_inf_nan="constants"`. diff --git a/packages/zarr-metadata/changes/4421.bugfix.md b/packages/zarr-metadata/changes/4421.bugfix.md deleted file mode 100644 index cb9d53847a..0000000000 --- a/packages/zarr-metadata/changes/4421.bugfix.md +++ /dev/null @@ -1,4 +0,0 @@ -`from_key_value` takes a mapping keyed by `str` or by any narrower key -type, so reading back what `to_key_value` wrote type-checks. `to_key_value` -keys its result by the store-key literals, and since a mapping's key type is -invariant, a `Mapping[str, bytes]` parameter refused it. diff --git a/packages/zarr-metadata/changes/4422.bugfix.md b/packages/zarr-metadata/changes/4422.bugfix.md deleted file mode 100644 index 7bf24c6d2b..0000000000 --- a/packages/zarr-metadata/changes/4422.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -`MetadataValidationError` pickles and copies with its message. A copy was -built from the error's message, and its message came out as the -original's characters, one per line. diff --git a/packages/zarr-metadata/changes/4422.feature.md b/packages/zarr-metadata/changes/4422.feature.md deleted file mode 100644 index 360b0b1c35..0000000000 --- a/packages/zarr-metadata/changes/4422.feature.md +++ /dev/null @@ -1,7 +0,0 @@ -**Breaking:** `ValidationProblem` and `MetadataValidationError` check at run -time what their annotations declare, and raise `TypeError` otherwise. A -problem's `loc` is a tuple of strings and integers, so a list, which was -accepted before, is now refused, as is a `bool`; its `message` is a string -and its `kind` one of the kinds; an error holds problems. A caller that is -not type-checked fails where it builds the value, not later, where a -location is read. diff --git a/packages/zarr-metadata/changes/4423.bugfix.md b/packages/zarr-metadata/changes/4423.bugfix.md deleted file mode 100644 index dd4fb93604..0000000000 --- a/packages/zarr-metadata/changes/4423.bugfix.md +++ /dev/null @@ -1,7 +0,0 @@ -A `bytes` or `bytearray` is not a JSON array. Where a document expects an -array, an empty one passed as an array with no items: the v3 parser -returned `storage_transformers=b""`, and the v2 readers took `dtype=b""`, -then failed when the model was written. The readers and models now refuse -one at `storage_transformers` and at a v2 `dtype` and its field records. At -`codecs`, `dimension_names` and v2 `filters`, which already refused one, -the problem now says the value has the wrong type. diff --git a/packages/zarr-metadata/changes/4426.bugfix.md b/packages/zarr-metadata/changes/4426.bugfix.md deleted file mode 100644 index 5f015b1533..0000000000 --- a/packages/zarr-metadata/changes/4426.bugfix.md +++ /dev/null @@ -1,5 +0,0 @@ -A model read by `from_json` or `from_key_value` shares no mutable state -with the document it was read from, as a document from `to_json` already -shares none with its model. A frozen model held the caller's own -dictionaries, so editing the input after reading it changed the model, -and could make `to_key_value` fail. diff --git a/packages/zarr-metadata/changes/4428.bugfix.md b/packages/zarr-metadata/changes/4428.bugfix.md deleted file mode 100644 index 44a9b94e74..0000000000 --- a/packages/zarr-metadata/changes/4428.bugfix.md +++ /dev/null @@ -1,4 +0,0 @@ -`from_key_value` checks that each store value it reads is `bytes`, as its -signature says: a `str` was decoded as a JSON document, and `None` or a -`memoryview` raised `TypeError`. Each is now a `MetadataValidationError` -with an `invalid_type` problem at the store key. diff --git a/packages/zarr-metadata/changes/4430.misc.md b/packages/zarr-metadata/changes/4430.misc.md deleted file mode 100644 index dac4e75304..0000000000 --- a/packages/zarr-metadata/changes/4430.misc.md +++ /dev/null @@ -1,4 +0,0 @@ -`ZarrV2ZGroupJSON` and `ZarrV2GroupMetadataJSON` are closed -(PEP 728), as the spec's "Other keys MUST NOT be present" and the v2 group -validator already said, and as its pydantic schema already was. A type -checker now refuses a v2 group document with a key it does not declare. diff --git a/packages/zarr-metadata/changes/4432.feature.md b/packages/zarr-metadata/changes/4432.feature.md deleted file mode 100644 index 159b92c1e0..0000000000 --- a/packages/zarr-metadata/changes/4432.feature.md +++ /dev/null @@ -1,14 +0,0 @@ -`zarr_metadata.typed_json.check(value, SomeTypedDict)` type-checks a JSON -value against any of the package's TypedDicts -- `ZarrV3ArrayMetadataJSON`, -`GzipCodecConfiguration`, or one of your own -- and gives back a value of -it, or None, with every problem located. A TypedDict reads as the typing -spec defines it, whatever the runtime's attributes say: a key is required -by its own `Required` or `NotRequired`, or else by the `total` of the class -that declared it, read from the evaluated annotations, so a module written -with `from __future__ import annotations` reads like any other; `closed`, -`extra_items` and openness come from the class, or from its bases when it -says neither; and each annotation is evaluated in the module of the class -that wrote it. A key a closed TypedDict does not declare is reported, as -the new problem kind `unknown_key`, and left out of the value, which still -comes back; a union of TypedDicts tagged by a `Literal` key is read by the -branch the tag names. diff --git a/packages/zarr-metadata/changes/4434.feature.1.md b/packages/zarr-metadata/changes/4434.feature.1.md deleted file mode 100644 index 021452493e..0000000000 --- a/packages/zarr-metadata/changes/4434.feature.1.md +++ /dev/null @@ -1,10 +0,0 @@ -**Breaking:** `must_understand: false` is refused at every extension point -of an array document, codecs and storage transformers included, not only -at `data_type`, `chunk_grid` and `chunk_key_encoding`: ignoring a codec -gives wrong bytes as surely as ignoring a data type gives wrong values, -and the spec naming only three points is read as an oversight. A document -that declares a codec or a storage transformer ignorable now has a problem -at `codecs.N.must_understand` or `storage_transformers.N.must_understand`, -so the readers refuse it, and no model writes one. The JSON schema `zarr_metadata.pydantic` generates for an array -document says the same, and so does `resolve`, which reads one field on -its own. diff --git a/packages/zarr-metadata/changes/4434.feature.md b/packages/zarr-metadata/changes/4434.feature.md deleted file mode 100644 index 0e8a1d7e32..0000000000 --- a/packages/zarr-metadata/changes/4434.feature.md +++ /dev/null @@ -1,50 +0,0 @@ -Each extension's metadata is a definition: a value holding the name the -metadata carries, the TypedDict its configuration is -- the one -declaration of that JSON -- and plain functions over that TypedDict for -the rules. `zarr_metadata.definition` and `zarr_metadata.v3.definition` are the -doors, and their module docstrings the guide. Every extension the package models is one: the -codecs `blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`, -`sharding_indexed`, `transpose` and `zstd`; every data type, raw bits -included (`r16` is read by the `r*` definition with `{"bits": 16}`, the size -its name carries); the `regular` and `rectilinear` chunk grids; and the `default` -and `v2` chunk key encodings. Each is a constant in its own module -- -`GZIP_CODEC`, `STRUCT_DATA_TYPE` -- and `CORE` holds what the -specification defines, `CORE_AND_EXTENSIONS` what `zarr-extensions` -registers besides. A third party writes a TypedDict, a rule function and -a definition, and adds it to a scope; a definition checks itself -when built, and nothing happens at class creation. - -Reading a field is three steps, each usable on its own by a caller -holding nothing but JSON. `check(value, SomeTypedDict)`, from -`zarr_metadata.typed_json`, type-checks JSON against a TypedDict and needs -nothing else; `definition.read_configuration(configuration)` is the check, each nested -field's envelope judged, and then the rules; `resolve(field, -CodecDefinition, scope)` reads a whole field in a scope -- its envelope -judged, its name related to a definition, its configuration judged, each -nested field read the same way -- and returns what the scope made of it, -with every problem located. A name nothing in scope claims is left -unjudged. A shard's pipelines, a cast's target -type and a struct's field types are nested fields, annotated `ZarrV3CodecField` -and `ZarrV3DataTypeField`, and read in the scope their field is read in. -A codec definition also says whether the size -of what it gives out is `static` or `dynamic`, and a member annotated -`ZarrV3StaticCodecField` takes codecs of static size only: a shard's -`index_codecs`, where the spec forbids a compressor, so a `gzip` there is a -problem at its place. - -The TypedDict reads as the typing spec defines it -- `total`, `Required`, -`NotRequired`, `closed` and `extra_items` mean what they mean to a type -checker, whether or not its module postpones annotations. A key a closed -TypedDict does not declare is reported, as `unknown_key`, and left out of -what the rules are handed. A definition refuses a TypedDict that says -nothing of the keys it does not declare, and a member typed -`ZarrV3MetadataFieldJSON`, which would check as plain JSON. - -`canonical_of(field, problems)` gives a field without problems in its -simplest equivalent spelling: each nested field in its own simplest -spelling, then its definition's own `canonical`, and the envelope in the -fewest words every reader takes, with no `must_understand`. A field with -any problem, an unknown key included, has -none, since a simpler spelling would erase what its author wrote. What -`canonical` gives is judged again, and one that does not hold is a -`ValueError`, a fault in the definition. diff --git a/packages/zarr-metadata/changes/4436.feature.md b/packages/zarr-metadata/changes/4436.feature.md deleted file mode 100644 index 0b8fab9962..0000000000 --- a/packages/zarr-metadata/changes/4436.feature.md +++ /dev/null @@ -1,12 +0,0 @@ -**Breaking:** the v3 readers read each extension point through its -definition: `read_array_metadata_v3` judges the data type, the chunk -grid, the chunk key encoding, each codec and each storage transformer -against the definition that claims its name in a scope, -`CORE_AND_EXTENSIONS` unless a `context` is passed, so a gzip `level` of -99, or a key a codec does not declare, is a problem located in the -configuration, and a document holding one, accepted before, is refused. -So do the v3 group readers, and the v3 model classes' `from_json` and -`from_key_value` take the same `context`. -A name nothing in scope claims is left unjudged. A problem in a field -that a configuration holds, such as a codec inside a shard, is reported -at that field's own location, and the field that holds it is still read. diff --git a/packages/zarr-metadata/changes/4443.bugfix.md b/packages/zarr-metadata/changes/4443.bugfix.md deleted file mode 100644 index 645f8ec603..0000000000 --- a/packages/zarr-metadata/changes/4443.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -`ZarrV3ArrayMetadata.create_default(shape=...)` derives a chunk length of 1 -for a dimension of length 0, where it wrote 0: the regular grid asks for -chunk lengths greater than zero, and `zarr` does not open a grid with one. diff --git a/packages/zarr-metadata/changes/4443.feature.2.md b/packages/zarr-metadata/changes/4443.feature.2.md deleted file mode 100644 index 004658702b..0000000000 --- a/packages/zarr-metadata/changes/4443.feature.2.md +++ /dev/null @@ -1,10 +0,0 @@ -**Breaking:** a v3 array's `fill_value` is judged against its `data_type`. -Each data type's definition declares the JSON shape of its fill value, -`fill_value`, and the rules for one of that shape, `fill_value_rules`: an -`int8` fill value of 300, a `float32` hex string of another width, and a -struct fill value missing a field are each a problem at `fill_value`, -where the package accepted them before. `fill_value_problems(data_type, -value)` judges a fill value against a data type field a scope read, and -a field that is read keeps the fields it read inside as -`nested`, a `Nested` mapping by location. A data type nothing -in scope claims leaves its fill value unjudged. diff --git a/packages/zarr-metadata/changes/4443.feature.3.md b/packages/zarr-metadata/changes/4443.feature.3.md deleted file mode 100644 index f7c8bcd78b..0000000000 --- a/packages/zarr-metadata/changes/4443.feature.3.md +++ /dev/null @@ -1,6 +0,0 @@ -**Breaking:** a v3 array's `chunk_grid` is judged against its `shape`, by -the grid's definition. A regular grid whose `chunk_shape` does not have -one length per dimension of the shape, and a rectilinear grid whose -`chunk_shapes` does not have one entry per dimension, or whose chunk -lengths fall short of their dimension, each have a problem in -`chunk_grid.configuration`, where the package accepted them before. diff --git a/packages/zarr-metadata/changes/4443.feature.4.md b/packages/zarr-metadata/changes/4443.feature.4.md deleted file mode 100644 index 3684877d9d..0000000000 --- a/packages/zarr-metadata/changes/4443.feature.4.md +++ /dev/null @@ -1,20 +0,0 @@ -**Breaking:** a v3 array's `codecs` are read as a pipeline: in order -- -array -> array codecs, then one array -> bytes codec, then bytes -> bytes -codecs -- and each codec judged against the chunk it is handed, which is -the grid's chunks of the array's data type, as the array -> array codecs -before it hand them on. A codec out of order, a second array -> bytes -codec or none at all; a `bytes` codec without an `endian` handed numbers -of several bytes, or handed values that vary in size; a `transpose` whose -`order` has another number of axes than its chunk; a `cast_value` to or -from a data type that models no real numbers, wrapping to one that is -not integral, or mapping a scalar that is not a fill value of the data -type on its side; a `scale_offset` handed values that are no numbers, or -with an `offset` or `scale` that is not a value of its data type; and a -struct field whose values vary in size are each a problem where they -sit, where the package accepted them before. A reading records each -codec with the chunk it is handed, as a `Stage`, and the lengths a -grid's chunks take; `storage_of(data_type)` says how a data type's -values are stored. A -definition's `rules` are handed the fields its configuration holds as -the scope read them, and the kinds gain `chunk_lengths` (grids), -`storage` (data types), and `chunk_rules` and `transition` (codecs). diff --git a/packages/zarr-metadata/changes/4443.feature.md b/packages/zarr-metadata/changes/4443.feature.md deleted file mode 100644 index db1618e3ec..0000000000 --- a/packages/zarr-metadata/changes/4443.feature.md +++ /dev/null @@ -1,13 +0,0 @@ -**Breaking:** a v3 array's sharding codec is judged against the shard it -is handed, and its two pipelines are read as the array's is. Its -`chunk_shape` has an inner chunk length for each axis of the shard, -dividing each length the shards take along it -- the shard being the -chunk the codec is handed, so a transposed shard is divided along its -transposed axes. Its inner codecs are handed the inner chunks, of the -shard's data type, and its index codecs the shard index, of `uint64`, -with an axis more than the shard. A shard that does not divide, or a -pipeline that does not fit what it is handed -- an index `bytes` codec -without an `endian` -- has a problem where it sits, where the package -accepted it before. A codec definition -says what the pipelines it holds are handed, `pipelines`, and each -codec's `Stage` keeps the stages of those pipelines as `inner`. diff --git a/packages/zarr-metadata/changes/4490.feature.md b/packages/zarr-metadata/changes/4490.feature.md deleted file mode 100644 index 792c349520..0000000000 --- a/packages/zarr-metadata/changes/4490.feature.md +++ /dev/null @@ -1,17 +0,0 @@ -**Breaking:** metadata is now read against the definitions in a scope, a -`ZarrV3Context` or `ZarrV2Context` that a reader builds from the codecs, -data types and other fields it supports, and a model is the document -together with the scope it was read in. The `read_*` readers report every -problem at its location, `unclaimed_fields` lists each field the scope -does not know, so a reader can tell whether it can open a node, repair -readers undo known zarr-python writer bugs, and JSON Schemas are exported. -Code written for 0.5.0 must change: models are no longer dataclasses, -their field members hold `AcceptedField` or `UnclaimedField` values, -`update` takes JSON and takes members the spec does not define through -`extra_fields`, the `validate_*`, `is_*` and `parse_*` functions for -documents and the `...Partial` TypedDicts are removed, some names moved to -another module, the v2 constants and models follow the v3 names and -shapes, and a v2 time dtype without a unit is refused. `to_json` now -writes each field as it was read, rather than shortening a codec to a bare -name that zarr-python could not open, and zarr-metadata now depends on -`annotated-types`. From a15b3d75aeb141aa2fff1d43bb60895a6627d61a Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 26 Oct 2026 14:17:47 +0100 Subject: [PATCH 2/3] fix(zarr-metadata): render each release note as one list item on the docs site The docs render `CHANGELOG.md` with Python-Markdown, which needs four spaces to keep a second paragraph or a nested list inside a list item, and a blank line before the next item. towncrier's default template indents by two and puts no blank line between entries, so the 0.6.0 Features list ended at the first entry with a second paragraph, and the entries after it ran together as one paragraph on the release notes page. `changes/template.md`, this package's copy of towncrier's default Markdown template, indents each entry's continuation lines by four spaces and separates entries by a blank line. The 0.6.0 section is built again with it: each of its 23 entries is one list item, with its nested lists inside it, both on the docs site and in CommonMark, as GitHub renders it. The earlier releases' sections are unchanged. Assisted-by: ClaudeCode:claude-opus-5-5 --- packages/zarr-metadata/CHANGELOG.md | 489 +++++++++++---------- packages/zarr-metadata/changes/template.md | 130 ++++++ packages/zarr-metadata/pyproject.toml | 2 + 3 files changed, 386 insertions(+), 235 deletions(-) create mode 100644 packages/zarr-metadata/changes/template.md diff --git a/packages/zarr-metadata/CHANGELOG.md b/packages/zarr-metadata/CHANGELOG.md index 657e56373c..f7f3f3fa0d 100644 --- a/packages/zarr-metadata/CHANGELOG.md +++ b/packages/zarr-metadata/CHANGELOG.md @@ -7,272 +7,291 @@ ### Features - **Breaking:** `ValidationProblem` and `MetadataValidationError` check at run - time what their annotations declare, and raise `TypeError` otherwise. A - problem's `loc` is a tuple of strings and integers, so a list, which was - accepted before, is now refused, as is a `bool`; its `message` is a string - and its `kind` one of the kinds; an error holds problems. A caller that is - not type-checked fails where it builds the value, not later, where a - location is read. ([#4422](https://github.com/zarr-developers/zarr-python/pull/4422)) + time what their annotations declare, and raise `TypeError` otherwise. A + problem's `loc` is a tuple of strings and integers, so a list, which was + accepted before, is now refused, as is a `bool`; its `message` is a string + and its `kind` one of the kinds; an error holds problems. A caller that is + not type-checked fails where it builds the value, not later, where a + location is read. ([#4422](https://github.com/zarr-developers/zarr-python/pull/4422)) + - `zarr_metadata.typed_json.check(value, SomeTypedDict)` type-checks a JSON - value against any of the package's TypedDicts -- `ZarrV3ArrayMetadataJSON`, - `GzipCodecConfiguration`, or one of your own -- and gives back a value of - it, or None, with every problem located. A TypedDict reads as the typing - spec defines it, whatever the runtime's attributes say: a key is required - by its own `Required` or `NotRequired`, or else by the `total` of the class - that declared it, read from the evaluated annotations, so a module written - with `from __future__ import annotations` reads like any other; `closed`, - `extra_items` and openness come from the class, or from its bases when it - says neither; and each annotation is evaluated in the module of the class - that wrote it. A key a closed TypedDict does not declare is reported, as - the new problem kind `unknown_key`, and left out of the value, which still - comes back; a union of TypedDicts tagged by a `Literal` key is read by the - branch the tag names. ([#4432](https://github.com/zarr-developers/zarr-python/pull/4432)) + value against any of the package's TypedDicts -- `ZarrV3ArrayMetadataJSON`, + `GzipCodecConfiguration`, or one of your own -- and gives back a value of + it, or None, with every problem located. A TypedDict reads as the typing + spec defines it, whatever the runtime's attributes say: a key is required + by its own `Required` or `NotRequired`, or else by the `total` of the class + that declared it, read from the evaluated annotations, so a module written + with `from __future__ import annotations` reads like any other; `closed`, + `extra_items` and openness come from the class, or from its bases when it + says neither; and each annotation is evaluated in the module of the class + that wrote it. A key a closed TypedDict does not declare is reported, as + the new problem kind `unknown_key`, and left out of the value, which still + comes back; a union of TypedDicts tagged by a `Literal` key is read by the + branch the tag names. ([#4432](https://github.com/zarr-developers/zarr-python/pull/4432)) + - **Breaking:** `must_understand: false` is refused at every extension point - of an array document, codecs and storage transformers included, not only - at `data_type`, `chunk_grid` and `chunk_key_encoding`: ignoring a codec - gives wrong bytes as surely as ignoring a data type gives wrong values, - and the spec naming only three points is read as an oversight. A document - that declares a codec or a storage transformer ignorable now has a problem - at `codecs.N.must_understand` or `storage_transformers.N.must_understand`, - so the readers refuse it, and no model writes one. The JSON schema `zarr_metadata.pydantic` generates for an array - document says the same, and so does `resolve`, which reads one field on - its own. ([#4434](https://github.com/zarr-developers/zarr-python/pull/4434)) + of an array document, codecs and storage transformers included, not only + at `data_type`, `chunk_grid` and `chunk_key_encoding`: ignoring a codec + gives wrong bytes as surely as ignoring a data type gives wrong values, + and the spec naming only three points is read as an oversight. A document + that declares a codec or a storage transformer ignorable now has a problem + at `codecs.N.must_understand` or `storage_transformers.N.must_understand`, + so the readers refuse it, and no model writes one. The JSON schema `zarr_metadata.pydantic` generates for an array + document says the same, and so does `resolve`, which reads one field on + its own. ([#4434](https://github.com/zarr-developers/zarr-python/pull/4434)) + - Each extension's metadata is a definition: a value holding the name the - metadata carries, the TypedDict its configuration is -- the one - declaration of that JSON -- and plain functions over that TypedDict for - the rules. `zarr_metadata.definition` and `zarr_metadata.v3.definition` are the - doors, and their module docstrings the guide. Every extension the package models is one: the - codecs `blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`, - `sharding_indexed`, `transpose` and `zstd`; every data type, raw bits - included (`r16` is read by the `r*` definition with `{"bits": 16}`, the size - its name carries); the `regular` and `rectilinear` chunk grids; and the `default` - and `v2` chunk key encodings. Each is a constant in its own module -- - `GZIP_CODEC`, `STRUCT_DATA_TYPE` -- and `CORE` holds what the - specification defines, `CORE_AND_EXTENSIONS` what `zarr-extensions` - registers besides. A third party writes a TypedDict, a rule function and - a definition, and adds it to a scope; a definition checks itself - when built, and nothing happens at class creation. - - Reading a field is three steps, each usable on its own by a caller - holding nothing but JSON. `check(value, SomeTypedDict)`, from - `zarr_metadata.typed_json`, type-checks JSON against a TypedDict and needs - nothing else; `definition.read_configuration(configuration)` is the check, each nested - field's envelope judged, and then the rules; `resolve(field, - CodecDefinition, scope)` reads a whole field in a scope -- its envelope - judged, its name related to a definition, its configuration judged, each - nested field read the same way -- and returns what the scope made of it, - with every problem located. A name nothing in scope claims is left - unjudged. A shard's pipelines, a cast's target - type and a struct's field types are nested fields, annotated `ZarrV3CodecField` - and `ZarrV3DataTypeField`, and read in the scope their field is read in. - A codec definition also says whether the size - of what it gives out is `static` or `dynamic`, and a member annotated - `ZarrV3StaticCodecField` takes codecs of static size only: a shard's - `index_codecs`, where the spec forbids a compressor, so a `gzip` there is a - problem at its place. - - The TypedDict reads as the typing spec defines it -- `total`, `Required`, - `NotRequired`, `closed` and `extra_items` mean what they mean to a type - checker, whether or not its module postpones annotations. A key a closed - TypedDict does not declare is reported, as `unknown_key`, and left out of - what the rules are handed. A definition refuses a TypedDict that says - nothing of the keys it does not declare, and a member typed - `ZarrV3MetadataFieldJSON`, which would check as plain JSON. - - `canonical_of(field, problems)` gives a field without problems in its - simplest equivalent spelling: each nested field in its own simplest - spelling, then its definition's own `canonical`, and the envelope in the - fewest words every reader takes, with no `must_understand`. A field with - any problem, an unknown key included, has - none, since a simpler spelling would erase what its author wrote. What - `canonical` gives is judged again, and one that does not hold is a - `ValueError`, a fault in the definition. ([#4434](https://github.com/zarr-developers/zarr-python/pull/4434)) + metadata carries, the TypedDict its configuration is -- the one + declaration of that JSON -- and plain functions over that TypedDict for + the rules. `zarr_metadata.definition` and `zarr_metadata.v3.definition` are the + doors, and their module docstrings the guide. Every extension the package models is one: the + codecs `blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`, + `sharding_indexed`, `transpose` and `zstd`; every data type, raw bits + included (`r16` is read by the `r*` definition with `{"bits": 16}`, the size + its name carries); the `regular` and `rectilinear` chunk grids; and the `default` + and `v2` chunk key encodings. Each is a constant in its own module -- + `GZIP_CODEC`, `STRUCT_DATA_TYPE` -- and `CORE` holds what the + specification defines, `CORE_AND_EXTENSIONS` what `zarr-extensions` + registers besides. A third party writes a TypedDict, a rule function and + a definition, and adds it to a scope; a definition checks itself + when built, and nothing happens at class creation. + + Reading a field is three steps, each usable on its own by a caller + holding nothing but JSON. `check(value, SomeTypedDict)`, from + `zarr_metadata.typed_json`, type-checks JSON against a TypedDict and needs + nothing else; `definition.read_configuration(configuration)` is the check, each nested + field's envelope judged, and then the rules; `resolve(field, + CodecDefinition, scope)` reads a whole field in a scope -- its envelope + judged, its name related to a definition, its configuration judged, each + nested field read the same way -- and returns what the scope made of it, + with every problem located. A name nothing in scope claims is left + unjudged. A shard's pipelines, a cast's target + type and a struct's field types are nested fields, annotated `ZarrV3CodecField` + and `ZarrV3DataTypeField`, and read in the scope their field is read in. + A codec definition also says whether the size + of what it gives out is `static` or `dynamic`, and a member annotated + `ZarrV3StaticCodecField` takes codecs of static size only: a shard's + `index_codecs`, where the spec forbids a compressor, so a `gzip` there is a + problem at its place. + + The TypedDict reads as the typing spec defines it -- `total`, `Required`, + `NotRequired`, `closed` and `extra_items` mean what they mean to a type + checker, whether or not its module postpones annotations. A key a closed + TypedDict does not declare is reported, as `unknown_key`, and left out of + what the rules are handed. A definition refuses a TypedDict that says + nothing of the keys it does not declare, and a member typed + `ZarrV3MetadataFieldJSON`, which would check as plain JSON. + + `canonical_of(field, problems)` gives a field without problems in its + simplest equivalent spelling: each nested field in its own simplest + spelling, then its definition's own `canonical`, and the envelope in the + fewest words every reader takes, with no `must_understand`. A field with + any problem, an unknown key included, has + none, since a simpler spelling would erase what its author wrote. What + `canonical` gives is judged again, and one that does not hold is a + `ValueError`, a fault in the definition. ([#4434](https://github.com/zarr-developers/zarr-python/pull/4434)) + - **Breaking:** the v3 readers read each extension point through its - definition: `read_array_metadata_v3` judges the data type, the chunk - grid, the chunk key encoding, each codec and each storage transformer - against the definition that claims its name in a scope, - `CORE_AND_EXTENSIONS` unless a `context` is passed, so a gzip `level` of - 99, or a key a codec does not declare, is a problem located in the - configuration, and a document holding one, accepted before, is refused. - So do the v3 group readers, and the v3 model classes' `from_json` and - `from_key_value` take the same `context`. - A name nothing in scope claims is left unjudged. A problem in a field - that a configuration holds, such as a codec inside a shard, is reported - at that field's own location, and the field that holds it is still read. ([#4436](https://github.com/zarr-developers/zarr-python/pull/4436)) + definition: `read_array_metadata_v3` judges the data type, the chunk + grid, the chunk key encoding, each codec and each storage transformer + against the definition that claims its name in a scope, + `CORE_AND_EXTENSIONS` unless a `context` is passed, so a gzip `level` of + 99, or a key a codec does not declare, is a problem located in the + configuration, and a document holding one, accepted before, is refused. + So do the v3 group readers, and the v3 model classes' `from_json` and + `from_key_value` take the same `context`. + A name nothing in scope claims is left unjudged. A problem in a field + that a configuration holds, such as a codec inside a shard, is reported + at that field's own location, and the field that holds it is still read. ([#4436](https://github.com/zarr-developers/zarr-python/pull/4436)) + - **Breaking:** a v3 array's `fill_value` is judged against its `data_type`. - Each data type's definition declares the JSON shape of its fill value, - `fill_value`, and the rules for one of that shape, `fill_value_rules`: an - `int8` fill value of 300, a `float32` hex string of another width, and a - struct fill value missing a field are each a problem at `fill_value`, - where the package accepted them before. `fill_value_problems(data_type, - value)` judges a fill value against a data type field a scope read, and - a field that is read keeps the fields it read inside as - `nested`, a `Nested` mapping by location. A data type nothing - in scope claims leaves its fill value unjudged. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + Each data type's definition declares the JSON shape of its fill value, + `fill_value`, and the rules for one of that shape, `fill_value_rules`: an + `int8` fill value of 300, a `float32` hex string of another width, and a + struct fill value missing a field are each a problem at `fill_value`, + where the package accepted them before. `fill_value_problems(data_type, + value)` judges a fill value against a data type field a scope read, and + a field that is read keeps the fields it read inside as + `nested`, a `Nested` mapping by location. A data type nothing + in scope claims leaves its fill value unjudged. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + - **Breaking:** a v3 array's sharding codec is judged against the shard it - is handed, and its two pipelines are read as the array's is. Its - `chunk_shape` has an inner chunk length for each axis of the shard, - dividing each length the shards take along it -- the shard being the - chunk the codec is handed, so a transposed shard is divided along its - transposed axes. Its inner codecs are handed the inner chunks, of the - shard's data type, and its index codecs the shard index, of `uint64`, - with an axis more than the shard. A shard that does not divide, or a - pipeline that does not fit what it is handed -- an index `bytes` codec - without an `endian` -- has a problem where it sits, where the package - accepted it before. A codec definition - says what the pipelines it holds are handed, `pipelines`, and each - codec's `Stage` keeps the stages of those pipelines as `inner`. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + is handed, and its two pipelines are read as the array's is. Its + `chunk_shape` has an inner chunk length for each axis of the shard, + dividing each length the shards take along it -- the shard being the + chunk the codec is handed, so a transposed shard is divided along its + transposed axes. Its inner codecs are handed the inner chunks, of the + shard's data type, and its index codecs the shard index, of `uint64`, + with an axis more than the shard. A shard that does not divide, or a + pipeline that does not fit what it is handed -- an index `bytes` codec + without an `endian` -- has a problem where it sits, where the package + accepted it before. A codec definition + says what the pipelines it holds are handed, `pipelines`, and each + codec's `Stage` keeps the stages of those pipelines as `inner`. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + - **Breaking:** a v3 array's `chunk_grid` is judged against its `shape`, by - the grid's definition. A regular grid whose `chunk_shape` does not have - one length per dimension of the shape, and a rectilinear grid whose - `chunk_shapes` does not have one entry per dimension, or whose chunk - lengths fall short of their dimension, each have a problem in - `chunk_grid.configuration`, where the package accepted them before. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + the grid's definition. A regular grid whose `chunk_shape` does not have + one length per dimension of the shape, and a rectilinear grid whose + `chunk_shapes` does not have one entry per dimension, or whose chunk + lengths fall short of their dimension, each have a problem in + `chunk_grid.configuration`, where the package accepted them before. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + - **Breaking:** a v3 array's `codecs` are read as a pipeline: in order -- - array -> array codecs, then one array -> bytes codec, then bytes -> bytes - codecs -- and each codec judged against the chunk it is handed, which is - the grid's chunks of the array's data type, as the array -> array codecs - before it hand them on. A codec out of order, a second array -> bytes - codec or none at all; a `bytes` codec without an `endian` handed numbers - of several bytes, or handed values that vary in size; a `transpose` whose - `order` has another number of axes than its chunk; a `cast_value` to or - from a data type that models no real numbers, wrapping to one that is - not integral, or mapping a scalar that is not a fill value of the data - type on its side; a `scale_offset` handed values that are no numbers, or - with an `offset` or `scale` that is not a value of its data type; and a - struct field whose values vary in size are each a problem where they - sit, where the package accepted them before. A reading records each - codec with the chunk it is handed, as a `Stage`, and the lengths a - grid's chunks take; `storage_of(data_type)` says how a data type's - values are stored. A - definition's `rules` are handed the fields its configuration holds as - the scope read them, and the kinds gain `chunk_lengths` (grids), - `storage` (data types), and `chunk_rules` and `transition` (codecs). ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + array -> array codecs, then one array -> bytes codec, then bytes -> bytes + codecs -- and each codec judged against the chunk it is handed, which is + the grid's chunks of the array's data type, as the array -> array codecs + before it hand them on. A codec out of order, a second array -> bytes + codec or none at all; a `bytes` codec without an `endian` handed numbers + of several bytes, or handed values that vary in size; a `transpose` whose + `order` has another number of axes than its chunk; a `cast_value` to or + from a data type that models no real numbers, wrapping to one that is + not integral, or mapping a scalar that is not a fill value of the data + type on its side; a `scale_offset` handed values that are no numbers, or + with an `offset` or `scale` that is not a value of its data type; and a + struct field whose values vary in size are each a problem where they + sit, where the package accepted them before. A reading records each + codec with the chunk it is handed, as a `Stage`, and the lengths a + grid's chunks take; `storage_of(data_type)` says how a data type's + values are stored. A + definition's `rules` are handed the fields its configuration holds as + the scope read them, and the kinds gain `chunk_lengths` (grids), + `storage` (data types), and `chunk_rules` and `transition` (codecs). ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + - **Breaking:** metadata is now read against the definitions in a scope, a - `ZarrV3Context` or `ZarrV2Context` that a reader builds from the codecs, - data types and other fields it supports, and a model is the document - together with the scope it was read in. The `read_*` readers report every - problem at its location, `unclaimed_fields` lists each field the scope - does not know, so a reader can tell whether it can open a node, repair - readers undo known zarr-python writer bugs, and JSON Schemas are exported. - Code written for 0.5.0 must change: models are no longer dataclasses, - their field members hold `AcceptedField` or `UnclaimedField` values, - `update` takes JSON and takes members the spec does not define through - `extra_fields`, the `validate_*`, `is_*` and `parse_*` functions for - documents and the `...Partial` TypedDicts are removed, some names moved to - another module, the v2 constants and models follow the v3 names and - shapes, and a v2 time dtype without a unit is refused. `to_json` now - writes each field as it was read, rather than shortening a codec to a bare - name that zarr-python could not open, and zarr-metadata now depends on - `annotated-types`. ([#4490](https://github.com/zarr-developers/zarr-python/pull/4490)) + `ZarrV3Context` or `ZarrV2Context` that a reader builds from the codecs, + data types and other fields it supports, and a model is the document + together with the scope it was read in. The `read_*` readers report every + problem at its location, `unclaimed_fields` lists each field the scope + does not know, so a reader can tell whether it can open a node, repair + readers undo known zarr-python writer bugs, and JSON Schemas are exported. + Code written for 0.5.0 must change: models are no longer dataclasses, + their field members hold `AcceptedField` or `UnclaimedField` values, + `update` takes JSON and takes members the spec does not define through + `extra_fields`, the `validate_*`, `is_*` and `parse_*` functions for + documents and the `...Partial` TypedDicts are removed, some names moved to + another module, the v2 constants and models follow the v3 names and + shapes, and a v2 time dtype without a unit is refused. `to_json` now + writes each field as it was read, rather than shortening a codec to a bare + name that zarr-python could not open, and zarr-metadata now depends on + `annotated-types`. ([#4490](https://github.com/zarr-developers/zarr-python/pull/4490)) ### Bugfixes - The v2 array validator follows the spec on two points it was stricter than: members outside the `.zarray` definition are accepted when they are JSON values with string keys, and are kept and written back by `to_json` and `to_key_value`; any other member is a problem, as in a v3 document (the spec says other keys ["SHOULD NOT be present ... and SHOULD be ignored"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L91-L92), unlike `.zgroup`'s ["MUST NOT"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L313)), and `filters: []` is accepted (["a list of JSON objects providing codec configurations, or null"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L76-L79) sets no minimum). The JSON Schema that `zarr_metadata.pydantic` generates follows: it accepts other members, and the filter pipeline has no minimum length. ([#4365](https://github.com/zarr-developers/zarr-python/pull/4365)) + - Concrete v3 entity types are now assignable to the fields they describe. - Previously, none of the package's canonical codec / chunk-grid / - chunk-key-encoding / data-type types (e.g. `BloscCodecMetadata`, - `RegularChunkGridMetadata`) satisfied `ZarrV3MetadataFieldJSON`, so a - type checker rejected putting them into the very fields they document - (`codecs`, `chunk_grid`, `data_type`, ...). Three changes fix this: - - - `ZarrV3NamedConfigJSON.name` and `.configuration` are now `ReadOnly` - (PEP 705), making them covariant so concrete `name: Literal[...]` and - required-`configuration` shapes are accepted. - - `ZarrV3NamedConfigJSON` is now `closed` (PEP 728): the spec's - named-configuration envelope has exactly `name` / `configuration` / - `must_understand`, and closing the type also makes it usable as a - `JSONValue` (needed for e.g. the `sharding_indexed` inner `codecs`). - - Every concrete `*Object` / `*Configuration` TypedDict is now `closed`, - and object forms declare `must_understand: NotRequired[bool]` (any v3 - metadata field may carry the extension member). - - **Soft-breaking** for type-checking consumers: dicts with keys beyond the - declared shape no longer satisfy the closed types, and `name` / - `configuration` can no longer be mutated through `ZarrV3NamedConfigJSON`. - Both were previously accepted by type checkers but produced documents - outside the spec's shapes. Generating a pydantic JSON Schema for - the `zarr_metadata.pydantic` field types no longer emits warnings. - - ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) + Previously, none of the package's canonical codec / chunk-grid / + chunk-key-encoding / data-type types (e.g. `BloscCodecMetadata`, + `RegularChunkGridMetadata`) satisfied `ZarrV3MetadataFieldJSON`, so a + type checker rejected putting them into the very fields they document + (`codecs`, `chunk_grid`, `data_type`, ...). Three changes fix this: + + - `ZarrV3NamedConfigJSON.name` and `.configuration` are now `ReadOnly` + (PEP 705), making them covariant so concrete `name: Literal[...]` and + required-`configuration` shapes are accepted. + - `ZarrV3NamedConfigJSON` is now `closed` (PEP 728): the spec's + named-configuration envelope has exactly `name` / `configuration` / + `must_understand`, and closing the type also makes it usable as a + `JSONValue` (needed for e.g. the `sharding_indexed` inner `codecs`). + - Every concrete `*Object` / `*Configuration` TypedDict is now `closed`, + and object forms declare `must_understand: NotRequired[bool]` (any v3 + metadata field may carry the extension member). + + **Soft-breaking** for type-checking consumers: dicts with keys beyond the + declared shape no longer satisfy the closed types, and `name` / + `configuration` can no longer be mutated through `ZarrV3NamedConfigJSON`. + Both were previously accepted by type checkers but produced documents + outside the spec's shapes. Generating a pydantic JSON Schema for + the `zarr_metadata.pydantic` field types no longer emits warnings. + + ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) + - Attributes are user data, and a non-finite number in them is read, - validated and written back. zarr-python writes attributes with the - defaults of Python's `json` module, so an xarray `_FillValue` or a CF - `missing_value` of NaN is stored as a bare `NaN`, which RFC 8259 lacks; - the model refused such a document and could not read the store at all. - A node's `attributes`, a v2 `.zattrs`, the `.zattrs` entries of a v2 - `.zmetadata`, and the attributes of the nodes a v3 group's inline - `consolidated_metadata` holds may now hold `NaN`, `Infinity` and - `-Infinity`, in every reader, such as `read_array_metadata_v3`, and in - each model's `from_json`, `from_key_value` and `to_key_value`. Wherever - the spec interprets a value it spells those numbers as strings, so - anywhere else a non-finite number is still refused, and the store reader - now says where one is instead of failing to decode the document. - - A model whose document the reader refuses raises - `MetadataValidationError` with every problem, instead of being written: - - - a non-finite number outside attributes, which was refused before too; - - a non-string attribute key, which was written as a string; - - a value that is not JSON, which raised `TypeError`; - - anything else the reader refuses, such as `dimension_names` longer than - `shape`, which was written and then refused on read. - - Any sequence is written as an array. A number too large for a float, such - as `1e400`, is read as infinity, so in attributes it is now accepted and - written back as `Infinity`. - - Through `zarr_metadata.pydantic`, pydantic writes a non-finite attribute - as `null` unless the enclosing model sets `ser_json_inf_nan="constants"`. - - ([#4420](https://github.com/zarr-developers/zarr-python/pull/4420)) + validated and written back. zarr-python writes attributes with the + defaults of Python's `json` module, so an xarray `_FillValue` or a CF + `missing_value` of NaN is stored as a bare `NaN`, which RFC 8259 lacks; + the model refused such a document and could not read the store at all. + A node's `attributes`, a v2 `.zattrs`, the `.zattrs` entries of a v2 + `.zmetadata`, and the attributes of the nodes a v3 group's inline + `consolidated_metadata` holds may now hold `NaN`, `Infinity` and + `-Infinity`, in every reader, such as `read_array_metadata_v3`, and in + each model's `from_json`, `from_key_value` and `to_key_value`. Wherever + the spec interprets a value it spells those numbers as strings, so + anywhere else a non-finite number is still refused, and the store reader + now says where one is instead of failing to decode the document. + + A model whose document the reader refuses raises + `MetadataValidationError` with every problem, instead of being written: + + - a non-finite number outside attributes, which was refused before too; + - a non-string attribute key, which was written as a string; + - a value that is not JSON, which raised `TypeError`; + - anything else the reader refuses, such as `dimension_names` longer than + `shape`, which was written and then refused on read. + + Any sequence is written as an array. A number too large for a float, such + as `1e400`, is read as infinity, so in attributes it is now accepted and + written back as `Infinity`. + + Through `zarr_metadata.pydantic`, pydantic writes a non-finite attribute + as `null` unless the enclosing model sets `ser_json_inf_nan="constants"`. + + ([#4420](https://github.com/zarr-developers/zarr-python/pull/4420)) + - `from_key_value` takes a mapping keyed by `str` or by any narrower key - type, so reading back what `to_key_value` wrote type-checks. `to_key_value` - keys its result by the store-key literals, and since a mapping's key type is - invariant, a `Mapping[str, bytes]` parameter refused it. ([#4421](https://github.com/zarr-developers/zarr-python/pull/4421)) + type, so reading back what `to_key_value` wrote type-checks. `to_key_value` + keys its result by the store-key literals, and since a mapping's key type is + invariant, a `Mapping[str, bytes]` parameter refused it. ([#4421](https://github.com/zarr-developers/zarr-python/pull/4421)) + - `MetadataValidationError` pickles and copies with its message. A copy was - built from the error's message, and its message came out as the - original's characters, one per line. ([#4422](https://github.com/zarr-developers/zarr-python/pull/4422)) + built from the error's message, and its message came out as the + original's characters, one per line. ([#4422](https://github.com/zarr-developers/zarr-python/pull/4422)) + - A `bytes` or `bytearray` is not a JSON array. Where a document expects an - array, an empty one passed as an array with no items: the v3 parser - returned `storage_transformers=b""`, and the v2 readers took `dtype=b""`, - then failed when the model was written. The readers and models now refuse - one at `storage_transformers` and at a v2 `dtype` and its field records. At - `codecs`, `dimension_names` and v2 `filters`, which already refused one, - the problem now says the value has the wrong type. ([#4423](https://github.com/zarr-developers/zarr-python/pull/4423)) + array, an empty one passed as an array with no items: the v3 parser + returned `storage_transformers=b""`, and the v2 readers took `dtype=b""`, + then failed when the model was written. The readers and models now refuse + one at `storage_transformers` and at a v2 `dtype` and its field records. At + `codecs`, `dimension_names` and v2 `filters`, which already refused one, + the problem now says the value has the wrong type. ([#4423](https://github.com/zarr-developers/zarr-python/pull/4423)) + - A model read by `from_json` or `from_key_value` shares no mutable state - with the document it was read from, as a document from `to_json` already - shares none with its model. A frozen model held the caller's own - dictionaries, so editing the input after reading it changed the model, - and could make `to_key_value` fail. ([#4426](https://github.com/zarr-developers/zarr-python/pull/4426)) + with the document it was read from, as a document from `to_json` already + shares none with its model. A frozen model held the caller's own + dictionaries, so editing the input after reading it changed the model, + and could make `to_key_value` fail. ([#4426](https://github.com/zarr-developers/zarr-python/pull/4426)) + - `from_key_value` checks that each store value it reads is `bytes`, as its - signature says: a `str` was decoded as a JSON document, and `None` or a - `memoryview` raised `TypeError`. Each is now a `MetadataValidationError` - with an `invalid_type` problem at the store key. ([#4428](https://github.com/zarr-developers/zarr-python/pull/4428)) + signature says: a `str` was decoded as a JSON document, and `None` or a + `memoryview` raised `TypeError`. Each is now a `MetadataValidationError` + with an `invalid_type` problem at the store key. ([#4428](https://github.com/zarr-developers/zarr-python/pull/4428)) + - `ZarrV3ArrayMetadata.create_default(shape=...)` derives a chunk length of 1 - for a dimension of length 0, where it wrote 0: the regular grid asks for - chunk lengths greater than zero, and `zarr` does not open a grid with one. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + for a dimension of length 0, where it wrote 0: the regular grid asks for + chunk lengths greater than zero, and `zarr` does not open a grid with one. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) ### Improved Documentation - The docs now render the release notes as a page instead of linking to - `CHANGELOG.md` on GitHub, so each docs version shows its own changelog. ([#4311](https://github.com/zarr-developers/zarr-python/pull/4311)) + `CHANGELOG.md` on GitHub, so each docs version shows its own changelog. ([#4311](https://github.com/zarr-developers/zarr-python/pull/4311)) ### Misc - `ZarrV2ConsolidatedMetadataJSON.zarr_consolidated_format` is typed - `Literal[1]` rather than `int`. Format 1 is the only defined `.zmetadata` - format and the runtime validator already rejected anything else, so the - type now carries the constraint the validator enforces instead of - contradicting it. ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) + `Literal[1]` rather than `int`. Format 1 is the only defined `.zmetadata` + format and the runtime validator already rejected anything else, so the + type now carries the constraint the validator enforces instead of + contradicting it. ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) + - **Breaking:** `MetadataValidationError.problems` is now a tuple instead of - a list. Iteration and indexing are unchanged; code that changes a report - must first copy it with `list(problems)`. `MetadataValidationError` still - accepts any problem sequence. ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) + a list. Iteration and indexing are unchanged; code that changes a report + must first copy it with `list(problems)`. `MetadataValidationError` still + accepts any problem sequence. ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) + - `ZarrV2ZGroupJSON` and `ZarrV2GroupMetadataJSON` are closed - (PEP 728), as the spec's "Other keys MUST NOT be present" and the v2 group - validator already said, and as its pydantic schema already was. A type - checker now refuses a v2 group document with a key it does not declare. ([#4430](https://github.com/zarr-developers/zarr-python/pull/4430)) + (PEP 728), as the spec's "Other keys MUST NOT be present" and the v2 group + validator already said, and as its pydantic schema already was. A type + checker now refuses a v2 group document with a key it does not declare. ([#4430](https://github.com/zarr-developers/zarr-python/pull/4430)) ## 0.5.0 (2026-08-14) diff --git a/packages/zarr-metadata/changes/template.md b/packages/zarr-metadata/changes/template.md new file mode 100644 index 0000000000..d3616fe8c5 --- /dev/null +++ b/packages/zarr-metadata/changes/template.md @@ -0,0 +1,130 @@ +{#- +This package's copy of towncrier's default Markdown template. The one +change: each entry's continuation lines are indented by four spaces, not +two, and entries are separated by a blank line, so an entry with a +second paragraph or a nested list stays one list item in Python-Markdown, +which renders the docs' release notes page, as well as in CommonMark. +Without both, Python-Markdown ends the list at the second paragraph and +runs the later entries together as one paragraph. +-#} +{#- +══════════════════════════════════════════════════════════════════════════════ +TOWNCRIER MARKDOWN TEMPLATE +══════════════════════════════════════════════════════════════════════════════ + +─── Macro: heading ───────────────────────────────────────────────────────── +Purpose: + Generates Markdown headings with the appropriate number of # characters. + Based on header_prefix (default: "#") and the level argument. + +Arguments: + level The relative heading level (1=#, 2=##, 3=###, etc.) +-#} +{%- macro heading(level) -%} + {{- "#" * ( header_prefix | length + level -1 ) }} +{%- endmacro -%} + +{#- +─── Variable: newline ────────────────────────────────────────────────────── +Purpose: + Consistent newline handling. -#} +{%- set newline = "\n" -%} + +{#- ════════════════════════ TEMPLATE GENERATION ════════════════════════ -#} +{#- ─── TITLE HEADING ─── #} +{#- render_title is false when title_format is specified in the config #} +{%- if render_title %} + {%- if versiondata.name %} + {{- heading(1) ~ " " ~ versiondata.name ~ " " ~ versiondata.version ~ " (" ~ versiondata.date ~ ")" ~ newline }} + {%- else %} + {{- heading(1) ~ " " ~ versiondata.version ~ " (" ~ versiondata.date ~ ")" ~ newline }} + {%- endif %} +{%- endif %} +{#- If title_format is specified, we start with a new line #} +{{- newline }} + +{%- for section, _ in sections.items() %} + {#- ─── SECTION HEADING ─── #} + {%- if section %} + {{- newline }} + {{- heading(2) ~ " " ~ section ~ newline }} + {{- newline }} + {%- endif %} + + {%- if sections[section] %} + + {%- for category, val in definitions.items() if category in sections[section] %} + {%- set issue_pks = [] %} + {#- ─── CATEGORY HEADING ─── #} + {#- Increase heading level if section is not present #} + {{- heading(3 if section else 2) ~" " ~ definitions[category]['name'] ~ newline }} + {{- newline }} + + {#- ─── RENDER ENTRIES ─── #} + {%- for text, values in sections[section][category].items() %} + {#- A blank line between entries: Python-Markdown starts a new list + item after an entry of several paragraphs only after one. #} + {%- if not loop.first %} + {{- newline }} + {%- endif %} + {#- Prepare the string of issue numbers (e.g., "#1, #9, #142") #} + {%- set issue_pks = [] %} + {%- for v_issue in values %} + {%- set _= issue_pks.append(v_issue.split(": ", 1)[0]) %} + {%- endfor %} + {%- set issues_list = issue_pks | join(", ") %} + + {#- Check if text contains a sublist #} + {%- set text_has_sublist = (("\n - " in text) or ("\n * " in text)) %} + + {#- CASE 1: No text, only issues #} + {#- Output: - #1, #9, #142 #} + {%- if not text and issues_list %} + {{- "- " ~ issues_list ~ newline }} + + {#- Cases where both text and issues exist #} + {%- elif text and issues_list %} + {%- if text_has_sublist %} + {#- CASE 3: Text with sublist #} + {#- Output: - TEXT\n\n (#1, #9, #142) #} + {{- "- " ~ (text | indent(2)) ~ newline ~ newline ~ " (" ~ issues_list ~ ")" ~ newline }} + {%- else %} + {#- CASE 2: Text, no sublist #} + {#- Output: - TEXT (#1, #9, #142) #} + {{- "- " ~ (text | indent(2)) ~ " (" ~ issues_list ~ ")" ~ newline }} + {%- endif %} + + {%- elif text %} + {#- Implicit Case: Text, but no issues #} + {#- Output: - TEXT #} + {{- "- " ~ (text | indent(2)) ~ newline }} + {%- endif %} + {%- endfor %} + + {#- New line between list and link references #} + {{- newline }} + + {#- Link references #} + {%- if issues_by_category[section][category] and "]: " in issues_by_category[section][category][0] %} + {%- for issue in issues_by_category[section][category] %} + {{- issue ~ newline }} + {%- endfor %} + {{- newline }} + {%- endif %} + + {#- No changes in this category #} + {%- if sections[section][category]|length == 0 %} + {{- newline }} + {{- "No significant changes." ~ newline * 2 }} + {%- endif %} + {%- endfor %} + {%- else %} + {#- No changes in this section #} + {{- "No significant changes." ~ newline * 2 }} + {%- endif %} +{%- endfor %} +{#- +Newline at the end of the rendered newsfile content. +In this way the there are 2 newlines between the latest release and the previous release content. +-#} +{{- newline -}} diff --git a/packages/zarr-metadata/pyproject.toml b/packages/zarr-metadata/pyproject.toml index 831050c569..fa30dee92d 100644 --- a/packages/zarr-metadata/pyproject.toml +++ b/packages/zarr-metadata/pyproject.toml @@ -199,6 +199,8 @@ underlines = ["", "", ""] title_format = "## {version} ({project_date})" issue_format = "[#{issue}](https://github.com/zarr-developers/zarr-python/pull/{issue})" start_string = "\n" +# Indents entries by four spaces, so the docs render multi-paragraph entries as one list item. +template = "changes/template.md" # Declaring any type replaces towncrier's built-in set, so all five the # `changes/README.md` menu offers are restated here. They are the defaults From bcfaaae6a1abe4a99195a5e913ddf420ac63274e Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 26 Oct 2026 14:21:27 +0100 Subject: [PATCH 3/3] fix(zarr-metadata): keep the towncrier template out of the changes directory The changelog check accepts only fragments, `.gitignore` and `README.md` in a changes directory, so `changes/template.md` failed it. The template is now `towncrier-template.md` in the package directory. Its name still ends in `.md`, which is how towncrier knows it is a Markdown template. Built from the 0.6.0 fragments, it gives the same `CHANGELOG.md`. Assisted-by: ClaudeCode:claude-opus-5-5 --- packages/zarr-metadata/pyproject.toml | 6 ++++-- .../{changes/template.md => towncrier-template.md} | 0 2 files changed, 4 insertions(+), 2 deletions(-) rename packages/zarr-metadata/{changes/template.md => towncrier-template.md} (100%) diff --git a/packages/zarr-metadata/pyproject.toml b/packages/zarr-metadata/pyproject.toml index fa30dee92d..abd93bc729 100644 --- a/packages/zarr-metadata/pyproject.toml +++ b/packages/zarr-metadata/pyproject.toml @@ -199,8 +199,10 @@ underlines = ["", "", ""] title_format = "## {version} ({project_date})" issue_format = "[#{issue}](https://github.com/zarr-developers/zarr-python/pull/{issue})" start_string = "\n" -# Indents entries by four spaces, so the docs render multi-paragraph entries as one list item. -template = "changes/template.md" +# Indents entries by four spaces, so the docs render multi-paragraph entries +# as one list item. It lives outside `changes/`, which holds only fragments, +# and its name ends in `.md`, which is how towncrier knows it is Markdown. +template = "towncrier-template.md" # Declaring any type replaces towncrier's built-in set, so all five the # `changes/README.md` menu offers are restated here. They are the defaults diff --git a/packages/zarr-metadata/changes/template.md b/packages/zarr-metadata/towncrier-template.md similarity index 100% rename from packages/zarr-metadata/changes/template.md rename to packages/zarr-metadata/towncrier-template.md