diff --git a/packages/zarr-metadata/CHANGELOG.md b/packages/zarr-metadata/CHANGELOG.md index c1e9f81a61..f7f3f3fa0d 100644 --- a/packages/zarr-metadata/CHANGELOG.md +++ b/packages/zarr-metadata/CHANGELOG.md @@ -2,6 +2,298 @@ +## 0.6.0 (2026-10-26) + +### Features + +- **Breaking:** `ValidationProblem` and `MetadataValidationError` check at run + time what their annotations declare, and raise `TypeError` otherwise. A + problem's `loc` is a tuple of strings and integers, so a list, which was + accepted before, is now refused, as is a `bool`; its `message` is a string + and its `kind` one of the kinds; an error holds problems. A caller that is + not type-checked fails where it builds the value, not later, where a + location is read. ([#4422](https://github.com/zarr-developers/zarr-python/pull/4422)) + +- `zarr_metadata.typed_json.check(value, SomeTypedDict)` type-checks a JSON + value against any of the package's TypedDicts -- `ZarrV3ArrayMetadataJSON`, + `GzipCodecConfiguration`, or one of your own -- and gives back a value of + it, or None, with every problem located. A TypedDict reads as the typing + spec defines it, whatever the runtime's attributes say: a key is required + by its own `Required` or `NotRequired`, or else by the `total` of the class + that declared it, read from the evaluated annotations, so a module written + with `from __future__ import annotations` reads like any other; `closed`, + `extra_items` and openness come from the class, or from its bases when it + says neither; and each annotation is evaluated in the module of the class + that wrote it. A key a closed TypedDict does not declare is reported, as + the new problem kind `unknown_key`, and left out of the value, which still + comes back; a union of TypedDicts tagged by a `Literal` key is read by the + branch the tag names. ([#4432](https://github.com/zarr-developers/zarr-python/pull/4432)) + +- **Breaking:** `must_understand: false` is refused at every extension point + of an array document, codecs and storage transformers included, not only + at `data_type`, `chunk_grid` and `chunk_key_encoding`: ignoring a codec + gives wrong bytes as surely as ignoring a data type gives wrong values, + and the spec naming only three points is read as an oversight. A document + that declares a codec or a storage transformer ignorable now has a problem + at `codecs.N.must_understand` or `storage_transformers.N.must_understand`, + so the readers refuse it, and no model writes one. The JSON schema `zarr_metadata.pydantic` generates for an array + document says the same, and so does `resolve`, which reads one field on + its own. ([#4434](https://github.com/zarr-developers/zarr-python/pull/4434)) + +- Each extension's metadata is a definition: a value holding the name the + metadata carries, the TypedDict its configuration is -- the one + declaration of that JSON -- and plain functions over that TypedDict for + the rules. `zarr_metadata.definition` and `zarr_metadata.v3.definition` are the + doors, and their module docstrings the guide. Every extension the package models is one: the + codecs `blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`, + `sharding_indexed`, `transpose` and `zstd`; every data type, raw bits + included (`r16` is read by the `r*` definition with `{"bits": 16}`, the size + its name carries); the `regular` and `rectilinear` chunk grids; and the `default` + and `v2` chunk key encodings. Each is a constant in its own module -- + `GZIP_CODEC`, `STRUCT_DATA_TYPE` -- and `CORE` holds what the + specification defines, `CORE_AND_EXTENSIONS` what `zarr-extensions` + registers besides. A third party writes a TypedDict, a rule function and + a definition, and adds it to a scope; a definition checks itself + when built, and nothing happens at class creation. + + Reading a field is three steps, each usable on its own by a caller + holding nothing but JSON. `check(value, SomeTypedDict)`, from + `zarr_metadata.typed_json`, type-checks JSON against a TypedDict and needs + nothing else; `definition.read_configuration(configuration)` is the check, each nested + field's envelope judged, and then the rules; `resolve(field, + CodecDefinition, scope)` reads a whole field in a scope -- its envelope + judged, its name related to a definition, its configuration judged, each + nested field read the same way -- and returns what the scope made of it, + with every problem located. A name nothing in scope claims is left + unjudged. A shard's pipelines, a cast's target + type and a struct's field types are nested fields, annotated `ZarrV3CodecField` + and `ZarrV3DataTypeField`, and read in the scope their field is read in. + A codec definition also says whether the size + of what it gives out is `static` or `dynamic`, and a member annotated + `ZarrV3StaticCodecField` takes codecs of static size only: a shard's + `index_codecs`, where the spec forbids a compressor, so a `gzip` there is a + problem at its place. + + The TypedDict reads as the typing spec defines it -- `total`, `Required`, + `NotRequired`, `closed` and `extra_items` mean what they mean to a type + checker, whether or not its module postpones annotations. A key a closed + TypedDict does not declare is reported, as `unknown_key`, and left out of + what the rules are handed. A definition refuses a TypedDict that says + nothing of the keys it does not declare, and a member typed + `ZarrV3MetadataFieldJSON`, which would check as plain JSON. + + `canonical_of(field, problems)` gives a field without problems in its + simplest equivalent spelling: each nested field in its own simplest + spelling, then its definition's own `canonical`, and the envelope in the + fewest words every reader takes, with no `must_understand`. A field with + any problem, an unknown key included, has + none, since a simpler spelling would erase what its author wrote. What + `canonical` gives is judged again, and one that does not hold is a + `ValueError`, a fault in the definition. ([#4434](https://github.com/zarr-developers/zarr-python/pull/4434)) + +- **Breaking:** the v3 readers read each extension point through its + definition: `read_array_metadata_v3` judges the data type, the chunk + grid, the chunk key encoding, each codec and each storage transformer + against the definition that claims its name in a scope, + `CORE_AND_EXTENSIONS` unless a `context` is passed, so a gzip `level` of + 99, or a key a codec does not declare, is a problem located in the + configuration, and a document holding one, accepted before, is refused. + So do the v3 group readers, and the v3 model classes' `from_json` and + `from_key_value` take the same `context`. + A name nothing in scope claims is left unjudged. A problem in a field + that a configuration holds, such as a codec inside a shard, is reported + at that field's own location, and the field that holds it is still read. ([#4436](https://github.com/zarr-developers/zarr-python/pull/4436)) + +- **Breaking:** a v3 array's `fill_value` is judged against its `data_type`. + Each data type's definition declares the JSON shape of its fill value, + `fill_value`, and the rules for one of that shape, `fill_value_rules`: an + `int8` fill value of 300, a `float32` hex string of another width, and a + struct fill value missing a field are each a problem at `fill_value`, + where the package accepted them before. `fill_value_problems(data_type, + value)` judges a fill value against a data type field a scope read, and + a field that is read keeps the fields it read inside as + `nested`, a `Nested` mapping by location. A data type nothing + in scope claims leaves its fill value unjudged. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + +- **Breaking:** a v3 array's sharding codec is judged against the shard it + is handed, and its two pipelines are read as the array's is. Its + `chunk_shape` has an inner chunk length for each axis of the shard, + dividing each length the shards take along it -- the shard being the + chunk the codec is handed, so a transposed shard is divided along its + transposed axes. Its inner codecs are handed the inner chunks, of the + shard's data type, and its index codecs the shard index, of `uint64`, + with an axis more than the shard. A shard that does not divide, or a + pipeline that does not fit what it is handed -- an index `bytes` codec + without an `endian` -- has a problem where it sits, where the package + accepted it before. A codec definition + says what the pipelines it holds are handed, `pipelines`, and each + codec's `Stage` keeps the stages of those pipelines as `inner`. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + +- **Breaking:** a v3 array's `chunk_grid` is judged against its `shape`, by + the grid's definition. A regular grid whose `chunk_shape` does not have + one length per dimension of the shape, and a rectilinear grid whose + `chunk_shapes` does not have one entry per dimension, or whose chunk + lengths fall short of their dimension, each have a problem in + `chunk_grid.configuration`, where the package accepted them before. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + +- **Breaking:** a v3 array's `codecs` are read as a pipeline: in order -- + array -> array codecs, then one array -> bytes codec, then bytes -> bytes + codecs -- and each codec judged against the chunk it is handed, which is + the grid's chunks of the array's data type, as the array -> array codecs + before it hand them on. A codec out of order, a second array -> bytes + codec or none at all; a `bytes` codec without an `endian` handed numbers + of several bytes, or handed values that vary in size; a `transpose` whose + `order` has another number of axes than its chunk; a `cast_value` to or + from a data type that models no real numbers, wrapping to one that is + not integral, or mapping a scalar that is not a fill value of the data + type on its side; a `scale_offset` handed values that are no numbers, or + with an `offset` or `scale` that is not a value of its data type; and a + struct field whose values vary in size are each a problem where they + sit, where the package accepted them before. A reading records each + codec with the chunk it is handed, as a `Stage`, and the lengths a + grid's chunks take; `storage_of(data_type)` says how a data type's + values are stored. A + definition's `rules` are handed the fields its configuration holds as + the scope read them, and the kinds gain `chunk_lengths` (grids), + `storage` (data types), and `chunk_rules` and `transition` (codecs). ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + +- **Breaking:** metadata is now read against the definitions in a scope, a + `ZarrV3Context` or `ZarrV2Context` that a reader builds from the codecs, + data types and other fields it supports, and a model is the document + together with the scope it was read in. The `read_*` readers report every + problem at its location, `unclaimed_fields` lists each field the scope + does not know, so a reader can tell whether it can open a node, repair + readers undo known zarr-python writer bugs, and JSON Schemas are exported. + Code written for 0.5.0 must change: models are no longer dataclasses, + their field members hold `AcceptedField` or `UnclaimedField` values, + `update` takes JSON and takes members the spec does not define through + `extra_fields`, the `validate_*`, `is_*` and `parse_*` functions for + documents and the `...Partial` TypedDicts are removed, some names moved to + another module, the v2 constants and models follow the v3 names and + shapes, and a v2 time dtype without a unit is refused. `to_json` now + writes each field as it was read, rather than shortening a codec to a bare + name that zarr-python could not open, and zarr-metadata now depends on + `annotated-types`. ([#4490](https://github.com/zarr-developers/zarr-python/pull/4490)) + +### Bugfixes + +- The v2 array validator follows the spec on two points it was stricter than: members outside the `.zarray` definition are accepted when they are JSON values with string keys, and are kept and written back by `to_json` and `to_key_value`; any other member is a problem, as in a v3 document (the spec says other keys ["SHOULD NOT be present ... and SHOULD be ignored"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L91-L92), unlike `.zgroup`'s ["MUST NOT"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L313)), and `filters: []` is accepted (["a list of JSON objects providing codec configurations, or null"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L76-L79) sets no minimum). The JSON Schema that `zarr_metadata.pydantic` generates follows: it accepts other members, and the filter pipeline has no minimum length. ([#4365](https://github.com/zarr-developers/zarr-python/pull/4365)) + +- Concrete v3 entity types are now assignable to the fields they describe. + Previously, none of the package's canonical codec / chunk-grid / + chunk-key-encoding / data-type types (e.g. `BloscCodecMetadata`, + `RegularChunkGridMetadata`) satisfied `ZarrV3MetadataFieldJSON`, so a + type checker rejected putting them into the very fields they document + (`codecs`, `chunk_grid`, `data_type`, ...). Three changes fix this: + + - `ZarrV3NamedConfigJSON.name` and `.configuration` are now `ReadOnly` + (PEP 705), making them covariant so concrete `name: Literal[...]` and + required-`configuration` shapes are accepted. + - `ZarrV3NamedConfigJSON` is now `closed` (PEP 728): the spec's + named-configuration envelope has exactly `name` / `configuration` / + `must_understand`, and closing the type also makes it usable as a + `JSONValue` (needed for e.g. the `sharding_indexed` inner `codecs`). + - Every concrete `*Object` / `*Configuration` TypedDict is now `closed`, + and object forms declare `must_understand: NotRequired[bool]` (any v3 + metadata field may carry the extension member). + + **Soft-breaking** for type-checking consumers: dicts with keys beyond the + declared shape no longer satisfy the closed types, and `name` / + `configuration` can no longer be mutated through `ZarrV3NamedConfigJSON`. + Both were previously accepted by type checkers but produced documents + outside the spec's shapes. Generating a pydantic JSON Schema for + the `zarr_metadata.pydantic` field types no longer emits warnings. + + ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) + +- Attributes are user data, and a non-finite number in them is read, + validated and written back. zarr-python writes attributes with the + defaults of Python's `json` module, so an xarray `_FillValue` or a CF + `missing_value` of NaN is stored as a bare `NaN`, which RFC 8259 lacks; + the model refused such a document and could not read the store at all. + A node's `attributes`, a v2 `.zattrs`, the `.zattrs` entries of a v2 + `.zmetadata`, and the attributes of the nodes a v3 group's inline + `consolidated_metadata` holds may now hold `NaN`, `Infinity` and + `-Infinity`, in every reader, such as `read_array_metadata_v3`, and in + each model's `from_json`, `from_key_value` and `to_key_value`. Wherever + the spec interprets a value it spells those numbers as strings, so + anywhere else a non-finite number is still refused, and the store reader + now says where one is instead of failing to decode the document. + + A model whose document the reader refuses raises + `MetadataValidationError` with every problem, instead of being written: + + - a non-finite number outside attributes, which was refused before too; + - a non-string attribute key, which was written as a string; + - a value that is not JSON, which raised `TypeError`; + - anything else the reader refuses, such as `dimension_names` longer than + `shape`, which was written and then refused on read. + + Any sequence is written as an array. A number too large for a float, such + as `1e400`, is read as infinity, so in attributes it is now accepted and + written back as `Infinity`. + + Through `zarr_metadata.pydantic`, pydantic writes a non-finite attribute + as `null` unless the enclosing model sets `ser_json_inf_nan="constants"`. + + ([#4420](https://github.com/zarr-developers/zarr-python/pull/4420)) + +- `from_key_value` takes a mapping keyed by `str` or by any narrower key + type, so reading back what `to_key_value` wrote type-checks. `to_key_value` + keys its result by the store-key literals, and since a mapping's key type is + invariant, a `Mapping[str, bytes]` parameter refused it. ([#4421](https://github.com/zarr-developers/zarr-python/pull/4421)) + +- `MetadataValidationError` pickles and copies with its message. A copy was + built from the error's message, and its message came out as the + original's characters, one per line. ([#4422](https://github.com/zarr-developers/zarr-python/pull/4422)) + +- A `bytes` or `bytearray` is not a JSON array. Where a document expects an + array, an empty one passed as an array with no items: the v3 parser + returned `storage_transformers=b""`, and the v2 readers took `dtype=b""`, + then failed when the model was written. The readers and models now refuse + one at `storage_transformers` and at a v2 `dtype` and its field records. At + `codecs`, `dimension_names` and v2 `filters`, which already refused one, + the problem now says the value has the wrong type. ([#4423](https://github.com/zarr-developers/zarr-python/pull/4423)) + +- A model read by `from_json` or `from_key_value` shares no mutable state + with the document it was read from, as a document from `to_json` already + shares none with its model. A frozen model held the caller's own + dictionaries, so editing the input after reading it changed the model, + and could make `to_key_value` fail. ([#4426](https://github.com/zarr-developers/zarr-python/pull/4426)) + +- `from_key_value` checks that each store value it reads is `bytes`, as its + signature says: a `str` was decoded as a JSON document, and `None` or a + `memoryview` raised `TypeError`. Each is now a `MetadataValidationError` + with an `invalid_type` problem at the store key. ([#4428](https://github.com/zarr-developers/zarr-python/pull/4428)) + +- `ZarrV3ArrayMetadata.create_default(shape=...)` derives a chunk length of 1 + for a dimension of length 0, where it wrote 0: the regular grid asks for + chunk lengths greater than zero, and `zarr` does not open a grid with one. ([#4443](https://github.com/zarr-developers/zarr-python/pull/4443)) + +### Improved Documentation + +- The docs now render the release notes as a page instead of linking to + `CHANGELOG.md` on GitHub, so each docs version shows its own changelog. ([#4311](https://github.com/zarr-developers/zarr-python/pull/4311)) + +### Misc + +- `ZarrV2ConsolidatedMetadataJSON.zarr_consolidated_format` is typed + `Literal[1]` rather than `int`. Format 1 is the only defined `.zmetadata` + format and the runtime validator already rejected anything else, so the + type now carries the constraint the validator enforces instead of + contradicting it. ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) + +- **Breaking:** `MetadataValidationError.problems` is now a tuple instead of + a list. Iteration and indexing are unchanged; code that changes a report + must first copy it with `list(problems)`. `MetadataValidationError` still + accepts any problem sequence. ([#4378](https://github.com/zarr-developers/zarr-python/pull/4378)) + +- `ZarrV2ZGroupJSON` and `ZarrV2GroupMetadataJSON` are closed + (PEP 728), as the spec's "Other keys MUST NOT be present" and the v2 group + validator already said, and as its pydantic schema already was. A type + checker now refuses a v2 group document with a key it does not declare. ([#4430](https://github.com/zarr-developers/zarr-python/pull/4430)) + + ## 0.5.0 (2026-08-14) ### Bugfixes diff --git a/packages/zarr-metadata/changes/4311.doc.md b/packages/zarr-metadata/changes/4311.doc.md deleted file mode 100644 index 674389780c..0000000000 --- a/packages/zarr-metadata/changes/4311.doc.md +++ /dev/null @@ -1,2 +0,0 @@ -The docs now render the release notes as a page instead of linking to -`CHANGELOG.md` on GitHub, so each docs version shows its own changelog. diff --git a/packages/zarr-metadata/changes/4365.bugfix.md b/packages/zarr-metadata/changes/4365.bugfix.md deleted file mode 100644 index 09a76e3b71..0000000000 --- a/packages/zarr-metadata/changes/4365.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -The v2 array validator follows the spec on two points it was stricter than: members outside the `.zarray` definition are accepted when they are JSON values with string keys, and are kept and written back by `to_json` and `to_key_value`; any other member is a problem, as in a v3 document (the spec says other keys ["SHOULD NOT be present ... and SHOULD be ignored"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L91-L92), unlike `.zgroup`'s ["MUST NOT"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L313)), and `filters: []` is accepted (["a list of JSON objects providing codec configurations, or null"](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v2/v2.0.rst#L76-L79) sets no minimum). The JSON Schema that `zarr_metadata.pydantic` generates follows: it accepts other members, and the filter pipeline has no minimum length. diff --git a/packages/zarr-metadata/changes/4378.bugfix.md b/packages/zarr-metadata/changes/4378.bugfix.md deleted file mode 100644 index 9142a6fe6e..0000000000 --- a/packages/zarr-metadata/changes/4378.bugfix.md +++ /dev/null @@ -1,24 +0,0 @@ -Concrete v3 entity types are now assignable to the fields they describe. -Previously, none of the package's canonical codec / chunk-grid / -chunk-key-encoding / data-type types (e.g. `BloscCodecMetadata`, -`RegularChunkGridMetadata`) satisfied `ZarrV3MetadataFieldJSON`, so a -type checker rejected putting them into the very fields they document -(`codecs`, `chunk_grid`, `data_type`, ...). Three changes fix this: - -- `ZarrV3NamedConfigJSON.name` and `.configuration` are now `ReadOnly` - (PEP 705), making them covariant so concrete `name: Literal[...]` and - required-`configuration` shapes are accepted. -- `ZarrV3NamedConfigJSON` is now `closed` (PEP 728): the spec's - named-configuration envelope has exactly `name` / `configuration` / - `must_understand`, and closing the type also makes it usable as a - `JSONValue` (needed for e.g. the `sharding_indexed` inner `codecs`). -- Every concrete `*Object` / `*Configuration` TypedDict is now `closed`, - and object forms declare `must_understand: NotRequired[bool]` (any v3 - metadata field may carry the extension member). - -**Soft-breaking** for type-checking consumers: dicts with keys beyond the -declared shape no longer satisfy the closed types, and `name` / -`configuration` can no longer be mutated through `ZarrV3NamedConfigJSON`. -Both were previously accepted by type checkers but produced documents -outside the spec's shapes. Generating a pydantic JSON Schema for -the `zarr_metadata.pydantic` field types no longer emits warnings. diff --git a/packages/zarr-metadata/changes/4378.misc.1.md b/packages/zarr-metadata/changes/4378.misc.1.md deleted file mode 100644 index 5f5eb1f238..0000000000 --- a/packages/zarr-metadata/changes/4378.misc.1.md +++ /dev/null @@ -1,5 +0,0 @@ -`ZarrV2ConsolidatedMetadataJSON.zarr_consolidated_format` is typed -`Literal[1]` rather than `int`. Format 1 is the only defined `.zmetadata` -format and the runtime validator already rejected anything else, so the -type now carries the constraint the validator enforces instead of -contradicting it. diff --git a/packages/zarr-metadata/changes/4378.misc.md b/packages/zarr-metadata/changes/4378.misc.md deleted file mode 100644 index 83d55866d4..0000000000 --- a/packages/zarr-metadata/changes/4378.misc.md +++ /dev/null @@ -1,4 +0,0 @@ -**Breaking:** `MetadataValidationError.problems` is now a tuple instead of -a list. Iteration and indexing are unchanged; code that changes a report -must first copy it with `list(problems)`. `MetadataValidationError` still -accepts any problem sequence. diff --git a/packages/zarr-metadata/changes/4420.bugfix.md b/packages/zarr-metadata/changes/4420.bugfix.md deleted file mode 100644 index 138dcd1a31..0000000000 --- a/packages/zarr-metadata/changes/4420.bugfix.md +++ /dev/null @@ -1,29 +0,0 @@ -Attributes are user data, and a non-finite number in them is read, -validated and written back. zarr-python writes attributes with the -defaults of Python's `json` module, so an xarray `_FillValue` or a CF -`missing_value` of NaN is stored as a bare `NaN`, which RFC 8259 lacks; -the model refused such a document and could not read the store at all. -A node's `attributes`, a v2 `.zattrs`, the `.zattrs` entries of a v2 -`.zmetadata`, and the attributes of the nodes a v3 group's inline -`consolidated_metadata` holds may now hold `NaN`, `Infinity` and -`-Infinity`, in every reader, such as `read_array_metadata_v3`, and in -each model's `from_json`, `from_key_value` and `to_key_value`. Wherever -the spec interprets a value it spells those numbers as strings, so -anywhere else a non-finite number is still refused, and the store reader -now says where one is instead of failing to decode the document. - -A model whose document the reader refuses raises -`MetadataValidationError` with every problem, instead of being written: - -- a non-finite number outside attributes, which was refused before too; -- a non-string attribute key, which was written as a string; -- a value that is not JSON, which raised `TypeError`; -- anything else the reader refuses, such as `dimension_names` longer than - `shape`, which was written and then refused on read. - -Any sequence is written as an array. A number too large for a float, such -as `1e400`, is read as infinity, so in attributes it is now accepted and -written back as `Infinity`. - -Through `zarr_metadata.pydantic`, pydantic writes a non-finite attribute -as `null` unless the enclosing model sets `ser_json_inf_nan="constants"`. diff --git a/packages/zarr-metadata/changes/4421.bugfix.md b/packages/zarr-metadata/changes/4421.bugfix.md deleted file mode 100644 index cb9d53847a..0000000000 --- a/packages/zarr-metadata/changes/4421.bugfix.md +++ /dev/null @@ -1,4 +0,0 @@ -`from_key_value` takes a mapping keyed by `str` or by any narrower key -type, so reading back what `to_key_value` wrote type-checks. `to_key_value` -keys its result by the store-key literals, and since a mapping's key type is -invariant, a `Mapping[str, bytes]` parameter refused it. diff --git a/packages/zarr-metadata/changes/4422.bugfix.md b/packages/zarr-metadata/changes/4422.bugfix.md deleted file mode 100644 index 7bf24c6d2b..0000000000 --- a/packages/zarr-metadata/changes/4422.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -`MetadataValidationError` pickles and copies with its message. A copy was -built from the error's message, and its message came out as the -original's characters, one per line. diff --git a/packages/zarr-metadata/changes/4422.feature.md b/packages/zarr-metadata/changes/4422.feature.md deleted file mode 100644 index 360b0b1c35..0000000000 --- a/packages/zarr-metadata/changes/4422.feature.md +++ /dev/null @@ -1,7 +0,0 @@ -**Breaking:** `ValidationProblem` and `MetadataValidationError` check at run -time what their annotations declare, and raise `TypeError` otherwise. A -problem's `loc` is a tuple of strings and integers, so a list, which was -accepted before, is now refused, as is a `bool`; its `message` is a string -and its `kind` one of the kinds; an error holds problems. A caller that is -not type-checked fails where it builds the value, not later, where a -location is read. diff --git a/packages/zarr-metadata/changes/4423.bugfix.md b/packages/zarr-metadata/changes/4423.bugfix.md deleted file mode 100644 index dd4fb93604..0000000000 --- a/packages/zarr-metadata/changes/4423.bugfix.md +++ /dev/null @@ -1,7 +0,0 @@ -A `bytes` or `bytearray` is not a JSON array. Where a document expects an -array, an empty one passed as an array with no items: the v3 parser -returned `storage_transformers=b""`, and the v2 readers took `dtype=b""`, -then failed when the model was written. The readers and models now refuse -one at `storage_transformers` and at a v2 `dtype` and its field records. At -`codecs`, `dimension_names` and v2 `filters`, which already refused one, -the problem now says the value has the wrong type. diff --git a/packages/zarr-metadata/changes/4426.bugfix.md b/packages/zarr-metadata/changes/4426.bugfix.md deleted file mode 100644 index 5f015b1533..0000000000 --- a/packages/zarr-metadata/changes/4426.bugfix.md +++ /dev/null @@ -1,5 +0,0 @@ -A model read by `from_json` or `from_key_value` shares no mutable state -with the document it was read from, as a document from `to_json` already -shares none with its model. A frozen model held the caller's own -dictionaries, so editing the input after reading it changed the model, -and could make `to_key_value` fail. diff --git a/packages/zarr-metadata/changes/4428.bugfix.md b/packages/zarr-metadata/changes/4428.bugfix.md deleted file mode 100644 index 44a9b94e74..0000000000 --- a/packages/zarr-metadata/changes/4428.bugfix.md +++ /dev/null @@ -1,4 +0,0 @@ -`from_key_value` checks that each store value it reads is `bytes`, as its -signature says: a `str` was decoded as a JSON document, and `None` or a -`memoryview` raised `TypeError`. Each is now a `MetadataValidationError` -with an `invalid_type` problem at the store key. diff --git a/packages/zarr-metadata/changes/4430.misc.md b/packages/zarr-metadata/changes/4430.misc.md deleted file mode 100644 index dac4e75304..0000000000 --- a/packages/zarr-metadata/changes/4430.misc.md +++ /dev/null @@ -1,4 +0,0 @@ -`ZarrV2ZGroupJSON` and `ZarrV2GroupMetadataJSON` are closed -(PEP 728), as the spec's "Other keys MUST NOT be present" and the v2 group -validator already said, and as its pydantic schema already was. A type -checker now refuses a v2 group document with a key it does not declare. diff --git a/packages/zarr-metadata/changes/4432.feature.md b/packages/zarr-metadata/changes/4432.feature.md deleted file mode 100644 index 159b92c1e0..0000000000 --- a/packages/zarr-metadata/changes/4432.feature.md +++ /dev/null @@ -1,14 +0,0 @@ -`zarr_metadata.typed_json.check(value, SomeTypedDict)` type-checks a JSON -value against any of the package's TypedDicts -- `ZarrV3ArrayMetadataJSON`, -`GzipCodecConfiguration`, or one of your own -- and gives back a value of -it, or None, with every problem located. A TypedDict reads as the typing -spec defines it, whatever the runtime's attributes say: a key is required -by its own `Required` or `NotRequired`, or else by the `total` of the class -that declared it, read from the evaluated annotations, so a module written -with `from __future__ import annotations` reads like any other; `closed`, -`extra_items` and openness come from the class, or from its bases when it -says neither; and each annotation is evaluated in the module of the class -that wrote it. A key a closed TypedDict does not declare is reported, as -the new problem kind `unknown_key`, and left out of the value, which still -comes back; a union of TypedDicts tagged by a `Literal` key is read by the -branch the tag names. diff --git a/packages/zarr-metadata/changes/4434.feature.1.md b/packages/zarr-metadata/changes/4434.feature.1.md deleted file mode 100644 index 021452493e..0000000000 --- a/packages/zarr-metadata/changes/4434.feature.1.md +++ /dev/null @@ -1,10 +0,0 @@ -**Breaking:** `must_understand: false` is refused at every extension point -of an array document, codecs and storage transformers included, not only -at `data_type`, `chunk_grid` and `chunk_key_encoding`: ignoring a codec -gives wrong bytes as surely as ignoring a data type gives wrong values, -and the spec naming only three points is read as an oversight. A document -that declares a codec or a storage transformer ignorable now has a problem -at `codecs.N.must_understand` or `storage_transformers.N.must_understand`, -so the readers refuse it, and no model writes one. The JSON schema `zarr_metadata.pydantic` generates for an array -document says the same, and so does `resolve`, which reads one field on -its own. diff --git a/packages/zarr-metadata/changes/4434.feature.md b/packages/zarr-metadata/changes/4434.feature.md deleted file mode 100644 index 0e8a1d7e32..0000000000 --- a/packages/zarr-metadata/changes/4434.feature.md +++ /dev/null @@ -1,50 +0,0 @@ -Each extension's metadata is a definition: a value holding the name the -metadata carries, the TypedDict its configuration is -- the one -declaration of that JSON -- and plain functions over that TypedDict for -the rules. `zarr_metadata.definition` and `zarr_metadata.v3.definition` are the -doors, and their module docstrings the guide. Every extension the package models is one: the -codecs `blosc`, `bytes`, `cast_value`, `crc32c`, `gzip`, `scale_offset`, -`sharding_indexed`, `transpose` and `zstd`; every data type, raw bits -included (`r16` is read by the `r*` definition with `{"bits": 16}`, the size -its name carries); the `regular` and `rectilinear` chunk grids; and the `default` -and `v2` chunk key encodings. Each is a constant in its own module -- -`GZIP_CODEC`, `STRUCT_DATA_TYPE` -- and `CORE` holds what the -specification defines, `CORE_AND_EXTENSIONS` what `zarr-extensions` -registers besides. A third party writes a TypedDict, a rule function and -a definition, and adds it to a scope; a definition checks itself -when built, and nothing happens at class creation. - -Reading a field is three steps, each usable on its own by a caller -holding nothing but JSON. `check(value, SomeTypedDict)`, from -`zarr_metadata.typed_json`, type-checks JSON against a TypedDict and needs -nothing else; `definition.read_configuration(configuration)` is the check, each nested -field's envelope judged, and then the rules; `resolve(field, -CodecDefinition, scope)` reads a whole field in a scope -- its envelope -judged, its name related to a definition, its configuration judged, each -nested field read the same way -- and returns what the scope made of it, -with every problem located. A name nothing in scope claims is left -unjudged. A shard's pipelines, a cast's target -type and a struct's field types are nested fields, annotated `ZarrV3CodecField` -and `ZarrV3DataTypeField`, and read in the scope their field is read in. -A codec definition also says whether the size -of what it gives out is `static` or `dynamic`, and a member annotated -`ZarrV3StaticCodecField` takes codecs of static size only: a shard's -`index_codecs`, where the spec forbids a compressor, so a `gzip` there is a -problem at its place. - -The TypedDict reads as the typing spec defines it -- `total`, `Required`, -`NotRequired`, `closed` and `extra_items` mean what they mean to a type -checker, whether or not its module postpones annotations. A key a closed -TypedDict does not declare is reported, as `unknown_key`, and left out of -what the rules are handed. A definition refuses a TypedDict that says -nothing of the keys it does not declare, and a member typed -`ZarrV3MetadataFieldJSON`, which would check as plain JSON. - -`canonical_of(field, problems)` gives a field without problems in its -simplest equivalent spelling: each nested field in its own simplest -spelling, then its definition's own `canonical`, and the envelope in the -fewest words every reader takes, with no `must_understand`. A field with -any problem, an unknown key included, has -none, since a simpler spelling would erase what its author wrote. What -`canonical` gives is judged again, and one that does not hold is a -`ValueError`, a fault in the definition. diff --git a/packages/zarr-metadata/changes/4436.feature.md b/packages/zarr-metadata/changes/4436.feature.md deleted file mode 100644 index 0b8fab9962..0000000000 --- a/packages/zarr-metadata/changes/4436.feature.md +++ /dev/null @@ -1,12 +0,0 @@ -**Breaking:** the v3 readers read each extension point through its -definition: `read_array_metadata_v3` judges the data type, the chunk -grid, the chunk key encoding, each codec and each storage transformer -against the definition that claims its name in a scope, -`CORE_AND_EXTENSIONS` unless a `context` is passed, so a gzip `level` of -99, or a key a codec does not declare, is a problem located in the -configuration, and a document holding one, accepted before, is refused. -So do the v3 group readers, and the v3 model classes' `from_json` and -`from_key_value` take the same `context`. -A name nothing in scope claims is left unjudged. A problem in a field -that a configuration holds, such as a codec inside a shard, is reported -at that field's own location, and the field that holds it is still read. diff --git a/packages/zarr-metadata/changes/4443.bugfix.md b/packages/zarr-metadata/changes/4443.bugfix.md deleted file mode 100644 index 645f8ec603..0000000000 --- a/packages/zarr-metadata/changes/4443.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -`ZarrV3ArrayMetadata.create_default(shape=...)` derives a chunk length of 1 -for a dimension of length 0, where it wrote 0: the regular grid asks for -chunk lengths greater than zero, and `zarr` does not open a grid with one. diff --git a/packages/zarr-metadata/changes/4443.feature.2.md b/packages/zarr-metadata/changes/4443.feature.2.md deleted file mode 100644 index 004658702b..0000000000 --- a/packages/zarr-metadata/changes/4443.feature.2.md +++ /dev/null @@ -1,10 +0,0 @@ -**Breaking:** a v3 array's `fill_value` is judged against its `data_type`. -Each data type's definition declares the JSON shape of its fill value, -`fill_value`, and the rules for one of that shape, `fill_value_rules`: an -`int8` fill value of 300, a `float32` hex string of another width, and a -struct fill value missing a field are each a problem at `fill_value`, -where the package accepted them before. `fill_value_problems(data_type, -value)` judges a fill value against a data type field a scope read, and -a field that is read keeps the fields it read inside as -`nested`, a `Nested` mapping by location. A data type nothing -in scope claims leaves its fill value unjudged. diff --git a/packages/zarr-metadata/changes/4443.feature.3.md b/packages/zarr-metadata/changes/4443.feature.3.md deleted file mode 100644 index f7c8bcd78b..0000000000 --- a/packages/zarr-metadata/changes/4443.feature.3.md +++ /dev/null @@ -1,6 +0,0 @@ -**Breaking:** a v3 array's `chunk_grid` is judged against its `shape`, by -the grid's definition. A regular grid whose `chunk_shape` does not have -one length per dimension of the shape, and a rectilinear grid whose -`chunk_shapes` does not have one entry per dimension, or whose chunk -lengths fall short of their dimension, each have a problem in -`chunk_grid.configuration`, where the package accepted them before. diff --git a/packages/zarr-metadata/changes/4443.feature.4.md b/packages/zarr-metadata/changes/4443.feature.4.md deleted file mode 100644 index 3684877d9d..0000000000 --- a/packages/zarr-metadata/changes/4443.feature.4.md +++ /dev/null @@ -1,20 +0,0 @@ -**Breaking:** a v3 array's `codecs` are read as a pipeline: in order -- -array -> array codecs, then one array -> bytes codec, then bytes -> bytes -codecs -- and each codec judged against the chunk it is handed, which is -the grid's chunks of the array's data type, as the array -> array codecs -before it hand them on. A codec out of order, a second array -> bytes -codec or none at all; a `bytes` codec without an `endian` handed numbers -of several bytes, or handed values that vary in size; a `transpose` whose -`order` has another number of axes than its chunk; a `cast_value` to or -from a data type that models no real numbers, wrapping to one that is -not integral, or mapping a scalar that is not a fill value of the data -type on its side; a `scale_offset` handed values that are no numbers, or -with an `offset` or `scale` that is not a value of its data type; and a -struct field whose values vary in size are each a problem where they -sit, where the package accepted them before. A reading records each -codec with the chunk it is handed, as a `Stage`, and the lengths a -grid's chunks take; `storage_of(data_type)` says how a data type's -values are stored. A -definition's `rules` are handed the fields its configuration holds as -the scope read them, and the kinds gain `chunk_lengths` (grids), -`storage` (data types), and `chunk_rules` and `transition` (codecs). diff --git a/packages/zarr-metadata/changes/4443.feature.md b/packages/zarr-metadata/changes/4443.feature.md deleted file mode 100644 index db1618e3ec..0000000000 --- a/packages/zarr-metadata/changes/4443.feature.md +++ /dev/null @@ -1,13 +0,0 @@ -**Breaking:** a v3 array's sharding codec is judged against the shard it -is handed, and its two pipelines are read as the array's is. Its -`chunk_shape` has an inner chunk length for each axis of the shard, -dividing each length the shards take along it -- the shard being the -chunk the codec is handed, so a transposed shard is divided along its -transposed axes. Its inner codecs are handed the inner chunks, of the -shard's data type, and its index codecs the shard index, of `uint64`, -with an axis more than the shard. A shard that does not divide, or a -pipeline that does not fit what it is handed -- an index `bytes` codec -without an `endian` -- has a problem where it sits, where the package -accepted it before. A codec definition -says what the pipelines it holds are handed, `pipelines`, and each -codec's `Stage` keeps the stages of those pipelines as `inner`. diff --git a/packages/zarr-metadata/changes/4490.feature.md b/packages/zarr-metadata/changes/4490.feature.md deleted file mode 100644 index 792c349520..0000000000 --- a/packages/zarr-metadata/changes/4490.feature.md +++ /dev/null @@ -1,17 +0,0 @@ -**Breaking:** metadata is now read against the definitions in a scope, a -`ZarrV3Context` or `ZarrV2Context` that a reader builds from the codecs, -data types and other fields it supports, and a model is the document -together with the scope it was read in. The `read_*` readers report every -problem at its location, `unclaimed_fields` lists each field the scope -does not know, so a reader can tell whether it can open a node, repair -readers undo known zarr-python writer bugs, and JSON Schemas are exported. -Code written for 0.5.0 must change: models are no longer dataclasses, -their field members hold `AcceptedField` or `UnclaimedField` values, -`update` takes JSON and takes members the spec does not define through -`extra_fields`, the `validate_*`, `is_*` and `parse_*` functions for -documents and the `...Partial` TypedDicts are removed, some names moved to -another module, the v2 constants and models follow the v3 names and -shapes, and a v2 time dtype without a unit is refused. `to_json` now -writes each field as it was read, rather than shortening a codec to a bare -name that zarr-python could not open, and zarr-metadata now depends on -`annotated-types`. diff --git a/packages/zarr-metadata/pyproject.toml b/packages/zarr-metadata/pyproject.toml index 831050c569..abd93bc729 100644 --- a/packages/zarr-metadata/pyproject.toml +++ b/packages/zarr-metadata/pyproject.toml @@ -199,6 +199,10 @@ underlines = ["", "", ""] title_format = "## {version} ({project_date})" issue_format = "[#{issue}](https://github.com/zarr-developers/zarr-python/pull/{issue})" start_string = "\n" +# Indents entries by four spaces, so the docs render multi-paragraph entries +# as one list item. It lives outside `changes/`, which holds only fragments, +# and its name ends in `.md`, which is how towncrier knows it is Markdown. +template = "towncrier-template.md" # Declaring any type replaces towncrier's built-in set, so all five the # `changes/README.md` menu offers are restated here. They are the defaults diff --git a/packages/zarr-metadata/towncrier-template.md b/packages/zarr-metadata/towncrier-template.md new file mode 100644 index 0000000000..d3616fe8c5 --- /dev/null +++ b/packages/zarr-metadata/towncrier-template.md @@ -0,0 +1,130 @@ +{#- +This package's copy of towncrier's default Markdown template. The one +change: each entry's continuation lines are indented by four spaces, not +two, and entries are separated by a blank line, so an entry with a +second paragraph or a nested list stays one list item in Python-Markdown, +which renders the docs' release notes page, as well as in CommonMark. +Without both, Python-Markdown ends the list at the second paragraph and +runs the later entries together as one paragraph. +-#} +{#- +══════════════════════════════════════════════════════════════════════════════ +TOWNCRIER MARKDOWN TEMPLATE +══════════════════════════════════════════════════════════════════════════════ + +─── Macro: heading ───────────────────────────────────────────────────────── +Purpose: + Generates Markdown headings with the appropriate number of # characters. + Based on header_prefix (default: "#") and the level argument. + +Arguments: + level The relative heading level (1=#, 2=##, 3=###, etc.) +-#} +{%- macro heading(level) -%} + {{- "#" * ( header_prefix | length + level -1 ) }} +{%- endmacro -%} + +{#- +─── Variable: newline ────────────────────────────────────────────────────── +Purpose: + Consistent newline handling. -#} +{%- set newline = "\n" -%} + +{#- ════════════════════════ TEMPLATE GENERATION ════════════════════════ -#} +{#- ─── TITLE HEADING ─── #} +{#- render_title is false when title_format is specified in the config #} +{%- if render_title %} + {%- if versiondata.name %} + {{- heading(1) ~ " " ~ versiondata.name ~ " " ~ versiondata.version ~ " (" ~ versiondata.date ~ ")" ~ newline }} + {%- else %} + {{- heading(1) ~ " " ~ versiondata.version ~ " (" ~ versiondata.date ~ ")" ~ newline }} + {%- endif %} +{%- endif %} +{#- If title_format is specified, we start with a new line #} +{{- newline }} + +{%- for section, _ in sections.items() %} + {#- ─── SECTION HEADING ─── #} + {%- if section %} + {{- newline }} + {{- heading(2) ~ " " ~ section ~ newline }} + {{- newline }} + {%- endif %} + + {%- if sections[section] %} + + {%- for category, val in definitions.items() if category in sections[section] %} + {%- set issue_pks = [] %} + {#- ─── CATEGORY HEADING ─── #} + {#- Increase heading level if section is not present #} + {{- heading(3 if section else 2) ~" " ~ definitions[category]['name'] ~ newline }} + {{- newline }} + + {#- ─── RENDER ENTRIES ─── #} + {%- for text, values in sections[section][category].items() %} + {#- A blank line between entries: Python-Markdown starts a new list + item after an entry of several paragraphs only after one. #} + {%- if not loop.first %} + {{- newline }} + {%- endif %} + {#- Prepare the string of issue numbers (e.g., "#1, #9, #142") #} + {%- set issue_pks = [] %} + {%- for v_issue in values %} + {%- set _= issue_pks.append(v_issue.split(": ", 1)[0]) %} + {%- endfor %} + {%- set issues_list = issue_pks | join(", ") %} + + {#- Check if text contains a sublist #} + {%- set text_has_sublist = (("\n - " in text) or ("\n * " in text)) %} + + {#- CASE 1: No text, only issues #} + {#- Output: - #1, #9, #142 #} + {%- if not text and issues_list %} + {{- "- " ~ issues_list ~ newline }} + + {#- Cases where both text and issues exist #} + {%- elif text and issues_list %} + {%- if text_has_sublist %} + {#- CASE 3: Text with sublist #} + {#- Output: - TEXT\n\n (#1, #9, #142) #} + {{- "- " ~ (text | indent(2)) ~ newline ~ newline ~ " (" ~ issues_list ~ ")" ~ newline }} + {%- else %} + {#- CASE 2: Text, no sublist #} + {#- Output: - TEXT (#1, #9, #142) #} + {{- "- " ~ (text | indent(2)) ~ " (" ~ issues_list ~ ")" ~ newline }} + {%- endif %} + + {%- elif text %} + {#- Implicit Case: Text, but no issues #} + {#- Output: - TEXT #} + {{- "- " ~ (text | indent(2)) ~ newline }} + {%- endif %} + {%- endfor %} + + {#- New line between list and link references #} + {{- newline }} + + {#- Link references #} + {%- if issues_by_category[section][category] and "]: " in issues_by_category[section][category][0] %} + {%- for issue in issues_by_category[section][category] %} + {{- issue ~ newline }} + {%- endfor %} + {{- newline }} + {%- endif %} + + {#- No changes in this category #} + {%- if sections[section][category]|length == 0 %} + {{- newline }} + {{- "No significant changes." ~ newline * 2 }} + {%- endif %} + {%- endfor %} + {%- else %} + {#- No changes in this section #} + {{- "No significant changes." ~ newline * 2 }} + {%- endif %} +{%- endfor %} +{#- +Newline at the end of the rendered newsfile content. +In this way the there are 2 newlines between the latest release and the previous release content. +-#} +{{- newline -}}