Add unified JSON metadata validation via msgspec (#3285) (#4063)

* Add unified parse_json runtime type checker (#3285)

Introduce zarr.core.json_parse.parse_json, a single type-annotation-driven validator that consolidates the scattered per-field parse_* helpers. Handles primitives, Literal, unions/Optional, fixed and variadic tuples, Sequence/list (coerced to tuple), Mapping/dict, and TypedDict, with a bool-vs-int safe primitive check. Adds tests/test_json_parse.py (94 tests) and a changelog fragment. No existing call sites migrated yet; this is the proof-of-direction module.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Migrate parse_order, parse_bool, parse_zarr_format to parse_json (#3285)

Pilot migration delegating three representative helpers to the unified parse_json validator. Public signatures and return types are unchanged. parse_zarr_format re-wraps parse_json's ValueError/TypeError as MetadataValidationError with the original message to preserve observable behavior. parse_json is imported function-locally to avoid circular imports. Focused suites green: test_common/test_config/test_metadata/test_json_parse = 430 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Fix lint and make TypedDict NotRequired robust under future annotations (#3285)

Apply ruff check/format to satisfy the Lint CI hook. Keep typing.Union/Optional spellings in tests (with noqa) to cover that origin path alongside X | Y. Rework _parse_typeddict to derive required/optional from get_type_hints(include_extras=True) + __total__ instead of __required_keys__, so class-syntax NotRequired is detected even when 'from __future__ import annotations' stringizes the hints (as in zarr's metadata modules). test_json_parse + test_common + test_metadata = 393 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Annotate origin as Any to satisfy mypy in _parse_typeddict (#3285)

get_origin(hint) is Required/NotRequired tripped mypy's comparison-overlap and unreachable checks; typing origin as Any keeps the runtime check while satisfying mypy. Full 'uv run --frozen mypy' is clean (190 files); ruff check/format clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Migrate Batch 1 literal parsers to parse_json (#3285)

Delegate the literal/type check of parse_indexing_order, parse_node_type, parse_node_type_array, parse_separator, two parse_zarr_format variants (group, v2), and parse_name to the unified parse_json. Public signatures and return types unchanged; each preserves its original exception type and message (wrapping parse_json's ValueError/TypeError into MetadataValidationError / NodeTypeValidationError / the original ValueError/TypeError where tests assert on them). parse_json imported function-locally to avoid circular imports. Focused suites + full mypy green (1196 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Migrate Batch 2 codec primitive parsers to parse_json (#3285)

Delegate the int/bool type check of parse_checksum, parse_clevel, parse_blocksize, parse_typesize, parse_gzip_level, and parse_zstd_level to parse_json, keeping each helper's range/bound check and exact error messages. Original exception types are preserved by wrapping parse_json's ValueError/TypeError. Note: parse_json rejects bool where the old isinstance(data, int) accepted it; verified no caller/test passes a bool to these, so this is a deliberate, more-correct strictening. parse_json imported function-locally. Codec suite + full mypy green (794 passed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Replace hand-written parse_json with msgspec.convert (#3285)

Delete the bespoke parse_json runtime type checker and route JSON metadata
validation through msgspec.convert, which handles the type coercions zarr needs
(Literal membership, int/bool strictness, list-to-tuple). A small hand-written
fallback (validate_json_value) covers the recursive JSON values msgspec cannot
build a schema for, and adds an explicit nesting-depth limit. The
registry/dtype/numcodec parsers stay hand-written since they need runtime
lookups.

Also fixes a latent generator-exhaustion bug in parse_storage_transformers,
adds msgspec as a dependency (pinned in the min_deps env), and preserves the
exception types and messages every migrated helper raised.

Net ~555 lines removed. Tests, mypy and ruff all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Encapsulate convert + field-context error into parse_field (#3285)

Address review feedback: factor the repeated convert-then-re-raise pattern out
of each per-field parser. convert now raises a field-agnostic
ValueError("Expected instance of TYPE, got DATA"); the new parse_field wraps it
and re-raises with field context ("Failed to parse input for FIELD") using the
caller's chosen exception type, chaining the original error. Every per-field
parser collapses to a one-liner.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: replace Sphinx roles in json_parse with plain literals (#3285)

The new ci/lint_docs.py check flags :func:/:class: roles, which pass through
as literal text under MkDocs/mkdocstrings instead of becoming links. msgspec
is not in the configured inventories, so cross-references would not resolve;
using inline literals matches how the codebase already writes np.nonzero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: keep literal members and the offending value in parse errors (#3285)

The msgspec rewrite regressed error quality for Literal types: _type_name fell
back to __name__, rendering Literal[3] as bare "Literal", and parse_field
dropped the value entirely. The old per-field parsers reported both, e.g.
"Invalid value for 'zarr_format'. Expected '3'. Got '3.0'."

_type_name now renders parameterized types via str(), so members survive, and
parse_field reports expected type and received value:
  Failed to parse input for 'zarr_format': expected Literal[3], got 3.0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor: hoist json_parse imports to module level (#3285)

These were function-local to dodge import cycles under the old hand-written
parse_json. After the msgspec rewrite json_parse's only zarr import is JSON
under TYPE_CHECKING, so it has no runtime zarr dependency and cannot form a
cycle. Verified each touched module still imports standalone.

Hoisting exposed a latent packaging gap: msgspec was added to pyproject.toml
but never locked, so uv.lock lacked it. That stayed hidden only because
json_parse was imported lazily; at module level it broke `uv run --frozen`
(ModuleNotFoundError in the docs job). Regenerated the lock, which adds
msgspec 0.21.1 and nothing else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: note the stricter metadata parsing in the changelog (#3285)

The old per-field checks compared with ==, so numerically equal values passed:
zarr_format=2.0 was accepted. msgspec requires an actual int, so such inputs
are now rejected. Spec-conforming metadata is unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: link msgspec symbols via its inventory (#3285)

Adds msgspec's Sphinx inventory so the json_parse docstrings can reference
msgspec.convert and msgspec.ValidationError as real cross-references instead
of inert literals. Verified the inventory is served at msgspec.dev (the
jcristharif.com path in the package metadata is stale) and that both symbols
resolve in it.

Same-module names stay plain literals: zarr.core.json_parse is internal and
not rendered in the API reference, so a cross-reference to it would not
resolve and would fail `mkdocs build --strict`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>

Author: 3 people

changes/3285.feature.md

@@ -0,0 +1,13 @@
+JSON metadata validation now delegates to ``msgspec.convert`` for the type
+coercions it supports (``Literal`` membership, ``int`` / ``bool`` strictness,
+list-to-tuple), replacing the per-field hand-written ``parse_*`` logic. A small
+fallback validates the recursive JSON values msgspec cannot, now with an
+explicit nesting-depth limit, and a latent generator-exhaustion bug in
+``parse_storage_transformers`` is fixed. See #3285.
+
+As a result some metadata inputs are now parsed more strictly. The previous
+per-field checks compared values with ``==``, which accepts any numerically
+equal object, so a float such as ``2.0`` was accepted as ``zarr_format``; it is
+now rejected because it is not an ``int``. Booleans are likewise no longer
+accepted where an ``int`` is expected, since ``bool`` is an ``int`` subclass.
+Metadata that conforms to the Zarr specification is unaffected.

mkdocs.yml

@@ -199,6 +199,7 @@ plugins:
             - https://docs.xarray.dev/en/stable/objects.inv
             - https://numpy.org/doc/stable/objects.inv
             - https://numcodecs.readthedocs.io/en/stable/objects.inv
+            - https://msgspec.dev/objects.inv
             - https://developmentseed.org/obstore/latest/objects.inv
             - https://filesystem-spec.readthedocs.io/en/latest/objects.inv
             - https://requests.readthedocs.io/en/latest/objects.inv

pyproject.toml

@@ -49,6 +49,7 @@ dependencies = [
     'google-crc32c>=1.5',
     'typing_extensions>=4.14',
     'donfig>=0.8',
+    'msgspec>=0.19',
 ]
 
 dynamic = [
@@ -281,6 +282,7 @@ extra-dependencies = [
     'typing_extensions==4.14.*',
     'donfig==0.8.*',
     'obstore==0.5.*',
+    'msgspec==0.19.*',
 ]
 
 [tool.hatch.envs.default]

src/zarr/codecs/blosc.py

@@ -14,6 +14,7 @@
 from zarr.core.buffer.cpu import as_numpy_array_wrapper
 from zarr.core.common import JSON, NamedRequiredConfig, parse_named_configuration
 from zarr.core.dtype.common import HasItemSize
+from zarr.core.json_parse import parse_field
 
 if TYPE_CHECKING:
     from typing import Self
@@ -104,27 +105,24 @@ class BloscCname(metaclass=_DeprecatedStrEnumMeta):
 
 
 def parse_typesize(data: JSON) -> int:
-    if isinstance(data, int):
-        if data > 0:
-            return data
-        else:
-            raise ValueError(
-                f"Value must be greater than 0. Got {data}, which is less or equal to 0."
-            )
-    raise TypeError(f"Value must be an int. Got {type(data)} instead.")
+    parsed: int = parse_field(data, int, "typesize", error=TypeError)
+    if parsed > 0:
+        return parsed
+    else:
+        raise ValueError(
+            f"Value must be greater than 0. Got {parsed}, which is less or equal to 0."
+        )
 
 
 # todo: real validation
 def parse_clevel(data: JSON) -> int:
-    if isinstance(data, int):
-        return data
-    raise TypeError(f"Value should be an int. Got {type(data)} instead.")
+    parsed: int = parse_field(data, int, "clevel", error=TypeError)
+    return parsed
 
 
 def parse_blocksize(data: JSON) -> int:
-    if isinstance(data, int):
-        return data
-    raise TypeError(f"Value should be an int. Got {type(data)} instead.")
+    parsed: int = parse_field(data, int, "blocksize", error=TypeError)
+    return parsed
 
 
 def _parse_cname(data: object) -> BloscCnameLiteral:

src/zarr/codecs/gzip.py

@@ -10,6 +10,7 @@
 from zarr.abc.codec import BytesBytesCodec
 from zarr.core.buffer.cpu import as_numpy_array_wrapper
 from zarr.core.common import JSON, parse_named_configuration
+from zarr.core.json_parse import parse_field
 
 if TYPE_CHECKING:
     from typing import Self
@@ -19,13 +20,12 @@
 
 
 def parse_gzip_level(data: JSON) -> int:
-    if not isinstance(data, (int)):
-        raise TypeError(f"Expected int, got {type(data)}")
-    if data not in range(10):
+    parsed: int = parse_field(data, int, "level", error=TypeError)
+    if parsed not in range(10):
         raise ValueError(
-            f"Expected an integer from the inclusive range (0, 9). Got {data} instead."
+            f"Expected an integer from the inclusive range (0, 9). Got {parsed} instead."
         )
-    return data
+    return parsed
 
 
 @dataclass(frozen=True)

src/zarr/codecs/zstd.py

@@ -12,6 +12,7 @@
 from zarr.abc.codec import BytesBytesCodec
 from zarr.core.buffer.cpu import as_numpy_array_wrapper
 from zarr.core.common import JSON, parse_named_configuration
+from zarr.core.json_parse import parse_field
 
 if TYPE_CHECKING:
     from typing import Self
@@ -21,17 +22,15 @@
 
 
 def parse_zstd_level(data: JSON) -> int:
-    if isinstance(data, int):
-        if data >= 23:
-            raise ValueError(f"Value must be less than or equal to 22. Got {data} instead.")
-        return data
-    raise TypeError(f"Got value with type {type(data)}, but expected an int.")
+    parsed: int = parse_field(data, int, "level", error=TypeError)
+    if parsed >= 23:
+        raise ValueError(f"Value must be less than or equal to 22. Got {parsed} instead.")
+    return parsed
 
 
 def parse_checksum(data: JSON) -> bool:
-    if isinstance(data, bool):
-        return data
-    raise TypeError(f"Expected bool. Got {type(data)}.")
+    parsed: bool = parse_field(data, bool, "checksum", error=TypeError)
+    return parsed
 
 
 @dataclass(frozen=True)

src/zarr/core/chunk_key_encodings.py

@@ -13,15 +13,14 @@
     NamedConfig,
     parse_named_configuration,
 )
+from zarr.core.json_parse import parse_field
 from zarr.registry import get_chunk_key_encoding_class, register_chunk_key_encoding
 
 SeparatorLiteral = Literal[".", "/"]
 
 
 def parse_separator(data: JSON) -> SeparatorLiteral:
-    if data not in (".", "/"):
-        raise ValueError(f"Expected an '.' or '/' separator. Got {data} instead.")
-    return cast("SeparatorLiteral", data)
+    return cast("SeparatorLiteral", parse_field(data, Literal[".", "/"], "separator"))
 
 
 class ChunkKeyEncodingParams(TypedDict):

src/zarr/core/common.py

@@ -20,6 +20,7 @@
 from typing_extensions import ReadOnly
 
 from zarr.core.config import config as zarr_config
+from zarr.core.json_parse import convert, parse_field
 from zarr.errors import ZarrRuntimeWarning
 
 if TYPE_CHECKING:
@@ -147,12 +148,13 @@ def parse_enum[E: Enum](data: object, cls: type[E]) -> E:
 
 
 def parse_name(data: JSON, expected: str | None = None) -> str:
-    if isinstance(data, str):
-        if expected is None or data == expected:
-            return data
-        raise ValueError(f"Expected '{expected}'. Got {data} instead.")
-    else:
-        raise TypeError(f"Expected a string, got an instance of {type(data)}.")
+    try:
+        data = cast("str", convert(data, str))
+    except (ValueError, TypeError) as exc:
+        raise TypeError(f"Expected a string, got an instance of {type(data)}.") from exc
+    if expected is None or data == expected:
+        return data
+    raise ValueError(f"Expected '{expected}'. Got {data} instead.")
 
 
 def parse_configuration(data: JSON) -> JSON:
@@ -227,15 +229,11 @@ def parse_fill_value(data: Any) -> Any:
 
 
 def parse_order(data: Any) -> Literal["C", "F"]:
-    if data in ("C", "F"):
-        return cast("Literal['C', 'F']", data)
-    raise ValueError(f"Expected one of ('C', 'F'), got {data} instead.")
+    return cast("Literal['C', 'F']", parse_field(data, Literal["C", "F"], "order"))
 
 
 def parse_bool(data: Any) -> bool:
-    if isinstance(data, bool):
-        return data
-    raise ValueError(f"Expected bool, got {data} instead.")
+    return cast("bool", convert(data, bool))
 
 
 def parse_int(data: Any) -> int:

src/zarr/core/config.py

@@ -33,6 +33,8 @@
 
 from donfig import Config as DConfig
 
+from zarr.core.json_parse import parse_field
+
 if TYPE_CHECKING:
     from donfig.config_obj import ConfigSet
 
@@ -159,7 +161,4 @@ def enable_gpu(self) -> ConfigSet:
 
 
 def parse_indexing_order(data: Any) -> Literal["C", "F"]:
-    if data in ("C", "F"):
-        return cast("Literal['C', 'F']", data)
-    msg = f"Expected one of ('C', 'F'), got {data} instead."
-    raise ValueError(msg)
+    return cast("Literal['C', 'F']", parse_field(data, Literal["C", "F"], "order"))

src/zarr/core/group.py

@@ -46,6 +46,7 @@
 )
 from zarr.core.config import config
 from zarr.core.dtype import parse_data_type
+from zarr.core.json_parse import parse_field
 from zarr.core.metadata import ArrayV2Metadata, ArrayV3Metadata
 from zarr.core.metadata.io import save_metadata
 from zarr.core.sync import SyncMixin, sync
@@ -85,18 +86,15 @@
 
 def parse_zarr_format(data: Any) -> ZarrFormat:
     """Parse the zarr_format field from metadata."""
-    if data in (2, 3):
-        return cast("ZarrFormat", data)
-    msg = f"Invalid zarr_format. Expected one of 2 or 3. Got {data}."
-    raise ValueError(msg)
+    return cast("ZarrFormat", parse_field(data, Literal[2, 3], "zarr_format"))
 
 
 def parse_node_type(data: Any) -> NodeType:
     """Parse the node_type field from metadata."""
-    if data in ("array", "group"):
-        return cast("Literal['array', 'group']", data)
-    msg = f"Invalid value for 'node_type'. Expected 'array' or 'group'. Got '{data}'."
-    raise MetadataValidationError(msg)
+    return cast(
+        "Literal['array', 'group']",
+        parse_field(data, Literal["array", "group"], "node_type", error=MetadataValidationError),
+    )
 
 
 # todo: convert None to empty dict