Repository navigation
Conversation
Implements URL pipeline support (https://github.com/jbms/url-pipeline): '|'-chained URLs resolve through pluggable adapters registered under the 'zarr.url_adapters' entry-point group (entry-point name = URL scheme). - zarr.abc.url_pipeline: PipelineSegment, AdapterResolution, PipelineContext, URLPipelineAdapter (single-classmethod contract) - zarr.storage._url_pipeline: parse_pipeline / resolve_pipeline; the root sub-URL delegates to make_store so existing file/memory/fsspec routing is unchanged - registry: register_url_adapter / get_url_adapter / list_url_adapter_schemes (name check only; no adapter imports) - make_store/make_store_path route strings containing '|' (or a registered root scheme) through the resolver; residual store paths combine with the user-supplied path - StorePath gains a zarr_format attribute (populated by format segments in a follow-up) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e-core # Conflicts: # src/zarr/errors.py # src/zarr/registry.py
…thoring - make_store validates the access mode before routing a URL pipeline string to adapters, matching every other StoreLike branch - make_store docstring lists URL pipeline strings; StorePath.open documents the invalid-mode ValueError - user guide: StoreLike bullet for pipeline strings and an adapter authoring section in extending.md (runnable example) - changelog: drop PR-relative wording, name the public parser entry points Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
get_url_adapter took a non-reentrant lock across entry_point.load(), so an adapter module that resolved another scheme at import time deadlocked the process. Matching entry points are now taken off the pending list under the lock and imported outside it; import failures are wrapped in URLPipelineError and leave the entry point discoverable for a retry. A same-named entry point is discarded with a ZarrUserWarning when the scheme is already registered (e.g. by a builtin), instead of staying pending forever; duplicate entry-point names warn and the first wins. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ent contracts - The resolver closes the store an adapter returned when it cannot be made read-only for mode 'r', instead of leaking it. - resolve_pipeline documents every error type it lets through; the resolve_preceding docstring states the local-file-root limitation. - AdapterResolution and PipelineContext compare and hash by identity, since a Store is not hashable and frozen=True implied otherwise. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… caller The zarr2:/zarr3: format carried on StorePath.zarr_format was merged only in zarr.api.asynchronous; create_array, from_array, Group.from_store, Group.open, Array.open and the deprecated AsyncArray._create dropped it and swallowed explicit conflicts. The merge now lives on StorePath.resolve_zarr_format and is applied at each site. To let the pipeline's format apply when the caller does not specify one, create_array (sync and async) and Group.from_store default zarr_format to None, resolving to the configured default (3) as before; Array.open gains a zarr_format parameter mirroring Group.open. Tests: registry deadlock/race/import-failure/shadowing, resolver leak, zarr_format merge at each core site, and pins for the documented memory:/ file: divergences and the local-file-root limitation (xfail, strict). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- The extending guide's wrapper example dropped the preceding residual path while its comment claimed the opposite; it now joins the two and a tested snippet shows root|a|b resolving to a/b. - Changelog and storage guide state the memory:// routing split with fsspec, the undecoded percent-escapes in file: roots, the local-file-root limitation, the entry-point collision warnings, and the zarr_format default changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ature/url-pipeline-core # Conflicts: # src/zarr/api/asynchronous.py # src/zarr/core/group.py
…leSystemWrapper fsspec < 2024.12.0 (the min_deps job) cannot open a plain memory:// URL at all, so the routing comparison only applies with fsspec>=2024.12.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Group.open`, `Array.open` and their async variants defaulted to `zarr_format=3`, so a pipeline with a `zarr2:` segment raised a conflict unless the caller passed `zarr_format=None`. The default is now a private "unspecified" sentinel. When the caller does not pass a format, a URL pipeline decides: its format segment if any, otherwise auto-detection. Every other store keeps the default of 3, so the metadata probing of non-pipeline callers is unchanged. An explicit format (including None) behaves as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The format adapters from the URL pipeline spec (schemes/zarr.md). The segment body is a path within the preceding resource and is joined onto the preceding residual path; `zarr2:` / `zarr3:` record the Zarr format and `zarr:` leaves it to auto-detection (keeping a format pinned by an earlier segment). Every other field of the preceding resolution is carried forward with `dataclasses.replace`, so a format segment also works as an intermediate segment. The builtins are registered as lazily loaded entry points ahead of the third-party ones in `_collect_entrypoints`, so importing zarr does not import them, and a third-party `zarr.url_adapters` entry point with the same name is ignored with a warning. They are adapter-only schemes: a string without `|` such as `zarr3:foo` is never routed to them as a pipeline root, so its meaning is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uide Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`zip:` (schemes/zip.md) addresses an entry or directory inside the ZIP
archive that the preceding pipeline resolves to. The preceding pipeline is
resolved with mode "r", and the archive is read through that store with
async byte-range requests only: the central directory is fetched from the
end of the archive (parsed in a worker thread), and each member is fetched
and decompressed on demand, with true range reads for stored members.
A thin read-only reader over the preceding store's bytes is used instead
of `ZipStore`, because `ZipStore` does blocking file I/O and needs a local
path or a sync file object. The reader works the same way for local files,
fsspec objects, `memory:` entries and nested `zip:` archives, and never
blocks the IO event loop.
Reading a root that *is* the archive relies on `get("")` addressing the
root object (follow-up F2); that is isolated in `_read_resource_range` /
`_resource_size` so the file-resource primitive can replace it.
Writable modes ("w", "w-", "r+") raise `URLPipelineError`; "a" and None
open read-only. One leading "/" in the body is ignored and more than one is
invalid. The preceding store is closed when the archive cannot be opened,
and the reader closes it on `close()`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-zip-v2 # Conflicts: # changes/4192.feature.md # docs/user-guide/extending.md # docs/user-guide/storage.md # src/zarr/abc/url_pipeline.py # src/zarr/api/asynchronous.py # src/zarr/core/array.py # src/zarr/core/group.py # src/zarr/errors.py # src/zarr/registry.py # src/zarr/storage/__init__.py # src/zarr/storage/_common.py # src/zarr/storage/_url_pipeline.py # tests/package_with_entrypoint/__init__.py # tests/test_url_pipeline/test_parser.py # tests/test_url_pipeline/test_resolver.py # tests/test_url_pipeline/test_spec_examples.py
ZipAdapter resolves the preceding pipeline read-only and reads the ZIP archive it points to through ZipReaderStore, a read-only store that uses async byte-range requests on the preceding store. The same path serves local files, fsspec objects, memory: entries and nested archives. The segment body is the path inside the archive. Every mode except "r" is rejected with URLPipelineError, as is zip: in the root position. A missing or unreadable archive raises URLPipelineError and closes the preceding store. The adapter is registered as an entry point under zarr.url_adapters in pyproject.toml, so it is loaded lazily like a third-party adapter. A core registry test that assumed no installed entry points now clears the pending list first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Groups and arrays through file:, memory: and fsspec roots (skipped when fsspec lacks AsyncFileSystemWrapper), nested paths and archives, zarr2: and zarr3: format segments, compression methods, mode rejection, a root-position zip:, missing and invalid archives, and the reader store's byte ranges and listing. Uses only the public API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4477 +/- ##
==========================================
+ Coverage 94.84% 94.92% +0.08%
==========================================
Files 96 97 +1
Lines 13998 14218 +220
==========================================
+ Hits 13276 13497 +221
+ Misses 722 721 -1
🚀 New features to boost your workflow:
|
Opening a directory raises PermissionError on Windows instead of reading as a missing key, so the adapter reports "could not open the ZIP archive". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A store that returns fewer bytes than its reported size made the central-directory loop refetch the same range forever; it now raises BadZipFile, which the adapter reports as a URLPipelineError. The stored- and compressed-member paths share one byte-range helper. New tests cover truncated archives, unexpected errors closing the preceding store, a bad local header, encrypted members, threaded decompression and the sparse file view, so the module is fully covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a builtin
zip:URL pipeline adapter, built on the adapter API from #4192. It reads a ZIP archive that the preceding pipeline resolves to: a local file, an fsspec object, amemory:entry, or a member of another archive. Reads use async byte-range requests on the preceding store, so a remote archive isn't downloaded whole.mode="r"is accepted."a"(theopen_groupdefault),"w","w-","r+"andNoneraiseURLPipelineError. To write an archive, useZipStore.zip:at the root of a pipeline raisesURLPipelineError, and so does a missing, truncated or non-ZIP archive. On error, the preceding store is closed../..components, or a leading//are rejected.zarr.url_adaptersin zarr'spyproject.toml, so it loads lazily, like a third-party adapter. The implementation is private:zarr.storage._zip_url_adapter.For reviewers
LocalStore/FsspecStoretreating the empty key as the root object. That reliance is isolated in_read_resource_range/_resource_size, so a future file-resource primitive can replace them.zipfileover a sparse view of the archive. Any range the view lacks is fetched and the parse retried.Author attestation
TODO
docs/user-guide/*.mdchanges/🤖 Generated with Claude Code