Skip to content

feat: read-only builtin zip: URL pipeline adapter - #4477

Draft
jhamman wants to merge 23 commits into
zarr-developers:mainfrom
jhamman:feature/url-pipeline-zip
Draft

jhamman wants to merge 23 commits into
zarr-developers:mainfrom
jhamman:feature/url-pipeline-zip

Conversation

@jhamman

@jhamman jhamman commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds a builtin zip: URL pipeline adapter, built on the adapter API from #4192. It reads a ZIP archive that the preceding pipeline resolves to: a local file, an fsspec object, a memory: entry, or a member of another archive. Reads use async byte-range requests on the preceding store, so a remote archive isn't downloaded whole.

from zarr.storage import URLPipeline

group = zarr.open_group(URLPipeline.from_url("s3://bucket/data.zip|zip:inner|zarr3:"), mode="r")
  • Read-only: only mode="r" is accepted. "a" (the open_group default), "w", "w-", "r+" and None raise URLPipelineError. To write an archive, use ZipStore.
  • zip: at the root of a pipeline raises URLPipelineError, and so does a missing, truncated or non-ZIP archive. On error, the preceding store is closed.
  • The segment body is the path inside the archive. A query, ./.. components, or a leading // are rejected.
  • Registered as an entry point under zarr.url_adapters in zarr's pyproject.toml, so it loads lazily, like a third-party adapter. The implementation is private: zarr.storage._zip_url_adapter.

For reviewers

  • Reading an archive that is the pipeline root relies on LocalStore / FsspecStore treating the empty key as the root object. That reliance is isolated in _read_resource_range / _resource_size, so a future file-resource primitive can replace them.
  • The central directory is parsed with the stdlib zipfile over a sparse view of the archive. Any range the view lacks is fetched and the parse retried.

Author attestation

  • I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.md
  • Changes documented as a new file in changes/
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

🤖 Generated with Claude Code

jhamman and others added 16 commits August 19, 2026 09:16
Implements URL pipeline support (https://github.com/jbms/url-pipeline):
'|'-chained URLs resolve through pluggable adapters registered under the
'zarr.url_adapters' entry-point group (entry-point name = URL scheme).

- zarr.abc.url_pipeline: PipelineSegment, AdapterResolution,
  PipelineContext, URLPipelineAdapter (single-classmethod contract)
- zarr.storage._url_pipeline: parse_pipeline / resolve_pipeline; the root
  sub-URL delegates to make_store so existing file/memory/fsspec routing
  is unchanged
- registry: register_url_adapter / get_url_adapter /
  list_url_adapter_schemes (name check only; no adapter imports)
- make_store/make_store_path route strings containing '|' (or a
  registered root scheme) through the resolver; residual store paths
  combine with the user-supplied path
- StorePath gains a zarr_format attribute (populated by format segments
  in a follow-up)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e-core

# Conflicts:
#	src/zarr/errors.py
#	src/zarr/registry.py
…thoring

- make_store validates the access mode before routing a URL pipeline
  string to adapters, matching every other StoreLike branch
- make_store docstring lists URL pipeline strings; StorePath.open
  documents the invalid-mode ValueError
- user guide: StoreLike bullet for pipeline strings and an adapter
  authoring section in extending.md (runnable example)
- changelog: drop PR-relative wording, name the public parser entry points

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
get_url_adapter took a non-reentrant lock across entry_point.load(), so an
adapter module that resolved another scheme at import time deadlocked the
process. Matching entry points are now taken off the pending list under the
lock and imported outside it; import failures are wrapped in URLPipelineError
and leave the entry point discoverable for a retry.

A same-named entry point is discarded with a ZarrUserWarning when the scheme
is already registered (e.g. by a builtin), instead of staying pending forever;
duplicate entry-point names warn and the first wins.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ent contracts

- The resolver closes the store an adapter returned when it cannot be made
  read-only for mode 'r', instead of leaking it.
- resolve_pipeline documents every error type it lets through; the
  resolve_preceding docstring states the local-file-root limitation.
- AdapterResolution and PipelineContext compare and hash by identity, since
  a Store is not hashable and frozen=True implied otherwise.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… caller

The zarr2:/zarr3: format carried on StorePath.zarr_format was merged only in
zarr.api.asynchronous; create_array, from_array, Group.from_store,
Group.open, Array.open and the deprecated AsyncArray._create dropped it and
swallowed explicit conflicts. The merge now lives on
StorePath.resolve_zarr_format and is applied at each site.

To let the pipeline's format apply when the caller does not specify one,
create_array (sync and async) and Group.from_store default zarr_format to
None, resolving to the configured default (3) as before; Array.open gains a
zarr_format parameter mirroring Group.open.

Tests: registry deadlock/race/import-failure/shadowing, resolver leak,
zarr_format merge at each core site, and pins for the documented memory:/
file: divergences and the local-file-root limitation (xfail, strict).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- The extending guide's wrapper example dropped the preceding residual path
  while its comment claimed the opposite; it now joins the two and a tested
  snippet shows root|a|b resolving to a/b.
- Changelog and storage guide state the memory:// routing split with fsspec,
  the undecoded percent-escapes in file: roots, the local-file-root
  limitation, the entry-point collision warnings, and the zarr_format
  default changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ature/url-pipeline-core

# Conflicts:
#	src/zarr/api/asynchronous.py
#	src/zarr/core/group.py
…leSystemWrapper

fsspec < 2024.12.0 (the min_deps job) cannot open a plain memory:// URL at
all, so the routing comparison only applies with fsspec>=2024.12.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`Group.open`, `Array.open` and their async variants defaulted to
`zarr_format=3`, so a pipeline with a `zarr2:` segment raised a conflict
unless the caller passed `zarr_format=None`.

The default is now a private "unspecified" sentinel. When the caller does
not pass a format, a URL pipeline decides: its format segment if any,
otherwise auto-detection. Every other store keeps the default of 3, so the
metadata probing of non-pipeline callers is unchanged. An explicit format
(including None) behaves as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The format adapters from the URL pipeline spec (schemes/zarr.md). The
segment body is a path within the preceding resource and is joined onto
the preceding residual path; `zarr2:` / `zarr3:` record the Zarr format and
`zarr:` leaves it to auto-detection (keeping a format pinned by an earlier
segment). Every other field of the preceding resolution is carried forward
with `dataclasses.replace`, so a format segment also works as an
intermediate segment.

The builtins are registered as lazily loaded entry points ahead of the
third-party ones in `_collect_entrypoints`, so importing zarr does not
import them, and a third-party `zarr.url_adapters` entry point with the
same name is ignored with a warning. They are adapter-only schemes: a
string without `|` such as `zarr3:foo` is never routed to them as a
pipeline root, so its meaning is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uide

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`zip:` (schemes/zip.md) addresses an entry or directory inside the ZIP
archive that the preceding pipeline resolves to. The preceding pipeline is
resolved with mode "r", and the archive is read through that store with
async byte-range requests only: the central directory is fetched from the
end of the archive (parsed in a worker thread), and each member is fetched
and decompressed on demand, with true range reads for stored members.

A thin read-only reader over the preceding store's bytes is used instead
of `ZipStore`, because `ZipStore` does blocking file I/O and needs a local
path or a sync file object. The reader works the same way for local files,
fsspec objects, `memory:` entries and nested `zip:` archives, and never
blocks the IO event loop.

Reading a root that *is* the archive relies on `get("")` addressing the
root object (follow-up F2); that is isolated in `_read_resource_range` /
`_resource_size` so the file-resource primitive can replace it.

Writable modes ("w", "w-", "r+") raise `URLPipelineError`; "a" and None
open read-only. One leading "/" in the body is ignored and more than one is
invalid. The preceding store is closed when the archive cannot be opened,
and the reader closes it on `close()`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jhamman and others added 4 commits October 31, 2026 08:58
…e-zip-v2

# Conflicts:
#	changes/4192.feature.md
#	docs/user-guide/extending.md
#	docs/user-guide/storage.md
#	src/zarr/abc/url_pipeline.py
#	src/zarr/api/asynchronous.py
#	src/zarr/core/array.py
#	src/zarr/core/group.py
#	src/zarr/errors.py
#	src/zarr/registry.py
#	src/zarr/storage/__init__.py
#	src/zarr/storage/_common.py
#	src/zarr/storage/_url_pipeline.py
#	tests/package_with_entrypoint/__init__.py
#	tests/test_url_pipeline/test_parser.py
#	tests/test_url_pipeline/test_resolver.py
#	tests/test_url_pipeline/test_spec_examples.py
ZipAdapter resolves the preceding pipeline read-only and reads the ZIP
archive it points to through ZipReaderStore, a read-only store that uses
async byte-range requests on the preceding store. The same path serves
local files, fsspec objects, memory: entries and nested archives. The
segment body is the path inside the archive.

Every mode except "r" is rejected with URLPipelineError, as is zip: in the
root position. A missing or unreadable archive raises URLPipelineError and
closes the preceding store.

The adapter is registered as an entry point under zarr.url_adapters in
pyproject.toml, so it is loaded lazily like a third-party adapter. A
core registry test that assumed no installed entry points now clears the
pending list first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Groups and arrays through file:, memory: and fsspec roots (skipped when
fsspec lacks AsyncFileSystemWrapper), nested paths and archives, zarr2:
and zarr3: format segments, compression methods, mode rejection, a
root-position zip:, missing and invalid archives, and the reader store's
byte ranges and listing. Uses only the public API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.92%. Comparing base (01fa6d9) to head (65eaae7).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4477      +/-   ##
==========================================
+ Coverage   94.84%   94.92%   +0.08%     
==========================================
  Files          96       97       +1     
  Lines       13998    14218     +220     
==========================================
+ Hits        13276    13497     +221     
+ Misses        722      721       -1     
Files with missing lines Coverage Δ
src/zarr/storage/_zip_url_adapter.py 100.00% <100.00%> (ø)

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jhamman and others added 3 commits October 31, 2026 09:29
Opening a directory raises PermissionError on Windows instead of reading
as a missing key, so the adapter reports "could not open the ZIP archive".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A store that returns fewer bytes than its reported size made the
central-directory loop refetch the same range forever; it now raises
BadZipFile, which the adapter reports as a URLPipelineError. The stored-
and compressed-member paths share one byte-range helper. New tests cover
truncated archives, unexpected errors closing the preceding store, a bad
local header, encrypted members, threaded decompression and the sparse
file view, so the module is fully covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant