dunders like __array__ and __dlpack__ are useful for extracting an array into a numpy array or a dlpack capsule. I'm not aware of a community-supported analog for extracting a chunked array into a chunked array container, like a dask array or a cubed array. This means we need to define an abstract chunked array, and then define a protocol that an implementation of that abstract chunked array would use to instantiate a real chunked array instance.
A few of my assumptions about the abstract chunked array:
- each chunk has the same data type
- the chunks are represented on a finite rectilinear grid (regular is a special case)
- each chunk decodes to the same in-memory array type(s)
- the grid of chunks must not be dense, so a fill value is required
As data, these assumptions are basically a subset of the Zarr array metadata document, with information about what in-memory array(s) the chunks decode into (e.g., which dunder methods does each decoded chunk define).
So if If we define __chunked_array__ and __async_chunked_array__ protocols for an object that decodes into an arbitrary chunked array, what should their signatures be? Something like __chunked_array__(self) -> Mapping[tuple[int, ...], Callable[[], ArrayLike]], i.e. it returns a mapping from chunk indices to functions that produce chunks? And for the async case, the callable would return a coroutine?
dunders like
__array__and__dlpack__are useful for extracting an array into a numpy array or a dlpack capsule. I'm not aware of a community-supported analog for extracting a chunked array into a chunked array container, like a dask array or a cubed array. This means we need to define an abstract chunked array, and then define a protocol that an implementation of that abstract chunked array would use to instantiate a real chunked array instance.A few of my assumptions about the abstract chunked array:
As data, these assumptions are basically a subset of the Zarr array metadata document, with information about what in-memory array(s) the chunks decode into (e.g., which dunder methods does each decoded chunk define).
So if If we define
__chunked_array__and__async_chunked_array__protocols for an object that decodes into an arbitrary chunked array, what should their signatures be? Something like__chunked_array__(self) -> Mapping[tuple[int, ...], Callable[[], ArrayLike]], i.e. it returns a mapping from chunk indices to functions that produce chunks? And for the async case, the callable would return a coroutine?