Dev

.github/workflows/publish.yml

@@ -0,0 +1,55 @@
+name: Publish to PyPI
+
+# Runs when you publish a GitHub Release (Releases -> Draft a new release -> Publish),
+# OR manually via the "Run workflow" button on the Actions tab (workflow_dispatch).
+# Builds the sdist + wheel and uploads them to PyPI via Trusted Publishing (OIDC) -
+# no API token/secret required (configure the trusted publisher on PyPI once, see README).
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+jobs:
+  build:
+    name: Build distributions
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.x"
+
+      - name: Build sdist and wheel
+        run: |
+          python -m pip install --upgrade build
+          python -m build
+
+      - name: Check metadata
+        run: |
+          python -m pip install --upgrade twine
+          python -m twine check dist/*
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    name: Publish to PyPI
+    needs: build
+    runs-on: ubuntu-latest
+    # The GitHub environment must match the one set on the PyPI trusted publisher.
+    environment:
+      name: pypi
+      url: https://pypi.org/project/dyna-zarr/
+    permissions:
+      id-token: write        # OIDC token for Trusted Publishing
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+      - name: Publish
+        uses: pypa/gh-action-pypi-publish@release/v1

.gitignore

@@ -61,3 +61,7 @@ env/
 # Project-specific
 _archive/
 .coverage
+
+# Non-shipped / local-only (bench + perf report; tests ARE tracked and CI-run)
+benchmarks/
+reports/

README.md

@@ -1,172 +1,182 @@
-# dyna_zarr
+# dyna-zarr
 
-A lightweight Python library for lazy operations on Zarr arrays.
+A lightweight, dask-free Python library for lazy, memory-bounded operations on large Zarr (and TIFF) arrays, with an optional GPU path.
 
 ## Overview
 
-**dyna_zarr** provides a thin layer on top of [Zarr](https://zarr-python.readthedocs.io/) for lazy and dynamic array processing. It is designed to simplify working with large, multidimensional datasets by enabling memory-efficient, region-wise I/O and computation.
+dyna-zarr is a thin, pull-based array layer over [Zarr](https://zarr-python.readthedocs.io/). Instead of building a task graph, every operation is a lazy *transform* whose `read(key)` maps an output slice back to a bounded input read, ending at a direct zarr/TensorStore read. Slicing a result pulls only that region through the whole operation chain, so no intermediates are materialized.
+
+The practical consequence is memory-boundedness. When you stream a result to disk with `io.write`, the array is processed region by region, so peak RAM is a function of the region and worker budget rather than the array size. This makes it possible to read, transform, and write arrays far larger than memory.
+
+## Memory-boundedness
+
+There are two ways to run a lazy result, with different memory behavior:
+
+- `io.write(result, path)` streams the result to disk region by region. Peak RAM is roughly `region_size_mb * max_workers`, independent of the array size. This is the memory-bounded path.
+- `result.compute()` returns a single in-memory NumPy array. It materializes the whole result by design (mirroring `dask.array.compute`), so it is not memory-bounded. Use it only for results that fit in RAM.
+
+Every operation is memory-bounded on the `io.write` path except `median`, `argmin`, and `argmax`, which are flagged in the operations catalog below.
 
 ## Features
 
-- **Lazy evaluation** – Operations are deferred until explicitly computed
-- **Multi-format I/O** – Read from TIFF, Zarr v2, and Zarr v3; write to Zarr v2 or Zarr v3
-- **Efficient region-wise processing** – Data is processed in regions, where each region may span multiple chunks
-- **Minimal dependencies** – Requires only `zarr`, `numpy`, `tensorstore`, and `tifffile`
+- **Pull-based and lazy.** Operations defer until `.compute()` (materialize) or `io.write` (stream to disk).
+- **Memory-bounded streaming.** Region-wise `io.write` with per-worker memory and worker-count knobs. Even reshape, flatten, and rechunk of incompatibly-chunked data stay bounded, by staging through disk.
+- **NumPy-like.** Operator overloads, array methods (`.astype`, `.clip`, `.round`), and the NumPy ufunc protocol (`np.sqrt(a)`, `np.add(a, 2)`) all work on a `DynamicArray`.
+- **Rich op set.** About 90 operations: pointwise ufuncs, streaming reductions, neighborhood (halo) filters, structural reshaping, differences, and array creation.
+- **Multi-format I/O.** Read TIFF, Zarr v2, and Zarr v3 (local, S3/GCS, HTTP); write Zarr v2/v3 with optional sharding.
+- **Optional GPU.** Run an op chain on CUDA via CuPy, with a single host-to-device transfer per region.
 
 ## Installation
 
 ```bash
-pip install git+https://github.com/bugraoezdemir/dyna_zarr.git
+pip install dyna-zarr
 ```
 
-For development:
+Optional GPU support (pick the extra matching your CUDA toolkit from `nvidia-smi`):
 
 ```bash
-git clone https://github.com/bugraoezdemir/dyna_zarr.git
-cd dyna_zarr
-pip install -e ".[dev]"
+pip install "dyna-zarr[gpu-cu12]"   # CUDA 12.x  ([gpu] is an alias for this)
+pip install "dyna-zarr[gpu-cu11]"   # CUDA 11.x
+pip install "dyna-zarr[gpu-cu13]"   # CUDA 13.x (e.g. Blackwell)
 ```
 
-## Quick Start
+## Quick start
 
-### Reading Arrays
+### Read
 
 ```python
 from dyna_zarr import io
 
-# Read a TIFF file as a DynamicArray
-arr = io.read("image.tiff")
-print(arr.shape)   # (100, 256, 256)
-print(arr.dtype)   # dtype('uint16')
-
-# Read Zarr v2
-arr = io.read("array_v2.zarr")
-
-# Read Zarr v3
-arr = io.read("array_v3.zarr")
+arr = io.read("image.tiff")      # TIFF via tifffile's zarr bridge
+arr = io.read("array_v2.zarr")   # Zarr v2
+arr = io.read("array_v3.zarr")   # Zarr v3  (also s3://, gs://, http://)
 
-# Compute the full array into memory
-data = arr.compute()
+print(arr.shape, arr.dtype, arr.chunks)
 
-# Compute a region of interest
-region = arr[10:20, 50:150, 100:200].compute()
+data   = arr.compute()                           # materialize the whole array
+region = arr[10:20, 50:150, 100:200].compute()   # pull just this region
 ```
 
-### Writing Arrays
+### Write (memory-bounded streaming)
 
 ```python
 from dyna_zarr import io, Codecs
 
-# Write to Zarr v3
-io.write(arr, "output_v3.zarr", zarr_format=3)
+io.write(arr, "out_v3.zarr", zarr_format=3)
+io.write(arr, "out.zarr", chunks=(64, 64, 64), zarr_format=3)
+io.write(arr, "out.zarr", dtype="float32", zarr_format=3)       # cast on write
+io.write(arr, "out.zarr", compressor=Codecs(compressor="zstd", clevel=5), zarr_format=3)
 
-# Write to Zarr v2
-io.write(arr, "output_v2.zarr", zarr_format=2)
-
-# Specify custom chunks
-io.write(arr, "output.zarr", chunks=(64, 64, 64), zarr_format=3)
-
-# Enable compression
-codecs = Codecs(compressor="zstd", clevel=5)
-io.write(arr, "output.zarr", compressor=codecs, zarr_format=3)
-
-# Convert dtype during write
-io.write(arr, "output.zarr", dtype="float32", zarr_format=3)
+# memory and parallelism controls (peak RAM is roughly region_size_mb * max_workers)
+io.write(arr, "out.zarr", region_size_mb=64, max_workers=4)
 ```
 
-### Lazy Operations
+### Lazy operation chains
 
 ```python
-from dyna_zarr import io, operations
+from dyna_zarr import io, operations as ops
 
-# Read array
 arr = io.read("input.zarr")
 
-# Apply lazy transformations (no computation yet)
-result = operations.abs(arr)
-result = operations.clip(result, 0, 1)
-result = operations.sqrt(result)
+result = ops.sqrt(ops.clip(ops.abs(arr), 0, 1))   # nothing computed yet
+io.write(result, "output.zarr", zarr_format=3)     # streamed, region by region
+# ...or result.compute() to materialize
+```
 
-# Write result using region-wise processing
-io.write(result, "output.zarr", zarr_format=3)
+### NumPy-like interface
+
+A `DynamicArray` behaves like a NumPy or dask array. Operators, methods, and ufuncs are all lazy:
+
+```python
+import numpy as np
 
-# Or fully materialize the result
-final_data = result.compute()
+masked = (arr > 3) & (arr < 100)     # elementwise operators build a lazy mask
+scaled = (arr.astype("float32") / 255).clip(0, 1)
+out    = np.sqrt(np.abs(arr))        # NumPy ufunc protocol dispatches to lazy ops
 ```
 
-### Multi-source Operations
+### Neighborhood filters
+
+Neighborhood (halo) filters wrap `scipy.ndimage`. Each read pulls its own halo, so results are chunk-invariant and exact, and stay memory-bounded when streamed.
 
 ```python
-from dyna_zarr import io, operations
+import numpy as np
+from dyna_zarr import io, operations as ops
+
+img = io.read("volume.zarr")   # e.g. (z, y, x)
 
-# Read multiple sources
-arr1 = io.read("input1.zarr")
-arr2 = io.read("input2.zarr")
+# LoG filtering
+log    = ops.gaussian_laplace(img, sigma=2)
+io.write(log, "log.zarr", zarr_format=3)   # halo handled per region
 
-# Chain operations
-result = operations.concatenate([arr1, arr2], axis=0)
-result = operations.clip(result, -1, 1)
+# median denoise
+denoised = ops.median_filter(img, size=3)
+io.write(denoised, "denoised.zarr")
 
-# Write result
-io.write(result, "concatenated_output.zarr", zarr_format=3)
+# a custom per-plane kernel
+kernel   = np.ones((1, 3, 3), dtype="float32") / 9   # 3x3 mean within each z-plane
+blurred  = ops.convolve(img, kernel)
+io.write(blurred, "blurred.zarr")
 ```
 
-## Core Components
+## Operations catalog
 
-- **`io.read()`** – Read TIFF, Zarr v2, or Zarr v3 sources and return a
-  `DynamicArray`
-- **`io.write()`** – Write a `DynamicArray` to Zarr v2 or v3 with optional
-  region-wise execution
-- **`operations`** – Lazy transformation functions such as `abs`,
-  `clip`, `sqrt`, `concatenate`, `reshape`, and `slice`
-- **`DynamicArray`** – Core lazy array abstraction supporting slicing,
-  shape/dtype inspection, and `.compute()`
-- **`Codecs`** – Compression configuration for Zarr v2 and v3
+Every operation is lazy, and memory-bounded on the `io.write` path except `median`, `argmin`, and `argmax` (see Memory-boundedness). All are available flat on `dyna_zarr.operations`, and also grouped by category submodule.
 
-## Further Examples
+- **Pointwise / ufuncs.** `abs`, `negative`, `sign`, `sqrt`, `square`, `exp`, `log`, `log2`, `log10`, `floor`, `ceil`, `reciprocal`, `round`, `clip`, `astype`; binary `add`, `subtract`, `multiply`, `divide`, `floor_divide`, `mod`, `power`, `maximum`, `minimum`; comparisons `greater(_equal)`, `less(_equal)`, `equal`, `not_equal`; logical `and`, `or`, `xor`, `not`; `where`, `isin`, `digitize`.
+- **Reductions.** Streaming and memory-bounded: `min`, `max`, `sum`, `prod`, `mean`, `any`, `all`, `var`, `std`, `histogram` (with `axis=` and `keepdims=`). Not fully bounded (hold the full reduced axis): `median`, `argmin`, `argmax`.
+- **Neighborhood (halo/overlap).** `gaussian_filter`, `uniform_filter`, `median_filter`, `minimum_filter`, `maximum_filter`, `grey_erosion`, `grey_dilation`, `convolve`, `correlate`, `laplace`, `gaussian_laplace`, `gaussian_gradient_magnitude`.
+- **Structural.** `concatenate`, `stack`, `transpose`, `swap_axes`, `reshape`, `flatten`, `squeeze`, `expand_dims`, `pad`, `tile`, `roll`, `flip`, `rot90`, `slice_array`.
+- **Differences.** `diff`, `gradient`.
+- **Scan (prefix, along one axis).** `cumsum`, `cumprod`, `cummax`, `cummin`. Streamed with a bounded carry on the `io.write` path, so memory-bounded despite the sequential dependency.
+- **Creation.** `zeros`, `ones`, `full`, `empty`, `random` (and the `*_like` variants). `random` is position-deterministic, so the result is independent of chunking.
+- **Primitives.** `map_blocks` (pointwise), `map_overlap` (neighborhood with a halo), `reduce` (streaming). Use these to build your own ops.
 
-### Manual Configuration of Region Size
+## Memory-bounded reshape, flatten, and rechunk
 
-```python
-from dyna_zarr import io
+C-order reshape and flatten conflict with n-dimensional chunk layout, so a naive implementation blows up. dyna-zarr stages these through disk (a Rechunker-style two-phase, read-once/write-once copy), so peak RAM stays a function of the per-worker budget rather than the array size. When you `io.write` an outermost `reshape` or `flatten`, this path is used automatically:
 
-arr = io.read("large_array.zarr")
+```python
+from dyna_zarr import io, operations as ops
 
-# Process approximately 64 MB regions at a time
-io.write(arr, "output.zarr", region_size_mb=64)
+arr = io.read("big_4d.zarr")                 # e.g. 5 GB, awkward chunks
+io.write(ops.flatten(arr), "flat.zarr", region_size_mb=128, max_workers=2)
+io.write(ops.reshape(arr, (a, b)), "reshaped.zarr")   # (a, b) is any target shape of the same size
 ```
 
-### Zarr Sharding (Zarr v3 only)
+## GPU (optional)
+
+With a CuPy install, run a chain on the GPU. Setting `device='cuda'` on a terminal call (`compute` or `io.write`) makes device-inheriting ops run on the GPU. A single host-to-device transfer happens at the first CUDA op and the data stays resident up the chain. Results are returned or written from the host.
 
 ```python
-io.write(
-    arr,
-    "sharded_output.zarr",
-    chunks=(64, 64, 64),            # Inner chunk size
-    shard_coefficients=(4, 4, 4),   # Shard size = 4x chunks in each dimension
-    zarr_format=3,
-)
+result = ops.gaussian_filter(arr, sigma=3)
+out = result.compute(device="cuda")          # whole chain on the GPU
+io.write(result, "out.zarr", device="cuda")  # per-region GPU compute, streamed write
 ```
 
-## Requirements
+## Relationship to dask
 
-- Python >= 3.11
-- zarr >= 3.0.0
-- numpy >= 1.20.0
-- tensorstore
-- tifffile
+dyna-zarr is not a general replacement for `dask.array`. It targets one job: memory-bounded read, transform, and write of large Zarr/TIFF arrays.
 
-## Testing
+The core idea is to drop the task graph. Because every operation is a pull-based chain, where each output slice maps back to a bounded input read, there is no graph to build and no scheduler to run it. That keeps the engine small, keeps peak RAM bounded by `region_size_mb * max_workers` on the `io.write` path, and avoids scheduling overhead, which makes the read-transform-write pipeline efficient.
 
-Run the test suite:
+The tradeoff is that only operations that fit this slice-pushdown model belong in the chain: pointwise math, neighborhood/halo filters, streaming reductions, and structural reshaping. These are operations that are commonly used in image processing, which is what dyna-zarr is mainly built for. Operations that would need a global, data-dependent graph do not fit directly, and a few that do (such as non-associative reductions) trade extra reads or memory to stay correct.
 
-```bash
-pytest tests/ -v
-```
+Two more differences worth knowing:
 
-Run tests with coverage:
+- **Single machine, for now.** Parallelism today is threaded I/O within one process, plus the optional GPU path. There is no cluster or distributed execution yet; better and process-based parallelism is a possible future direction.
+- **Narrower surface.** About 90 operations today, extended where the slice-pushdown model permits. Binary ops also need equal-shaped operands (no general broadcasting between differently shaped lazy arrays yet).
 
-```bash
-pytest tests/ --cov=src/dyna_zarr --cov-report=html
-```
+## Core components
+
+- `io.read(source)` reads TIFF, Zarr v2, or Zarr v3 (local or remote) into a `DynamicArray`.
+- `io.write(array, path, ...)` streams a `DynamicArray` to Zarr v2/v3 (chunks, sharding, compression, dtype cast, `region_size_mb`, `max_workers`, `device`).
+- `operations` is the lazy op set above.
+- `DynamicArray` is the pull-based lazy array (slicing, `.compute()`, operators, `.astype`/`.clip`/`.round`, ufunc protocol).
+- `Codecs` is the compression configuration for Zarr v2 and v3.
+
+## Requirements
 
+- Python 3.11 or newer
+- zarr 3.0.0+, numpy 1.20+, scipy 1.6+, tensorstore, tifffile
+- Optional: CuPy (via the `gpu-cuXX` extras) for the GPU path

pyproject.toml

@@ -9,8 +9,8 @@ packages = ["dyna_zarr"]
 "" = "src"
 
 [project]
-name = "dyna_zarr"
-version = "0.0.1"
+name = "dyna-zarr"
+version = "0.0.2"
 description = "A lightweight library for lazy operations on Zarr arrays without task graph overhead"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -32,6 +32,7 @@ classifiers = [
 dependencies = [
     "zarr>=3.0.0",
     "numpy>=1.20.0",
+    "scipy>=1.6.0",
     "tensorstore",
     "tifffile"
 ]
@@ -49,6 +50,18 @@ docs = [
     "sphinx>=5.0",
     "sphinx-rtd-theme>=1.0",
 ]
+# GPU execution (optional). CuPy ships CUDA-version-specific wheels and pip CANNOT
+# auto-detect your CUDA, so pick the extra that matches `nvidia-smi` (CUDA Version):
+#   pip install dyna-zarr[gpu-cu11]   # CUDA 11.x
+#   pip install dyna-zarr[gpu-cu12]   # CUDA 12.x
+#   pip install dyna-zarr[gpu-cu13]   # CUDA 13.x (e.g. Blackwell)
+# `[gpu]` is a convenience alias for CUDA 12.x. The [ctk] extra ships the NVRTC headers
+# CuPy needs to JIT kernels (without it every kernel raises "Failed to find CUDA headers").
+# Auto-detect alternative (may lag new CUDA releases): pip install cupy-wheel
+gpu = ["cupy-cuda12x[ctk]"]
+gpu-cu11 = ["cupy-cuda11x[ctk]"]
+gpu-cu12 = ["cupy-cuda12x[ctk]"]
+gpu-cu13 = ["cupy-cuda13x[ctk]"]
 
 [project.urls]
 Homepage = "https://github.com/bugraoezdemir/dyna_zarr"