Skip to content

Commit e93292f

Browse files
lguerardclaude
lguerard
and
claude
committed
docs: 📝 add a step-by-step cluster-workflow guide
Add a thorough 'Cluster workflow (Snakemake + SLURM)' guide page (install → configure every field → dry-run → local vs SLURM → monitor → outputs → troubleshooting), wire it into the nav, and link it from the workflow README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent a361ed2 commit e93292f

3 files changed

Lines changed: 218 additions & 0 deletions

File tree

‎docs/guide/snakemake.md‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
# Cluster workflow (Snakemake + SLURM)
2+
3+
`tile_process` runs every tile **serially on one GPU**. For a large 3-D image
4+
that can be days. The bundled Snakemake workflow instead submits **one GPU job
5+
per tile**, so with *N* GPUs the segmentation is ~*N*× faster. This page walks
6+
through running it from scratch.
7+
8+
```text
9+
convert ──▶ prepare (checkpoint) ──▶ segment {tile} ──▶ merge
10+
one GPU SLURM job per tile
11+
```
12+
13+
## 1. Get the workflow
14+
15+
The workflow lives in the `workflow/` directory of the patchworks repository
16+
(it is not shipped inside the pip package — it is a set of Snakemake files you
17+
run):
18+
19+
```bash
20+
git clone https://github.com/imcf/patchworks
21+
cd patchworks/workflow
22+
```
23+
24+
## 2. Install the dependencies
25+
26+
You need patchworks with the workflow + reader + segmentation extras, in the
27+
environment Snakemake will use:
28+
29+
```bash
30+
pip install "patchworks[workflow,cellpose,imaris,bioio]"
31+
```
32+
33+
- `workflow` → Snakemake + the SLURM executor plugin
34+
- `cellpose` → the segmentation model
35+
- `imaris` / `bioio` → read your input format (`.ims`, `.czi`, `.lif`, …)
36+
37+
On a cluster, do this inside a conda/venv that the compute nodes can see, or let
38+
each rule activate a conda env (see *Conda*, below).
39+
40+
## 3. Configure the run
41+
42+
Copy and edit `config/config.yaml`. Every field:
43+
44+
```yaml
45+
# input / output
46+
input: "/data/scan.ims" # .ims/.czi/.lif/.nd2/ome-tiff/.zarr
47+
work_dir: "/scratch/results" # everything is written here
48+
49+
# conversion (input → pyramidal OME-ZARR)
50+
reuse_pyramid: true # .ims: copy its own pyramid (fast)
51+
convert_chunks: null # null → bounded auto chunks; or [c,z,y,x]
52+
shard: false # true → pack chunks into shards (fewer files)
53+
54+
# tiling
55+
channel: 0 # channel to segment (null = keep all)
56+
level: 0 # pyramid level (0 = full resolution)
57+
tile_shape: "auto" # "auto", or e.g. [16, 1024, 1024] (zyx)
58+
overlap: 30 # halo ≈ one object diameter
59+
skip_empty: true # skip background tiles
60+
empty_threshold: null # null → Otsu
61+
62+
# segmentation
63+
method: "cellpose" # "cellpose" (GPU) or "threshold" (no GPU)
64+
label_name: "cellpose" # name under image.zarr/labels/
65+
cellpose:
66+
model: "cyto3"
67+
diameter: 30
68+
do_3D: true
69+
gpu: true
70+
# extra model.eval() kwargs, e.g. flow_threshold: 0.4
71+
72+
# label pyramid
73+
pyramid_levels: 5
74+
pyramid_downscale: 2
75+
sequential_labels: true # renumber labels to a contiguous 1..N
76+
```
77+
78+
!!! tip "Tile size vs runtime"
79+
`tile_shape: "auto"` sizes each tile to your GPU's VRAM. Smaller tiles =
80+
more (faster) jobs; very large 3-D tiles are slow. Keep `do_3D: false` (2-D
81+
per slice) if your objects segment fine per slice — it is much faster.
82+
83+
## 4. Dry-run (always do this first)
84+
85+
Check the plan without running anything:
86+
87+
```bash
88+
python -m snakemake -s Snakefile --configfile config/config.yaml -n -p
89+
```
90+
91+
You should see `convert`, `prepare`, and a note that the **checkpoint** will add
92+
the `segment` jobs after `prepare` runs. (The number of segment jobs is only
93+
known after `prepare` decides which tiles are non-empty.)
94+
95+
## 5a. Run locally (single machine)
96+
97+
```bash
98+
python -m snakemake -s Snakefile --configfile config/config.yaml --cores 8
99+
```
100+
101+
Tiles run on the local machine (one at a time on the GPU). Good for a small
102+
image or a smoke test.
103+
104+
## 5b. Run on SLURM (one GPU job per tile)
105+
106+
Edit `profile/slurm/config.yaml` for **your** cluster — partitions, account,
107+
and the GPU request:
108+
109+
```yaml
110+
executor: slurm
111+
jobs: 64 # max concurrent SLURM jobs ≈ GPUs you can grab
112+
default-resources:
113+
slurm_partition: "cpu" # your CPU partition
114+
# slurm_account: "my_account"
115+
mem_mb: 16000
116+
cpus_per_task: 4
117+
runtime: 60
118+
set-resources:
119+
segment: # the GPU step
120+
slurm_partition: "gpu" # your GPU partition
121+
slurm_extra: "'--gres=gpu:1'"
122+
mem_mb: 32000
123+
runtime: 120
124+
merge:
125+
mem_mb: 128000
126+
runtime: 240
127+
```
128+
129+
Then launch (from a login node — Snakemake submits and watches the jobs):
130+
131+
```bash
132+
python -m snakemake --workflow-profile profile/slurm \
133+
--configfile config/config.yaml
134+
```
135+
136+
Snakemake submits `convert`, then `prepare`, then **one `segment` job per
137+
non-empty tile** (up to `jobs:` at once → that many GPUs in parallel), then
138+
`merge`. Raise `jobs:` to use more GPUs.
139+
140+
!!! note "GPU request flag"
141+
Clusters differ. `--gres=gpu:1` is common; some need `--gpus=1` or a
142+
specific gres name (`--gres=gpu:a100:1`). Put whatever `sbatch` flag your
143+
cluster needs in `slurm_extra`.
144+
145+
## 6. Monitor
146+
147+
- **Snakemake** prints each job as it submits/finishes and a `X of Y steps`
148+
counter.
149+
- **SLURM**: `squeue --me` shows your queued/running jobs (`smk-segment`, …);
150+
logs land where your profile/cluster sends them.
151+
- **patchworks** logs (`processing tile k/N`, ETA) are inside each job's stdout.
152+
153+
## 7. Output
154+
155+
Everything is under `work_dir`:
156+
157+
```text
158+
results/
159+
image.zarr/ # converted, pyramidal OME-ZARR
160+
image.zarr/labels/<name>/ # the segmentation (multi-scale, calibrated)
161+
```
162+
163+
The labels live **inside** the image store. View image + labels together:
164+
165+
```python
166+
from patchworks.plugins.napari import view_in_napari
167+
view_in_napari("/scratch/results/image.zarr") # auto-loads the labels
168+
```
169+
170+
## 8. Re-running and resuming
171+
172+
Snakemake is resumable — if jobs fail or you cancel, just relaunch the same
173+
command and it picks up only the missing tiles. To force a clean rerun, delete
174+
`work_dir` (or the relevant outputs).
175+
176+
## Conda (optional)
177+
178+
To have each rule run in a named conda env instead of the active one, add
179+
`--use-conda` and point the rules at an env; or activate your env in a SLURM
180+
prologue. The simplest path is a single shared env that the compute nodes see.
181+
182+
## Troubleshooting
183+
184+
| Symptom | Fix |
185+
|---------|-----|
186+
| `snakemake: command not found` | use `python -m snakemake` |
187+
| Segment jobs pend forever | wrong `slurm_partition`/`slurm_extra` GPU flag for your cluster |
188+
| `cellpose is not installed` in a job | the job's env lacks `patchworks[cellpose]` |
189+
| Reading the input fails | install the matching reader (`patchworks[imaris]`/`[bioio]` + a `bioio-*`) |
190+
| Out of GPU memory | smaller `tile_shape`, or `do_3D: false` |
191+
| Very slow | confirm GPU is used (`nvidia-smi`); try 2-D or a lower `level` |
192+
193+
## How it works (for the curious)
194+
195+
The rule scripts are thin wrappers over patchworks' public API, so you can build
196+
the same per-tile distribution yourself:
197+
198+
```python
199+
from patchworks import (
200+
load_ome_zarr, spatial_tiles, create_stage, stage_tile, merge_tile_labels
201+
)
202+
from patchworks.plugins.ome_zarr import write_labels
203+
204+
img = load_ome_zarr("image.zarr", channel=0)
205+
tiles = spatial_tiles(img.shape, tile_shape=(16, 1024, 1024))
206+
create_stage("stage.zarr", img.shape, (16, 1024, 1024))
207+
# (distribute these across jobs:)
208+
for i in range(len(tiles)):
209+
stage_tile(img, my_fn, "stage.zarr", i, tile_shape=(16, 1024, 1024), overlap=30)
210+
merged = merge_tile_labels("stage.zarr", input_component="staged",
211+
write_to="merged.zarr", sequential_labels=True)
212+
write_labels("image.zarr", merged, name="cells")
213+
```

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ nav:
3939
- Empty tile skipping: guide/skip_empty.md
4040
- GPU & distributed: guide/gpu_distributed.md
4141
- Performance & memory: guide/performance.md
42+
- Cluster workflow (Snakemake): guide/snakemake.md
4243
- OME-ZARR & napari: guide/ome_zarr_napari.md
4344
- Pitfalls: guide/pitfalls.md
4445
- Examples:

‎workflow/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,10 @@
33
A SLURM-ready pipeline that segments an arbitrarily large image and spreads the
44
expensive Cellpose step across **many GPUs** — one tile per SLURM job.
55

6+
> **Full step-by-step guide:**
7+
> <https://imcf.one/patchworks/guide/snakemake/> — install, configure every
8+
> field, dry-run, local vs SLURM, monitoring, outputs and troubleshooting.
9+
610
```text
711
convert ──▶ prepare (checkpoint) ──▶ segment {tile} ──▶ merge
812
one GPU job/tile

0 commit comments

Comments
 (0)