Skip to content

fix(workflow): skip OME-ZARR conversion when the store already exists #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lguerard merged 1 commit into from
Jun 24, 2026
Merged

fix(workflow): skip OME-ZARR conversion when the store already exists #26

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/guide/snakemake.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,11 @@ Snakemake is resumable — if jobs fail or you cancel, just relaunch the same
command and it picks up only the missing tiles. To force a clean rerun, delete
`work_dir` (or the relevant outputs).

The OME-ZARR conversion is **not redone** once `image.zarr` exists: the
`convert` rule's output is a marker file inside the store, so Snakemake skips it
on every later run. To force a fresh conversion, delete `image.zarr` or run
`snakemake --forcerun convert`.

## pixi (instead of conda)

Conda is **not** required — Snakemake runs in whatever environment launches it.
Expand Down
1 change: 0 additions & 1 deletion workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L2ltYWdlLnphcnI=
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L2xhYmVscy5kb25l
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3N0YWdlLnphcnI=
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3NlZy80LmRvbmU=
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3NlZy8wLmRvbmU=
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3NlZy8xLmRvbmU=
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3NlZy8zLmRvbmU=
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3RpbGVzLmpzb24=
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvbGFiZWxzLmRvbmU=
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "merge", "input": ["/tmp/pw_wf/out/seg/0.done", "/tmp/pw_wf/out/seg/1.done", "/tmp/pw_wf/out/seg/2.done", "/tmp/pw_wf/out/seg/3.done", "/tmp/pw_wf/out/seg/4.done", "/tmp/pw_wf/out/seg/5.done", "/tmp/pw_wf/out/seg/6.done", "/tmp/pw_wf/out/seg/7.done"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/merge.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289946198395, "starttime": 1782306826.05809, "endtime": 1782306826.057891, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/seg/0.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/pw_wf/out/seg/1.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/pw_wf/out/seg/2.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/pw_wf/out/seg/3.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/pw_wf/out/seg/4.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/pw_wf/out/seg/5.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/pw_wf/out/seg/6.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/pw_wf/out/seg/7.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvc2VnLzAuZG9uZQ==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "segment", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json", "/tmp/pw_wf/out/stage.zarr", "/tmp/pw_wf/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289938895482, "starttime": 1782306823.2564273, "endtime": 1782306823.255227, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/tiles.json": "sha256:d5abe0a9301e6274b02cdae545a8b5e414d3b51c1396d18ae9bcb1ea935bfe67", "/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvc2VnLzEuZG9uZQ==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "segment", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json", "/tmp/pw_wf/out/stage.zarr", "/tmp/pw_wf/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289938895635, "starttime": 1782306823.2361996, "endtime": 1782306823.2352273, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/tiles.json": "sha256:d5abe0a9301e6274b02cdae545a8b5e414d3b51c1396d18ae9bcb1ea935bfe67", "/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvc2VnLzIuZG9uZQ==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "segment", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json", "/tmp/pw_wf/out/stage.zarr", "/tmp/pw_wf/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289938895788, "starttime": 1782306820.7251186, "endtime": 1782306820.724245, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/tiles.json": "sha256:d5abe0a9301e6274b02cdae545a8b5e414d3b51c1396d18ae9bcb1ea935bfe67", "/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvc2VnLzMuZG9uZQ==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "segment", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json", "/tmp/pw_wf/out/stage.zarr", "/tmp/pw_wf/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289938792521, "starttime": 1782306820.8450873, "endtime": 1782306820.8442442, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/tiles.json": "sha256:d5abe0a9301e6274b02cdae545a8b5e414d3b51c1396d18ae9bcb1ea935bfe67", "/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvc2VnLzQuZG9uZQ==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "segment", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json", "/tmp/pw_wf/out/stage.zarr", "/tmp/pw_wf/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289938792674, "starttime": 1782306820.7828398, "endtime": 1782306820.7822447, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/tiles.json": "sha256:d5abe0a9301e6274b02cdae545a8b5e414d3b51c1396d18ae9bcb1ea935bfe67", "/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvc2VnLzUuZG9uZQ==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "segment", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json", "/tmp/pw_wf/out/stage.zarr", "/tmp/pw_wf/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289938792827, "starttime": 1782306823.5588164, "endtime": 1782306823.558225, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/tiles.json": "sha256:d5abe0a9301e6274b02cdae545a8b5e414d3b51c1396d18ae9bcb1ea935bfe67", "/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvc2VnLzYuZG9uZQ==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "segment", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json", "/tmp/pw_wf/out/stage.zarr", "/tmp/pw_wf/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289938792980, "starttime": 1782306823.0890763, "endtime": 1782306823.0882282, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/tiles.json": "sha256:d5abe0a9301e6274b02cdae545a8b5e414d3b51c1396d18ae9bcb1ea935bfe67", "/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvc2VnLzcuZG9uZQ==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "segment", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json", "/tmp/pw_wf/out/stage.zarr", "/tmp/pw_wf/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289938793133, "starttime": 1782306821.0048127, "endtime": 1782306821.0042431, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/tiles.json": "sha256:d5abe0a9301e6274b02cdae545a8b5e414d3b51c1396d18ae9bcb1ea935bfe67", "/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvc3RhZ2UuemFycg==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "prepare", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/prepare_tiles.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289946375343, "starttime": 1782306818.2703326, "endtime": 1782306818.2701185, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
1 change: 1 addition & 0 deletions workflow/.snakemake/metadata/L3RtcC9wd193Zi9vdXQvdGlsZXMuanNvbg==
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"rule": "prepare", "input": ["/tmp/pw_wf/out/image.zarr/zarr.json"], "log": [], "shellcmd": null, "params": [], "code": " \"../scripts/prepare_tiles.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 8289946375343, "starttime": 1782306818.2703326, "endtime": 1782306818.2692626, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/pw_wf/out/image.zarr/zarr.json": "sha256:8baa9bb439818b6e4a5358464300e6239dae6fddb2b31fa26717a7c56eaf1458"}}
2 changes: 2 additions & 0 deletions workflow/config/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ work_dir: "/path/to/results" # everything is written under here
reuse_pyramid: true # for .ims: copy its own pyramid (fast); else rebuild
convert_chunks: null # null → patchworks' bounded auto chunks; or [c,z,y,x]
shard: false # true → pack chunks into shards (zarr v3; fewer files)
# Re-running reuses an existing image.zarr automatically (skips conversion).
# To force a rebuild: delete image.zarr, or `snakemake --forcerun convert`.

# ---- tiling -----------------------------------------------------------------
channel: 0 # channel to segment
Expand Down
5 changes: 5 additions & 0 deletions workflow/rules/common.smk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

WORK = config["work_dir"]
IMAGE = f"{WORK}/image.zarr"
# A single file inside the store, used as the convert rule's output and as the
# dependency marker for downstream rules. Tracking a leaf file (not the
# directory) lets Snakemake skip conversion when the store already exists and
# avoids wiping the whole store on a re-run (same trick as imcf/sopa).
IMAGE_OK = f"{IMAGE}/zarr.json"
TILES = f"{WORK}/tiles.json"
STAGE = f"{WORK}/stage.zarr"

Expand Down
3 changes: 2 additions & 1 deletion workflow/rules/convert.smk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

rule convert:
output:
directory(IMAGE),
# marker file inside the store; existence => skip re-conversion.
IMAGE_OK,
script:
"../scripts/convert.py"
4 changes: 2 additions & 2 deletions workflow/rules/segment.smk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

checkpoint prepare:
input:
IMAGE,
IMAGE_OK,
output:
tiles=TILES,
stage=directory(STAGE),
Expand All @@ -15,7 +15,7 @@ rule segment:
input:
tiles=TILES,
stage=STAGE,
image=IMAGE,
image=IMAGE_OK,
output:
f"{WORK}/seg/{{index}}.done",
script:
Expand Down
10 changes: 8 additions & 2 deletions workflow/scripts/convert.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
"""Snakemake script: convert the input to a pyramidal OME-ZARR."""
"""Snakemake script: convert the input to a pyramidal OME-ZARR.

The rule's output is a marker file inside the store (``image.zarr/zarr.json``),
so Snakemake skips this step entirely when the store already exists — the
conversion is not redone. To force a fresh conversion, delete ``image.zarr``
(or run ``snakemake --forcerun convert``).
"""

from patchworks.plugins.ome_zarr import to_ome_zarr

cfg = snakemake.config # noqa: F821 (injected by Snakemake)
chunks = cfg.get("convert_chunks")
to_ome_zarr(
cfg["input"],
snakemake.output[0], # noqa: F821
str(snakemake.output[0]).removesuffix("/zarr.json"), # noqa: F821
chunks=tuple(chunks) if chunks else None,
shard=bool(cfg.get("shard", False)),
reuse_pyramid=bool(cfg.get("reuse_pyramid", False)),
Expand Down
Loading