@silvbarb has been running Cellpose SAM on some quite large images (e.g. i.e. 4, 169, 42k, 43k pixel size) and running in a bunch of places where we should profile/improve the memory footprint. The actual ROI loading has been working very well, but finding ROIs in label images & building pyramdis on such large images at the end has cause out of memory errors that I'll describe below.
The full .zarray of resolution 0 is:
{
"chunks": [
1,
10,
2000,
2000
],
"compressor": {
"blocksize": 0,
"clevel": 5,
"cname": "lz4",
"id": "blosc",
"shuffle": 1
},
"dimension_separator": "/",
"dtype": "<u2",
"fill_value": 0,
"filters": null,
"order": "C",
"shape": [
4,
169,
42318,
43111
],
"zarr_format": 2
}
Masking iterators based on label (not table)
The first failure case was when using a label image to define the ROIs, not a masking ROI table. i.e.
This ran into an out-of-memory error with 32 GB of memory. We should check if this iterator initialization based on label images can be done more memory efficiently.
Post-processing memory error
I suspect this is a memory error during pyramid building (I'll follow up with more info if a rerun with verbose logging shows anything specific).
The logs show the following before the out of memory error:
2026-03-24 07:55:18,456; cellpose_sam_task; INFO; Processed ROI 1/1 (avg time per ROI: 75.24 s)
/path/to/FRACTAL_TASK_DIR/7/fractal-cellpose-sam-task/0.1.10/source_dir/project_python.sh: line 3: 2390773 Killed
/path/to/FRACTAL_TASK_DIR/7/fractal-cellpose-sam-task/0.1.10/source_dir/.pixi/envs/default/bin/python "$@"
/path/to/FRACTAL_TASK_DIR/7/fractal-cellpose-sam-task/0.1.10/source_dir/.pixi/envs/default/lib/python3.14/multiprocessing/resource_tracker.py:396: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown: {'/mp-j93or4yq', '/mp-5gof4549'}
warnings.warn(
Thus, the actual segmentation successfully finished, but after the segmentation, the task ran into an out-of-memory error. Given that a relatively small ROI (maybe 1k x 1k x 169 pixels) was processed for segmentation, the pyramid building here is obviously a much larger image that needs to be generated.
This failed with 12 CPUs, 120 GB of memory. Now rerunning with 240 GB of memory & 1 CPU (to avoid dask trying too many things at once), I'll report back when we have results on that
=> We should profile where in post-processing the memory spike happens. If it's actually in pyramid building, we should optimize pyramid building to stay within the memory envelope more robustly.
@silvbarb has been running Cellpose SAM on some quite large images (e.g. i.e. 4, 169, 42k, 43k pixel size) and running in a bunch of places where we should profile/improve the memory footprint. The actual ROI loading has been working very well, but finding ROIs in label images & building pyramdis on such large images at the end has cause out of memory errors that I'll describe below.
The full .zarray of resolution 0 is:
Masking iterators based on label (not table)
The first failure case was when using a label image to define the ROIs, not a masking ROI table. i.e.
This ran into an out-of-memory error with 32 GB of memory. We should check if this iterator initialization based on label images can be done more memory efficiently.
Post-processing memory error
I suspect this is a memory error during pyramid building (I'll follow up with more info if a rerun with verbose logging shows anything specific).
The logs show the following before the out of memory error:
Thus, the actual segmentation successfully finished, but after the segmentation, the task ran into an out-of-memory error. Given that a relatively small ROI (maybe 1k x 1k x 169 pixels) was processed for segmentation, the pyramid building here is obviously a much larger image that needs to be generated.
This failed with 12 CPUs, 120 GB of memory. Now rerunning with 240 GB of memory & 1 CPU (to avoid dask trying too many things at once), I'll report back when we have results on that
=> We should profile where in post-processing the memory spike happens. If it's actually in pyramid building, we should optimize pyramid building to stay within the memory envelope more robustly.