Feature: Parallelize shard writes across scenes with a single shared pool#68
Open
BrianWhitneyAI wants to merge 4 commits into
Open
Feature: Parallelize shard writes across scenes with a single shared pool#68BrianWhitneyAI wants to merge 4 commits into
BrianWhitneyAI wants to merge 4 commits into
Conversation
Lift the ProcessPoolExecutor out of the per-scene write path so one pool spans the whole conversion. Per-scene planning moves into _plan_and_dispatch_scene, which initializes each store and streams its shards into the shared pool; _write_auto_layout_shards is replaced by the pure _scene_shard_bounds enumerator. Previously a pool was built and torn down per scene, so a scene with fewer shards than workers left cores idle until it finished (e.g. 47 scenes x 2 shards on 4 cores never used more than 2). Now idle cores immediately pick up the next scene's shards. Output paths are resolved and existence-checked up front so a late FileExistsError cannot fire mid-stream. Verified on a 3-scene file: 4 workers reach peak concurrency 4 with shards from different scenes running simultaneously (was capped at the per-scene shard count before), and average concurrency tracks the theoretical max. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BrianWhitneyAI
changed the title
Drop the pool/scheduling narrative (which describes the caller, not this enumerator) and keep only what a caller needs: one region per shard, disjoint and shard-aligned, writable independently and in any order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BrianWhitneyAI
marked this pull request as ready for review
July 22, 2026 22:36
BrianWhitneyAI
requested review from
SeanDuHare,
aswallace,
kmitcham,
pgarrison,
saeliddp and
toloudis
July 22, 2026 22:36
SeanDuHare
reviewed
Jul 23, 2026
| def _output_base_for_scene(self, scene_index: int) -> str: | ||
| """Sanitized output basename (no extension) for a scene's store. | ||
|
|
||
| Single-scene conversions use the base name as-is; multi-scene runs |
Contributor
There was a problem hiding this comment.
Who defines the illegal characters for these? Or do you just mean characters difficult to store in a path?
Contributor
Author
There was a problem hiding this comment.
Its just path sanitization moved from lower in the writer and added to a function
| and ome_dims | ||
| ) | ||
|
|
||
| if self._writer_chunk_shape is not None: |
Contributor
There was a problem hiding this comment.
What do you think about breaking this if/else block into a method? Feels like its lengthy
Contributor
Author
There was a problem hiding this comment.
yeah we can do that
Contributor
|
Need to think about the changes more - but LGTM so far |
Conflicts were in ome_zarr_converter.py between the parallel pool refactor and main's fsspec remote-URI fixes (PR #70). - Kept HEAD's single cross-scene ProcessPoolExecutor and the _plan_and_dispatch_scene / _scene_shard_bounds factoring - Restored main's fsspec-based path building and existence check in convert() so remote URIs (S3, GCS, etc.) are still supported - _plan_and_dispatch_scene now receives an out_path: str consistent with both local and remote path representations Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #69
Lifts the
ProcessPoolExecutorout of the per-scene write path so one pool spans the whole conversion.Previously a pool was built and torn down per scene, so a scene with fewer shards than workers left cores idle until it finished (e.g. 47 scenes × 2 shards on 4 cores never used more than 2). Now idle cores immediately pick up the next scene's shards. Output paths are resolved and existence-checked up front so a late
FileExistsErrorcannot fire mid-stream.Verification
Verified on a 3-scene file: 4 workers reach peak concurrency 4 with shards from different scenes running simultaneously (was capped at the per-scene shard count before), and average concurrency tracks the theoretical max.
TODO
🤖 Generated with Claude Code