Skip to content

Bugfix/allow remote dest#70

Merged
BrianWhitneyAI merged 6 commits into
mainfrom
bugfix/allow-remote-dest
Jul 22, 2026
Merged

Bugfix/allow remote dest#70
BrianWhitneyAI merged 6 commits into
mainfrom
bugfix/allow-remote-dest

Conversation

@saeliddp

@saeliddp saeliddp commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Link to Relevant Issue

Follows bioio-devs/bioio-ome-zarr#157 to allow the conversion package to write directly to S3 stores. See related issue: https://github.com/AllenCell/bioio-conversion-pipeline/issues/16

Description of Changes

Replace path building logic that assumed local destinations with logic that allows for remote destinations as well.

Manual Tests

  • create environment using this branch + the related bioio-ome-zarr branch
  • run zarr conversion on slurm, using bff-pipeline output bucket as destination
  • validate that using extant S3 path as destination causes failure

@saeliddp
saeliddp marked this pull request as ready for review July 21, 2026 20:49
@saeliddp
saeliddp requested a review from a team as a code owner July 21, 2026 20:49
@BrianWhitneyAI
BrianWhitneyAI merged commit 332646f into main Jul 22, 2026
13 checks passed
@BrianWhitneyAI
BrianWhitneyAI deleted the bugfix/allow-remote-dest branch July 22, 2026 22:19
BrianWhitneyAI added a commit that referenced this pull request Jul 24, 2026
Conflicts were in ome_zarr_converter.py between the parallel pool
refactor and main's fsspec remote-URI fixes (PR #70).

- Kept HEAD's single cross-scene ProcessPoolExecutor and the
  _plan_and_dispatch_scene / _scene_shard_bounds factoring
- Restored main's fsspec-based path building and existence check in
  convert() so remote URIs (S3, GCS, etc.) are still supported
- _plan_and_dispatch_scene now receives an out_path: str consistent
  with both local and remote path representations

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants