Add opt-in source provenance to OME-Zarr conversion#62
Closed
BrianWhitneyAI wants to merge 5 commits into
Closed
Conversation
Add bioio_conversion.channel_colors with get_channel_colors(), which derives an OME-Zarr display color per channel, and wire it into OmeZarrConverter so channels are colored automatically instead of all defaulting to white. Color policy (per channel): - Brightfield / transmitted-light channels -> white, and do not consume a fluorescent palette slot. Detected via OME contrast_method / illumination_type with a channel-name fallback. - Recognized fluorophores -> their true emission color via the FLUOROPHORE_COLORS table (e.g. GFP -> green, AF647 -> red). - Remaining channels -> a count-based, perceptually-distinct palette (green / green+magenta / cyan-yellow-magenta / +Okabe-Ito), ordered by emission wavelength when available. Black is never assigned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…colors # Conflicts: # bioio_conversion/__init__.py # bioio_conversion/converters/ome_zarr_converter.py
Lift the ProcessPoolExecutor out of the per-scene write path so one pool spans the whole conversion. Per-scene planning moves into _plan_and_dispatch_scene, which initializes each store and streams its shards into the shared pool; _write_auto_layout_shards is replaced by the pure _scene_shard_bounds enumerator. Previously a pool was built and torn down per scene, so a scene with fewer shards than workers left cores idle until it finished (e.g. 47 scenes x 2 shards on 4 cores never used more than 2). Now idle cores immediately pick up the next scene's shards. Output paths are resolved and existence-checked up front so a late FileExistsError cannot fire mid-stream. Verified on a 3-scene file: 4 workers reach peak concurrency 4 with shards from different scenes running simultaneously (was capped at the per-scene shard count before), and average concurrency tracks the theoretical max. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an include_provenance flag (default off) that writes a top-level "bioio" attribute block beside the NGFF "ome" block in each scene's store, plus the source metadata XML as sidecars under bioio/. The block records the source file name, UTC conversion timestamp, reader plugin, bioio package versions (including bioio-conversion), the cross-format StandardMetadata for that scene, and pointers to the native/OME XML sidecars. Provenance assembly lives in a new bioio_conversion/provenance.py (ProvenanceBuilder), kept out of the converter proper. It is built once per conversion and cached: the metadata reader and whole-file XML are scene-independent, so only StandardMetadata is re-read per scene. For formats that surface richer metadata under non-default reader options, provenance opens a dedicated metadata reader (pixel I/O is unaffected): CZI uses an aicspylibczi reader with subblock metadata (embedding <Subblocks> into the native XML and deriving extra StandardMetadata such as total_time_duration), and ND2 uses the standard 96-well plate geometry so each scene's stage position is assigned a well row/column. The dedicated reader is best-effort and falls back to the pixel reader if it can't open or its scene names diverge. Provenance is keyed by scene name, not index, so a different metadata backend can't misattach a scene. Add a "provenance" install extra (bioio-czi, bioio-nd2) and pin bioio-nd2 in the test extras. Tests in tests/converters/test_provenance.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
include_provenanceflag (default off) toOmeZarrConverterthat captures where a converted store came from. When enabled, each scene's store gets:"bioio"attribute block (sibling of the NGFF"ome"block) with: source file name, UTC conversion timestamp, reader plugin, bioio package versions (incl.bioio-conversion), the scene's cross-formatStandardMetadata, and pointers to the metadata sidecars.bioio/: the source's native metadata and normalized OME-XML (deduped to one file when byte-identical, e.g. OME-TIFF).Design
bioio_conversion/provenance.py(ProvenanceBuilder), kept out of the converter. Built once per conversion and cached — onlyStandardMetadatais re-read per scene; the metadata reader and whole-file XML are scene-independent.aicspylibczireader with subblock metadata (embeds<Subblocks>into native XML, derives extra fields liketotal_time_duration); ND2 uses the standard 96-well plate so each scene's stage position gets a well row/column. Best-effort — falls back to the pixel reader if the dedicated reader can't open or its scene names diverge.Packaging
provenanceinstall extra (bioio-czi,bioio-nd2).bioio-nd2>=1.6.0pinned in thetestextras.Tests
tests/converters/test_provenance.py— block assembly + per-scene metadata across CZI/OME-TIFF at multiple scene indices, sidecar dedup/distinct, CZI subblock embedding, and unit tests for the reader-kwargs dispatch and_json_safecoercion.Note on base
main— the provenance code builds on_plan_and_dispatch_sceneintroduced there. Only the final commit (Add opt-in source provenance…) is new to this PR; review that commit for the provenance change.🤖 Generated with Claude Code