Exploration: Relax the "always close file handles" policy to improve reader performance

[BrianWhitneyAI]

Background

bioio currently enforces a strict policy that readers must never hold an open file handle between operations. This policy originated to ensure that Reader objects are picklable for use with dask distributed computing (so dask workers can serialize and ship reader tasks across a cluster).

This came to our attention via bioio-devs/bioio-bioformats#51, where a user reported 200s load times for a 384-scene .stk file. The root cause: bioio's policy forces the bioformats reader to fully close and reinitialize the underlying Java reader object on every scene transition. At ~1 second per reinitialization × 384 scenes, the overhead is enormous. The same file loads in 1–2 seconds in Fiji.

As noted by contributors in that issue:

"bioio enforces that readers never hold an open file handle while they aren't actively reading image data, so the bioformats reader is forced to close the underlying java reader object and reopen it every time the scene is changed."

"Bioformats (java) is just fundamentally not particularly ready to support the model that bioio wants to enforce, where every operation leaves zero file handles open."

Expected Impact

This change should yield significant performance improvements for any reader that currently pays a per-operation reopen cost, not just bioio-bioformats. Any reader backed by a resource that is expensive to open (JVM-based readers, network file handles, etc.) would benefit.

References

• Slow image reading bioio-bioformats#51 — performance report and discussion thread that surfaced this issue

[toloudis]

Might require re-implementation of many readers. Might have implications or limitations on how parallelizable reads are. Might also have benefits of simpler code in addition to performance in some scenarios...

[jmuhlich]

Since the real requirement is that readers can survive a pickle/unpickle round trip, that's really what the test suite should be validating instead. All existing readers that don't maintain persistent open filehandles will already pass that test, so no readers would even need to be reimplemented. Pickle support for future readers that decide to keep filehandles open can be achieved with the "pickle protocol" __getstate__/__setstate__ methods. Unpickling will still incur the overhead of reinitialization but at least it will only happen when it needs to.