Parallel file read

[jcumby]

Problem

When reading highly compressed image files a fairly large amount of time is spent decompressing the images, rather than performing the reconstruction.

Potential solution

Using multiprocessing, it's possible to split the work of reading files and reconstructing asynchronously across processors. Ideally this would parallelise both the reading and reconstruction steps, but given that h5py cannot perform parallel writes I opted to just parallelise the read operation.

This fork uses a Pool of workers (default 4) to read image files and yield the results as needed to a single worker which does the reconstruction. To avoid i/o out-running the reconstruction (and filling memory) the Pool is restricted to storing a maximum of 8 images. On the few test-cases I've run, it seems to give a fairly good speedup.

To use the parallel processing, meerkat must be called as an importable script - I've used argparse to collect (some of) the parameters for reconstruct_data from the command line, and the whole lot can be called with python -m meerkat.meerkat --help

Possible Downsides

• In order to implement parallel reading, I've had to modify reconstuct_data to take an iterable of (image_number, data) pairs instead of the previous filename_template, and also a semaphore object (default None) to work with multiprocessing. This change would change the API, but could also allow the use of fabio.frame_series which has some (potential) speed advantages.
• Parallel processing can only be performed as a script. This is a limitation of multiprocessing, and difficult to overcome...