Ways to share the hdf5 files using amazon web services

[tomsing1]

Some random thoughts on how to potentially make the hdf5 files / data available within Denali.

Note: Every time data is retrieved from AWS, there is a small transfer fee (per Gb). Probably not an issue right now, but good to know.

• Keep a copy of the hdf5 files on AWS S3 and provide functions to download them the first time they are needed (and then keep them cached on the user's computer). This is similar to downloading them from the authors' website, but probably faster because of our fast uplink to AWS. The SRAdb package is an example of providing a function to download the required data.
• AWS Elastic File System (EFS) that would make the data available to EC2 instances. (There seem to be workarounds to mount EFS on non EC2 computers, but that might be too much of a hassle.)
• HDF Server implements a REST service. This slide deck is interesting, too, and points to a github repo from the hdfgroup. Not sure if h5serv and hdfserver refer to the same thing....

[trahloff]

Hi Thomas! Have you found an approach to conveniently store HDF5 files on AWS and work with them?

[tomsing1]

@trahloff : I toyed with using the HDF5Array Delayed Arrays from Bioconductor and got to the proof-of-concept stage with help from folks on the Bioconductor support forum: https://support.bioconductor.org/p/9135005/ and https://support.bioconductor.org/p/9134972/ . But I never got to use it for anything real. Right now, I am still copying the HDF5 files from S3 to a local volume.