To enable fast reads of ice surface elevation time-series data for analytical and/or visualization purposes!
Current state
ICESat-2 ATL11 data is stored in an HDF5 format (see https://nsidc.org/data/atl11/versions/5), in a cumbersome nested hierarchical format (one 'dataset' per laser pair track). While the ICESat-2 HDF5 files are now on AWS S3 object storage (https://nsidc.org/data/user-resources/data-announcements/data-set-updates-new-earthdata-cloud-access-option-icesat-2-and-icesat-data-sets) as of 29 Sep 2022, which streamlines data read speeds (as long as compute is next to the data on AWS us-west2), the nested HDF5 structure of ATL11 can still be a pain to handle.
Desired state
Remove the nested pair-track data structure (pt1, pt2, pt3), i.e. flatten it into a single non-nested structure. There has been some work on this before to convert the ICESat-2 HDF5 format to:
There's a good blog post about cloud-native vector formats at https://cholmes.medium.com/an-overview-of-cloud-native-vector-c223845638e0, not point-cloud specific, but it discusses about the analytics/visualization trade-offs. Also check out new developments on cloud-optimized ICESat-2 data, for example, see what @tsnow03 is doing at https://github.com/CryoInTheCloud/IS2CloudOptimizedData
Action points
References
To enable fast reads of ice surface elevation time-series data for analytical and/or visualization purposes!
Current state
ICESat-2 ATL11 data is stored in an HDF5 format (see https://nsidc.org/data/atl11/versions/5), in a cumbersome nested hierarchical format (one 'dataset' per laser pair track). While the ICESat-2 HDF5 files are now on AWS S3 object storage (https://nsidc.org/data/user-resources/data-announcements/data-set-updates-new-earthdata-cloud-access-option-icesat-2-and-icesat-data-sets) as of 29 Sep 2022, which streamlines data read speeds (as long as compute is next to the data on AWS us-west2), the nested HDF5 structure of ATL11 can still be a pain to handle.
Desired state
Remove the nested pair-track data structure (
pt1,pt2,pt3), i.e. flatten it into a single non-nested structure. There has been some work on this before to convert the ICESat-2 HDF5 format to:xarray, optimized for analytics, less so for visualization@abarciauskas-bgse, though it's for ATL08 dataThere's a good blog post about cloud-native vector formats at https://cholmes.medium.com/an-overview-of-cloud-native-vector-c223845638e0, not point-cloud specific, but it discusses about the analytics/visualization trade-offs. Also check out new developments on cloud-optimized ICESat-2 data, for example, see what @tsnow03 is doing at https://github.com/CryoInTheCloud/IS2CloudOptimizedData
Action points
icepyxto read ATL11 properly, both for local and cloud-hosted files (update read-in module for ATL11 icesat2py/icepyx#398, and for cloud?)References