🔗 Synced from upstream: NOAA-GSL/zyra#19
This issue is mirrored from the upstream repository. Status changes here will be synced back.
Now that the acquisition managers (S3, HTTP, FTP) support partial byte-range downloads and .idx-based GRIB2 subsetting, we need to extend the Processing module so it can directly work with these subsets.
The goal is to provide utilities that take the raw bytes returned from acquisition and:
- Decode GRIB2 data into usable Python objects.
- Extract specific variables or levels.
- Convert to standardized formats for downstream visualization.
- Validate subsets and extract metadata.
Proposed Features:
Decoding
- grib_decode(data: bytes) -> Dict[str, Any]
- Use cfgrib (preferred) or pygrib to decode GRIB2 messages from a bytes object.
- Handle multi-message GRIB2 files and return a structure keyed by variable name.
Variable Extraction
- extract_variable(decoded_data, var_name: str)
- Filter and return only the requested variable(s).
Format Conversion
- convert_to_format(decoded_data, format_type: str)
- Supported formats: Pandas DataFrame, xarray Dataset, NetCDF, GeoTIFF (where applicable).
Validation
- validate_subset(decoded_data, expected_fields: List[str])
- Ensure subset contains all requested variables, with correct shapes and timestamps.
Metadata Extraction
- extract_metadata(decoded_data)
- Return dict of model run time, forecast hour, variables available, grid info, etc.
Notes & Dependencies:
- Should integrate cleanly with acquisition output (download_byteranges returns bytes).
- Recommend cfgrib + eccodes as default decode path; fallback to pygrib if installed.
- Keep processing utilities self-contained in src/datavizhub/processing/grib_utils.py.
- Include docstrings with minimal examples (matching the “Advanced Acquisition” README section).
Testing:
- Mock GRIB2 subsets for unit tests (small fixture files).
- Tests for decoding, variable extraction, format conversion, metadata, and validation.
Benefit:
This will complete the end-to-end workflow: acquisition → subsetting → processing → visualization, all within DataVizHub.
Now that the acquisition managers (S3, HTTP, FTP) support partial byte-range downloads and .idx-based GRIB2 subsetting, we need to extend the Processing module so it can directly work with these subsets.
The goal is to provide utilities that take the raw bytes returned from acquisition and:
Proposed Features:
Decoding
Variable Extraction
Format Conversion
Validation
Metadata Extraction
Notes & Dependencies:
Testing:
Benefit:
This will complete the end-to-end workflow: acquisition → subsetting → processing → visualization, all within DataVizHub.