Skip to content

Add GRIB2 Decoding and Transformation Utilities to Processing Module #217

Description

@github-actions

🔗 Synced from upstream: NOAA-GSL/zyra#19

This issue is mirrored from the upstream repository. Status changes here will be synced back.


Now that the acquisition managers (S3, HTTP, FTP) support partial byte-range downloads and .idx-based GRIB2 subsetting, we need to extend the Processing module so it can directly work with these subsets.

The goal is to provide utilities that take the raw bytes returned from acquisition and:

  • Decode GRIB2 data into usable Python objects.
  • Extract specific variables or levels.
  • Convert to standardized formats for downstream visualization.
  • Validate subsets and extract metadata.

Proposed Features:

Decoding

  • grib_decode(data: bytes) -> Dict[str, Any]
  • Use cfgrib (preferred) or pygrib to decode GRIB2 messages from a bytes object.
  • Handle multi-message GRIB2 files and return a structure keyed by variable name.

Variable Extraction

  • extract_variable(decoded_data, var_name: str)
  • Filter and return only the requested variable(s).

Format Conversion

  • convert_to_format(decoded_data, format_type: str)
  • Supported formats: Pandas DataFrame, xarray Dataset, NetCDF, GeoTIFF (where applicable).

Validation

  • validate_subset(decoded_data, expected_fields: List[str])
  • Ensure subset contains all requested variables, with correct shapes and timestamps.

Metadata Extraction

  • extract_metadata(decoded_data)
  • Return dict of model run time, forecast hour, variables available, grid info, etc.

Notes & Dependencies:

  • Should integrate cleanly with acquisition output (download_byteranges returns bytes).
  • Recommend cfgrib + eccodes as default decode path; fallback to pygrib if installed.
  • Keep processing utilities self-contained in src/datavizhub/processing/grib_utils.py.
  • Include docstrings with minimal examples (matching the “Advanced Acquisition” README section).

Testing:

  • Mock GRIB2 subsets for unit tests (small fixture files).
  • Tests for decoding, variable extraction, format conversion, metadata, and validation.

Benefit:
This will complete the end-to-end workflow: acquisition → subsetting → processing → visualization, all within DataVizHub.

Metadata

Metadata

Assignees

No one assigned

    Labels

    upstream-syncIssue mirrored from NOAA-GSL/zyra

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions