Skip to content

[Exploratory] Add chunked mesh/results readers and geometry-free result extraction #307

Description

@gpt-cmdr

Intent

Bring the useful chunked mesh and results-reader pattern directly into ras-commander, while keeping the core API simple and useful to downstream projects such as ras2cng.

Proposed API direction

Start with internal primitives and narrowly scoped public methods instead of introducing an HdfResultsBlock abstraction.

Core additions

  • A private iterator such as _iter_dataset_time_slices(...) for bounded HDF reads and reductions
  • A geometry-free get_mesh_summary_values(...) returning a pandas DataFrame
  • Optional public iterators only where downstream callers need streaming:
    • iter_mesh_timeseries(...)
    • iter_xsec_timeseries(...)

Public iterators could yield bounded xarray.DataArray or DataFrame batches with explicit time and spatial coordinates. That reuses established return types and avoids a custom block type until real use cases justify one.

Why geometry-free extraction matters

Heavy analytics and database-loading workflows often need values, cell/face identifiers, timestamps, and mesh names—but not Shapely objects. Building geometry can dominate CPU and memory and is unnecessary for SQL ingestion or aggregate analysis.

ras2cng currently contains a geometry-free results extraction path. The reusable HEC-RAS semantics belong upstream so downstream connectors do not have to reproduce HDF paths, metadata handling, or version compatibility.

Compatibility goal

  • Transparently refactor existing eager methods to use the shared reader where safe
  • Preserve existing outputs and defaults
  • Add streaming/chunking as opt-in API surface
  • Keep DuckDB optional and outside this first increment

Proposed stages

  1. Implement and test the internal HDF slice iterator.
  2. Add geometry-free mesh summary extraction.
  3. Refactor selected existing methods onto the shared primitive.
  4. Add a public time-series iterator only after validating a concrete ras2cng consumer.
  5. Benchmark direct aggregation and database ingestion.

Acceptance direction

  • Bounded memory for large time-series reads
  • Correct handling of multiple meshes, timestamps, identifiers, missing values, and HEC-RAS version differences
  • Output-equivalence coverage against current APIs
  • A documented ras2cng example showing no duplicated HDF traversal logic
  • No mandatory DuckDB or Arrow dependency

Open questions

  • Should batches be time-major to match common HDF chunking?
  • Should the iterator yield xarray objects, DataFrames, Arrow record batches, or support a small set?
  • Which summaries should use HEC-RAS-native summary datasets versus time-series reduction?

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestquestionFurther information is requested

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions