Intent
Bring the useful chunked mesh and results-reader pattern directly into ras-commander, while keeping the core API simple and useful to downstream projects such as ras2cng.
Proposed API direction
Start with internal primitives and narrowly scoped public methods instead of introducing an HdfResultsBlock abstraction.
Core additions
- A private iterator such as
_iter_dataset_time_slices(...) for bounded HDF reads and reductions
- A geometry-free
get_mesh_summary_values(...) returning a pandas DataFrame
- Optional public iterators only where downstream callers need streaming:
iter_mesh_timeseries(...)
iter_xsec_timeseries(...)
Public iterators could yield bounded xarray.DataArray or DataFrame batches with explicit time and spatial coordinates. That reuses established return types and avoids a custom block type until real use cases justify one.
Why geometry-free extraction matters
Heavy analytics and database-loading workflows often need values, cell/face identifiers, timestamps, and mesh names—but not Shapely objects. Building geometry can dominate CPU and memory and is unnecessary for SQL ingestion or aggregate analysis.
ras2cng currently contains a geometry-free results extraction path. The reusable HEC-RAS semantics belong upstream so downstream connectors do not have to reproduce HDF paths, metadata handling, or version compatibility.
Compatibility goal
- Transparently refactor existing eager methods to use the shared reader where safe
- Preserve existing outputs and defaults
- Add streaming/chunking as opt-in API surface
- Keep DuckDB optional and outside this first increment
Proposed stages
- Implement and test the internal HDF slice iterator.
- Add geometry-free mesh summary extraction.
- Refactor selected existing methods onto the shared primitive.
- Add a public time-series iterator only after validating a concrete
ras2cng consumer.
- Benchmark direct aggregation and database ingestion.
Acceptance direction
- Bounded memory for large time-series reads
- Correct handling of multiple meshes, timestamps, identifiers, missing values, and HEC-RAS version differences
- Output-equivalence coverage against current APIs
- A documented
ras2cng example showing no duplicated HDF traversal logic
- No mandatory DuckDB or Arrow dependency
Open questions
- Should batches be time-major to match common HDF chunking?
- Should the iterator yield xarray objects, DataFrames, Arrow record batches, or support a small set?
- Which summaries should use HEC-RAS-native summary datasets versus time-series reduction?
Intent
Bring the useful chunked mesh and results-reader pattern directly into
ras-commander, while keeping the core API simple and useful to downstream projects such asras2cng.Proposed API direction
Start with internal primitives and narrowly scoped public methods instead of introducing an
HdfResultsBlockabstraction.Core additions
_iter_dataset_time_slices(...)for bounded HDF reads and reductionsget_mesh_summary_values(...)returning a pandas DataFrameiter_mesh_timeseries(...)iter_xsec_timeseries(...)Public iterators could yield bounded
xarray.DataArrayor DataFrame batches with explicit time and spatial coordinates. That reuses established return types and avoids a custom block type until real use cases justify one.Why geometry-free extraction matters
Heavy analytics and database-loading workflows often need values, cell/face identifiers, timestamps, and mesh names—but not Shapely objects. Building geometry can dominate CPU and memory and is unnecessary for SQL ingestion or aggregate analysis.
ras2cngcurrently contains a geometry-free results extraction path. The reusable HEC-RAS semantics belong upstream so downstream connectors do not have to reproduce HDF paths, metadata handling, or version compatibility.Compatibility goal
Proposed stages
ras2cngconsumer.Acceptance direction
ras2cngexample showing no duplicated HDF traversal logicOpen questions