Intent
Explore adding a new return type for very heavy result APIs without disrupting existing pandas/xarray/GeoPandas callers.
Proposal
Add an opt-in result view, tentatively HdfResultView or RasResultView:
result = HdfResultsMesh.get_mesh_cells_timeseries(
plan_hdf,
mesh_name="2D Area",
return_type="view",
)
The default would remain the existing eager return type. The view would describe the source path, dataset, selections, dimensions, coordinates, and metadata without immediately materializing the values.
Candidate operations:
.to_xarray()
.to_numpy()
.to_pandas()
.to_arrow() when the optional Arrow dependency is installed
.iter_batches()
.reduce("max")
.to_duckdb() or .sql(...) when the optional DuckDB integration is installed
Why a view instead of HdfResultsBlock
A block is useful as one batch in an iterator, but it does not by itself solve query planning, deferred selection, format conversion, or connection lifetime. A source-backed view is a clearer user-facing abstraction; its batches can remain familiar xarray/DataFrame/Arrow objects.
Design constraints
- Existing return types and defaults remain unchanged
- DuckDB and PyArrow remain optional dependencies
- The view must have explicit file/handle lifetime behavior
- Selection should be pushed down to HDF slicing before materialization
- A raw
DuckDBPyRelation should not be the primary public return type because its connection ownership can be surprising
- Typing and documentation must make the eager/lazy distinction obvious
Proposed first increment
- Prototype the view around one mesh time-series API.
- Support selection, batching, reduction, and
.to_xarray().
- Compare ergonomics and performance with a plain public iterator.
- Add Arrow/DuckDB adapters only if the core abstraction proves worthwhile.
- Decide whether to generalize across mesh, face, cross-section, and hydrograph results.
Acceptance direction
- No material read occurs when the view is created
- Selections are applied before loading data
- Batches have bounded memory and stable labeled dimensions
- Eager output matches the existing API
- Optional integrations fail with clear install guidance
- Tests cover closed files, moved files, concurrent readers, and serialization expectations
Open questions
- Is
return_type="view" clearer than a separate query_... method?
- Should views reopen the HDF per operation or own a context-managed handle?
- Is the iterator-only API sufficient, making a view unnecessary?
- Which metadata belongs in the view versus a separate schema object?
Intent
Explore adding a new return type for very heavy result APIs without disrupting existing pandas/xarray/GeoPandas callers.
Proposal
Add an opt-in result view, tentatively
HdfResultVieworRasResultView:The default would remain the existing eager return type. The view would describe the source path, dataset, selections, dimensions, coordinates, and metadata without immediately materializing the values.
Candidate operations:
.to_xarray().to_numpy().to_pandas().to_arrow()when the optional Arrow dependency is installed.iter_batches().reduce("max").to_duckdb()or.sql(...)when the optional DuckDB integration is installedWhy a view instead of
HdfResultsBlockA block is useful as one batch in an iterator, but it does not by itself solve query planning, deferred selection, format conversion, or connection lifetime. A source-backed view is a clearer user-facing abstraction; its batches can remain familiar xarray/DataFrame/Arrow objects.
Design constraints
DuckDBPyRelationshould not be the primary public return type because its connection ownership can be surprisingProposed first increment
.to_xarray().Acceptance direction
Open questions
return_type="view"clearer than a separatequery_...method?