Skip to content

[Exploratory] Add an optional lazy/queryable result return type #308

Description

@gpt-cmdr

Intent

Explore adding a new return type for very heavy result APIs without disrupting existing pandas/xarray/GeoPandas callers.

Proposal

Add an opt-in result view, tentatively HdfResultView or RasResultView:

result = HdfResultsMesh.get_mesh_cells_timeseries(
    plan_hdf,
    mesh_name="2D Area",
    return_type="view",
)

The default would remain the existing eager return type. The view would describe the source path, dataset, selections, dimensions, coordinates, and metadata without immediately materializing the values.

Candidate operations:

  • .to_xarray()
  • .to_numpy()
  • .to_pandas()
  • .to_arrow() when the optional Arrow dependency is installed
  • .iter_batches()
  • .reduce("max")
  • .to_duckdb() or .sql(...) when the optional DuckDB integration is installed

Why a view instead of HdfResultsBlock

A block is useful as one batch in an iterator, but it does not by itself solve query planning, deferred selection, format conversion, or connection lifetime. A source-backed view is a clearer user-facing abstraction; its batches can remain familiar xarray/DataFrame/Arrow objects.

Design constraints

  • Existing return types and defaults remain unchanged
  • DuckDB and PyArrow remain optional dependencies
  • The view must have explicit file/handle lifetime behavior
  • Selection should be pushed down to HDF slicing before materialization
  • A raw DuckDBPyRelation should not be the primary public return type because its connection ownership can be surprising
  • Typing and documentation must make the eager/lazy distinction obvious

Proposed first increment

  1. Prototype the view around one mesh time-series API.
  2. Support selection, batching, reduction, and .to_xarray().
  3. Compare ergonomics and performance with a plain public iterator.
  4. Add Arrow/DuckDB adapters only if the core abstraction proves worthwhile.
  5. Decide whether to generalize across mesh, face, cross-section, and hydrograph results.

Acceptance direction

  • No material read occurs when the view is created
  • Selections are applied before loading data
  • Batches have bounded memory and stable labeled dimensions
  • Eager output matches the existing API
  • Optional integrations fail with clear install guidance
  • Tests cover closed files, moved files, concurrent readers, and serialization expectations

Open questions

  • Is return_type="view" clearer than a separate query_... method?
  • Should views reopen the HDF per operation or own a context-managed handle?
  • Is the iterator-only API sufficient, making a view unnecessary?
  • Which metadata belongs in the view versus a separate schema object?

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestquestionFurther information is requested

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions