Skip to content

scanColumn aggregates: coalesce N aggregates on one column into a single stream #43

Description

@philcunliffe

Surfaced during review of icebird PR #28 (the scanColumn streaming hook), part of the hypaware bounded-query-execution effort.

Observation

tryColumnScanAggregate / scanColumnAggregate calls the source scanColumn(column) hook once per aggregate spec. So a query with multiple aggregates over the same column — e.g. SELECT COUNT(c), MIN(c), MAX(c), SUM(c), AVG(c) FROM t — opens N independent scanColumn streams, re-reading and re-decoding that column N times (N× metadata/IO, costly for remote/parquet sources). Memory stays O(1) per stream, but the IO is multiplied.

This is a memory-for-IO tradeoff that only appears once a source implements scanColumn (icebird now does). It is not a correctness bug and does not affect the memory-bounding goal.

Suggested fix

In tryColumnScanAggregate, group the column-scan aggregate specs by { column, limit, offset } and drive each group from a single scanColumn stream, feeding all aggregates for that column from the one pass.

Refs

  • icebird PR Execute cleanup #28 (the hook this concerns)
  • hypaware bounded-query-execution: design LLP 0058, decision LLP 0055 (streaming aggregates via scanColumn)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions