You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Surfaced during review of icebird PR #28 (the scanColumn streaming hook), part of the hypaware bounded-query-execution effort.
Observation
tryColumnScanAggregate / scanColumnAggregate calls the source scanColumn(column) hook once per aggregate spec. So a query with multiple aggregates over the same column — e.g. SELECT COUNT(c), MIN(c), MAX(c), SUM(c), AVG(c) FROM t — opens N independent scanColumn streams, re-reading and re-decoding that column N times (N× metadata/IO, costly for remote/parquet sources). Memory stays O(1) per stream, but the IO is multiplied.
This is a memory-for-IO tradeoff that only appears once a source implements scanColumn (icebird now does). It is not a correctness bug and does not affect the memory-bounding goal.
Suggested fix
In tryColumnScanAggregate, group the column-scan aggregate specs by { column, limit, offset } and drive each group from a singlescanColumn stream, feeding all aggregates for that column from the one pass.
Surfaced during review of icebird PR #28 (the
scanColumnstreaming hook), part of the hypaware bounded-query-execution effort.Observation
tryColumnScanAggregate/scanColumnAggregatecalls the sourcescanColumn(column)hook once per aggregate spec. So a query with multiple aggregates over the same column — e.g.SELECT COUNT(c), MIN(c), MAX(c), SUM(c), AVG(c) FROM t— opens N independentscanColumnstreams, re-reading and re-decoding that column N times (N× metadata/IO, costly for remote/parquet sources). Memory stays O(1) per stream, but the IO is multiplied.This is a memory-for-IO tradeoff that only appears once a source implements
scanColumn(icebird now does). It is not a correctness bug and does not affect the memory-bounding goal.Suggested fix
In
tryColumnScanAggregate, group the column-scan aggregate specs by{ column, limit, offset }and drive each group from a singlescanColumnstream, feeding all aggregates for that column from the one pass.Refs