Is there an existing issue for this?
What would you like to be added?
Add an opt-in, hot-reloadable queryNode.segcore.tieredStorage.lazyColumnGroupEnabled setting, disabled by default, to defer the projected reader, translator, and cache-slot initialization of eligible Storage V3 column-group tasks until first access.
- Apply the same mechanism to internal manifests and externally mapped columns.
- Keep the existing
ProxyChunkColumn interface and initialize its shared ChunkedColumnGroup once per task, including concurrent first access.
- Share column-size estimates across tasks for the same physical column group within a load batch, and release factory captures after successful initialization.
- Preserve existing warmup and interim-index decisions. Primary keys, system fields, and other fields that require excluded load-time preparation stay on the regular loading path. Nullable vectors use the common eligibility checks.
Why is this needed?
Disabling data warmup currently does not defer construction of projected chunk readers, translators, and cache slots. Segment loading can still read Parquet footers and allocate metadata/cache structures for columns that no request subsequently accesses.
Deferring that per-task work can avoid unnecessary initialization and its retained structures for cold columns. This is distinct from delaying data-page loading alone or caching the segment manifest itself; no quantified memory or latency reduction is claimed here.
Anything else?
Is there an existing issue for this?
What would you like to be added?
Add an opt-in, hot-reloadable
queryNode.segcore.tieredStorage.lazyColumnGroupEnabledsetting, disabled by default, to defer the projected reader, translator, and cache-slot initialization of eligible Storage V3 column-group tasks until first access.ProxyChunkColumninterface and initialize its sharedChunkedColumnGrouponce per task, including concurrent first access.Why is this needed?
Disabling data warmup currently does not defer construction of projected chunk readers, translators, and cache slots. Segment loading can still read Parquet footers and allocate metadata/cache structures for columns that no request subsequently accesses.
Deferring that per-task work can avoid unnecessary initialization and its retained structures for cold columns. This is distinct from delaying data-page loading alone or caching the segment manifest itself; no quantified memory or latency reduction is claimed here.
Anything else?