Add native async columnar batch execution - #54
Merged
Conversation
This was referenced Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prepareScan()source contract with phased column demands, authoritative residual work, lazy batch production, and scan cardinality metadatascan()/scanColumn()data sources and row consumersPerformance
Headline: production-shaped prepared-source queries are up to 2.38x faster, with 12–20% lower sampled peak heap, while the full compatibility suite is effectively flat overall.
Compared
origin/master(c8fa31e) with this branch (7dbc135) on Node 26.5.1. Every benchmark performed an exact result comparison before timing.Prepared-source Hypaware workload
This comparison used the same two local Icebird partitions (339,209 rows, 57 fields) for both engines. The source exposes both APIs:
masterconsumes its legacy row scan, while this branch consumes a benchmarkprepareScan()bridge in 4,000-row batches. Queries mirror recent Hypaware overview and analysis shapes found in recorded usage, especially grouped JSON token rollups and distinct-session counts.One correctness pass was followed by three alternating timed samples with GC before each run; values below are medians.
The bridge still constructs Icebird rows before packing them into vectors, so it isolates executor overhead but understates the benefit expected from a connector that emits prepared batches directly.
Full
squirreling-perfsuiteThe current 21-case Wikipedia + deterministic synthetic Hypaware suite used one correctness pass, one warm-up, and five alternating timed samples per case.
0.997x, -0.3%)0.999x, -0.1%)0.95xBoth suite sources currently implement only
scan()/scanColumn(), notprepareScan(), so this is primarily a compatibility/regression comparison. Currentmasteralso already includes the internal batch execution introduced in #52, which is why the earlier v0.15.3-era gains do not appear again in this baseline.Validation
npm test— 65 files, 1,979 tests passednpm run lintnpx tscgit diff --check origin/master...HEAD