Skip to content

Add native async columnar batch execution - #54

Merged
platypii merged 7 commits into
masterfrom
columnar-design
Aug 18, 2026
Merged

Add native async columnar batch execution#54
platypii merged 7 commits into
masterfrom
columnar-design

Conversation

@platypii

Copy link
Copy Markdown
Contributor

Summary

  • expose async batch, vector, row-selection, deferred-column, and row/batch adapter primitives as public API
  • add a schema-based prepareScan() source contract with phased column demands, authoritative residual work, lazy batch production, and scan cardinality metadata
  • preserve native batches through supported scan, filter, projection, distinct, limit/offset, collection, and streaming aggregate paths, with explicit row fallbacks for unsupported work
  • harden column ownership across CTEs, nested/correlated subqueries, qualified struct access, and duplicate projection aliases
  • preserve abort handling, laziness, restartability, and compatibility with existing scan() / scanColumn() data sources and row consumers

Performance

Headline: production-shaped prepared-source queries are up to 2.38x faster, with 12–20% lower sampled peak heap, while the full compatibility suite is effectively flat overall.

Compared origin/master (c8fa31e) with this branch (7dbc135) on Node 26.5.1. Every benchmark performed an exact result comparison before timing.

Prepared-source Hypaware workload

This comparison used the same two local Icebird partitions (339,209 rows, 57 fields) for both engines. The source exposes both APIs: master consumes its legacy row scan, while this branch consumes a benchmark prepareScan() bridge in 4,000-row batches. Queries mirror recent Hypaware overview and analysis shapes found in recorded usage, especially grouped JSON token rollups and distinct-session counts.

One correctness pass was followed by three alternating timed samples with GC before each run; values below are medians.

Query master branch Speedup Sampled peak heap
Model/provider JSON token rollup 2176.7 ms 914.6 ms 2.38x 296.2 → 259.3 MiB (-12%)
Daily distinct-session + token rollup 1383.2 ms 853.3 ms 1.62x 290.3 → 253.4 MiB (-13%)
Tool calls + distinct sessions 256.8 ms 221.9 ms 1.16x 32.6 → 27.2 MiB (-17%)
Date/count overview probe 827.1 ms 769.0 ms 1.08x 35.0 → 28.1 MiB (-20%)

The bridge still constructs Icebird rows before packing them into vectors, so it isolates executor overhead but understates the benefit expected from a connector that emits prepared batches directly.

Full squirreling-perf suite

The current 21-case Wikipedia + deterministic synthetic Hypaware suite used one correctness pass, one warm-up, and five alternating timed samples per case.

  • all cases: 7724.0 ms on master vs 7748.7 ms on the branch (0.997x, -0.3%)
  • nine production-derived Hypaware cases: 4025.8 ms vs 4031.0 ms (0.999x, -0.1%)
  • multi-scalar aggregate: 19.66 ms vs 17.96 ms (1.10x)
  • 18 of 21 cases were within ±2%; the slowest outlier was the row-boundary self-join at 0.95x
  • exact result parity held for all 21 cases and every timed repetition

Both suite sources currently implement only scan() / scanColumn(), not prepareScan(), so this is primarily a compatibility/regression comparison. Current master also already includes the internal batch execution introduced in #52, which is why the earlier v0.15.3-era gains do not appear again in this baseline.

Validation

  • npm test — 65 files, 1,979 tests passed
  • npm run lint
  • npx tsc
  • git diff --check origin/master...HEAD

@platypii
platypii merged commit d02c4c8 into master Aug 18, 2026
6 checks passed
@platypii
platypii deleted the columnar-design branch August 18, 2026 06:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant