Skip to content

feat(experimental): add dense luDF group aggregation - #2953

Open
ibgreen-openai wants to merge 1 commit into
codex/ludf-derived-columnsfrom
codex/ludf-group-aggregation
Open

feat(experimental): add dense luDF group aggregation#2953
ibgreen-openai wants to merge 1 commit into
codex/ludf-derived-columnsfrom
codex/ludf-group-aggregation

Conversation

@ibgreen-openai

Copy link
Copy Markdown
Collaborator

Goals

Add GPU-native dense categorical grouping to the optional luDF dataframe while preserving immutable query planning, explicit null semantics, source batches, graph ownership, and direct visualization-ready output.

Changes

  • Add precisely typed frame.groupBy(key).aggregate(definitions) and composable filtered/derived query variants for unsigned dense category keys.
  • Reuse GPUGroupAggregation, GPUMask, and the existing dataframe command graph for count, sum, min, max, and mean across all original source batches.
  • Infer complete dense domains from explicit categorical dictionaries or accept an explicit positive group count; preserve ordered output aliases, category labels, and source metadata.
  • Exclude null category keys; count all remaining rows while floating-point statistics additionally exclude null, NaN, and infinite values.
  • Publish explicit per-statistic GPU validity masks, distinguish empty groups from zero-valued sums, cache shared input masks, and keep every result resident in owned GPU buffers.
  • Preserve per-batch row masks/indices/counts, reusable encoder-ordered filter parameters, derived-column validity, empty source tables, caller-owned submission, and source destruction leases.
  • Add focused Node/API contracts and real-WebGPU coverage for mixed empty batches, all five metrics, null keys/values, NaN/infinity, repeated encodes, derived inputs, empty dictionary domains, ownership, and exact CPU-equivalent results.

Verification

  • nvm use — Node 22.22.1, matching .nvmrc.
  • yarn install — enterprise registry returns HTTP 403 for an existing upstream alpha dev-tool dependency; reused the previously verified ignored dependency tree.
  • yarn lint fix — passed across the complete repository after final source changes.
  • yarn build — passed across every workspace package after final formatting.
  • Focused Node: all existing and new luDF suites — 46 tests passed.
  • Focused real WebGPU: dense grouping, ordered parameters, derived values, nonfinite filtering, and schema-only sources — five tests passed.
  • CI=1 yarn test — Node: 589 passed and one skipped; real Chromium/WebGPU: 1,570 passed and 25 skipped.
  • yarn website:build — passed, including validation of 461 documentation pages.
  • (cd website && yarn build) — passed, including validation of 461 documentation pages.
  • yarn examples:typecheck — passed across all 46 example workspaces.
  • yarn bundle-size — passed all seven existing bundle budgets.

Scope and risks

  • Initial groups require dense uint32 keys and float32 numeric statistics; signed dictionary indices, sparse keys, and CPU-side domain inference are intentionally unsupported.
  • Counts are unsigned 32-bit and reject source row counts beyond that representable range; floating-point aggregation follows the existing WebGPU primitive's float32 precision and explicitly rejects nonfinite contributions.
  • Group result rows are globally merged into one dense output batch; original source selection masks and stable source-row identities remain batch-aligned.
  • Global reductions, numeric histograms, stable sorting, joins, and integrated examples remain separate follow-on changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants