Skip to content

feat(experimental): add luDF reductions and histograms - #2955

Open
ibgreen-openai wants to merge 2 commits into
codex/ludf-group-aggregationfrom
codex/ludf-reductions-histograms
Open

feat(experimental): add luDF reductions and histograms#2955
ibgreen-openai wants to merge 2 commits into
codex/ludf-group-aggregationfrom
codex/ludf-reductions-histograms

Conversation

@ibgreen-openai

Copy link
Copy Markdown
Collaborator

Goals

Add GPU-resident global dataframe reductions and filtered numeric histograms to luDF without implicit batch packing, command submission, or CPU row readback.

Changes

  • Add typed frame.aggregate(definitions) / query.aggregate(definitions) global count, sum, min, max, and mean operations.
  • Preserve float32, sint32, and uint32 source types for sum/min/max; expose float32 means and uint32 counts with one-row GPU-owned result tables.
  • Add frame.histogram(column, {bins, domain}) and {edges} variants; keep domains explicit so filters and nullable rows cannot corrupt inferred bounds.
  • Compose filters, chained derived columns, encoder-ordered mutable parameters, preserved source batch/chunk masks, and canonical source ownership with the existing graph extension.
  • Reuse GPUReduction, GPUHistogram, and GPUMask; exclude null/NaN/infinite statistic inputs and expose explicit GPU validity flags for every nullable scalar result.
  • Preserve integer wrap semantics, reject unsupported count/buffer limits, retain native signed/unsigned histogram domains, and safely synthesize empty float NaN payloads at runtime for Dawn/WebGPU compatibility.
  • Add strict immutable Node API/type/validation contracts and real-WebGPU CPU-equivalence cases for all scalar formats, zero batches, filtered/derived inputs, NaN/infinity, irregular edges, repeated encoding, and destruction ordering.

Verification

  • nvm use — Node 22.22.1.
  • yarn install — the existing enterprise registry returns HTTP 403 for an upstream alpha dev-tool package; reused the previously verified ignored workspace dependency tree.
  • yarn lint fix — passed after final source changes.
  • yarn build — passed across every workspace package after final formatting.
  • Focused luDF Node suites — 54 tests passed.
  • Focused real Chromium/WebGPU — six integration scenarios passed.
  • CI=1 yarn test — Node: 597 passed and one skipped; real Chromium/WebGPU: 1,576 passed and 25 skipped.
  • yarn website:build — passed, including validation of 461 documentation pages.
  • (cd website && yarn build) — passed, including validation of 461 documentation pages.
  • yarn examples:typecheck — passed across all 46 example workspaces.
  • yarn bundle-size — passed all seven existing bundle budgets.

Risks and limitations

  • Existing GPU reduction and histogram primitives operate on native 32-bit scalars; floating-point sums and means have float32 precision, and signed/unsigned sums preserve native 32-bit overflow semantics.
  • Domains and irregular edges must be explicit; automatic filtered extents are deliberately unsupported.
  • Scalar result tables contain one merged row, histogram tables contain one dense result batch, and original row masks/indices/counts retain source batch topology.
  • Stable sorting, top-K, limited joins, and interactive documentation/examples follow independently.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 030dece020

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modules/experimental/src/ludf/lu-global-aggregation-compiler.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants