Skip to content

feat(experimental): add batch-preserving luDF joins and lookups - #2962

Open
ibgreen-openai wants to merge 4 commits into
codex/ludf-sorting-topkfrom
codex/ludf-joins
Open

feat(experimental): add batch-preserving luDF joins and lookups#2962
ibgreen-openai wants to merge 4 commits into
codex/ludf-sorting-topkfrom
codex/ludf-joins

Conversation

@ibgreen-openai

Copy link
Copy Markdown
Collaborator

Goals

Expose documented unique-right-key uint32 inner joins and bounded left lookups through the optional luDF API while preserving independently streamed left/right source batches, explicit ownership, stable row identity, and GPU-resident diagnostics.

Changes

  • Add immutable, precisely typed LuDataFrame.innerJoin, LuDataFrame.lookup, and corresponding filtered/derived query methods with explicit key names and bounded index/output capacities.
  • Compose the independently published preserved-batch GPUBatchHashIndex prerequisite with existing GPU hash join/query primitives; build one index from independently chunked right tables without concatenation, repacking, CPU row access, or hidden submission.
  • Preserve discontinuous source-row offsets and every empty left/right batch, exclude explicit nullable keys, and retain both owned source leases until compiled results are destroyed.
  • Publish stable matched left/right row identifiers, exact required counts, capacity-limited published counts, explicit per-batch overflow, source-aligned lookup masks/probes, six-word index diagnostics, and a GPU-native uniqueness/reserved-key/overflow contract-violation flag.
  • Suppress all matches for duplicate right keys, valid reserved keys, or incomplete indexes; reject incompatible categorical dictionaries, invalid formats/layouts, unsupported capacities, and unsafe cumulative probe bounds before GPU work.
  • Reuse one caller-owned command graph across ordered parameter updates and preserve visualization-ready selection masks without CPU readback.
  • Add focused planning and real-WebGPU coverage for mismatched [3,0,5]/[2,0,3] topologies, nullable rows, discontinuous offsets, bounded outputs, duplicate/reserved/overflow contracts, repeated encodes, schema-only right tables, and source destruction.

Dependencies

Verification

  • nvm use: Node v22.22.1.
  • yarn install: not rerun; reused a verified dependency tree because the existing enterprise alpha registry dependency returns HTTP 403.
  • Official @vis.gl/dev-tools alpha lint configuration and final yarn lint fix passed.
  • Final yarn build passed.
  • Final yarn test: Node 621 passed/1 skipped; real Chromium/WebGPU 1,590 passed/25 skipped.
  • Focused luDF Node suite: 8 files/71 tests.
  • Focused real-WebGPU dataframe joins: 5 cases; combined preserved-batch index and join integration: 9 cases.
  • yarn website:build and (cd website && yarn build) passed.
  • yarn examples:typecheck and all seven yarn bundle-size budgets passed.

Limitations

  • Supports unique-right unsigned scalar inner joins and source-aligned bounded lookups only.
  • Many-to-many, outer, multi-key, string, and signed-key joins remain intentionally unsupported.
  • 0xffffffff remains reserved; valid reserved right keys explicitly invalidate the join.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants