feat(experimental): document luDF with Arrow-native GPU benchmarks

[ibgreen-openai]

Goals

• Complete the browser-native luDF integration as an optional @luma.gl/experimental/ludf dataframe facade over existing GPU tables, with Apache Arrow ingestion remaining in @luma.gl/arrow.
• Show the complete workflow in the existing GPU Data Analysis example, document the exact supported semantics, and validate real WebGPU results against independent CPU references.
• Integrate the separately reviewed graph-safety, bounded-dispatch, chunk-preserving hash-index, Arrow-ingestion, and dataframe feature prerequisites as actual Git ancestors.

Changes

• Add a dedicated luDF API reference and register it in both experimental navigation trees, the experimental overview/tab bar, and the v10 release notes.
• Document immutable borrowed/owned dataframes; preserved record batches and stable source identifiers; sliced Arrow validity masks and dictionary labels; SQL-style nullable expressions and reusable graph parameters; derived columns; dense grouped and global reductions; explicit-domain histograms; stable per-batch sorting/top-K; and bounded unique-right uint32 inner joins/lookups with explicit GPU overflow, validity, and contract diagnostics.
• Extend the existing /examples/experimental/gpu-data-analysis route without adding a workspace, dependency, lockfile entry, or standalone application.
• Add an explicitly opt-in Arrow-backed benchmark that creates sliced nullable Float32/dictionary columns, independently batched right-hand rows, and preserved empty source batches; exercises filtering, dense grouping, stable batch-local top-K, and unique-right joins; and compares bounded GPU outputs with independent CPU oracles.
• Report upload, graph compilation, equivalent standalone index construction, fenced WebGPU execution, explicit bounded result readback, and CPU reference durations separately. The standalone index measurement does not replace or subtract the join's own index build.
• Add real-browser/WebGPU coverage for the complete benchmark, interactive click-to-run behavior, all six timing phases, preserved batch topology, and graceful unsupported-WebGPU handling; add focused Node coverage for documentation, package boundaries, routes, and opt-in behavior.
• Merge the real prerequisite histories instead of copying prerequisite source files: renderer-independent Arrow ingestion, physical writable-buffer alias protection, bounded visibility/compaction and sort primitive dispatch, chunk-preserving hash indexing, and the complete luDF feature stack.

Verification

• nvm use: passed, selecting Node v22.22.1 from .nvmrc.
• yarn install: not completed because the registry returned HTTP 403 for required @vis.gl/dev-tools@2.0.0-alpha.4. Verification reused already-installed workspace dependencies from the existing checkout; the exact upstream alpha.4 Biome configuration was installed only under ignored node_modules. This change adds no packages or lockfile changes.
• yarn lint and yarn lint fix: passed, checking all 1,568 files with the exact official alpha.4 Biome configuration.
• yarn build: passed for every workspace package after final formatting and integration changes.
• yarn test-node: passed, with 672 passing Node tests and one existing skipped test across 127 files.
• CI=1 yarn test / yarn test: passed, with 672 passing Node tests (one existing skipped) and 1,598 passing real Chromium/WebGPU tests (25 existing skipped) across 297 browser files.
• Focused integration browser smoke: 3/3 passed on actual Chromium/WebGPU, including direct Arrow-to-GPU correctness, the interactive opt-in UI, and unsupported-WebGPU fallback.
• Combined prerequisite regression after merging all three safety/dispatch branches: 22/22 real-WebGPU tests across four files and 29/29 focused Node tests across four files passed.
• Focused documentation/integration Node suite: 4/4 passed.
• The new luDF reference compiles through the real MDX compiler; both experimental navigation trees include its page; the added docs tab independently transpiles.
• yarn website:build: passed, validating all 462 raw documentation pages.
• (cd website && yarn build): passed independently, validating all 462 raw documentation pages.
• yarn examples:typecheck: passed across all 46 configured example workspaces.
• yarn workspace luma.gl-examples-experimental-gpu-data-analysis build: passed, including standalone TypeScript compilation and the real production Vite bundle for the updated Arrow/luDF example.
• yarn bundle-size: passed, with all seven production bundle fixtures below their enforced ceilings.

Prerequisites and review order

This integration branch contains the following independently reviewable pull requests as actual Git ancestors:

• feat(experimental): add immutable luDF dataframe foundation #2933: immutable optional luDF dataframe foundation.
• feat(arrow): add renderer-independent GPU analytics ingestion #2936: renderer-independent Arrow analytics ingestion, sliced validity, and dictionary metadata.
• fix(experimental): reject writable physical command-graph buffer aliases #2939: reject writable physical GPU command-graph aliases while allowing shared read-only aliases.
• feat(experimental): add GPU-resident luDF expression filtering #2941: safe immutable nullable expressions and reusable GPU filtering.
• feat(experimental): add nullable GPU-resident luDF derived columns #2945: nullable, batch-preserving derived columns.
• fix(experimental): scale visibility workflow dispatches #2951: bounded multi-dimensional dispatch in visibility, mask, and compaction primitives.
• feat(experimental): add dense luDF group aggregation #2953: dense nullable dictionary/category grouping.
• feat(experimental): add luDF reductions and histograms #2955: global scalar reductions and explicit-domain/edges histograms.
• fix(experimental): scale GPU sort dispatches #2956: bounded multi-dimensional dispatch in GPU sort primitives.
• feat(experimental): add stable luDF sorting and top-K #2959: stable batch-local dataframe sorting and top-K.
• feat(experimental): build shared GPU hash indexes from preserved batches #2960: chunk-preserving right-side GPU hash-index construction.
• feat(experimental): add batch-preserving luDF joins and lookups #2962: bounded unique-right inner joins and source-aligned lookups.

All twelve prerequisite PRs had successful GitHub checks and zero unresolved non-outdated review threads when this description was prepared.

Risks and limitations

• luDF is experimental and WebGPU-only. It supports packed float32, sint32, and uint32 scalars; dense grouping and hash joins require uint32 category/key storage. It does not provide native GPU float64/int64, arbitrary GPU strings, full SQL/cuDF parity, many-to-many/outer/multi-key joins, or implicit global cross-batch sorting/top-K.
• Bounded multi-dimensional dispatch is verified for the merged reusable visibility/compaction and sort primitives. This PR does not claim that every dataframe-specific expression, aggregation, or join shader supports workloads beyond one-dimensional device dispatch limits; the interactive benchmark intentionally uses bounded data sizes.
• Right-side duplicate keys, reserved valid 0xffffffff keys, or incomplete index construction set GPU-resident contractViolation and suppress matches. Per-batch publication capacity and overflow remain explicit application responsibilities.
• Integer sums retain native 32-bit overflow behavior, floating-point reductions retain float32 precision, and grouped atomic sum comparisons use a documented floating-point tolerance.
• GPU command submission, any optional output readback, device fallback, and lifecycle management remain application-controlled; the benchmark performs explicit bounded readback only for correctness validation.