feat(experimental): document luDF with Arrow-native GPU benchmarks

docs/api-reference/experimental/README.md

@@ -147,6 +147,20 @@ exact force-layout coordinates, neighborhood highlighting, stable GPU picking, d
 An opt-in live benchmark compares six actual CPU and WebGPU graph workloads across five graph
 families while reporting command encoding, completion fences, setup costs, and layout accuracy.
 
+## GPU-Resident Dataframes
+
+<p class="badges">
+  <img src="https://img.shields.io/badge/WebGPU-required-blueviolet.svg?style=flat-square" alt="WebGPU required" />
+</p>
+
+[`luDF`](/docs/api-reference/experimental/ludf) adds immutable, GPU-resident dataframe queries on
+top of existing `GPUTable` batches. Its optional `@luma.gl/experimental/ludf` entry point provides
+nullable expressions, derived columns, categorical and global aggregation, histograms, stable
+per-batch sorting, top-K selection, and bounded unique-right joins without hidden GPU submission,
+source repacking, or CPU readback. The existing
+[GPU Data Analysis example](/examples/experimental/gpu-data-analysis) demonstrates Arrow ingestion
+and opt-in, explicitly fenced GPU-versus-CPU benchmarks.
+
 ## GPU-resident Linked Crossfiltering
 
 <p class="badges">

docs/table-of-contents.json

@@ -213,6 +213,7 @@
           "api-reference/experimental/luraster/README",
           "api-reference/experimental/luproj",
           "api-reference/experimental/lugraph",
+          "api-reference/experimental/ludf",
           "api-reference/experimental/luxfilter",
           "api-reference/experimental/lutrace",
           "api-reference/experimental/g-buffer",
@@ -347,6 +348,7 @@
           "api-reference/experimental/luraster/README",
           "api-reference/experimental/luproj",
           "api-reference/experimental/lugraph",
+          "api-reference/experimental/ludf",
           "api-reference/experimental/luxfilter",
           "api-reference/experimental/lutrace",
           "api-reference/experimental/g-buffer",

docs/whats-new.md

@@ -54,6 +54,11 @@ Target Release Date: Q3, 2026
 - **Shared interactive GPU-accelerated ray tracing** - `RayTracingSceneRenderer` composes world-space instance bounds, dirty-only Morton-sorted object/instance TLAS construction, retained-permutation transform refits, topology-only Morton-sorted per-mesh triangle BLAS construction, nearest-hit traversal, bounded direct-light shadows, adaptive internal resolution, interleaved frame-budget coverage, stable-identity temporal reprojection, progressive accumulation, and upsampled HDR presentation through WebGPU compute/command graphs. Frame pacing uses ordinary animation intervals, the tracing pass uses exactly eight storage buffers, every TLAS or BLAS construction pass stays within the eight-storage-buffer WebGPU CORE limit, and command submission stays application-owned.
 - **[Generated physical lighting environments](/docs/api-reference/experimental/pbr-environment)** - `PBREnvironmentGenerator` and `preparePBREnvironment()` integrate equirectangular source textures into GGX-prefiltered specular cubemap mip chains, diffuse irradiance cubemaps, and split-sum BRDF lookup textures on both WebGL and WebGPU.
 - **Scene-color transmission and volume attenuation** - The shared forward renderer captures opaque scene color automatically for transmissive surfaces, then applies screen-space refraction, roughness, Fresnel response, index of refraction, thickness, and Beer-Lambert attenuation while preserving physically opaque output.
+- **[`luDF` GPU-resident dataframes](/docs/api-reference/experimental/ludf)** - The optional
+  `@luma.gl/experimental/ludf` entry point adds immutable nullable expressions, derived columns,
+  dense categorical and global aggregations, explicit-domain histograms, stable per-batch sorting
+  and top-K, and bounded unique-right joins over existing `GPUTable` batches. Applications retain
+  ownership of GPU command submission, source lifetimes, and optional result readback.
 - **`HTMLTexture`** - Experimental copied texture binding source copies HTML-in-Canvas DOM subtrees into GPU textures while the browser API is still experimental.
 - **OIT resolve pipelines** - A-buffer and weighted-blended order-independent transparency now
   resolve captured fragments through exported `ShaderPassPipeline` factories, allowing WBOIT to
@@ -109,6 +114,9 @@ Target Release Date: Q3, 2026
 
 **@luma.gl/arrow** NEW MODULE
 
+- **Renderer-independent Arrow analytics upload** - `makeGPUAnalyticsTableFromArrowTable()`
+  preserves Arrow record batches, sliced validity bitmaps, nullable column masks, and explicit
+  dictionary labels while uploading portable scalar columns without requiring a `ShaderLayout`.
 - **Arrow shader layouts** - `getArrowBufferLayout()` maps Arrow scalar and `FixedSizeList` columns to shader attribute formats from a shader-first layout, including direct `arrow.Vector` sources and Arrow table path mappings.
 - **Arrow GPU adapters** - Arrow factories, append helpers, and readback helpers bridge Apache Arrow inputs into `@luma.gl/tables` objects and preserve chunked UTF-8 GPU vector input for text workflows.
 - **Variable-length Arrow attribute lists** - `GPUVector` can retain chunked nested list columns whose elements contain one to four numeric components, covering scalar streams plus tuple-style data such as XY, XYZ, and XYZM coordinates for future path-rendering workflows.

examples/experimental/gpu-data-analysis/src/app-shell.ts

@@ -124,6 +124,33 @@ export const GPU_DATA_ANALYSIS_TEMPLATE = `
       </p>
     </section>
 
+    <section class="benchmark-lab" aria-label="Arrow-native CPU and WebGPU benchmark">
+      <div class="benchmark-heading">
+        <div>
+          <p class="eyebrow">ARROW / CPU + GPU BENCHMARK</p>
+          <h2>Measure the complete dataframe pipeline.</h2>
+          <p>
+            Compare filtering, grouped aggregation, stable top-K, and hash joins over genuine
+            nullable, dictionary-encoded Arrow batches. Nothing runs until you ask.
+          </p>
+        </div>
+        <button id="analysis-ludf-benchmark-run" class="benchmark-button" data-ludf-benchmark disabled>
+          Run verified benchmark <span>→</span>
+        </button>
+      </div>
+      <p id="analysis-ludf-benchmark-status" class="benchmark-status" data-ludf-benchmark-status>
+        Filter, group, stable top-K, and unique-key joins remain GPU-resident.
+      </p>
+      <div
+        id="analysis-ludf-benchmark-results"
+        class="benchmark-results"
+        data-ludf-benchmark-phases
+        data-state="idle"
+        data-validated="false"
+        aria-live="polite"
+      ></div>
+    </section>
+
     <section class="visualizations" aria-label="GPU-computed analytics">
       <article class="visualization-card histogram-card">
         <p class="eyebrow">DISTRIBUTION / HISTOGRAM + CDF</p>
@@ -413,6 +440,33 @@ export const GPU_DATA_ANALYSIS_STYLES = `
 
   .query-status[data-state="verified"] { color: var(--green); }
 
+  .benchmark-lab {
+    margin-top: 23px;
+    padding: 23px;
+    border: 1px solid rgba(115, 170, 250, 0.26);
+    border-radius: 12px;
+    background: linear-gradient(135deg, rgba(16, 24, 40, 0.95), rgba(10, 18, 29, 0.98));
+  }
+
+  .benchmark-heading {
+    display: flex;
+    align-items: center;
+    justify-content: space-between;
+    gap: 20px;
+  }
+
+  .benchmark-heading .eyebrow { color: var(--blue); }
+  .benchmark-heading h2 { margin: 0 0 7px; font-size: 21px; letter-spacing: -0.4px; }
+  .benchmark-heading > div > p:last-child { max-width: 680px; margin: 0; color: var(--muted); }
+  .analysis-example .benchmark-button { width: 230px; border-color: rgba(115, 170, 250, 0.5); color: var(--blue); }
+  .benchmark-status { margin: 16px 0 0; color: var(--muted); font-size: 11px; }
+  .benchmark-results table { width: 100%; margin-top: 13px; border-collapse: collapse; }
+  .benchmark-results th, .benchmark-results td { padding: 10px; border-bottom: 1px solid var(--border); text-align: left; }
+  .benchmark-results thead th { color: var(--muted); font-size: 10px; letter-spacing: 0.5px; }
+  .benchmark-results tbody th { color: var(--text); font-size: 12px; font-weight: 500; }
+  .benchmark-results td { color: var(--green); text-align: right; font-variant-numeric: tabular-nums; }
+  .benchmark-results[data-state="error"] { color: #ff8585; }
+
   .visualizations {
     display: grid;
     grid-template-columns: 1fr 1fr;
@@ -485,6 +539,9 @@ export const GPU_DATA_ANALYSIS_STYLES = `
     .pipeline > span:last-child { border-bottom: 0; }
     .pipeline strong { display: inline; margin-left: 7px; }
     .dataframe-lab { padding: 15px; }
+    .benchmark-lab { padding: 15px; }
+    .benchmark-heading { align-items: flex-start; flex-direction: column; }
+    .analysis-example .benchmark-button { width: 100%; }
     .visualizations { grid-template-columns: 1fr; }
     .heatmap-card { grid-column: auto; }
     .expression-code p { font-size: 10px; }

examples/experimental/gpu-data-analysis/src/app.ts

@@ -25,6 +25,7 @@ import {GPURecordBatch, GPUTable, type GPUVector} from '@luma.gl/tables';
 import {webgpuAdapter} from '@luma.gl/webgpu';
 import * as arrow from 'apache-arrow';
 import {GPU_DATA_ANALYSIS_STYLES, GPU_DATA_ANALYSIS_TEMPLATE} from './app-shell';
+import {runLuDataFrameBenchmark, type LuDataFrameBenchmarkResult} from './ludf-benchmark';
 
 const APP_ID = 'gpu-data-analysis-app';
 const STYLE_ID = 'gpu-data-analysis-style';
@@ -61,6 +62,9 @@ type ExampleElements = {
   luDataFrameRun: HTMLButtonElement;
   luDataFrameSelected: HTMLElement;
   luDataFrameThreshold: HTMLInputElement;
+  ludfBenchmark: HTMLButtonElement;
+  ludfBenchmarkResults: HTMLElement;
+  ludfBenchmarkStatus: HTMLElement;
   nodes: HTMLElement;
   reuse: HTMLElement;
   run: HTMLButtonElement;
@@ -86,6 +90,7 @@ class GPUDataAnalysisExample {
   private readonly elements: ExampleElements;
   private device: Device | null = null;
   private resources: ExampleResources | null = null;
+  private benchmarkController: AbortController | null = null;
   private destroyed = false;
   private hasRunLuDataFrameDemo = false;
   private runVersion = 0;
@@ -96,6 +101,7 @@ class GPUDataAnalysisExample {
   private readonly handleLuDataFrameChange = (): void => {
     if (this.hasRunLuDataFrameDemo) void this.runLuDataFrameDemo();
   };
+  private readonly handleLuDataFrameBenchmark = (): void => void this.runBenchmark();
 
   constructor(root: HTMLElement) {
     this.elements = getElements(root);
@@ -115,6 +121,7 @@ class GPUDataAnalysisExample {
       control.addEventListener('change', this.handleLuDataFrameChange);
     }
     this.updateLuDataFrameExpression();
+    this.elements.ludfBenchmark.addEventListener('click', this.handleLuDataFrameBenchmark);
   }
 
   async initialize(): Promise<void> {
@@ -130,20 +137,26 @@ class GPUDataAnalysisExample {
       }
       this.device = device;
       await this.run();
+      if (!this.destroyed) {
+        this.elements.ludfBenchmark.disabled = false;
+      }
     } catch (error) {
       this.setStatus(getErrorMessage(error), true);
+      this.elements.ludfBenchmarkStatus.textContent = 'WebGPU is unavailable on this device.';
     }
   }
 
   destroy(): void {
     if (this.destroyed) return;
     this.destroyed = true;
+    this.benchmarkController?.abort();
     this.elements.run.removeEventListener('click', this.handleRun);
     this.elements.luDataFrameRun.removeEventListener('click', this.handleLuDataFrameRun);
     for (const control of this.getLuDataFrameControls()) {
       control.removeEventListener('input', this.handleLuDataFrameInput);
       control.removeEventListener('change', this.handleLuDataFrameChange);
     }
+    this.elements.ludfBenchmark.removeEventListener('click', this.handleLuDataFrameBenchmark);
     for (const element of [
       this.elements.dataset,
       this.elements.bins,
@@ -718,6 +731,38 @@ class GPUDataAnalysisExample {
     this.elements.status.textContent = message;
     this.elements.status.dataset.state = error ? 'error' : 'ok';
   }
+
+  /** Executes optional, bounded CPU/GPU comparisons only after an explicit user request. */
+  private async runBenchmark(): Promise<void> {
+    if (!this.device || this.destroyed || this.benchmarkController) return;
+    const controller = new AbortController();
+    this.benchmarkController = controller;
+    this.elements.ludfBenchmark.disabled = true;
+    this.elements.ludfBenchmarkResults.dataset.state = 'running';
+    this.elements.ludfBenchmarkResults.dataset.validated = 'false';
+    this.elements.ludfBenchmarkStatus.textContent =
+      'Uploading a nullable Arrow dictionary dataset and validating GPU dataframe queries...';
+
+    try {
+      const result = await runLuDataFrameBenchmark(this.device, {
+        rowCount: 384,
+        signal: controller.signal
+      });
+      if (this.destroyed || controller.signal.aborted) return;
+      renderLuDataFrameBenchmark(this.elements, result);
+    } catch (error) {
+      if (this.destroyed || controller.signal.aborted) return;
+      this.elements.ludfBenchmarkResults.dataset.state = 'error';
+      this.elements.ludfBenchmarkStatus.textContent = getErrorMessage(error);
+    } finally {
+      if (this.benchmarkController === controller) {
+        this.benchmarkController = null;
+      }
+      if (!this.destroyed) {
+        this.elements.ludfBenchmark.disabled = false;
+      }
+    }
+  }
 }
 
 function makeDataset(length: number): {
@@ -1001,6 +1046,9 @@ function getElements(root: HTMLElement): ExampleElements {
     luDataFrameRun: get('[data-ludf-run]'),
     luDataFrameSelected: get('[data-ludf-selected]'),
     luDataFrameThreshold: get('[data-ludf-threshold]'),
+    ludfBenchmark: get('[data-ludf-benchmark]'),
+    ludfBenchmarkResults: get('[data-ludf-benchmark-phases]'),
+    ludfBenchmarkStatus: get('[data-ludf-benchmark-status]'),
     nodes: get('[data-nodes]'),
     reuse: get('[data-reuse]'),
     run: get('[data-run]'),
@@ -1017,6 +1065,33 @@ function ensureStyles(): void {
   document.head.appendChild(style);
 }
 
+/** Renders only bounded, independently fenced phase timings and explicit CPU-oracle validation. */
+function renderLuDataFrameBenchmark(
+  elements: ExampleElements,
+  result: LuDataFrameBenchmarkResult
+): void {
+  const timingRows = [
+    ['upload', 'Arrow upload', result.timings.uploadMilliseconds],
+    ['compile', 'Graph compilation', result.timings.compileMilliseconds],
+    ['index', 'Standalone hash-index build', result.timings.indexMilliseconds],
+    ['execution', 'Fenced WebGPU execution', result.timings.executionMilliseconds],
+    ['readback', 'Bounded result readback', result.timings.readbackMilliseconds],
+    ['cpu', 'Equivalent CPU reference', result.timings.cpuMilliseconds]
+  ] as const;
+  elements.ludfBenchmarkResults.innerHTML = `<table><thead><tr><th scope="col">Phase</th><th scope="col">Milliseconds</th></tr></thead><tbody>${timingRows
+    .map(
+      ([phase, label, milliseconds]) =>
+        `<tr data-ludf-phase="${phase}"><th scope="row">${label}</th><td>${milliseconds.toFixed(2)}</td></tr>`
+    )
+    .join('')}</tbody></table>`;
+  const validated = Object.values(result.validation).every(Boolean);
+  elements.ludfBenchmarkResults.dataset.state = validated ? 'ok' : 'error';
+  elements.ludfBenchmarkResults.dataset.validated = String(validated);
+  elements.ludfBenchmarkStatus.textContent = validated
+    ? `${result.rowCount.toLocaleString()} Arrow rows · batches ${result.batchRowCounts.join(' / ')} · filter, grouping, sorting, and joins match the CPU reference · ${result.readbackBytes.toLocaleString()} summary bytes read`
+    : 'GPU dataframe results did not match their equivalent CPU reference.';
+}
+
 function getErrorMessage(error: unknown): string {
   return error instanceof Error ? error.message : String(error);
 }