Skip to content

feat(parquet): prepare range and worker foundations - #3539

Merged
ibgreen merged 1 commit into
masterfrom
codex/parquet-source-prep
Aug 4, 2026
Merged

feat(parquet): prepare range and worker foundations#3539
ibgreen merged 1 commit into
masterfrom
codex/parquet-source-prep

Conversation

@ibgreen-openai

@ibgreen-openai ibgreen-openai commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Goals

Changes

  • Refactors HttpFile around exact validated byte ranges, pinned ETag/Last-Modified identity, strict or best-effort consistency, custom fetch options, cancellation, and immutable transport telemetry.
  • Separates the 293-line HttpFile lifecycle/scheduler façade from the focused HTTP transport, response-validation, telemetry, and public-type modules.
  • Recognizes valid empty HTTP objects from an opening 416 response with Content-Range: bytes */0, preserving validators and zero-length reads.
  • Extends RangeRequestScheduler with explicit transport-context isolation, exact response validation, safe EOF clamping through authoritative source length, prompt per-caller cancellation, and network timing.
  • Routes GeoTIFF and PMTiles range reads through the clarified scheduler contract while preserving per-source request isolation.
  • Adds immutable ParquetSource capability reporting that reflects the current range-backed TypeScript source: statistics, custom transport, object validation, and network/decode telemetry are supported; source worker decoding remains deferred.
  • Deep-freezes cached Parquet schema state, telemetry snapshots, and batch provenance, and preserves caller AbortSignal.reason.
  • Publishes the bundler-safe @loaders.gl/parquet/wasm URL helper and raw WASM asset export, and updates parquet-wasm to 0.7.2.
  • Adds focused coverage for identity changes, malformed/short responses, valid empty-object discovery, EOF clamping, coalescing, sibling cancellation, callback/source isolation, capability exports, immutable source state, abort reasons, and local WASM resolution.
  • Moves browser-neutral HttpFile coverage into the shared test project and verifies it in Node and Chromium.
  • Adds a dedicated HttpFile API reference covering opening, options, consistency modes, response requirements, methods, telemetry, cancellation, CORS, and scheduler sharing.
  • Links the API page from the loader-utils overview/sidebar and corrects stale ReadableFile examples that used a nonexistent slice() method.
  • Documents validated HTTP ranges, scheduler isolation, Parquet capabilities, and explicit package-local WASM workflows.

Current boundary

ParquetSource continues to use the selective TypeScript range backend merged in #3531, #3533, and #3535. It does not introduce the obsolete whole-file WASM source/session alternative from the pre-stack prototype. Worker-backed source decoding remains advertised as unavailable; the canonical WASM ParquetLoader worker path is already provided by #3538.

Validation

  • yarn install
  • yarn lint fix
  • yarn build
  • cd website && yarn build
  • Focused Node suites — 8 files, 68 tests passed
  • Focused Chromium suites — 6 files, 39 tests passed
  • Shared HttpFile suite — 14 tests passed in Node and 14 in Chromium
  • yarn test-node — 403 files passed, 3 skipped; 1,915 tests passed, 81 skipped
  • yarn test-headless — 406 files passed, 3 skipped; 1,881 tests passed, 60 skipped
  • GitHub Actions test workflow — 7 jobs passed
  • git diff --check

Stack

Part of #3525.

@ibgreen
ibgreen marked this pull request as ready for review August 3, 2026 01:29
@ibgreen
ibgreen force-pushed the codex/parquet-source-prep branch from ea1c0e9 to a154fc1 Compare August 3, 2026 01:34

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ea1c0e9053

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread modules/loader-utils/test/lib/readable-file/http-file.spec.ts
Comment thread modules/parquet/src/lib/sources/parquet-source-session.ts Outdated
Comment thread modules/loader-utils/src/lib/files/http-file.ts Outdated
@ibgreen
ibgreen force-pushed the codex/parquet-source-prep branch 2 times, most recently from e5d9e53 to 5de2429 Compare August 3, 2026 01:44
@ibgreen
ibgreen force-pushed the codex/parquet-source-prep branch from 5de2429 to 2cf901d Compare August 3, 2026 13:48
@ibgreen
ibgreen merged commit e26d47c into master Aug 4, 2026
10 checks passed
@ibgreen
ibgreen deleted the codex/parquet-source-prep branch August 4, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants