Skip to content

Add lightweight native decompression - #3461

Merged
ibgreen-openai merged 5 commits into
masterfrom
codex/native-decompression-stream-support
Aug 4, 2026
Merged

Add lightweight native decompression#3461
ibgreen-openai merged 5 commits into
masterfrom
codex/native-decompression-stream-support

Conversation

@ibgreen-openai

@ibgreen-openai ibgreen-openai commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Goals

  • Let async consumers use runtime DecompressionStream support without eagerly importing codec fallbacks.
  • Preserve deterministic codec-backed compression classes and existing synchronous behavior.

Changes

  • Added the zero-dependency @loaders.gl/compression/native-decompression subpath for gzip, deflate, raw deflate, Brotli, and forward-compatible Zstandard probing.
  • Added codec-specific subpaths so fallback implementations can be dynamically imported without pulling every codec into one chunk.
  • Made Parquet gzip/Brotli/Zstandard and SPZ Zstandard decompression native-first, with registered Brotli/Zstandard modules taking precedence and codec fallbacks loaded only when needed.
  • Removed automatic optional installation of Brotli, LZ4, and Zstandard codec packages from @loaders.gl/compression; Parquet no longer installs Brotli or zstd-codec as runtime dependencies.
  • Kept GZipCompression, DeflateCompression, BrotliCompression, and ZstdCompression codec-backed and deterministic; compression and sync APIs retain their behavior.
  • Added real-runtime atomic and batched tests for every native format when supported, plus mocked unsupported, failure, Parquet fallback/module-override, and SPZ native/module-override coverage.
  • Documented the lightweight entrypoint, lazy fallback behavior, future-facing Zstandard support, and v5 release note.

Validation

  • corepack yarn
  • corepack yarn build
  • corepack yarn test-node
  • corepack yarn lint fix
  • cd website; corepack yarn; corepack yarn build
  • Bundled the native entrypoint and Parquet path with esbuild splitting: the native entrypoint is ~2.8 KB with no imports, and pako is absent from the initial Parquet chunk.

@ibgreen-openai
ibgreen-openai marked this pull request as ready for review July 10, 2026 14:59
@ibgreen-openai ibgreen-openai changed the title Add native DecompressionStream support Add lightweight native decompression Aug 2, 2026
@ibgreen-openai
ibgreen-openai force-pushed the codex/native-decompression-stream-support branch from 337041f to e470746 Compare August 2, 2026 19:43
@ibgreen
ibgreen force-pushed the codex/native-decompression-stream-support branch from e470746 to b65e146 Compare August 4, 2026 13:24
@ibgreen-openai
ibgreen-openai merged commit eb6cc65 into master Aug 4, 2026
10 checks passed
@ibgreen-openai
ibgreen-openai deleted the codex/native-decompression-stream-support branch August 4, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant