Skip to content

Benchmark Parquet implementations against latest hyparquet - #3547

Merged
ibgreen-openai merged 6 commits into
masterfrom
codex/parquet-live-benchmarks
Aug 4, 2026
Merged

Benchmark Parquet implementations against latest hyparquet#3547
ibgreen-openai merged 6 commits into
masterfrom
codex/parquet-live-benchmarks

Conversation

@ibgreen

@ibgreen ibgreen commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Goals

  • Keep loaders.gl Parquet performance comparisons pinned to the latest verified hyparquet release.
  • Measure the loaders.gl TypeScript decoder, the default parquet-wasm backend, and hyparquet with equivalent in-memory object-row workloads.
  • Make the comparison reproducible through the existing benchmark command.

Stack position: #3461 → this PR. The PR targets codex/native-decompression-stream-support; compared with master, it builds on that native-decompression tranche and adds the benchmark changes below.

Actual changes

  • Replace the two-backend smoke benchmark with grouped cross-implementation scenarios for LZ4_RAW, Hadoop LZ4, DELTA_BYTE_ARRAY, and projected columns.
  • Label the comparator as the exact latest verified version, hyparquet 1.27.1, and use hyparquet-compressors 1.1.1.
  • Preload input bytes and loader implementations outside timed callbacks.
  • Warm each implementation twice, validate common row counts, and require at least five measured iterations.
  • Keep existing Arrow and GeoParquet benchmarks.
  • Remove the redundant ts-node/esm loader registration that conflicted with the benchmark loader.

Validation

  • yarn lint fix
  • yarn test bench parquet
  • yarn test-node test/bench-loader.node.spec.ts
  • yarn build
  • yarn test-node
  • yarn test-headless
  • git diff --check

Base automatically changed from codex/native-decompression-stream-support to master August 4, 2026 20:26
@ibgreen-openai
ibgreen-openai marked this pull request as ready for review August 4, 2026 20:27
@ibgreen-openai
ibgreen-openai merged commit e0bdfb7 into master Aug 4, 2026
10 checks passed
@ibgreen-openai
ibgreen-openai deleted the codex/parquet-live-benchmarks branch August 4, 2026 20:27

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 08a4127d62

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

function installMockDecompressionStream(options: MockDecompressionStreamOptions): () => void {
const originalDescriptor = Object.getOwnPropertyDescriptor(globalThis, 'DecompressionStream');

class MockDecompressionStream {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add TSDoc to the mock stream class

This introduces a new MockDecompressionStream class (with new readonly fields) without TSDoc; the repo's documented style guidance requires TSDoc on every new class, function, method, and field, so this test helper should get class/field docs before merge.

AGENTS.md reference: AGENTS.md:L33-L33

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants