feat: add Carquet reads through the existing IO interfaces - #1047
Open
luoxiaojian wants to merge 1 commit into
Open
feat: add Carquet reads through the existing IO interfaces#1047luoxiaojian wants to merge 1 commit into
luoxiaojian wants to merge 1 commit into
Conversation
Implement private scalar and nested chunk suppliers, schema sniffing, complete predicate filtering, physical projection, ordered multi-file scans and conservative row-group pruning. Preserve the public IO contract and current production Arrow registration. Extend the adjacent Carquet patch for nested projection and fixed-size-list metadata, and add supplier, scan, HTTP Range and feature documentation coverage. Validate Debug/Release, ASan/UBSan, existing Parquet/export regressions and a clean patched Carquet build without pkg-config.
luoxiaojian
force-pushed
the
codex/carquet-r03-complete-reader
branch
from
September 10, 2026 03:43
1ff38b8 to
702ff96
Compare
liulx20
reviewed
Sep 10, 2026
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| */ | ||
| #pragma once |
liulx20
reviewed
Sep 10, 2026
Comment on lines
+294
to
+307
| if (format == "c") { | ||
| return convertPrimitive<int8_t, int32_t>(schema, array, path); | ||
| } | ||
| if (format == "s") { | ||
| return convertPrimitive<int16_t, int32_t>(schema, array, path); | ||
| } | ||
| if (format == "i") { | ||
| return convertPrimitive<int32_t, int32_t>(schema, array, path); | ||
| } | ||
| if (format == "C") { | ||
| return convertPrimitive<uint8_t, uint32_t>(schema, array, path); | ||
| } | ||
| if (format == "S") { | ||
| return convertPrimitive<uint16_t, uint32_t>(schema, array, path); |
Collaborator
There was a problem hiding this comment.
[P1] 8/16 位整数的读取宽度与 Carquet 实际缓冲区不一致
新增转换器按 int8_t/int16_t 等类型读取，但固定版本 Carquet 的两个导出路径仍将这些 Parquet 整数按 INT32、每值 4 字节存放，仅 schema 标记为 8/16 位。因此读取步长错误，后续值会变成前一个整数的部分字节。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the private Carquet reading path in the Parquet extension. CarquetSniffer and CarquetChunkSupplier reuse InputStreamFactory and IDataChunkSupplier and return NeuG-owned DataChunk columns. The scan handles scalar/NULL/temporal values, LIST and ARRAY nesting, projection, complete predicates with runtime parameters, multiple files, batch/full reads and ordered parallel row-group tasks.
The reader selects output columns plus all predicate dependencies before decoding, applies conservative row-group pruning, evaluates the complete predicate using the shared R5 filter and restores the requested output projection. Reader, schema and batch ownership use RAII, including failed initialization and reads. The existing callback input adapter adds bounded caching. Carquet remains a pinned submodule with an adjacent patch; the additions restore fixed-size-list metadata and select nested top-level fields before decoding.
Production Parquet registration still uses Arrow. No public IO, compiler, execution, CSV/JSON or existing Arrow-reader behavior is changed. The new implementation remains in the test build until the backend-switch PR. MAP schema inspection is supported, but general MAP/STRUCT values are outside this reader's supported scope. Chunk size bounds returned chunks; nested/projected/pruned paths may decode a whole row group and full reads retain their complete result.
Review map
Validation
These are local macOS results. Linux runners and the system-Arrow configuration still require PR CI; no local system Arrow installation is available. No push or PR creation was performed.
Base: upstream/main fc9096b (R5 merged as #1023). Branch: codex/carquet-r03-complete-reader. One cohesive feature commit: 25 files, +5397/-8 (5405 changed lines). Logs: build/review-r03-expanded-mdfpz1ah/.