feat(pandas-gbq): add arrow decoder for read rows response - #17958
Open
shuoweil wants to merge 3 commits into
Open
feat(pandas-gbq): add arrow decoder for read rows response#17958shuoweil wants to merge 3 commits into
shuoweil wants to merge 3 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a new arrow integration submodule in pandas-gbq to decode ReadRowsResponse protobuf messages into pyarrow.RecordBatch objects, along with corresponding unit tests. The review feedback points out that pyarrow.Message does not have a schema attribute, which would cause decoding to fail or fall back incorrectly. Additionally, it is recommended to catch specific pyarrow.ArrowException errors instead of using broad except Exception blocks.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
pandas_gbq.arrow.from_read_rows_responseto decode BigQuery Storage Read APIReadRowsResponseprotobuf messages intopyarrow.RecordBatchinstances.Changes
pandas_gbq/arrow.py: Implementsfrom_read_rows_response(message, arrow_schema=None)to deserialize IPC stream and message buffers with empty-batch fallback.pandas_gbq/__init__.py: Exposesarrowin package__all__.tests/unit/test_arrow.py: Adds unit tests for valid batch parsing, empty message handling, and missingpyarrowerror handling.Fixes #<540939659> 🦕