Read Column With Null Data Page Header (#776) - #777
Open
mukunku wants to merge 3 commits into
Open
Conversation
mukunku
commented
Aug 4, 2026
| break; | ||
| case PageType.DATA_PAGE: | ||
| await ReadDataPageV1Async(ph, rc, cancellationToken); | ||
| allNullColumnProcessed = definedValuesCount == 0; |
Contributor
Author
There was a problem hiding this comment.
Unfortunately rc.ValuesRead always returns 0 for this column because it has a MaxDefinitionLevel of 1. See:
public int ValuesRead => Field.MaxDefinitionLevel > 0
? _definitionOffset
: _definedDataCount;So the rc.MarkValuesRead(allValueCount); call I added below isn't actually doing anything in this situation 🤷🏾♂️
This is my clunky attempt to get this to work. The loop will first read the dictionary page, then the data page and then will exit early.
This also doesn't make any changes to ReadDataPageV2Async so that's also an open question 😬
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is for #776.
One of my users reported an issue when reading a file created with PyArrow v25 where all the values in the column are null. It seems the
DataPageHeaderis null for this file which causes the library to throw an exception when reading it. However other providers like parquetreader.com and DuckDB are able to open this file.This PR is my clunky attempt to make this test file readable. I'm open to feedback or if you'd like to come up with your own solution 👍🏾