Skip to content

Read Column With Null Data Page Header (#776) #777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mukunku wants to merge 4 commits into from
Closed

Read Column With Null Data Page Header (#776) #777

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fec2414
add patch and test
mukunku Aug 3, 2026
98d9a9e
comment
mukunku Aug 3, 2026
3ad6804
fix comment
mukunku Aug 3, 2026
ea2c415
Merge branch 'master' into mukunku/read-column-with-no-data-page-header
aloneguid Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion src/Parquet.Test/ParquetReaderOnTestFilesTest.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -397,12 +397,25 @@ public async Task PyArrow23Async() {
Assert.Equal(39_366, rs.Data.Count);
}


[Fact]
public async Task UnshreddedVariantAsync() {
await using Stream s = OpenTestFile("variant_unshredded.parquet");
await using ParquetReader r = await ParquetReader.CreateAsync(s);
Assert.NotNull(r.Schema);
}

[Fact]
public async Task AllNullColumnPyArrowV25() {
using Stream s = OpenTestFile("all_null_column_pyarrow_v25.parquet");
await using ParquetReader r = await ParquetReader.CreateAsync(s);
using ParquetRowGroupReader groupReader = r.OpenRowGroupReader(0);
DataField[] fs = r.Schema.GetDataFields();
double?[] data = await ReadNullableValuesAsync<double>(groupReader, fs[0]);
Assert.Equal(46, data.Length);
Assert.All(data, d => Assert.Null(d));

data = await ReadNullableValuesAsync<double>(groupReader, fs[1]);
Assert.Equal(46, data.Length);
Assert.All(data, d => Assert.Equal(0, d));
}
}
Binary file added src/Parquet.Test/data/all_null_column_pyarrow_v25.parquet
View file
Open in desktop
Binary file not shown.
14 changes: 10 additions & 4 deletions src/Parquet/File/DataColumnReader.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,11 @@ public async ValueTask ReadAsync<T>(ReadingColumn<T> rc, CancellationToken cance
if(_stats?.NullCount != null)
definedValuesCount -= (int)_stats.NullCount.Value;

//using var pc = new PackedColumn(_dataField, totalValuesInChunk, definedValuesCount);
long fileOffset = GetFileOffset();
_inputStream.Seek(fileOffset, SeekOrigin.Begin);

while(rc.ValuesRead < totalValuesInChunk) {
bool allNullColumnProcessed = false;
while(rc.ValuesRead < totalValuesInChunk && !allNullColumnProcessed) {
PageHeader ph = PageHeader.Read(new ThriftCompactProtocolReader(_inputStream));

switch(ph.Type) {
Expand All @@ -79,6 +79,7 @@ public async ValueTask ReadAsync<T>(ReadingColumn<T> rc, CancellationToken cance
break;
case PageType.DATA_PAGE:
await ReadDataPageV1Async(ph, rc, cancellationToken);
allNullColumnProcessed = definedValuesCount == 0;

@mukunku mukunku Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately rc.ValuesRead always returns 0 for this column because it has a MaxDefinitionLevel of 1. See:

public int ValuesRead => Field.MaxDefinitionLevel > 0
    ? _definitionOffset
    : _definedDataCount;

So the rc.MarkValuesRead(allValueCount); call I added below isn't actually doing anything in this situation 🤷🏾‍♂️

This is my clunky attempt to get this to work. The loop will first read the dictionary page, then the data page and then will exit early.

This also doesn't make any changes to ReadDataPageV2Async so that's also an open question 😬

break;
case PageType.DATA_PAGE_V2:
await ReadDataPageV2Async(ph, rc, totalValuesInChunk, cancellationToken);
Expand Down Expand Up @@ -122,12 +123,17 @@ private long GetFileOffset() =>
private async ValueTask ReadDataPageV1Async<T>(PageHeader ph, ReadingColumn<T> rc, CancellationToken cancellationToken) where T : struct {
using IMemoryOwner<byte> bytes = await ReadPageDataAsync(ph);

int allValueCount = (int)_thriftColumnChunk.MetaData!.NumValues;
if(ph.DataPageHeader == null) {
throw new ParquetException($"column '{_dataField.Path}' is missing data page header, file is corrupt");
if (allValueCount != (int?)_stats?.NullCount) {
throw new ParquetException($"column '{_dataField.Path}' is missing data page header, file is corrupt");
}
// all values are meant to be null; mark as read and return.
rc.MarkValuesRead(allValueCount);
return;
}

int dataUsed = 0;
int allValueCount = (int)_thriftColumnChunk.MetaData!.NumValues;
int pageValueCount = ph.DataPageHeader.NumValues;

if(_dataField.MaxRepetitionLevel > 0) {
Expand Down
Loading