[BUG]: element type System.DateTime is not supported in DELTA_BINARY_PACKED

[perlun]

Library Version

6.0.3

.NET Runtime

10.0.302

OS and Architecture

Linux (64-bit)

How to reproduce?

(I'm not sure if this is technically a "bug"; it can be seen as a limitation in Parquet.Net. Feel free to relabel accordingly)

When attempting to read a Parquet file generated from a 3rd party, we ran into the following excption:

System.NotSupportedException: only INT32 and INT64 are supported in DELTA_BINARY_PACKED but element type passed is System.DateTime
  at int DeltaBinaryPackedEncoder.Decode(Span<byte> s, Array dest, int destOffset, int valueCount, out int consumedBytes)()
  at void DataColumnReader.ReadColumn(Span<byte> src, Encoding encoding, long totalValuesInChunk, int totalValuesInPage, PackedColumn pc)()
  at async Task DataColumnReader.ReadDataPageV1Async(PageHeader ph, PackedColumn pc)()
  at async Task<DataColumn> DataColumnReader.ReadAsync(CancellationToken cancellationToken)()

[...]

We were initially using a slightly older version of Parquet.Net (Parquet.Net), but analysing the problem (with Claude) suggested that this was reproducible on later versions as well. I've constructed (manually, worth pointing out) an MCVE that reproduces the problem on a more current 6.0.3 version of Parquet.Net. The failing Parquet file has also been included. I have not been able to generate a failing file with Parquet.Net itself.

With the MCVE included in this issue, the exception looks like this:

System.NotSupportedException : element type System.DateTime is not supported in DELTA_BINARY_PACKED
   at Parquet.Encodings.DeltaBinaryPackedEncoder.Decode[T](Span`1 s, Span`1 dest, Int32 valueCount, Int32& consumedBytes)
   at Parquet.File.DataColumnReader.ReadColumn[T](Span`1 src, Encoding encoding, Int64 totalValuesInChunk, Int32 totalValuesInPage, ReadingColumn`1 rc)
   at Parquet.File.DataColumnReader.ReadDataPageV1Async[T](PageHeader ph, ReadingColumn`1 rc, CancellationToken cancellationToken)
   at Parquet.File.DataColumnReader.ReadAsync[T](ReadingColumn`1 rc, CancellationToken cancellationToken)
   at Parquet.ParquetRowGroupReader.ReadRawAsync[T](DataField field, Memory`1 values, Nullable`1 definitionLevels, Nullable`1 repetitionLevels, CancellationToken cancellationToken)
   at Parquet.ParquetRowGroupReader.ReadRawColumnDataAsync[T](DataField field, CancellationToken cancellationToken)
   at CallSite.Target(Closure, CallSite, Object)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Parquet.ParquetRowGroupReader.ReadRawColumnDataBaseAsync(DataField field, CancellationToken cancellationToken)

[...]

Workaround

What we did was to detect fields of the appropriate type, read them as int instead, then manually construct DateTime instances based on these values. (I'm deliberately vague here because our workaround was inspired by LLM suggestions and I don't want to contaminate the project - it's better if someone writes a workaround by hand for inclusion in the project code base).

Here is a test file that reproduces the problem: parquet-file-with-binary-packed-date-column.parquet.gz (need to gzip-decompress locally, I had to gzip it to be able to attach it here on GitHub)

Failing test

using NUnit.Framework;

using Parquet;

using System.IO;
using System.Threading.Tasks;

namespace Tests;

public class ParquetReadDateColumnTests {
    [Test]
    public async Task ReadColumnAsync_can_read_delta_binary_packed_column_with_Date_value() {
        using (var stream = File.OpenRead("parquet-file-with-binary-packed-date-column.parquet")) {
            await using (var parquetReader = await ParquetReader.CreateAsync(stream)) {
                var dataFields = parquetReader.Schema.GetDataFields();

                for (int rowGroup = 0; rowGroup < parquetReader.RowGroupCount; rowGroup++) {
                    // Read all columns for this row group
                    using (ParquetRowGroupReader groupReader = parquetReader.OpenRowGroupReader(rowGroup)) {

                        for (int i = 0; i < dataFields.Length; i++) {
                            var field = dataFields[i];

                            await groupReader.ReadRawColumnDataBaseAsync(field);
                        }
                    }
                }
            }
        }
    }
}