Issue description
Hi,
currently the definitionLevels are automatically derived for nullable columns:
|
public static WritingColumn<T> NewWritingColumn(DataField field, ReadOnlyMemory<T?> nullableValues, ReadOnlyMemory<int>? repetitionLevels) { |
|
|
|
Validate(field, repetitionLevels); |
|
|
|
// Calculate number of nulls beforehand in a separate cycle in order to consume some memory (we know how much to allocate after that). |
|
// We can also build definition levels in the same cycle. |
|
IMemoryOwner<int> definitionLevelsOwner = MemoryOwner<int>.Allocate(nullableValues.Length); |
|
Span<int> span = definitionLevelsOwner.Memory.Span; |
|
int nullCount = FillDefinionsAndCountNulls(field, nullableValues.Span, ref span); |
|
|
|
// fill non-nulls |
|
int valueCount = nullableValues.Length - nullCount; |
|
IMemoryOwner<T> valuesOwner = MemoryOwner<T>.Allocate(valueCount); |
|
FillNonNullValues(nullableValues.Span, valuesOwner.Memory.Span); |
|
|
|
return new WritingColumn<T>(field, nullableValues.Length, valuesOwner.Memory, valuesOwner, definitionLevelsOwner.Memory, definitionLevelsOwner, repetitionLevels); |
It would be nice for memry alignment purposes to have the possibility to supply these separately, e.g. as a ReadOnlyMemory<T> so we could have e.g. tightly packed doubles in a plain buffer and supply the invalidation data and the null count separately.
API could look like this, for example:
public async Task WriteAsync<T>(DataField field,
ReadOnlyMemory<T> values, // dense values, null stripped
// total value count now would be the definitionLevels length
ReadOnlyMemory<int> definitionLevels, // not nullable so we can have a proper overload without breaking existing APIs
int nullCount, // not sure we need this, maybe for statistics?
ReadOnlyMemory<int>? repetitionLevels = null,
Dictionary<string, string>? customMetadata = null,
CancellationToken cancellationToken = default) where T : struct;
This would also save temporary allocations for the dense values.
What do you think?
Might be a duplicate of #755
Issue description
Hi,
currently the definitionLevels are automatically derived for nullable columns:
parquet-dotnet/src/Parquet/WritingColumn.cs
Lines 29 to 44 in 0b86ddb
It would be nice for memry alignment purposes to have the possibility to supply these separately, e.g. as a
ReadOnlyMemory<T>so we could have e.g. tightly packed doubles in a plain buffer and supply the invalidation data and the null count separately.API could look like this, for example:
This would also save temporary allocations for the dense values.
What do you think?
Might be a duplicate of #755