Skip to content

[Feature request] Allow definitionLevels / nullablity to be user provided #772

Description

@sneusse

Issue description

Hi,

currently the definitionLevels are automatically derived for nullable columns:

public static WritingColumn<T> NewWritingColumn(DataField field, ReadOnlyMemory<T?> nullableValues, ReadOnlyMemory<int>? repetitionLevels) {
Validate(field, repetitionLevels);
// Calculate number of nulls beforehand in a separate cycle in order to consume some memory (we know how much to allocate after that).
// We can also build definition levels in the same cycle.
IMemoryOwner<int> definitionLevelsOwner = MemoryOwner<int>.Allocate(nullableValues.Length);
Span<int> span = definitionLevelsOwner.Memory.Span;
int nullCount = FillDefinionsAndCountNulls(field, nullableValues.Span, ref span);
// fill non-nulls
int valueCount = nullableValues.Length - nullCount;
IMemoryOwner<T> valuesOwner = MemoryOwner<T>.Allocate(valueCount);
FillNonNullValues(nullableValues.Span, valuesOwner.Memory.Span);
return new WritingColumn<T>(field, nullableValues.Length, valuesOwner.Memory, valuesOwner, definitionLevelsOwner.Memory, definitionLevelsOwner, repetitionLevels);

It would be nice for memry alignment purposes to have the possibility to supply these separately, e.g. as a ReadOnlyMemory<T> so we could have e.g. tightly packed doubles in a plain buffer and supply the invalidation data and the null count separately.

API could look like this, for example:

public async Task WriteAsync<T>(DataField field,
    ReadOnlyMemory<T> values, // dense values, null stripped
                              // total value count now would be the definitionLevels length
    ReadOnlyMemory<int> definitionLevels, // not nullable so we can have a proper overload without breaking existing APIs
    int nullCount, // not sure we need this, maybe for statistics?
    ReadOnlyMemory<int>? repetitionLevels = null,
    Dictionary<string, string>? customMetadata = null,
    CancellationToken cancellationToken = default) where T : struct;

This would also save temporary allocations for the dense values.

What do you think?


Might be a duplicate of #755

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions