Perf bigdecimals

[rferraton]

Summary

Refactors BigDecimal encoding to use stack-allocated Span<byte> buffers and a new WriteBytes method,
eliminating intermediate byte[] allocations and Array.Copy calls in the hot encoding path.
Also adds comprehensive test coverage for BigDecimal values.

Changes

Parquet/Data/BigDecimal.cs

• Added WriteBytes(Span<byte>) method that writes the unscaled value directly into a caller-provided
  buffer using BigInteger.TryWriteBytes, avoiding the heap allocation from BigInteger.ToByteArray().
• Replaced the old GetBytes() implementation (which used ToByteArray() + Array.Copy + Enumerable.Reverse)
  with a thin wrapper over WriteBytes.
• Sign extension is now done via Span.Fill(0xFF) / Span.Fill(0) instead of a manual loop.
• Big-endian conversion is done in-place via Span.Reverse() instead of Enumerable.Reverse().ToArray().

Parquet/Encodings/ParquetPlainEncoder.cs

• Encode(ReadOnlySpan<BigDecimal>, Stream, SchemaElement) now uses a stackalloc byte[16] buffer
  and calls WriteBytes instead of GetBytes(), removing per-element heap allocations.
• Encode(ReadOnlySpan<decimal>, Stream, SchemaElement) for FIXED_LEN_BYTE_ARRAY case updated similarly.
• SchemaElement.TypeLength is set from WriteBytes return value.

Parquet.Test/DataTypes/BigDecimalTest.cs

• Expanded from 2 to 70+ test cases organized as [Theory] with [InlineData]:
  • GetBytes_roundtrip_preserves_value — 40 cases: encode → reverse → reconstruct BigInteger,
    covering zero, small/medium/large, positive/negative, no scale/with scale/high scale, all precision tiers (1–38).
  • WriteBytes_returns_correct_size_for_precision — 10 cases: validates buffer size for each precision tier.
  • WriteBytes_zero_produces_all_zero_bytes — 3 cases: zero encoding at different precisions.
  • WriteBytes_negative_value_has_sign_extension — 3 cases: leading byte has sign bit set.
  • WriteBytes_positive_value_has_no_sign_bit — 3 cases: leading byte has no sign bit.
  • WriteBytes_buffer_too_small_throws — error handling.
  • ToString_formats_correctly — 11 cases: string formatting with ±, zeros, scale/no-scale.

Parquet.Test/Types/EndToEndTypeTest.cs

• Expanded BigDecimal end-to-end tests to 30+ cases via a single [Theory]:
  • Type_bigdecimal_write_read_roundtrip — full parquet file write → read roundtrip covering the
    same value matrix (zero, positive/negative, small/large, all precision tiers, high scale).

Motivation

The original GetBytes() path performed 3 heap allocations per value during encoding
(ToByteArray(), new byte[] result, Reverse().ToArray()). For columns with millions of decimal
values this creates significant GC pressure. The new path uses a single stackalloc buffer per batch,
writing directly via TryWriteBytes and reversing in-place.

Test Results

All 161 related tests pass on both .NET 8 and .NET 10:

• 70+ unit tests in BigDecimalTest
• 30+ end-to-end parquet roundtrip tests in EndToEndTypeTest
• All pre-existing tests unaffected (590 total, 587 pass, 3 skipped as before)

[aloneguid]

Looks interesting. Out of interest, did you run performance benchmarks to see what we gain?

[rferraton]

Benchmark Results

Here's the key takeaway from the results:
WriteBytes(Span) vs GetBytes() — per-element comparison

Size	Precision	GetBytes (heap)	WriteBytes (stack)	Speedup	Memory saved
1M	10	16,573 µs / 32 MB	14,111 µs / 0 B	15% faster	100%
1M	38	15,255 µs / 40 MB	14,272 µs / 0 B	~7% faster	100%
100K	10	1,635 µs / 3.2 MB	1,407 µs / 0 B	14% faster	100%
100K	38	1,477 µs / 4 MB	1,224 µs / 0 B	17% faster	100%

Key findings

1. WriteBytes(Span) allocates exactly 0 bytes — zero GC pressure, no Gen0/Gen1/Gen2 collections
2. GetBytes() allocates 32–40 bytes per element (array for result), causing heavy GC churn at scale
3. Speed improvement is 7–17% depending on precision and data size
4. The full parquet write path (Encode()) benefits from this since it now uses WriteBytes(Span) internally with a stackalloc buffer instead of calling GetBytes() per element

Files created/modified

• New: BigDecimalBenchmarks.cs — 4 benchmarks × 6 parameter combos = 24 runs
• Modified: Program.cs — added "bigdecimal" command

[aloneguid]

Looks good, thank you very much!