Add write benchmark of NYC Taxi Dataset. - #685
Conversation
|
This is a really good benchmark and one that I'm going to start using in my own tuned version. I don't know how much I can share here as my version was written on company time and also has a slightly different API and drops support for anything lower than .NET9 but there is a lot of scope for performance improvements. Here are the results for the single dataset |
|
Did both ran with the same runtime ? |
|
Yep same runtime (.NET10) on the same machine at the same time using a long BenchmarkDotNet job. Here is the full text output from the large dataset as well. You mention going parallel as one of your tuning paths, one thing I tried but then ended up going back on was the idea of parallel creation of the columns. The reason for that was that my usage case is to make a lot of parquet files (400+) on a machine with very few CPU cores and I found that by going parallel I ended up jamming up the threadpool so whilst a single benchmark would be a bit faster when used in anger with lots of files it slowed things down. TLDR, these gains are all through coding changes as opposed to going parallel
|
|
Yeah we know there is a lot of room for improvement. Especially since delta encoding was enabled by default, which is CPU heavy on write. This would be my focus for the near future. The issue with these benchmarks is reliability. We don't know which compression is used, what encodings are utilised by each library, what's the file size, how efficient the target file is and so on. This is just testing defaults, which I can change for existing parquet.net and get at least 10x speed increase. It will be more useful to have benchmarks which test two libraries in exactly the same conditions. It's hard to achieve though. |
|
Good point, i'll set the same compression algo for each bench |
|
Agreed performance tuning is a complex subject and it is very easy to "thumb the scales" to get a good looking benchmark that is not reflective of the real world. I haven't focused on benchmarking other libraries just comparing the stock ParquetDotNet (which is excellent BTW) against my modified version. I haven't changed any of the defaults or encoding options so both versions make identical files down to the bit in fact that is one of my tests to make sure I haven't broken anything! |
|
I'd like to check what parquetsharp is using for integer compression as well, thanks for kicking this off. @Kevin-Ross-ECC is using some magic with SIMD. I haven't got as far as that (which I'm really excited about). |
|
Just did a quick test, also this lib generates files that are much smaller (263b vs 57.8kb) with identical physical compression (SNAPPY). ParquetSharp will use RLE_DICTIONARY for integer column: 
whereas this lib defaults to DELTA_BINARY_PACKED: 
So we are comparing two different logical compression algorithms. |
|
I added the logical compression algorithm as a bench parameter, sadly ParquetSharp is a bit uncooperative at outputing RleDictionary, so it will be tonight when I will have the time That's way better compared to ParquetSharp. |
|
@Kuinox it is possible to baseline benchmarks of local codebase to a released version, see https://benchmarkdotnet.org/articles/samples/IntroNuGet.html |
Yes, but it looks like this specific endpoint is deprecated (from what I see in Parquet.Net warnings). |
|
That's good, I always wanted to build something like this for Parquet.Net, which measures perf on each commit to master (inspired by https://github.com/microsoft/Microsoft.IO.RecyclableMemoryStream?tab=readme-ov-file#benchmark-tests): 
|
|
I don't know how it's going to work though without re-testing each commit at once. Action agents allocate random node with unpredictable performance. |
You or someone can host a github action runner on their machine. I do have a NAS at home, and a bare metal server at ovh. They will have less noise, but there are still things running in background, and use old or low power CPUs (and sometimes I do run kind of heavy stuff on it). |
|
I have a good news, and bad news: Now the bad news:
But that's works for another PR. |
# Conflicts: # src/Parquet.PerfRunner/Parquet.PerfRunner.csproj # src/Parquet.PerfRunner/Program.cs
@aloneguid I did some refactoring to handle safely parallelised writing, without changing the external API.
The benchmark rely on this PR, but I can start another PR without the benchmark if you want
|
I'm digging why when we add columns we are suddendly slower than ParquetSharp. |
|
The causes is that, on the read end, we assume that DateTime are local if unspecified, and on the write end, we always write utc DateTime. 
You can see that on the taxi dataset, without specifying any kind of DateTime, it's substracting my UTC offset of the original DateTime, so I think that's a bug. It's too late to dig deeper, but disabling the useless convertion here double the speed, I'll make a proper patch later. |
|
V6 preview (in master) had a lot of improvements, I'd recommend looking into it. |
|
@aloneguid On another project I have some benchmark and I see 50% perf regression for plain encoding (except bool). btw the other project in question is a .NET Parquet lib :p |
|
@Kuinox that's interesting, because all the tests I came up so far show massive performance win v5 vs v6. I'd be interested to see how you benchmark. |
|
this doesn't produce a parquet that use dictionary encoding: #:package Parquet.Net@6.0.0-pre.8
using Parquet;
using Parquet.Schema;
var field = new DataField<int>("value");
var schema = new Parquet.Schema.ParquetSchema(field);
var options = new ParquetOptions
{
CompressionMethod = CompressionMethod.None,
DictionaryEncodingThreshold = 1.0,
DictionaryEncodingSampleSize = 0
};
options.ColumnEncodingHints.Add(field.Path.ToString(), EncodingHint.Dictionary);
var values = Enumerable.Range(0, 4_096).Select(static i => i % 16).ToArray();
await using var stream = File.Create("parquetnet-int32-dictionary-hint.parquet");
await using var writer = await Parquet.ParquetWriter.CreateAsync(schema, stream, options, false);
using var rowGroup = writer.CreateRowGroup();
await rowGroup.WriteAsync<int>(field, values.AsMemory(), null, null, default); |
|
Huh, I asked codex to update & run benchs and it decided to puch the branch by itself 🤡 |
|
The self comparison bench added by codex in this commit have the same result that what I found:
|
|
API in v6 is not compatible with v5, so I'm not quite sure how LLM made those changes work. |
|
It sprinkled with #if a bit everywhere, and it very ugly. I can make it better, but first, do you even want to support benching against the previous version ? |
|
Not really, only wondering why your tests are slower on v6. My approach was to run benchmarks on v5 branch, then on master. Separately one after another on the same machine. |
|
Ok, do you want I keep digging into a more minimal benchmark/dataset to compare the two ? |
|
Not at all, I was just wondering how 5 is slower, I couldn't reproduce this with a test. |
3 participants
Since I want to do work to improve the performance of this library, I figured benching against real world data instead of synthetic data was needed.
This would avoid me writing optimisations for my specific use case and ruin performance for other scenarios.
Sadly the benchmark shows Parquet.Net is up to 2x slower than parquet sharp.
I see two way of improving the performance:
Parallelisation
I believe we can run mosts of the compute intensive stuff in parallel. The users would then just do Task.WhenAll.
Improving the encoding algos
When I profiled for my use case, most of the time was spent on encoding, and a significant portion on deduplicating the input.
I believe there is room for improvement here and easily earn 10-20%.