Allow custom length limit for `ElementChunk1<T>[]`

[eduherminio]

Hi Steve,

When implementing 'big arrays' via jagged arrays in the past, something I found useful was being able to test them in 'everyday' machines, aka machines with not enough RAM to be able to reach the Array.MaxLength before hitting the memory limit.

The way to do that was to customize that upper limit from which the 'array of arrays' was needed (which is normally Array.MaxLength), being able to somehow replicate the impact of having n jagged arrays in a 512GB RAM machine, but in machines with 32/64GB RAM.

Any chance you can add some way to configure (reduce) the length limit from which ElementChunk1<T>[] stops being used, to be able to replicate scenarios like the mentioned above?

Thanks for the project!

[hez2010]

Hi! Sorry for the delay.

BigArray<T> is not a jagged array, instead it uses one array whose element type is an inline chunk struct (i.e. the layout is purely flattened instead of nested). The ElementChunk1<T>[] cutoff is an implementation detail tied to whether a normal managed array backing store is enough. Even if I introduced the cutoff, it would only force a different representation to the length instead of being a "jagged array".

[eduherminio]

I'm aware, I was trying to make the parallelism in an attempt to justify my ask: being able to test the performance of your library as a drop-in solution for a jagged array implementation, without having to use too beefy machines RAM-wise to hit territory where length > Array.MaxLength

[hez2010]

I see. If you want to control the shape of underlying storage, you can use this private helper to allocate an array in your desired shape directly:

Hezium.Memory/Hezium.Memory/BigArray.Helpers.cs

Line 2367 in c2fb040

private static Array AllocateArray<TElement>(int chunks, bool pinned, bool uninitialized)

Then construct the BigArray with this constructor:

Hezium.Memory/Hezium.Memory/BigArray.cs

Line 42 in c2fb040

private BigArray(Array storage, nint length)

You will need to make them public so that you can reference them in your benchmark code directly.

[hez2010]

Hi! I've added the benchmark code and result. You can check the updated README.md now.

[eduherminio]

Awesome, benchmarks look very promising!
I'll try to find some time this week to test it in my specific scenario.

[eduherminio]

@hez2010 Could you take a look at lynx-chess/Lynx@197f5b5 and verify that those changes would be enough to mimic the chunks threshold?

---

In case you're curious, these are the actual changes to replace my dual-implementation with your library: lynx-chess/Lynx@13391d3 (+smaller fixes in lynx-chess/Lynx@de79008 and lynx-chess/Lynx@1d6e4de).

I also created this PR, since I'm missing support for pinning in BigArrays. Is that something you'd be willing to merge?

Running some performance tests right now to properly validate the changes on my end.

[hez2010]

Looks promising. I also plan to add a unified way to specify the chunk size together with your PR in the next version. Need a bit of time to come up with a design.

[hez2010]

As said above. The physical memory layout in BigArray will always be flattened and linear no matter what limit you put on the ElementChunk1, so I don't think it makes sense to add a limit to the length.

But for span-based algorithms, adding a configuration of the chunk size being used in processing data seems reasonable to me. For this I have added a Hezium.Memory.MaxProcessingChunkLength process-wide app config.

Now it's live in v1.1.0.

[eduherminio]

As said above. The physical memory layout in BigArray will always be flattened and linear no matter what limit you put on the ElementChunk1, so I don't think it makes sense to add a limit to the length.

I essentially agree, other than for benchmarking purposes. At least I can't think of another reason to ask for it.
And I'm happy with the solution of copying the files locally and changing the right values to benchmark it, so you can essentially close this GH issue.

You might want to create a static variable for it anyway and not expose it, for clarity purposes (to differentiate those Array.MaxLength uses vs the other existing ones) or to facilitate this kind of benchmarking in the future, but that's just some totally optional and extra tidying up.

I'll keep you posted about the adoption of the library here as I finish my tests, and might come up with more suggestions in the following weeks if all goes well (I'll open independent issues for them) 😊

PS: you might want to link the dev.to article in the README, that's how I found the project!

[eduherminio]

Hey, performance tests were successful, so I finally decided to use Hezium.Memory in my project Lynx (a chess engine).
You can see more details in lynx-chess/Lynx#2600 if you're interested.
Thanks for the library and the support!