Setting max_unflushed_gb too high creates PUTs that are too large

[jjsp556]

In an attempt to increase write performance, I tried setting max_unflushed_gb = 16.0 in version 1.0.8. I then saw error messages in the log that indicated that zerofs was attempting to perform a PUT that was a little larger than the 5 GiB S3 limit for a single PUT. It seems that either zerofs could attempt to keep individual PUTs at 5 GiB or less, or there should be a max value for max_unflushed_gb enforced so that this issue is avoided.

[Barre]

Can you share your zerofs configuration as well as logs?

[jjsp556]

Unfortunately, I am not in a position to directly copy config and logs. But this is a repeatable error for me. The steps to re-create are to set max_unflushed_gb = 16.0, mount an ndb block device, and then copy a large file to the nbd block device (e.g. aws s3 cp s3://bucket/large40GBfile.bin /path/to/mounted/nbd/device/). Here is some relevant info tht I've transcribed:

zerofs.toml
[cache]
dir = "/cache"
disk_size_gb = 800.0
memory_size_gb = 20.0

[storage]
url = "s3://mybucket/zerofs" # this is on AWS
encryption_password = "..."

[servers.nbd]
unix_socket = "/tmp/zerofs-nbd.sock"

[lsm]
max_unflushed_gb = 16.0
max_concurrent_compactions = 12
flush_interval_secs = 120

Error message:
INFO slatedb::retrying_object_store: retrying object store operation [error=Generic { store: "S3", source: RetryError(RetryErrorImpl { method: PUT uri: Some(https:??mybucket/zerofs/wal/000000000000000xxxx.sst), retries: 0, max_retries: 10, elapsed: xxxxs, retry_timeout: 180s, inner: Status { status: 400,body: Some(<Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed size</Message><ProposedSize>6xxxxxxxxx</ProposedSize><MaxAllowedSize>5368709120</MaxAllowedSize>.....)]

It looks like a wal file is larger than the max allowed size of 5GiB.