Update buffer/stage/commit strategy during writes#168
Merged
lnkuiper merged 2 commits intoMay 28, 2026
Merged
Conversation
benfleis
force-pushed
the
adjust-buffer-stage-commit-strategy
branch
from
May 12, 2026 05:47
eec6835 to
512d6f2
Compare
benfleis
marked this pull request as draft
May 12, 2026 09:24
benfleis
force-pushed
the
adjust-buffer-stage-commit-strategy
branch
4 times, most recently
from
May 13, 2026 10:36
7dec63d to
79654cc
Compare
benfleis
force-pushed
the
adjust-buffer-stage-commit-strategy
branch
2 times, most recently
from
May 20, 2026 12:56
12a1c66 to
a4a1b71
Compare
Previous update (commit every 10k stage calls) address 1 Azure limitation, but missed a second limitation. This fixes the full issue of file size limitations mentioned in duckdb/duckdb-delta#299. On top of earlier change to force Commit/Sync before hitting Azure blob limit of 100k staged blocks, go a step further and introduce proper write buffering with fixed-size blocks. - introduces a local fixed size write buffer (same as S3 already does) - by default allows files up to (50k * 8MB) ~= 390 GiB - add buffer size option, allowing Azure's max 4000MiB buffers and 190TiB files
benfleis
force-pushed
the
adjust-buffer-stage-commit-strategy
branch
from
May 20, 2026 14:59
a4a1b71 to
af09f6f
Compare
benfleis
marked this pull request as ready for review
May 20, 2026 15:08
lnkuiper
reviewed
May 28, 2026
lnkuiper
left a comment
lnkuiper
left a comment
Member
There was a problem hiding this comment.
Thanks for the PR! I had Codex cross-reference this with the Azure docs and it thinks there is breakage:
Contributor
|
@benfleis sorry for being a pain, how can i test this ? |
Member
Author
|
@djouallah no worries, it was on my list to test today, but I see now that I need to switch a build config to get nightly back to working in 1.5.3. I'll push that PR and ping you here once it's available! |
benfleis
added a commit
that referenced
this pull request
Jul 13, 2026
**Forward fill v1.5 variegata PRs -> main** Forward-ports: - #169 — force blob (write) sync every 10k appends (Azure 100k staged block limit) - #168 — buffer/stage/commit rewrite: fixed-size local write buffer (mirrors S3), configurable buffer size, fixes DFS sync-offset - #173 — actionlint CI fixes (ccache key, setup-python@v5) Not forward-ported (branch-specific to v1.5-variegata, don't apply to main): #166 (duckdb pin bump to variegata tag), #167 (CI refs pinned to v1.5-variegata / v1.5.3). #162 already on main via separate cherry-picks. **CI/Build** - Needed action version updates + Actionlint fixes - Windows build/image vcvars fix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update buffer/stage/commit strategy during writes
Previous update (commit every 10k stage calls) address 1 Azure
limitation, but missed a second limitation. This fixes the full issue
of file size limitations mentioned in duckdb/duckdb-delta#299.
On top of earlier change to force Commit/Sync before hitting Azure blob
limit of 100k staged blocks, go a step further and introduce proper
write buffering with fixed-size blocks.
190TiB files
Addresses (fully) duckdb/duckdb-delta#299