build: bake the DuckDB extensions into the image - #435
Merged
Conversation
A cold escurel pod downloaded ~137MB of DuckDB extensions before it could
serve anything, and against a Google Drive lake it never finished inside the
substrate's 8m deploy timeout. Measured in the cluster: a cold container ran
7m56s and was still going, while the next start in the SAME pod reached the
catalog in 7 SECONDS off a warm cache.
A PersistentVolume looks like the fix and is not: `helm upgrade --atomic`
deletes the PVC when it rolls the release back, so the cache is destroyed by
the very failure it exists to prevent. It could only help AFTER a deploy had
already succeeded, which was the thing that could not happen.
So the extensions ship in the image: ducklake, postgres, httpfs, fts, vss and
gdrive, installed in a separate stage and copied into the runtime image.
158MB, and a pod now downloads nothing at boot -- which also removes a
boot-time dependency on two extension repositories being reachable.
Verified against the constraints the substrate chart actually imposes, not in
a friendly shell:
* --read-only --network none --user 65532: INSTALL ducklake / postgres /
fts / vss and a path LOAD of gdrive all succeed. No network means the
baked cache is definitely what was used, and read-only means INSTALL
no-ops against a pre-installed extension rather than needing to write.
* The build itself asserts all six land under
$HOME/.duckdb/extensions/<version>/<platform>. DuckDB resolves by that
exact path, so a version or platform mismatch would silently download
again at runtime -- the failure that looks like success.
Two details that cost a build each and are now encoded:
* `INSTALL postgres` is an ALIAS; the artifact it lands is
postgres_scanner.duckdb_extension. Asserting the alias name failed the
build while all six were present.
* HOME is what decides where DuckDB looks. The runtime image sets
HOME=/opt/escurel to match the copy; a deployment that overrides HOME
elsewhere silently reverts to downloading.
gdrive comes from the erpl.io mirror rather than `community`, because
workload identity federation only exists in v2026.09.01 and the community
repository still serves v2026.08.07, whose credential_chain refuses
external_account outright. GDRIVE_REPO is a build arg; switch it once
duckdb/community-extensions#2588 is merged and built.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A cold escurel pod downloaded ~137 MB of DuckDB extensions before it could serve anything, and against a Google Drive lake it never finished inside the substrate's 8m deploy timeout.
Measured in the cluster:
Why a volume was not the fix
helm upgrade --atomicdeletes the PVC when it rolls the release back, so the cache is destroyed by the very failure it exists to prevent. It could only ever help after a deploy had already succeeded — which was the thing that could not happen. That was tried; this replaces it.What this does
ducklake,postgres,httpfs,fts,vssandgdriveare installed in a separate build stage and copied into the runtime image. 158 MB, and a pod downloads nothing at boot — which also removes a boot-time dependency on two extension repositories being reachable.Verified against the constraints the chart imposes, not in a friendly shell
--network noneproves the baked cache is what was used.--read-onlyprovesINSTALLno-ops against a pre-installed extension rather than needing to write — which was the one real risk in this design.The build also asserts all six land under
$HOME/.duckdb/extensions/<version>/<platform>. DuckDB resolves by that exact path, so a version or platform mismatch would silently download again at runtime — the failure that looks like success.Two details that each cost a build, now encoded
INSTALL postgresis an alias; the artifact it lands ispostgres_scanner.duckdb_extension. Asserting the alias name failed the build while all six were present.HOMEdecides where DuckDB looks. The runtime image setsHOME=/opt/escurelto match the copy; a deployment that overridesHOMEelsewhere silently reverts to downloading. (The heron-escurel values do exactly that today and will be updated alongside this.)gdrive source
From the erpl.io mirror, not
community: workload identity federation only exists in v2026.09.01, and the community repository still serves v2026.08.07, whosecredential_chainrefusesexternal_accountoutright.GDRIVE_REPOis a build arg — switch it once duckdb/community-extensions#2588 is merged and built.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.