feat(duckvfs): install the extension from the community repository - #433
Merged
Conversation
The duckvfs lane store could not be deployed. `extension_path` names a LOCALLY BUILT gdrive.duckdb_extension — fine on a developer's machine, impossible in a container — and it is the only way to register write_blob/remove_file/file_size, which live in the extension rather than DuckDB core and are needed for EVERY root, not only `gdrive://`. Leaving it unset does not help, and is the worse failure: `None` SKIPS the load and assumes the functions are present. The store then opens cleanly and the first WRITE fails with "Scalar Function with name write_blob does not exist". A boot that succeeds and a write path that cannot work is exactly the shape that reaches production looking healthy. gdrive now ships in the DuckDB community repository, which duckvfs.rs already anticipated in a comment. `extension_repo` is that: `community`, or a repository URL. Wired through as ESCUREL_STORAGE_DUCKVFS_EXTENSION_REPO. A NEW field rather than repurposing `extension_path: None`. Redefining None would change behaviour for existing `file://` roots and force a network install where none happens today; `extension_path` keeps its exact meaning and still wins, so an operator pointing at a local build gets that build. The test asserts a WRITE round trip, not that the constructor returns Ok: `new` performs no I/O against the root, so a store that never loaded the extension constructs perfectly. It takes no ESCUREL_TEST_GDRIVE_EXTENSION, because needing a prebuilt file is the thing being removed. Two checks, and the second is the one that mattered. Mutating extension_repo to None fails it with the missing-write_blob error, so it is load-bearing. And it first passed in 0.65s — too fast for a download, because gdrive.duckdb_extension was already in ~/.duckdb from local use. Re-run with a cold HOME it downloads, populates the fresh cache and passes, which is the container case; the deployment already sets HOME to a writable dir. Without that check this would have been green here and broken everywhere else. 1377 passed, 0 failed. clippy clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
duckvfslane store could not be deployed to a container. This makes it deployable.The problem
extension_pathnames a locally builtgdrive.duckdb_extension. It is the only way to registerwrite_blob/remove_file/file_size, which live in the extension rather than DuckDB core and are needed for every root, not onlygdrive://. A pod has no such file.Leaving it unset is the worse failure, not an escape:
Noneskips the load and assumes the functions are present. The store then opens cleanly and the first write fails withCatalog Error: Scalar Function with name write_blob does not exist!. A boot that succeeds and a write path that cannot work is the shape that reaches production looking healthy.The change
gdrivenow ships in the DuckDB community repository — whichduckvfs.rsalready anticipated in a comment ("Once gdrive is in the community repository this becomesINSTALL gdrive FROM community; LOAD gdrive;").extension_repois that, asESCUREL_STORAGE_DUCKVFS_EXTENSION_REPO:community, or a repository URL.A new field, not a redefinition of
extension_path: None. RepurposingNonewould change behaviour for existingfile://roots and force a network install where none happens today.extension_pathkeeps its exact meaning and still takes precedence, so an operator pointing at a local build always gets that build.One detail encoded in the code:
communityis a keyword in DuckDB's grammar and must not be quoted, while a URL must be. Quoting the keyword makes DuckDB look for a repository literally named"community"and fail with a message that says nothing about quoting.Verification
Red → green, plus two checks — the second is the one that mattered.
extension_repotoNonefails the test with the missing-write_bloberror. It asserts a write round trip, not that the constructor returnsOk—newperforms no I/O against the root, so a store that never loaded the extension constructs perfectly.gdrive.duckdb_extensionwas already in~/.duckdb/extensions/v1.5.5/linux_amd64/from local use. Re-run with a coldHOMEit genuinely downloads, populates the fresh cache and passes. That is the container case; the deployment already setsHOMEto a writable dir. Without this check the test would have been green here and broken everywhere it is actually needed.The new test deliberately takes no
ESCUREL_TEST_GDRIVE_EXTENSION— needing a prebuilt file is the thing being removed. It does need network access to the community repository.1377 passed, 0 failed, clippy clean. API contract unchanged.Why now
This unblocks running an Escurel in the cluster against the existing gdrive-backed lake (the
datazoo-loopscorpus) rather than moving that data. The remaining pieces are a WIF credential for the Drive, a catalog DSN secret, and the substrate app.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.