Link DuckDB statically and retire the self-extracting launcher (#78) - #83
Merged
Conversation
The launcher existed to put libraries next to the server. After #70/#76 that set was down to one — libduckdb — so linking it in leaves the launcher with nothing to do. The shipped artefact is now simply the server: no tar payload, no footer, no first-run unpack of ~84 MB into /tmp, no per-version temp directories accumulating there. DuckDB comes from a pinned `duckdb/` submodule at v1.5.5 and is built from source, because the prebuilt libduckdb_static.a cannot be linked at all: it leaves ExtensionHelper::LoadAllExtensions undefined *inside the archive*. flapi solved this first and this follows it. Deliberate choices, each one found by something failing: - NOT dummy_static_extension_loader. That target is for the prebuilt archive; here the real duckdb_generated_extension_loader is what registers the statically linked extensions. Linking the dummy instead satisfies the loader symbol with a no-op, so the extensions compile in and never register — which surfaces as 58 unit tests failing on icu operators, naming nothing about extensions. - An explicit extension set (core_functions, parquet, icu, json, autocomplete). A bare source build links only the first two, while the prebuilt libduckdb we used before also carried the rest. - BUILD_UNITTESTS OFF. Not just slow: with it ON, DuckDB exposes internals that are private in a normal build, so code compiles against members it has no business touching and breaks when the flag flips. - DUCKDB_EXPLICIT_VERSION. DuckDB derives its version from `git describe`, which in a shallow submodule clone yields "v1.6.0-dev82307" for a checkout sitting exactly on a release tag — and that string picks the extension repository. - v1.5.5, not the 1.5.4 we shipped before. At the v1.5.4 tag the in-tree headers disagree with DuckDB's own released amalgamation of the same version (private members, `Identifier` rather than `string`), and its prebuilt httpfs segfaults inside LoadInternal when loaded into a source-built engine. v1.5.5 is consistent, needs no source changes, and is what flapi already runs. Runtime extension loading still works: autoload and autoinstall stay on, and the quack tests — which install a prebuilt extension into the statically linked engine — pass. Verified: 16567 assertions in 107 test cases, zero skipped; no libduckdb in ldd; --smoke reports v1.5.5; e2e 13/13; `doctor` green against A4H from the static binary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
duckdb/CMakeLists.txt sets CMAKE_MSVC_RUNTIME_LIBRARY unconditionally as a plain variable, so /MT wins inside its own subdirectory and cannot be overridden from here. Mixing that with our /MD produced LNK2038 "mismatch detected for 'RuntimeLibrary'" for every DuckDB object. The x64-windows-static-md triplet existed to match the prebuilt DuckDB DLL, which no longer exists. x64-windows-static is now both correct and simpler: a fully static CRT is one fewer thing the target machine needs. Same resolution flapi reached. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jrosskopf
marked this pull request as ready for review
August 31, 2026 14:30
macOS had no CI job at all -- it was built only by the release workflow, on a tag, which is the worst place to find a platform break. That mattered less when DuckDB arrived prebuilt; it matters now that we compile it from source on every platform. Builds the SDK backend rather than the release's proto backend so it needs no erpl-proto token: the untested part is that the C++ and the DuckDB build work on arm64 macOS, not which RFC backend is linked. Also bumps the workflow's DUCKDB_VERSION to 1.5.5 to match the submodule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jrosskopf
added a commit
that referenced
this pull request
Sep 1, 2026
release.yml still said 1.5.4 while CMakeLists, the Makefile, CI and the submodule had all moved to 1.5.5. It is passed as DUCKDB_EXPLICIT_VERSION, so the release would have stamped the wrong engine version onto the binary AND sent it looking for extensions built for a version it is not -- which is precisely the mismatch that segfaults inside a prebuilt httpfs. Caught reading the workflow before tagging, not by CI: nothing builds release.yml on a normal push. Also drops two comments still describing the self-extracting bundle, which #83 removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #78.
CI is green on Linux and Windows (34m49s — Windows compiles DuckDB from
source). One Windows fix was needed:
duckdb/CMakeLists.txtsetsCMAKE_MSVC_RUNTIME_LIBRARYunconditionally as a plain variable, so/MTwinsinside its own subdirectory and cannot be overridden from here — mixing it with
our
/MDgaveLNK2038for every DuckDB object. Thex64-windows-static-mdtriplet existed to match the prebuilt DuckDB DLL, which no longer exists, so the
build now uses
x64-windows-static. A fully static CRT is also one fewer thingthe target machine needs. Same resolution flapi reached.
What
The launcher existed to put libraries next to the server. After #70/#76 that set
was down to one —
libduckdb— so linking it in leaves the launcher with nothingto do. The shipped artefact is now simply the server.
Gone:
launcher/,scripts/bundle.sh,scripts/bundle.ps1,scripts/stage_runtime.sh, the tar payload, the footer, the first-run unpack of~84 MB into
/tmp, and the per-version temp directories that accumulated there.Net −515 lines.
DuckDB now comes from a pinned
duckdb/submodule at v1.5.5, built fromsource. The prebuilt
libduckdb_static.acannot be linked at all: it leavesExtensionHelper::LoadAllExtensionsundefined inside the archive. flapi solvedthis first and this follows it.
Five traps, each found by something failing
dummy_static_extension_loader. That target is for the prebuiltarchive. Here
duckdb_generated_extension_loaderis what registers thestatically linked extensions; linking the dummy instead satisfies the loader
symbol with a no-op, so extensions compile in and never register — surfacing as
58 unit tests failing on icu operators, naming nothing about extensions.
core_functions, parquet, icu, json, autocomplete). A bare source build links only the first two; the prebuiltlibduckdb we shipped before carried the rest.
BUILD_UNITTESTS OFF. Not merely slow: with it ON, DuckDB exposesinternals that are private in a normal build, so code compiles against members
it has no business touching and breaks the moment the flag flips.
DUCKDB_EXPLICIT_VERSION. DuckDB derives its version fromgit describe,which in a shallow submodule clone yields
v1.6.0-dev82307for a checkoutsitting exactly on a release tag — and that string selects the extension
repository.
DuckDB's own released amalgamation of the same version (private members,
Identifierrather thanstring), and its prebuilthttpfssegfaults insideLoadInternalwhen loaded into a source-built engine. v1.5.5 is self-consistent,needs no source changes, and is what flapi already runs.
src/is untouchedby this PR.
Runtime extensions still work
This was the open question, and the answer is yes: autoload and autoinstall stay
on, and the two quack tests — which install a prebuilt extension into the
statically linked engine — pass.
ENABLE_EXPORTSis set on the server target. Measured: httpfs and quack bothload correctly without it, so it is not load-bearing for what erpl-rev uses
today. It is kept because flapi, which loads its whole extension set as prebuilt
binaries, does need it — and an extension that does resolve host symbols fails as
a segfault inside its own
LoadInternal, which reads as a broken extensionrather than a missing link flag.
Verification (Linux)
lddlibduckdb--smokev1.5.5scripts/e2e.shdoctoragainst A4H from the static binaryBinary is ~86 MB, against 11.8 MB + a 67 MB
libduckdbbefore — the same total,as #78 predicted. Size was never the argument; simplicity is.
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.