Skip to content

Link DuckDB statically and retire the self-extracting launcher (#78) - #83

Merged
jrosskopf merged 3 commits into
mainfrom
feat/static-duckdb
Aug 31, 2026
Merged

Link DuckDB statically and retire the self-extracting launcher (#78)#83
jrosskopf merged 3 commits into
mainfrom
feat/static-duckdb

Conversation

@jrosskopf

@jrosskopf jrosskopf commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Closes #78.

CI is green on Linux and Windows (34m49s — Windows compiles DuckDB from
source). One Windows fix was needed: duckdb/CMakeLists.txt sets
CMAKE_MSVC_RUNTIME_LIBRARY unconditionally as a plain variable, so /MT wins
inside its own subdirectory and cannot be overridden from here — mixing it with
our /MD gave LNK2038 for every DuckDB object. The x64-windows-static-md
triplet existed to match the prebuilt DuckDB DLL, which no longer exists, so the
build now uses x64-windows-static. A fully static CRT is also one fewer thing
the target machine needs. Same resolution flapi reached.

What

The launcher existed to put libraries next to the server. After #70/#76 that set
was down to one — libduckdb — so linking it in leaves the launcher with nothing
to do. The shipped artefact is now simply the server.

Gone: launcher/, scripts/bundle.sh, scripts/bundle.ps1,
scripts/stage_runtime.sh, the tar payload, the footer, the first-run unpack of
~84 MB into /tmp, and the per-version temp directories that accumulated there.
Net −515 lines.

DuckDB now comes from a pinned duckdb/ submodule at v1.5.5, built from
source. The prebuilt libduckdb_static.a cannot be linked at all: it leaves
ExtensionHelper::LoadAllExtensions undefined inside the archive. flapi solved
this first and this follows it.

Five traps, each found by something failing

  • Not dummy_static_extension_loader. That target is for the prebuilt
    archive. Here duckdb_generated_extension_loader is what registers the
    statically linked extensions; linking the dummy instead satisfies the loader
    symbol with a no-op, so extensions compile in and never register — surfacing as
    58 unit tests failing on icu operators, naming nothing about extensions.
  • An explicit extension set (core_functions, parquet, icu, json, autocomplete). A bare source build links only the first two; the prebuilt
    libduckdb we shipped before carried the rest.
  • BUILD_UNITTESTS OFF. Not merely slow: with it ON, DuckDB exposes
    internals that are private in a normal build, so code compiles against members
    it has no business touching and breaks the moment the flag flips.
  • DUCKDB_EXPLICIT_VERSION. DuckDB derives its version from git describe,
    which in a shallow submodule clone yields v1.6.0-dev82307 for a checkout
    sitting exactly on a release tag — and that string selects the extension
    repository.
  • v1.5.5, not v1.5.4. At the v1.5.4 tag the in-tree headers disagree with
    DuckDB's own released amalgamation of the same version (private members,
    Identifier rather than string), and its prebuilt httpfs segfaults inside
    LoadInternal when loaded into a source-built engine. v1.5.5 is self-consistent,
    needs no source changes, and is what flapi already runs. src/ is untouched
    by this PR.

Runtime extensions still work

This was the open question, and the answer is yes: autoload and autoinstall stay
on, and the two quack tests — which install a prebuilt extension into the
statically linked engine — pass.

ENABLE_EXPORTS is set on the server target. Measured: httpfs and quack both
load correctly without it, so it is not load-bearing for what erpl-rev uses
today. It is kept because flapi, which loads its whole extension set as prebuilt
binaries, does need it — and an extension that does resolve host symbols fails as
a segfault inside its own LoadInternal, which reads as a broken extension
rather than a missing link flag.

Verification (Linux)

Check Result
unit tests 16567 assertions, 107 cases, zero skipped
quack (prebuilt extension into static engine) passes
ldd no libduckdb
--smoke reports v1.5.5
scripts/e2e.sh 13/13
doctor against A4H from the static binary Ready
build cost ~2 min wall on 32 cores (46 min CPU)

Binary is ~86 MB, against 11.8 MB + a 67 MB libduckdb before — the same total,
as #78 predicted. Size was never the argument; simplicity is.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

jrosskopf and others added 2 commits August 31, 2026 13:56
The launcher existed to put libraries next to the server. After #70/#76
that set was down to one — libduckdb — so linking it in leaves the
launcher with nothing to do. The shipped artefact is now simply the
server: no tar payload, no footer, no first-run unpack of ~84 MB into
/tmp, no per-version temp directories accumulating there.

DuckDB comes from a pinned `duckdb/` submodule at v1.5.5 and is built
from source, because the prebuilt libduckdb_static.a cannot be linked at
all: it leaves ExtensionHelper::LoadAllExtensions undefined *inside the
archive*. flapi solved this first and this follows it.

Deliberate choices, each one found by something failing:

- NOT dummy_static_extension_loader. That target is for the prebuilt
  archive; here the real duckdb_generated_extension_loader is what
  registers the statically linked extensions. Linking the dummy instead
  satisfies the loader symbol with a no-op, so the extensions compile in
  and never register — which surfaces as 58 unit tests failing on icu
  operators, naming nothing about extensions.
- An explicit extension set (core_functions, parquet, icu, json,
  autocomplete). A bare source build links only the first two, while the
  prebuilt libduckdb we used before also carried the rest.
- BUILD_UNITTESTS OFF. Not just slow: with it ON, DuckDB exposes internals
  that are private in a normal build, so code compiles against members it
  has no business touching and breaks when the flag flips.
- DUCKDB_EXPLICIT_VERSION. DuckDB derives its version from `git describe`,
  which in a shallow submodule clone yields "v1.6.0-dev82307" for a
  checkout sitting exactly on a release tag — and that string picks the
  extension repository.
- v1.5.5, not the 1.5.4 we shipped before. At the v1.5.4 tag the in-tree
  headers disagree with DuckDB's own released amalgamation of the same
  version (private members, `Identifier` rather than `string`), and its
  prebuilt httpfs segfaults inside LoadInternal when loaded into a
  source-built engine. v1.5.5 is consistent, needs no source changes, and
  is what flapi already runs.

Runtime extension loading still works: autoload and autoinstall stay on,
and the quack tests — which install a prebuilt extension into the
statically linked engine — pass.

Verified: 16567 assertions in 107 test cases, zero skipped; no libduckdb
in ldd; --smoke reports v1.5.5; e2e 13/13; `doctor` green against A4H
from the static binary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
duckdb/CMakeLists.txt sets CMAKE_MSVC_RUNTIME_LIBRARY unconditionally as a
plain variable, so /MT wins inside its own subdirectory and cannot be
overridden from here. Mixing that with our /MD produced LNK2038
"mismatch detected for 'RuntimeLibrary'" for every DuckDB object.

The x64-windows-static-md triplet existed to match the prebuilt DuckDB
DLL, which no longer exists. x64-windows-static is now both correct and
simpler: a fully static CRT is one fewer thing the target machine needs.

Same resolution flapi reached.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jrosskopf
jrosskopf marked this pull request as ready for review August 31, 2026 14:30
macOS had no CI job at all -- it was built only by the release workflow,
on a tag, which is the worst place to find a platform break. That mattered
less when DuckDB arrived prebuilt; it matters now that we compile it from
source on every platform.

Builds the SDK backend rather than the release's proto backend so it needs
no erpl-proto token: the untested part is that the C++ and the DuckDB build
work on arm64 macOS, not which RFC backend is linked.

Also bumps the workflow's DUCKDB_VERSION to 1.5.5 to match the submodule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jrosskopf
jrosskopf merged commit b7e9850 into main Aug 31, 2026
4 checks passed
@jrosskopf
jrosskopf deleted the feat/static-duckdb branch August 31, 2026 15:50
jrosskopf added a commit that referenced this pull request Sep 1, 2026
release.yml still said 1.5.4 while CMakeLists, the Makefile, CI and the
submodule had all moved to 1.5.5. It is passed as DUCKDB_EXPLICIT_VERSION,
so the release would have stamped the wrong engine version onto the
binary AND sent it looking for extensions built for a version it is not
-- which is precisely the mismatch that segfaults inside a prebuilt
httpfs. Caught reading the workflow before tagging, not by CI: nothing
builds release.yml on a normal push.

Also drops two comments still describing the self-extracting bundle,
which #83 removed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Statically link DuckDB and retire the self-extracting launcher

1 participant