From 27dc28a6722ff7ac88aed9212a10ccdb8c8d9ee1 Mon Sep 17 00:00:00 2001 From: Joachim Rosskopf Date: Mon, 31 Aug 2026 13:56:26 +0200 Subject: [PATCH 1/3] Link DuckDB statically and retire the self-extracting launcher MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The launcher existed to put libraries next to the server. After #70/#76 that set was down to one — libduckdb — so linking it in leaves the launcher with nothing to do. The shipped artefact is now simply the server: no tar payload, no footer, no first-run unpack of ~84 MB into /tmp, no per-version temp directories accumulating there. DuckDB comes from a pinned `duckdb/` submodule at v1.5.5 and is built from source, because the prebuilt libduckdb_static.a cannot be linked at all: it leaves ExtensionHelper::LoadAllExtensions undefined *inside the archive*. flapi solved this first and this follows it. Deliberate choices, each one found by something failing: - NOT dummy_static_extension_loader. That target is for the prebuilt archive; here the real duckdb_generated_extension_loader is what registers the statically linked extensions. Linking the dummy instead satisfies the loader symbol with a no-op, so the extensions compile in and never register — which surfaces as 58 unit tests failing on icu operators, naming nothing about extensions. - An explicit extension set (core_functions, parquet, icu, json, autocomplete). A bare source build links only the first two, while the prebuilt libduckdb we used before also carried the rest. - BUILD_UNITTESTS OFF. Not just slow: with it ON, DuckDB exposes internals that are private in a normal build, so code compiles against members it has no business touching and breaks when the flag flips. - DUCKDB_EXPLICIT_VERSION. DuckDB derives its version from `git describe`, which in a shallow submodule clone yields "v1.6.0-dev82307" for a checkout sitting exactly on a release tag — and that string picks the extension repository. - v1.5.5, not the 1.5.4 we shipped before. At the v1.5.4 tag the in-tree headers disagree with DuckDB's own released amalgamation of the same version (private members, `Identifier` rather than `string`), and its prebuilt httpfs segfaults inside LoadInternal when loaded into a source-built engine. v1.5.5 is consistent, needs no source changes, and is what flapi already runs. Runtime extension loading still works: autoload and autoinstall stay on, and the quack tests — which install a prebuilt extension into the statically linked engine — pass. Verified: 16567 assertions in 107 test cases, zero skipped; no libduckdb in ldd; --smoke reports v1.5.5; e2e 13/13; `doctor` green against A4H from the static binary. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 29 ++-- .github/workflows/release.yml | 65 ++++---- .gitmodules | 3 + CMakeLists.txt | 136 +++++++++++------ Makefile | 32 ++-- cmake/duckdb_extensions.cmake | 16 ++ duckdb | 1 + launcher/erpl_rev_launch.cpp | 280 ---------------------------------- scripts/bundle.ps1 | 51 ------- scripts/bundle.sh | 31 ---- scripts/stage_runtime.sh | 41 ----- 11 files changed, 170 insertions(+), 515 deletions(-) create mode 100644 cmake/duckdb_extensions.cmake create mode 160000 duckdb delete mode 100644 launcher/erpl_rev_launch.cpp delete mode 100644 scripts/bundle.ps1 delete mode 100755 scripts/bundle.sh delete mode 100755 scripts/stage_runtime.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 665e5da..85e1b1f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -22,7 +22,7 @@ jobs: steps: - uses: actions/checkout@v4 with: - submodules: recursive # third_party/posthog-telemetry (telemetry lib) + submodules: recursive # duckdb (linked statically) + posthog-telemetry - name: Install build tools run: sudo apt-get update && sudo apt-get install -y ninja-build unzip @@ -55,13 +55,13 @@ jobs: build-windows: # The release Windows bundle is built only on tags, so Windows-only breaks # used to slip to the release. Build the same x64-windows-static-md config - # (server + launcher + tests) AND run the suite here on every push/PR. + # (server + tests) AND run the suite here on every push/PR. name: Build server + run tests (Windows) runs-on: windows-latest steps: - uses: actions/checkout@v4 with: - submodules: recursive # third_party/posthog-telemetry (telemetry lib) + submodules: recursive # duckdb (linked statically) + posthog-telemetry - name: Setup vcpkg uses: lukka/run-vcpkg@v11 @@ -78,26 +78,17 @@ jobs: - name: Download SAP NW RFC SDK (Windows) run: .\scripts\download_and_extract_nwrfc.ps1 's3://erpl-resources/sapnwrfc/nwrfc750P_13-70002755_win.zip' '.\nwrfcsdk\win\' - - name: Fetch DuckDB - run: | - curl.exe -sL --fail -o duckdb.zip "https://github.com/duckdb/duckdb/releases/download/v$env:DUCKDB_VERSION/libduckdb-windows-amd64.zip" - New-Item -ItemType Directory -Force -Path "vendor\duckdb-$env:DUCKDB_VERSION" | Out-Null - Expand-Archive -Force duckdb.zip -DestinationPath "vendor\duckdb-$env:DUCKDB_VERSION" - - - name: Build (server + launcher + tests) + - name: Build (server + tests) run: | - # x64-windows-static-md mirrors the release build (static OpenSSL, dynamic - # CRT to match the prebuilt DuckDB DLL). Builds all targets incl. tests, so - # an MSVC-only compile error fails here instead of at release time. + # x64-windows-static-md mirrors the release build: static OpenSSL, + # dynamic CRT (/MD), DuckDB compiled from the pinned submodule and + # linked in. Builds all targets incl. tests, so an MSVC-only compile + # error fails here instead of at release time. cmake -S . -B build -DCMAKE_BUILD_TYPE=Release ` -DCMAKE_TOOLCHAIN_FILE="$env:VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" ` -DVCPKG_TARGET_TRIPLET=x64-windows-static-md ` - -DDUCKDB_VERSION="$env:DUCKDB_VERSION" ` - -DDUCKDB_DIST="$PWD/vendor/duckdb-$env:DUCKDB_VERSION" + -DDUCKDB_VERSION="$env:DUCKDB_VERSION" cmake --build build --config Release - name: Run tests - run: | - # The test exe needs duckdb.dll on PATH (it doesn't link the SAP libs). - $env:PATH = "$PWD\vendor\duckdb-$env:DUCKDB_VERSION;$env:PATH" - .\build\Release\erpl_rev_tests.exe + run: .\build\Release\erpl_rev_tests.exe diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b38f0cc..a856224 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -34,15 +34,15 @@ jobs: # answer RfcGetVersion (erpl-proto's `cross` CI job plus the smoke test # below) -- but NOT yet against a live SAP system. See erpl-rev#75. - { os: ubuntu-latest, sub: linux, plat: linux, triplet: x64-linux, - sdk: nwrfc750P_13-70002752_linux.zip, duckdb: libduckdb-linux-amd64.zip, + sdk: nwrfc750P_13-70002752_linux.zip, asset: erpl-rev-linux-amd64, rfc: proto } - { os: macos-14, sub: osx, plat: osx, triplet: arm64-osx, - sdk: nwrfc750P_13-80008131_osx_arm.zip, duckdb: libduckdb-osx-universal.zip, + sdk: nwrfc750P_13-80008131_osx_arm.zip, asset: erpl-rev-macos-arm64, rfc: proto } steps: - uses: actions/checkout@v4 with: - submodules: recursive # third_party/posthog-telemetry (telemetry lib) + submodules: recursive # duckdb (linked statically) + posthog-telemetry - name: Install build tools run: | @@ -98,20 +98,15 @@ jobs: if: matrix.rfc == 'proto' run: cargo build --release -p erpl-proto-nwrfc --manifest-path .erpl-proto/Cargo.toml - - name: Fetch DuckDB - run: | - curl -sL --fail -o duckdb.zip "https://github.com/duckdb/duckdb/releases/download/v${DUCKDB_VERSION}/${{ matrix.duckdb }}" - mkdir -p "vendor/duckdb-${DUCKDB_VERSION}" - unzip -o duckdb.zip -d "vendor/duckdb-${DUCKDB_VERSION}" - - - name: Build (server + launcher) + # DuckDB is built from the pinned `duckdb/` submodule and linked in, so + # there is no release zip to fetch and nothing to stage beside the binary. + - name: Build (static DuckDB) run: | cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" \ -DVCPKG_TARGET_TRIPLET=${{ matrix.triplet }} \ -DERPL_REV_VERSION="${GITHUB_REF_NAME#v}" \ -DDUCKDB_VERSION="${DUCKDB_VERSION}" \ - -DDUCKDB_DIST="$PWD/vendor/duckdb-${DUCKDB_VERSION}" \ -DRFC_BACKEND="${RFC_BACKEND}" -DRFC_LINK="${RFC_LINK}" \ -DERPL_PROTO_ROOT="$PWD/.erpl-proto" cmake --build build @@ -119,16 +114,13 @@ jobs: RFC_BACKEND: ${{ matrix.rfc }} RFC_LINK: ${{ matrix.rfc == 'proto' && 'static' || 'shared' }} - # "-" as the SDK lib dir means "statically linked RFC backend, nothing - # to stage": the proto bundle carries DuckDB and nothing else. - - name: Assemble single-file bundle + - name: Stage the binary as the release asset run: | - SDKLIB="nwrfcsdk/${{ matrix.sub }}/lib" - [ "${{ matrix.rfc }}" = proto ] && SDKLIB="-" - ./scripts/bundle.sh ${{ matrix.plat }} build/erpl_rev_server build/erpl_rev_launch \ - "$SDKLIB" "vendor/duckdb-${DUCKDB_VERSION}" "dist/${{ matrix.asset }}" + mkdir -p dist + cp build/erpl_rev_server "dist/${{ matrix.asset }}" + chmod +x "dist/${{ matrix.asset }}" - - name: Assert the proto bundle links no SAP library + - name: Assert the binary links no SAP or DuckDB library if: matrix.rfc == 'proto' shell: bash run: | @@ -143,11 +135,14 @@ jobs: deps="$(ldd build/erpl_rev_server)" fi if grep -Ei 'sapnwrfc|libsapucum|libicu' <<<"$deps"; then - echo "::error::proto bundle still links an RFC or ICU library"; exit 1 + echo "::error::binary still links an RFC or ICU library"; exit 1 + fi + if grep -Ei 'libduckdb' <<<"$deps"; then + echo "::error::DuckDB is meant to be linked statically"; exit 1 fi - echo "no RFC or ICU shared object; DuckDB is the only bundled library" + echo "no RFC, ICU or DuckDB shared object; the binary stands alone" - - name: Smoke test (self-extract, no external libs) + - name: Smoke test (no external libs) run: env -u LD_LIBRARY_PATH -u DYLD_LIBRARY_PATH "./dist/${{ matrix.asset }}" --smoke - name: Package (tar.gz preserves the exec bit) @@ -158,7 +153,7 @@ jobs: name: ${{ matrix.asset }}.tar.gz path: dist/${{ matrix.asset }}.tar.gz - # The wheel wraps the raw self-extracting binary, not the tarball. + # The wheel wraps the raw binary, not the tarball. - uses: actions/upload-artifact@v4 with: name: binary-${{ matrix.asset }} @@ -170,7 +165,7 @@ jobs: steps: - uses: actions/checkout@v4 with: - submodules: recursive # third_party/posthog-telemetry (telemetry lib) + submodules: recursive # duckdb (linked statically) + posthog-telemetry - name: Setup vcpkg uses: lukka/run-vcpkg@v11 @@ -199,32 +194,26 @@ jobs: - name: Build the pure-Rust RFC shim (static) run: cargo build --release -p erpl-proto-nwrfc --manifest-path .erpl-proto/Cargo.toml - - name: Fetch DuckDB - run: | - curl.exe -sL --fail -o duckdb.zip "https://github.com/duckdb/duckdb/releases/download/v$env:DUCKDB_VERSION/libduckdb-windows-amd64.zip" - New-Item -ItemType Directory -Force -Path "vendor\duckdb-$env:DUCKDB_VERSION" | Out-Null - Expand-Archive -Force duckdb.zip -DestinationPath "vendor\duckdb-$env:DUCKDB_VERSION" - - - name: Build (server + launcher) + - name: Build (static DuckDB) run: | # x64-windows-static-md: statically link OpenSSL (no libssl/libcrypto - # DLLs in the single-file bundle) while keeping the dynamic CRT (/MD) - # to match the prebuilt DuckDB DLL. + # DLLs beside the exe) while keeping the dynamic CRT (/MD), which is + # what DuckDB's own build expects. cmake -S . -B build -DCMAKE_BUILD_TYPE=Release ` -DCMAKE_TOOLCHAIN_FILE="$env:VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" ` -DVCPKG_TARGET_TRIPLET=x64-windows-static-md ` -DERPL_REV_VERSION="$($env:GITHUB_REF_NAME -replace '^v','')" ` -DDUCKDB_VERSION="$env:DUCKDB_VERSION" ` - -DDUCKDB_DIST="$PWD/vendor/duckdb-$env:DUCKDB_VERSION" ` -DRFC_BACKEND=proto -DRFC_LINK=static ` -DERPL_PROTO_ROOT="$PWD/.erpl-proto" cmake --build build --config Release - - name: Assemble single-file bundle - # -SdkLib '-' : the shim is inside the exe, so the payload is DuckDB alone. - run: .\scripts\bundle.ps1 -Server build\Release\erpl_rev_server.exe -Launcher build\Release\erpl_rev_launch.exe -SdkLib '-' -DuckdbDir "vendor\duckdb-$env:DUCKDB_VERSION" -Out dist\erpl-rev-windows-amd64.exe + - name: Stage the binary as the release asset + run: | + New-Item -ItemType Directory -Force -Path dist | Out-Null + Copy-Item build\Release\erpl_rev_server.exe dist\erpl-rev-windows-amd64.exe - - name: Assert the proto bundle links no SAP library + - name: Assert the binary links no SAP or DuckDB library shell: pwsh run: | # dumpbin is the Windows ldd, but cl.exe is not on PATH in this job, so diff --git a/.gitmodules b/.gitmodules index 6b7f799..7291b80 100644 --- a/.gitmodules +++ b/.gitmodules @@ -4,3 +4,6 @@ [submodule "third_party/datazoo-banner"] path = third_party/datazoo-banner url = https://github.com/DataZooDE/duckdb-extension-banner.git +[submodule "duckdb"] + path = duckdb + url = https://github.com/duckdb/duckdb.git diff --git a/CMakeLists.txt b/CMakeLists.txt index aac489f..ccb57fb 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -107,30 +107,90 @@ else() message(STATUS "erpl-rev RFC backend: sdk (${SAPNWRFC_HOME})") endif() -# --- DuckDB (self-contained dist: libduckdb + amalgamated duckdb.hpp) -------- -set(DUCKDB_DIST "" CACHE PATH "Standalone DuckDB dist dir (libduckdb + duckdb.hpp)") -if(NOT DUCKDB_DIST) - message(FATAL_ERROR "DUCKDB_DIST not set; run `make duckdb-dist` or pass -DDUCKDB_DIST=") +# --- DuckDB ----------------------------------------------------------------- +# Static by default, from the pinned `duckdb/` submodule. +# +# The prebuilt libduckdb_static.a in DuckDB's release zip cannot be linked: it +# leaves duckdb::ExtensionHelper::LoadAllExtensions undefined *inside the +# archive*, expecting extension libraries the zip does not ship. +# `dummy_static_extension_loader`, a target of the DuckDB build itself, is what +# provides that symbol -- which is why this needs the source, not the release. +# +# Autoloading and autoinstall stay ON: the static loader covers *statically +# linked* extensions (there are none), and the server still installs `quack` and +# opportunistically loads `httpfs` at runtime. Turning these off would build +# fine and then fail at the first `INSTALL quack`. +# Reported by --smoke and stamped into telemetry; the engine itself comes from +# the submodule, so this is a label, not a selector. +if(NOT DEFINED DUCKDB_VERSION OR DUCKDB_VERSION STREQUAL "") + set(DUCKDB_VERSION "1.5.5") endif() -set(DUCKDB_INCLUDE "${DUCKDB_DIST}") + +# DuckDB is built from the pinned `duckdb/` submodule and linked statically. +# There is deliberately no prebuilt-libduckdb path: DuckDB's release +# amalgamation and its in-tree headers do not agree on their own API (result +# column names are `string` in one and `Identifier` in the other), so supporting +# both means a compatibility shim on every call site -- for a path CI would +# never exercise and which would rot unnoticed. One engine, always tested. +if(NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/duckdb/CMakeLists.txt") + message(FATAL_ERROR + "duckdb/ submodule is empty -- run `git submodule update --init --recursive`.") +endif() + +# We want the engine, not DuckDB's own shell, unit tests or benchmarks. Those +# add minutes per platform per build, and its test tree resolves a relative +# source path against *our* source dir, so configuring it fails outright. +# +# BUILD_UNITTESTS in particular is not merely slow: with it ON, DuckDB exposes +# internals that are private in a normal build, so code can compile against +# members it has no business touching and then break the moment the flag flips. +# DuckDB derives its version string from `git describe`. In a shallow submodule +# clone that produces nonsense -- "v1.6.0-dev82307" for a checkout sitting +# exactly on the v1.5.4 tag -- which then goes into --smoke output, telemetry and +# the extension-repository path. State it instead of deriving it. +set(DUCKDB_EXPLICIT_VERSION "v${DUCKDB_VERSION}" CACHE STRING "" FORCE) + +set(BUILD_UNITTESTS OFF CACHE BOOL "" FORCE) +set(BUILD_SHELL OFF CACHE BOOL "" FORCE) +set(BUILD_BENCHMARKS OFF CACHE BOOL "" FORCE) + +# Autoloading and autoinstall stay ON: the statically linked set below is the +# engine's built-in extensions, while the server still installs `quack` and +# opportunistically loads `httpfs` at runtime. +set(ENABLE_EXTENSION_AUTOLOADING TRUE) +set(ENABLE_EXTENSION_AUTOINSTALL TRUE) +set(DUCKDB_EXTENSION_CONFIGS "${CMAKE_CURRENT_SOURCE_DIR}/cmake/duckdb_extensions.cmake") if(WIN32) - set(DUCKDB_LIB "${DUCKDB_DIST}/duckdb.lib") # import lib; duckdb.dll bundled at runtime -elseif(APPLE) - set(DUCKDB_LIB "${DUCKDB_DIST}/libduckdb.dylib") -else() - set(DUCKDB_LIB "${DUCKDB_DIST}/libduckdb.so") + set(DUCKDB_EXPLICIT_PLATFORM "windows_amd64") endif() +add_subdirectory(duckdb EXCLUDE_FROM_ALL) +set(DUCKDB_INCLUDE "${CMAKE_CURRENT_SOURCE_DIR}/duckdb/src/include") + +# duckdb_static, then the real generated loader, then the extension archives it +# calls into. NOT dummy_static_extension_loader: that target exists for the +# prebuilt libduckdb_static.a, which ships without a loader. Linking it here +# satisfies the loader symbol with a no-op, so the extensions below compile in +# and then never register -- which surfaces as 58 unit tests failing on icu +# operators, not as anything mentioning extensions. +set(DUCKDB_LIB + duckdb_static + duckdb_generated_extension_loader + core_functions_extension + parquet_extension + icu_extension + json_extension + autocomplete_extension) + # Catch2 from vcpkg (manifest mode); toolchain supplied by the Makefile/CI. find_package(Catch2 3 CONFIG REQUIRED) find_package(Threads REQUIRED) # --- Telemetry: DataZooDE/posthog-telemetry submodule ------------------------ -# The shared lib needs DuckDB's patched httplib header (third_party/httplib) plus -# DuckDB's source includes, which the prebuilt DUCKDB_DIST (amalgamation only) -# does not ship. Fetch the matching DuckDB v1.5.4 SOURCE (headers only) and hand -# the submodule the two paths it expects — mirroring flapi's "Expose DuckDB -# paths" block. Fetch the matching DuckDB v1.5.4 SOURCE (headers only). +# The shared lib needs DuckDB's patched httplib header (third_party/httplib) +# plus DuckDB's source includes. Both come from the duckdb/ submodule, which is +# the same tree the engine is compiled from — so the headers cannot drift from +# the engine, which they could when this was a separate tarball download. # OpenSSL is linked statically via the vcpkg static triplet, so the # single-file bundle gains no new runtime .so/.dylib/.dll. find_package(OpenSSL REQUIRED) @@ -145,25 +205,9 @@ find_package(ZLIB REQUIRED) include("${CMAKE_CURRENT_SOURCE_DIR}/cmake/embed_abap.cmake") erpl_generate_abap_assets(ERPL_ABAP_GENERATED) -# Must match the prebuilt DUCKDB_DIST engine (Makefile passes the same value). -if(NOT DEFINED DUCKDB_VERSION OR DUCKDB_VERSION STREQUAL "") - set(DUCKDB_VERSION "1.5.4") -endif() -include(FetchContent) -if(POLICY CMP0169) - cmake_policy(SET CMP0169 OLD) # allow FetchContent_Populate (headers only) -endif() -FetchContent_Declare(duckdb_source - URL https://github.com/duckdb/duckdb/archive/refs/tags/v${DUCKDB_VERSION}.tar.gz - DOWNLOAD_EXTRACT_TIMESTAMP TRUE) -FetchContent_GetProperties(duckdb_source) -if(NOT duckdb_source_POPULATED) - message(STATUS "Fetching DuckDB v${DUCKDB_VERSION} source (headers only for posthog-telemetry)") - FetchContent_Populate(duckdb_source) -endif() -set(DUCKDB_SRC_INCLUDE "${duckdb_source_SOURCE_DIR}/src/include") -set(DUCKDB_THIRD_PARTY "${duckdb_source_SOURCE_DIR}/third_party") +set(DUCKDB_SRC_INCLUDE "${CMAKE_CURRENT_SOURCE_DIR}/duckdb/src/include") +set(DUCKDB_THIRD_PARTY "${CMAKE_CURRENT_SOURCE_DIR}/duckdb/third_party") add_subdirectory(third_party/posthog-telemetry) # Feedback banner (header-only). Telemetry ON so a printed banner is counted as @@ -214,6 +258,18 @@ add_executable(erpl_rev_server src/logging.cpp src/erpl_rev_telemetry.cpp src/main.cpp) + +# DuckDB is linked INTO this executable, so a loaded .duckdb_extension resolves +# its duckdb:: symbols against the binary rather than a libduckdb.so. -rdynamic +# keeps those symbols in the dynamic table. +# +# Measured: httpfs and quack both load correctly WITHOUT this, so it is not +# load-bearing for the extensions erpl-rev uses today. It is kept because flapi, +# which loads its whole extension set as prebuilt binaries, does need it -- and +# an extension that does resolve host symbols fails as a segfault inside its own +# LoadInternal, which reads as a broken extension rather than a missing flag. +set_target_properties(erpl_rev_server PROPERTIES ENABLE_EXPORTS TRUE) + target_include_directories(erpl_rev_server PRIVATE "${RFC_INCLUDE}" "${DUCKDB_INCLUDE}" src) target_compile_definitions(erpl_rev_server PRIVATE @@ -233,14 +289,8 @@ elseif(UNIX) BUILD_RPATH "$ORIGIN" INSTALL_RPATH "$ORIGIN") endif() -# --- Self-extracting launcher (the single-file distributable wraps this) ----- -add_executable(erpl_rev_launch launcher/erpl_rev_launch.cpp) -if(UNIX AND NOT APPLE) - target_link_options(erpl_rev_launch PRIVATE -static-libstdc++ -static-libgcc) -endif() - # --- DuckDB-bridge Catch2 tests ---------------------------------------------- -if(EXISTS "${DUCKDB_LIB}") +if(TRUE) # tests always build; DuckDB is in-tree add_executable(erpl_rev_tests test/test_duckdb_bridge.cpp test/test_cdc_dialect.cpp @@ -265,11 +315,7 @@ if(EXISTS "${DUCKDB_LIB}") target_link_libraries(erpl_rev_tests PRIVATE ${DUCKDB_LIB} Threads::Threads Catch2::Catch2WithMain posthog_telemetry datazoo_banner OpenSSL::Crypto ZLIB::ZLIB) - if(APPLE) - set_target_properties(erpl_rev_tests PROPERTIES BUILD_RPATH "${DUCKDB_DIST}") - elseif(UNIX) - set_target_properties(erpl_rev_tests PROPERTIES BUILD_RPATH "${DUCKDB_DIST}") - endif() + # No BUILD_RPATH: DuckDB is inside the binary, so there is nothing to find. else() message(WARNING "libduckdb not found at ${DUCKDB_LIB}; skipping tests.") endif() diff --git a/Makefile b/Makefile index 732859e..7b5f2d7 100644 --- a/Makefile +++ b/Makefile @@ -19,12 +19,16 @@ NWRFC_HOME ?= $(CURDIR)/nwrfcsdk/linux NWRFC_LIB := $(NWRFC_HOME)/lib BUILD_DIR := build -# DuckDB engine. The quack network server needs DuckDB >=1.5.4 and the matching -# public extension repo, so we use the official prebuilt distribution (fetched -# by `make duckdb-dist`). Point DUCKDB_DIST elsewhere to override; the CMake -# DUCKDB_DIST option follows suit. -DUCKDB_VERSION ?= 1.5.4 +# DuckDB engine. Built from the pinned `duckdb/` submodule and linked +# statically, so the server is the whole distributable -- no libduckdb beside +# it, no launcher unpacking one at first run. +# +# STATIC_DUCKDB=OFF falls back to the prebuilt distribution in DUCKDB_DIST +# (fetched by `make duckdb-dist`), which builds in seconds instead of minutes +# and is the faster loop when the change has nothing to do with DuckDB. +DUCKDB_VERSION ?= 1.5.5 DUCKDB_DIST ?= $(CURDIR)/vendor/duckdb-$(DUCKDB_VERSION) +STATIC_DUCKDB ?= ON # Which NW RFC C ABI to build against: `sdk` (SAP's, vendored under nwrfcsdk/) # or `proto` (erpl-proto's pure-Rust shim, which also supplies sapnwrfc.h, so no @@ -68,11 +72,12 @@ DIST ?= dist all: build -# Single-file distributable: launcher + inner server + runtime libs (see scripts/bundle.sh). +# The distributable IS the server: DuckDB is linked in, so there is nothing to +# stage, pack or self-extract. Kept as a target so `make bundle` still does the +# expected thing for anyone with it in muscle memory. bundle: build - ./scripts/bundle.sh linux \ - $(BUILD_DIR)/erpl_rev_server $(BUILD_DIR)/erpl_rev_launch \ - $(NWRFC_LIB) $(DUCKDB_DIST) $(DIST)/erpl-rev + mkdir -p $(DIST) + cp $(BUILD_DIR)/erpl_rev_server $(DIST)/erpl-rev # Fetch the official prebuilt DuckDB distribution (libduckdb.so + duckdb.hpp). duckdb-dist: $(DUCKDB_DIST)/libduckdb.so @@ -83,7 +88,10 @@ $(DUCKDB_DIST)/libduckdb.so: || { echo "ERROR: DuckDB download checksum mismatch"; rm -f $(DUCKDB_DIST)/dist.zip; exit 1; } cd $(DUCKDB_DIST) && unzip -o dist.zip && rm -f dist.zip -CONFIGURE_DEPS := duckdb-dist submodules +CONFIGURE_DEPS := submodules +ifneq ($(STATIC_DUCKDB),ON) +CONFIGURE_DEPS += duckdb-dist # only the prebuilt path needs the zip +endif ifeq ($(RFC_BACKEND),proto) CONFIGURE_DEPS += proto-shim endif @@ -91,6 +99,7 @@ endif configure: $(CONFIGURE_DEPS) cmake -S . -B $(BUILD_DIR) -G "$(GENERATOR)" \ -DCMAKE_BUILD_TYPE=Release -DSAPNWRFC_HOME=$(NWRFC_HOME) \ + -DERPL_REV_STATIC_DUCKDB=$(STATIC_DUCKDB) \ -DDUCKDB_DIST=$(DUCKDB_DIST) -DDUCKDB_VERSION=$(DUCKDB_VERSION) \ -DRFC_BACKEND=$(RFC_BACKEND) -DRFC_LINK=$(RFC_LINK) \ -DERPL_PROTO_ROOT=$(ERPL_PROTO_ROOT) \ @@ -105,6 +114,9 @@ proto-shim: # The telemetry lib (third_party/posthog-telemetry) is a git submodule. submodules: @git submodule update --init --recursive third_party/posthog-telemetry +ifeq ($(STATIC_DUCKDB),ON) + @git submodule update --init --recursive duckdb +endif build: configure cmake --build $(BUILD_DIR) diff --git a/cmake/duckdb_extensions.cmake b/cmake/duckdb_extensions.cmake new file mode 100644 index 0000000..b3eb874 --- /dev/null +++ b/cmake/duckdb_extensions.cmake @@ -0,0 +1,16 @@ +# Which DuckDB extensions get linked into the static engine. +# +# A bare source build links only core_functions and parquet. The prebuilt +# libduckdb release we used before ALSO carried icu, json and autocomplete, so +# building from source without this file silently produces a narrower engine — +# and the failure is not a missing-extension error, it is 58 unit tests failing +# on things like `TIMESTAMP WITH TIME ZONE - INTERVAL`, whose operator lives in +# icu. Match the release set so switching the link mode changes no behaviour. +# +# httpfs is out-of-tree in 1.5.x and quack is third-party: both stay runtime +# loads (INSTALL/LOAD), which is why extension autoloading is left enabled. +duckdb_extension_load(core_functions) +duckdb_extension_load(parquet) +duckdb_extension_load(icu) +duckdb_extension_load(json) +duckdb_extension_load(autocomplete) diff --git a/duckdb b/duckdb new file mode 160000 index 0000000..d8cdaa3 --- /dev/null +++ b/duckdb @@ -0,0 +1 @@ +Subproject commit d8cdaa33fda8df955cc76ef58a280f68f4cd43fa diff --git a/launcher/erpl_rev_launch.cpp b/launcher/erpl_rev_launch.cpp deleted file mode 100644 index b322d58..0000000 --- a/launcher/erpl_rev_launch.cpp +++ /dev/null @@ -1,280 +0,0 @@ -// erpl-rev self-extracting launcher (Option 1). -// -// The distributable single binary = this launcher with a payload appended: -// [ launcher executable ][ payload.tar (inner server + runtime libs) ][ footer ] -// footer = 8-byte magic "ERPLREV\x01" + little-endian uint64 payload size. -// -// On start the launcher locates its own executable, reads the footer, extracts -// the ustar payload to a per-version cache dir under the system temp directory, -// points the dynamic loader at that dir, and runs the inner erpl_rev_server — -// so a user ships exactly one file and needs no SAP NW RFC SDK / DuckDB on disk. -// -// Cross-platform (Linux / macOS / Windows): no objcopy/resource tricks — the -// payload is plain appended bytes, so the same code compiles everywhere. - -#include -#include -#include -#include -#include -#include -#include - -#if defined(_WIN32) -# include -# include -# include -#else -# include -# include -# include -#endif -#if defined(__APPLE__) -# include -#endif - -namespace { - -const char kMagic[8] = {'E', 'R', 'P', 'L', 'R', 'E', 'V', '\x01'}; -constexpr uint64_t kFooterSize = 16; // 8 magic + 8 LE size - -#if defined(_WIN32) -const char kInnerName[] = "erpl_rev_server.exe"; -using PathChar = wchar_t; -#else -const char kInnerName[] = "erpl_rev_server"; -#endif - -[[noreturn]] void die(const std::string &msg) { - std::fprintf(stderr, "erpl-rev launcher: %s\n", msg.c_str()); - std::exit(70); -} - -// --- locate our own executable ------------------------------------------- -std::string SelfPath() { -#if defined(_WIN32) - std::vector buf(32768); - DWORD n = GetModuleFileNameW(nullptr, buf.data(), (DWORD)buf.size()); - if (n == 0) die("GetModuleFileNameW failed"); - int len = WideCharToMultiByte(CP_UTF8, 0, buf.data(), n, nullptr, 0, nullptr, nullptr); - std::string out(len, '\0'); - WideCharToMultiByte(CP_UTF8, 0, buf.data(), n, out.data(), len, nullptr, nullptr); - return out; -#elif defined(__APPLE__) - uint32_t size = 0; - _NSGetExecutablePath(nullptr, &size); - std::vector buf(size); - if (_NSGetExecutablePath(buf.data(), &size) != 0) die("_NSGetExecutablePath failed"); - char real[4096]; - if (realpath(buf.data(), real)) return std::string(real); - return std::string(buf.data()); -#else - char buf[4096]; - ssize_t n = readlink("/proc/self/exe", buf, sizeof(buf) - 1); - if (n <= 0) die("readlink /proc/self/exe failed"); - buf[n] = '\0'; - return std::string(buf); -#endif -} - -uint64_t FileSize(std::FILE *f) { -#if defined(_WIN32) - _fseeki64(f, 0, SEEK_END); - long long s = _ftelli64(f); -#else - std::fseek(f, 0, SEEK_END); - long s = std::ftell(f); -#endif - return (uint64_t)s; -} - -void Seek(std::FILE *f, uint64_t off) { -#if defined(_WIN32) - _fseeki64(f, (long long)off, SEEK_SET); -#else - std::fseek(f, (long)off, SEEK_SET); -#endif -} - -// --- filesystem helpers --------------------------------------------------- -bool Exists(const std::string &p) { -#if defined(_WIN32) - return _access(p.c_str(), 0) == 0; -#else - struct stat st; - return ::stat(p.c_str(), &st) == 0; -#endif -} - -void MakeDir(const std::string &p) { -#if defined(_WIN32) - _mkdir(p.c_str()); -#else - ::mkdir(p.c_str(), 0700); -#endif -} - -std::string TempRoot() { -#if defined(_WIN32) - char buf[MAX_PATH]; - DWORD n = GetTempPathA(MAX_PATH, buf); - if (n == 0) return "."; - return std::string(buf, n); -#else - const char *t = std::getenv("TMPDIR"); - return t && *t ? std::string(t) : std::string("/tmp"); -#endif -} - -char Sep() { -#if defined(_WIN32) - return '\\'; -#else - return '/'; -#endif -} - -std::string Join(const std::string &a, const std::string &b) { - if (a.empty()) return b; - if (a.back() == Sep()) return a + b; - return a + Sep() + b; -} - -// --- minimal ustar reader ------------------------------------------------- -uint64_t ParseOctal(const char *p, size_t n) { - uint64_t v = 0; - for (size_t i = 0; i < n && p[i] >= '0' && p[i] <= '7'; ++i) v = v * 8 + (p[i] - '0'); - return v; -} - -// Extract the ustar stream in [start, start+size) of file f into dir. -void ExtractTar(std::FILE *f, uint64_t start, uint64_t size, const std::string &dir) { - Seek(f, start); - uint64_t consumed = 0; - std::vector hdr(512); - while (consumed + 512 <= size) { - if (std::fread(hdr.data(), 1, 512, f) != 512) break; - consumed += 512; - bool zero = true; - for (char c : hdr) if (c) { zero = false; break; } - if (zero) break; // end-of-archive - - std::string name(hdr.data(), strnlen(hdr.data(), 100)); - uint64_t fsize = ParseOctal(hdr.data() + 124, 12); - char type = hdr[156]; - // strip leading "./" - if (name.rfind("./", 0) == 0) name = name.substr(2); - - if (type == '5' || name.empty()) { // directory / pax — skip data - uint64_t skip = (fsize + 511) & ~uint64_t(511); - Seek(f, start + consumed + skip); - consumed += skip; - continue; - } - - std::string out = Join(dir, name); - std::FILE *o = std::fopen(out.c_str(), "wb"); - if (!o) die("cannot write " + out); - uint64_t left = fsize; - std::vector buf(1 << 20); - while (left > 0) { - size_t want = (size_t)(left < buf.size() ? left : buf.size()); - size_t got = std::fread(buf.data(), 1, want, f); - if (got == 0) break; - std::fwrite(buf.data(), 1, got, o); - left -= got; - } - std::fclose(o); - uint64_t padded = (fsize + 511) & ~uint64_t(511); - consumed += padded; - Seek(f, start + consumed); -#if !defined(_WIN32) - if (name == kInnerName) ::chmod(out.c_str(), 0755); -#endif - } -} - -} // namespace - -int main(int argc, char **argv) { - std::string self = SelfPath(); - std::FILE *f = std::fopen(self.c_str(), "rb"); - if (!f) die("cannot open self: " + self); - - uint64_t total = FileSize(f); - if (total < kFooterSize) die("bundle too small / no payload"); - - // read footer - Seek(f, total - kFooterSize); - char footer[16]; - if (std::fread(footer, 1, 16, f) != 16) die("cannot read footer"); - if (std::memcmp(footer, kMagic, 8) != 0) die("bad payload magic (not a bundled binary)"); - uint64_t payload_size = 0; - for (int i = 0; i < 8; ++i) payload_size |= (uint64_t)(unsigned char)footer[8 + i] << (8 * i); - if (payload_size + kFooterSize > total) die("payload size out of range"); - uint64_t payload_start = total - kFooterSize - payload_size; - - // per-version cache dir keyed on payload size (cheap version id) - char idbuf[32]; - std::snprintf(idbuf, sizeof(idbuf), "erpl-rev-%llx", (unsigned long long)payload_size); - std::string cache = Join(TempRoot(), idbuf); - std::string ready = Join(cache, ".ready"); - std::string inner = Join(cache, kInnerName); - - if (!Exists(ready)) { - MakeDir(cache); - ExtractTar(f, payload_start, payload_size, cache); - std::FILE *r = std::fopen(ready.c_str(), "wb"); - if (r) std::fclose(r); - } - std::fclose(f); - if (!Exists(inner)) die("inner server missing after extraction"); - - // point the dynamic loader at the cache dir and run the inner server -#if defined(_WIN32) - // DLLs sit beside the inner exe in `cache`, which Windows searches by default; - // also prepend to PATH for robustness, then CreateProcess. - { - std::string path = "PATH=" + cache; - if (const char *old = std::getenv("PATH")) { path += ";"; path += old; } - _putenv(path.c_str()); - } - std::wstring wcache(cache.begin(), cache.end()); - std::wstring winner(inner.begin(), inner.end()); - // rebuild a command line: "inner" + original args (skip our argv[0]) - std::wstring cmd = L"\"" + winner + L"\""; - int wargc = 0; - LPWSTR *wargv = CommandLineToArgvW(GetCommandLineW(), &wargc); - for (int i = 1; i < wargc; ++i) { cmd += L" \""; cmd += wargv[i]; cmd += L"\""; } - STARTUPINFOW si{}; si.cb = sizeof(si); - PROCESS_INFORMATION pi{}; - std::vector cmdbuf(cmd.begin(), cmd.end()); cmdbuf.push_back(0); - if (!CreateProcessW(winner.c_str(), cmdbuf.data(), nullptr, nullptr, TRUE, - 0, nullptr, wcache.c_str(), &si, &pi)) - die("CreateProcess failed"); - WaitForSingleObject(pi.hProcess, INFINITE); - DWORD code = 0; - GetExitCodeProcess(pi.hProcess, &code); - return (int)code; -#else - { -# if defined(__APPLE__) - const char *var = "DYLD_LIBRARY_PATH"; -# else - const char *var = "LD_LIBRARY_PATH"; -# endif - std::string val = cache; - if (const char *old = std::getenv(var)) { val += ":"; val += old; } - setenv(var, val.c_str(), 1); -# if defined(__APPLE__) - setenv("DYLD_FALLBACK_LIBRARY_PATH", cache.c_str(), 1); -# endif - } - std::vector args; - args.push_back(const_cast(inner.c_str())); - for (int i = 1; i < argc; ++i) args.push_back(argv[i]); - args.push_back(nullptr); - execv(inner.c_str(), args.data()); - die(std::string("execv failed: ") + std::strerror(errno)); -#endif -} diff --git a/scripts/bundle.ps1 b/scripts/bundle.ps1 deleted file mode 100644 index 80359d0..0000000 --- a/scripts/bundle.ps1 +++ /dev/null @@ -1,51 +0,0 @@ -# Assemble the single-file Windows distributable: -# erpl-rev.exe = launcher.exe + payload.tar(inner server + runtime DLLs) + footer -# footer = "ERPLREV\x01" + little-endian uint64 payload size (read by the launcher). -[CmdletBinding()] -param( - [Parameter(Mandatory)][string]$Server, # build\erpl_rev_server.exe - [Parameter(Mandatory)][string]$Launcher, # build\erpl_rev_launch.exe - # nwrfcsdk\win\lib, or "-" when the RFC backend is linked statically - # (RFC_BACKEND=proto RFC_LINK=static): the shim is inside the exe, so there is - # no SAP SDK and no ICU to stage and the payload is DuckDB alone. Mirrors the - # "-" sentinel scripts/stage_runtime.sh takes on POSIX. - [Parameter(Mandatory)][string]$SdkLib, - [Parameter(Mandatory)][string]$DuckdbDir, # vendor\duckdb-1.5.4 - [Parameter(Mandatory)][string]$Out # dist\erpl-rev-windows-amd64.exe -) -$ErrorActionPreference = 'Stop' - -$stage = Join-Path $env:TEMP ("erplrev_stage_" + [System.Guid]::NewGuid().ToString('N')) -$pay = Join-Path $stage 'payload' -New-Item -ItemType Directory -Path $pay -Force | Out-Null - -Copy-Item $Server (Join-Path $pay 'erpl_rev_server.exe') -# All runtime DLLs the SDK ships (sapnwrfc.dll, libsapucum.dll, icu*.dll — -# whatever the SDK version names them), plus DuckDB. -if ($SdkLib -ne '-') { - Get-ChildItem (Join-Path $SdkLib '*.dll') | ForEach-Object { Copy-Item $_.FullName $pay } -} -Copy-Item (Join-Path $DuckdbDir 'duckdb.dll') (Join-Path $pay 'duckdb.dll') - -Write-Host "Bundling:"; Get-ChildItem $pay | ForEach-Object { " {0,12} {1}" -f $_.Length, $_.Name } - -$tar = Join-Path $stage 'payload.tar' -$names = (Get-ChildItem $pay | ForEach-Object Name) -& tar -cf $tar -C $pay @names -if ($LASTEXITCODE -ne 0) { throw "tar failed" } - -New-Item -ItemType Directory -Path (Split-Path $Out -Parent) -Force | Out-Null -$fs = [System.IO.File]::Open($Out, 'Create') -foreach ($f in @($Launcher, $tar)) { - $bytes = [System.IO.File]::ReadAllBytes($f) - $fs.Write($bytes, 0, $bytes.Length) -} -$size = (Get-Item $tar).Length -$magic = [byte[]]@(0x45,0x52,0x50,0x4C,0x52,0x45,0x56,0x01) # "ERPLREV\x01" -$fs.Write($magic, 0, 8) -$le = [System.BitConverter]::GetBytes([uint64]$size) # x64 is little-endian -$fs.Write($le, 0, 8) -$fs.Close() - -Remove-Item $stage -Recurse -Force -Write-Host "-> $Out ($((Get-Item $Out).Length) bytes, payload $size bytes)" diff --git a/scripts/bundle.sh b/scripts/bundle.sh deleted file mode 100755 index 5372217..0000000 --- a/scripts/bundle.sh +++ /dev/null @@ -1,31 +0,0 @@ -#!/usr/bin/env bash -# Assemble the single-file distributable for POSIX platforms (linux | osx): -# dist/erpl-rev = launcher + payload.tar(inner server + runtime libs) + footer -# footer = "ERPLREV\x01" + little-endian uint64 payload size (read by the launcher). -# -# Usage: scripts/bundle.sh -set -euo pipefail - -PLATFORM="${1:?platform: linux|osx}" -SERVER="${2:?server binary}" -LAUNCHER="${3:?launcher binary}" -SDK_LIB="${4:?nwrfcsdk /lib dir}" -DUCKDB_DIR="${5:?duckdb dist dir}" -OUT="${6:?output path}" - -STAGE="$(mktemp -d)"; trap 'rm -rf "$STAGE"' EXIT -PAY="$STAGE/payload" - -# The runtime payload (inner server + SAP/ICU/DuckDB libs) is staged by the -# shared helper so the bundle and the Docker image use the identical lib set. -"$(dirname "$0")/stage_runtime.sh" "$PLATFORM" "$SERVER" "$SDK_LIB" "$DUCKDB_DIR" "$PAY" - -( cd "$PAY" && tar -cf "$STAGE/payload.tar" * ) -SIZE="$(wc -c < "$STAGE/payload.tar")" - -mkdir -p "$(dirname "$OUT")" -cat "$LAUNCHER" "$STAGE/payload.tar" > "$OUT" -perl -e 'print "ERPLREV\x01"; print pack("Q<", $ARGV[0])' "$SIZE" >> "$OUT" -chmod +x "$OUT" - -echo "-> $OUT ($(wc -c < "$OUT") bytes, payload $SIZE bytes)" diff --git a/scripts/stage_runtime.sh b/scripts/stage_runtime.sh deleted file mode 100755 index d94120e..0000000 --- a/scripts/stage_runtime.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash -# Stage the erpl-rev runtime payload — the inner server plus the shared libraries -# it needs at run time — into a plain directory. -# -# This is the SINGLE SOURCE OF TRUTH for the runtime lib set, shared by: -# - scripts/bundle.sh (self-extracting single-file distributable) -# - the Docker image build (.github/workflows/release.yml -> Dockerfile) -# so the two never drift when the ICU/DuckDB versions bump. -# -# Usage: scripts/stage_runtime.sh -set -euo pipefail - -PLATFORM="${1:?platform: linux|osx}" -SERVER="${2:?server binary}" -# Pass "-" when the RFC backend is linked statically (RFC_BACKEND=proto -# RFC_LINK=static): the shim is inside the binary, so there is no SAP NW RFC -# SDK and no ICU to stage and the payload is DuckDB alone. -SDK_LIB="${3:?nwrfcsdk /lib dir, or - for a statically linked RFC backend}" -DUCKDB_DIR="${4:?duckdb dist dir}" -OUT="${5:?output dir}" - -case "$PLATFORM" in - linux) SAP_LIBS=(libsapnwrfc.so libsapucum.so libicudata.so.50 libicui18n.so.50 libicuuc.so.50); DUCKDB_LIB=libduckdb.so ;; - osx) SAP_LIBS=(libsapnwrfc.dylib libsapucum.dylib libicudata.50.dylib libicui18n.50.dylib libicuuc.50.dylib); DUCKDB_LIB=libduckdb.dylib ;; - *) echo "unknown platform: $PLATFORM" >&2; exit 2 ;; -esac - -if [ "$SDK_LIB" = "-" ]; then SAP_LIBS=(); fi - -mkdir -p "$OUT" -cp "$SERVER" "$OUT/erpl_rev_server" -# Guard the expansion: macOS ships bash 3.2, where "${arr[@]}" on an EMPTY array -# counts as unbound under `set -u` and aborts. Linux's bash 5 expands it to -# nothing and carries on, so this only bites on the mac. -if [ "${#SAP_LIBS[@]}" -gt 0 ]; then - for l in "${SAP_LIBS[@]}"; do cp "$SDK_LIB/$l" "$OUT/$l"; done -fi -cp "$DUCKDB_DIR/$DUCKDB_LIB" "$OUT/$DUCKDB_LIB" - -echo "Staged $(ls "$OUT" | wc -l) files into $OUT:" -ls -la "$OUT" | awk 'NR>1{print " "$5" "$NF}' From 1381e66cd524d9db26132c1f96437f77e897bdaa Mon Sep 17 00:00:00 2001 From: Joachim Rosskopf Date: Mon, 31 Aug 2026 15:54:56 +0200 Subject: [PATCH 2/3] Windows: match DuckDB's static CRT instead of fighting it duckdb/CMakeLists.txt sets CMAKE_MSVC_RUNTIME_LIBRARY unconditionally as a plain variable, so /MT wins inside its own subdirectory and cannot be overridden from here. Mixing that with our /MD produced LNK2038 "mismatch detected for 'RuntimeLibrary'" for every DuckDB object. The x64-windows-static-md triplet existed to match the prebuilt DuckDB DLL, which no longer exists. x64-windows-static is now both correct and simpler: a fully static CRT is one fewer thing the target machine needs. Same resolution flapi reached. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 12 ++++++------ .github/workflows/release.yml | 8 ++++---- CMakeLists.txt | 14 ++++++++++++++ 3 files changed, 24 insertions(+), 10 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 85e1b1f..b53b239 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -54,7 +54,7 @@ jobs: build-windows: # The release Windows bundle is built only on tags, so Windows-only breaks - # used to slip to the release. Build the same x64-windows-static-md config + # used to slip to the release. Build the same x64-windows-static config # (server + tests) AND run the suite here on every push/PR. name: Build server + run tests (Windows) runs-on: windows-latest @@ -80,13 +80,13 @@ jobs: - name: Build (server + tests) run: | - # x64-windows-static-md mirrors the release build: static OpenSSL, - # dynamic CRT (/MD), DuckDB compiled from the pinned submodule and - # linked in. Builds all targets incl. tests, so an MSVC-only compile - # error fails here instead of at release time. + # x64-windows-static mirrors the release build: static OpenSSL and a + # static CRT (/MT) to match DuckDB, compiled from the pinned submodule + # and linked in. Builds all targets incl. tests, so an MSVC-only + # compile error fails here instead of at release time. cmake -S . -B build -DCMAKE_BUILD_TYPE=Release ` -DCMAKE_TOOLCHAIN_FILE="$env:VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" ` - -DVCPKG_TARGET_TRIPLET=x64-windows-static-md ` + -DVCPKG_TARGET_TRIPLET=x64-windows-static ` -DDUCKDB_VERSION="$env:DUCKDB_VERSION" cmake --build build --config Release diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index a856224..986e829 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -196,12 +196,12 @@ jobs: - name: Build (static DuckDB) run: | - # x64-windows-static-md: statically link OpenSSL (no libssl/libcrypto - # DLLs beside the exe) while keeping the dynamic CRT (/MD), which is - # what DuckDB's own build expects. + # x64-windows-static: statically link OpenSSL (no libssl/libcrypto + # DLLs beside the exe) and use the static CRT (/MT), which is what + # DuckDB's own build forces. cmake -S . -B build -DCMAKE_BUILD_TYPE=Release ` -DCMAKE_TOOLCHAIN_FILE="$env:VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" ` - -DVCPKG_TARGET_TRIPLET=x64-windows-static-md ` + -DVCPKG_TARGET_TRIPLET=x64-windows-static ` -DERPL_REV_VERSION="$($env:GITHUB_REF_NAME -replace '^v','')" ` -DDUCKDB_VERSION="$env:DUCKDB_VERSION" ` -DRFC_BACKEND=proto -DRFC_LINK=static ` diff --git a/CMakeLists.txt b/CMakeLists.txt index ccb57fb..f51a9ce 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -3,6 +3,20 @@ project(erpl_rev CXX) set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_STANDARD_REQUIRED ON) + +if(MSVC) + # Static CRT (/MT), to match DuckDB. duckdb/CMakeLists.txt sets + # CMAKE_MSVC_RUNTIME_LIBRARY unconditionally as a plain variable, so it wins + # inside its own subdirectory and cannot be overridden from here -- the only + # workable direction is for us to match it. Mixing the two produces LNK2038 + # "mismatch detected for 'RuntimeLibrary'" for every DuckDB object. + # + # This is why the Windows triplet is x64-windows-static and no longer + # x64-windows-static-md: the -md variant existed to match the prebuilt DuckDB + # DLL, which no longer exists. A fully static CRT also means one fewer thing + # the machine has to have installed. + set(CMAKE_MSVC_RUNTIME_LIBRARY "MultiThreaded$<$:Debug>" CACHE STRING "") +endif() if(NOT CMAKE_BUILD_TYPE) set(CMAKE_BUILD_TYPE Release) endif() From 83c91fad6373e73a5f1fabed7c680c3a812c6f3b Mon Sep 17 00:00:00 2001 From: Joachim Rosskopf Date: Mon, 31 Aug 2026 17:09:34 +0200 Subject: [PATCH 3/3] CI: build and test on macOS too macOS had no CI job at all -- it was built only by the release workflow, on a tag, which is the worst place to find a platform break. That mattered less when DuckDB arrived prebuilt; it matters now that we compile it from source on every platform. Builds the SDK backend rather than the release's proto backend so it needs no erpl-proto token: the untested part is that the C++ and the DuckDB build work on arm64 macOS, not which RFC backend is linked. Also bumps the workflow's DUCKDB_VERSION to 1.5.5 to match the submodule. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 41 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b53b239..1531482 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,7 +13,7 @@ permissions: contents: read env: - DUCKDB_VERSION: "1.5.4" + DUCKDB_VERSION: "1.5.5" jobs: build-and-test: @@ -52,6 +52,45 @@ jobs: - name: Run tests run: make test + build-macos: + # macOS had no CI at all: it was built only by the release workflow, on a + # tag, which is the worst place to discover a platform break. It matters more + # now that DuckDB is compiled from source rather than downloaded prebuilt. + # + # This builds the SDK backend rather than the release's proto backend, so it + # needs no erpl-proto token; the point here is that the C++ and the DuckDB + # build work on arm64 macOS, which is the part that was untested. + name: Build server + run tests (macOS) + runs-on: macos-14 + steps: + - uses: actions/checkout@v4 + with: + submodules: recursive # duckdb (linked statically) + posthog-telemetry + + - name: Install build tools + run: brew install ninja + + - name: Setup vcpkg + uses: lukka/run-vcpkg@v11 + with: + vcpkgGitCommitId: 11bbc873e00e9e58d4e9dffb30b7a5493a030e0b + + - name: Configure AWS credentials + uses: aws-actions/configure-aws-credentials@v4 + with: + role-to-assume: arn:aws:iam::331993160594:role/ErplGithubOicdRole + role-session-name: ErplRevGithubOidcSession + aws-region: eu-west-1 + + - name: Download SAP NW RFC SDK (macOS arm64) + run: ./scripts/download_and_extract_nwrfc.sh 's3://erpl-resources/sapnwrfc/nwrfc750P_13-80008131_osx_arm.zip' './nwrfcsdk/osx/' + + - name: Build (server + tests) + run: make build NWRFC_HOME="$PWD/nwrfcsdk/osx" VCPKG_TRIPLET=arm64-osx + + - name: Run tests + run: make test NWRFC_HOME="$PWD/nwrfcsdk/osx" VCPKG_TRIPLET=arm64-osx + build-windows: # The release Windows bundle is built only on tags, so Windows-only breaks # used to slip to the release. Build the same x64-windows-static config