ci: smoke-test the shipped artifact on every platform - #17
Merged
Conversation
Every existing job exercises build/release/duckdb -- a shell with the extension statically linked in. That never loads the .duckdb_extension file users install, so nothing in this pipeline could catch an artifact that builds green and then fails to instantiate. quack-oauth's CLAUDE.md already records that gap biting: its wasm side-module "builds green but won't instantiate in the browser". scripts/smoke_test.py downloads the OFFICIAL DuckDB CLI for the target version, installs the artifact into it, loads it, and calls a real function. The third step matters on its own -- loading proves the shared object resolves, and symbols can resolve lazily and only fail on call. Wired as needs: [build], with each deploy job taking the smoke job in its needs, so a broken artifact on any platform blocks distribution. The smoke query was verified against a real build of this extension rather than read off the source. Three of the fleet-wide candidates were wrong when checked that way. Piloted in DataZooDE/erpl-tunnel#3: six smoke jobs green across linux_amd64, osx_arm64 and windows_amd64 on two DuckDB lines.
The gate used a substring test against exclude_archs. Nearly every caller excludes `windows_amd64_mingw`, which also matches `windows_amd64` -- so the Windows smoke job printed "excluded by the build job" and reported success without ever downloading, loading or exercising the artifact. Green meant nothing ran, which is precisely what this workflow exists to prevent. Split the list on `;`/`,`/whitespace and compare whole tokens instead. The list is passed via env because some callers set exclude_archs from a GitHub expression whose value contains quotes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
Every job in this pipeline exercises
build/release/duckdb— a shell with the extension statically linked in. That never loads the.duckdb_extensionfile users install, so nothing here can catch an artifact that builds green and then fails to instantiate.Not hypothetical: quack-oauth's
CLAUDE.mdrecords its wasm side-module as "builds green but won't instantiate in the browser — a load-time failure CI can't see." Same gap, already known.11 of 15 DataZoo extensions had no load verification at all. This is part of closing that fleet-wide.
What this adds
scripts/smoke_test.pydownloads the official DuckDB CLI for the target version, installs the built artifact into it, loads it, and calls a real function:INSTALLLOAD+duckdb_extensions()The third step earns its place: loading proves the shared object resolves, not that its registered functions run — symbols can resolve lazily and only fail when called.
Wiring
needs: [build], with each deploy job taking the smoke job in itsneeds— a broken artifact on any platform now blocks distribution. Matrix:linux_amd64,osx_arm64,windows_amd64, honouring the build job'sexclude_archs.Verification
The smoke query was run against a real build of this extension, not read off the source. That mattered: three of the fleet-wide candidates were wrong when checked that way —
anofox_stats_ols_fittakes X column-major,context_parse_jsonis a table function rather than a scalar, andcontext_serialize_jsontakes a query string.Piloted in erpl-tunnel#3: six smoke jobs green across
linux_amd64,osx_arm64,windows_amd64on two DuckDB lines. The pilot also caught a real defect in the first draft — the arch gate was a job-levelif:referencingmatrix.arch, which is out of scope there; GitHub rejected the workflow at validation, producing a run with zero jobs and no logs that reads as "nothing failed". Fixed by gating at step level before this was propagated.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.