Skip to content

ci: smoke-test the shipped artifact on every platform - #19

Merged
jrosskopf merged 2 commits into
mainfrom
feat/smoke-test
Aug 9, 2026
Merged

ci: smoke-test the shipped artifact on every platform#19
jrosskopf merged 2 commits into
mainfrom
feat/smoke-test

Conversation

@jrosskopf

@jrosskopf jrosskopf commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

The gap

Every job in this pipeline exercises build/release/duckdb — a shell with the extension statically linked in. That never loads the .duckdb_extension file users install, so nothing here can catch an artifact that builds green and then fails to instantiate.

Not hypothetical: quack-oauth's CLAUDE.md records its wasm side-module as "builds green but won't instantiate in the browser — a load-time failure CI can't see." Same gap, already known.

11 of 15 DataZoo extensions had no load verification at all. This is part of closing that fleet-wide.

What this adds

scripts/smoke_test.py downloads the official DuckDB CLI for the target version, installs the built artifact into it, loads it, and calls a real function:

Step Catches
INSTALL truncated artifact, wrong platform, wrong DuckDB version
LOAD + duckdb_extensions() builds but will not instantiate
the smoke query loads but functions do not work

The third step earns its place: loading proves the shared object resolves, not that its registered functions run — symbols can resolve lazily and only fail when called.

Wiring

needs: [build], with each deploy job taking the smoke job in its needs — a broken artifact on any platform now blocks distribution. Matrix: linux_amd64, osx_arm64, windows_amd64, honouring the build job's exclude_archs.

Verification

The smoke query was run against a real build of this extension, not read off the source. That mattered: three of the fleet-wide candidates were wrong when checked that way — anofox_stats_ols_fit takes X column-major, context_parse_json is a table function rather than a scalar, and context_serialize_json takes a query string.

Piloted in erpl-tunnel#3: six smoke jobs green across linux_amd64, osx_arm64, windows_amd64 on two DuckDB lines. The pilot also caught a real defect in the first draft — the arch gate was a job-level if: referencing matrix.arch, which is out of scope there; GitHub rejected the workflow at validation, producing a run with zero jobs and no logs that reads as "nothing failed". Fixed by gating at step level before this was propagated.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

jrosskopf and others added 2 commits August 8, 2026 20:09
Every existing job exercises build/release/duckdb -- a shell with the
extension statically linked in. That never loads the .duckdb_extension
file users install, so nothing in this pipeline could catch an artifact
that builds green and then fails to instantiate. quack-oauth's CLAUDE.md
already records that gap biting: its wasm side-module "builds green but
won't instantiate in the browser".

scripts/smoke_test.py downloads the OFFICIAL DuckDB CLI for the target
version, installs the artifact into it, loads it, and calls a real
function. The third step matters on its own -- loading proves the shared
object resolves, and symbols can resolve lazily and only fail on call.

Wired as needs: [build], with each deploy job taking the smoke job in
its needs, so a broken artifact on any platform blocks distribution.

The smoke query was verified against a real build of this extension
rather than read off the source. Three of the fleet-wide candidates were
wrong when checked that way.

Piloted in DataZooDE/erpl-tunnel#3: six smoke jobs green across
linux_amd64, osx_arm64 and windows_amd64 on two DuckDB lines.
The gate used a substring test against exclude_archs. Nearly every caller
excludes `windows_amd64_mingw`, which also matches `windows_amd64` -- so the
Windows smoke job printed "excluded by the build job" and reported success
without ever downloading, loading or exercising the artifact. Green meant
nothing ran, which is precisely what this workflow exists to prevent.

Split the list on `;`/`,`/whitespace and compare whole tokens instead. The
list is passed via env because some callers set exclude_archs from a GitHub
expression whose value contains quotes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jrosskopf
jrosskopf merged commit 9937d6d into main Aug 9, 2026
74 checks passed
@jrosskopf
jrosskopf deleted the feat/smoke-test branch August 9, 2026 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant