Skip to content

Verify the extension actually loads & runs in DuckDB-Wasm (add a Node runtime test harness + CI gate) #255

Description

@sipemu

Summary

We build anofox_forecast for WASM, but nothing verifies the built .wasm actually loads and runs in DuckDB-Wasm — CI only proves it compiles and links. Per query.farm's Testing DuckDB-WASM Extensions ("compiling isn't running"), that's exactly the gap where WASM extensions silently break at LOAD or on first call.

The sister repo anofox-statistics just went through this end-to-end (DataZooDE/anofox-statistics#131). This issue ports the approach here.

What's already good here (verified from main)

So the load path should work — but it is unverified. This issue is about proving it and gating it.

Proposed work (mirrors anofox-statistics #131)

  1. Add a Node harness under test/wasm/:
    • Boots DuckDB-Wasm in Node (@duckdb/duckdb-wasm/dist/duckdb-node.cjs, eh bundle, web-worker@1.2.0 pinned, db.instantiate(mainModule, null), db.open({ allowUnsignedExtensions: true })).
    • Serves the locally-built anofox_forecast.duckdb_extension.wasm over a localhost HTTP server (version-agnostic: serve the file for any <version>/<platform>/ path) and FORCE INSTALL ... FROM + LOADs it.
    • Runs the existing test/sql/**/*.test via a minimal sqllogictest-subset runner and asserts.
  2. Gating CI job in MainDistributionPipeline.yml (needs: the wasm build) that downloads the wasm_eh artifact and runs the harness — fails the build on any WASM load/runtime error.
  3. A dedicated WASM workflow + README badge (green/red) so the badge reflects only WASM, not the whole pipeline.

Gotchas we hit in anofox-statistics (save yourself the loop)

  • duckdb-wasm engine version must ABI-match the extension's DuckDB version. The npm version ≠ engine version: @duckdb/duckdb-wasm@1.29.0 → engine v1.1.1, 1.32.0v1.4.3, 1.33.1-dev64.0v1.5.5. A mismatch fails LOAD with bad export type for '...SupportStatementCache...'. If forecast builds for DuckDB v1.5.5, pin @duckdb/duckdb-wasm@1.33.1-dev64.0 (currently the only published build bundling v1.5.5 — a dev tag; no stable duckdb-wasm ships 1.5.x yet). Verify a candidate's engine with:
    url=$(npm view @duckdb/duckdb-wasm@<version> dist.tarball)
    curl -sL "$url" | tar xz -C /tmp/dw
    strings -n4 /tmp/dw/package/dist/duckdb-eh.wasm | grep -oE 'v1\.[0-9]+\.[0-9]+' | sort -uV | tail -1
  • DECIMAL rendering in duckdb-wasm's Arrow-JS is unscaled1.0 (a DECIMAL(2,1) literal) reads back as 10 via .toArray().toJSON(), causing spurious test failures. Format results through DuckDB's own ::VARCHAR (SELECT COLUMNS(*)::VARCHAR FROM (<query>)) so the scale is applied and output matches native sqllogictest. (This bit us as a fake MIN(x1) "discrepancy".)
  • Per-file catalog isolation: re-open the DB + re-LOAD per .test file, or CREATE TABLE state leaks across files.
  • Node specifics: pin web-worker@1.2.0 (1.5.x throws module is not defined); pthreadWorker must be null for the eh bundle; use FORCE INSTALL (Node caches extensions on the real FS).
  • Optional tidy: make openssl a "!wasm32" dep in vcpkg.json (it's unused on WASM since telemetry is off there) to avoid building OpenSSL for Emscripten.

Reference implementation

anofox-statistics PR DataZooDE/anofox-statistics#131test/wasm/run.mjs, test/wasm/sqllogic.mjs, the wasm-runtime-test job, and WasmTest.yml. It runs the full suite (2095/2095) against the built .wasm.

Acceptance criteria

  • Node harness loads the built anofox_forecast.duckdb_extension.wasm in DuckDB-Wasm and runs test/sql.
  • Gating CI job fails on any WASM load/runtime error.
  • WASM status badge in README.
  • @duckdb/duckdb-wasm pinned to the engine version matching the built DuckDB version, documented.

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions