feat(wasix): build wasm_tests Rust fixtures with cargo-wasix - #6698
feat(wasix): build wasm_tests Rust fixtures with cargo-wasix#6698Arshia001 wants to merge 25 commits into
Conversation
Replace direct rustc invocation with cargo wasix build, using ephemeral Cargo.toml files for single-file fixtures and supporting explicit Cargo projects with ## directives and BuildBin selection. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the WASIX wasm_tests integration-test harness to build Rust fixtures via cargo-wasix instead of invoking rustc directly, and extends fixture discovery to support full Cargo projects.
Changes:
- Add fixture discovery support for
Cargo.toml(treat as a Cargo project) and*.rsas independent Rust fixtures, with explicit precedence rules. - Replace Rust single-file compilation (
rustc --target=wasm32-wasip1) withcargo wasix build, generating an ephemeralCargo.tomlfor single-file fixtures. - Add a
BuildBindirective to select which Cargo binary artifact to copy tomainfor multi-bin Cargo fixtures.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| lib/wasix/tests/wasm_tests/README.md | Documents new fixture discovery precedence and the switch to cargo-wasix for Rust/Cargo fixtures. |
| lib/wasix/tests/wasm_tests/mod.rs | Implements Cargo-project discovery, directive parsing for Cargo.toml, cargo-wasix build execution, and artifact selection/copying. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Drop BuildBin and multi-bin selection, use edition 2024, and let cargo wasix build surface workspace and artifact errors directly. Co-authored-by: Cursor <cursoragent@cursor.com>
marxin
left a comment
marxin
left a comment
There was a problem hiding this comment.
Much better, thank you for the changes!
Singlepass lacks exception-handling support, and the WASIX toolchain (cargo-wasix with the exnref sysroot) now emits EH opcodes even for fixtures that never touched exceptions under wasm32-wasip1. WASIX and Singlepass aren't expected to play nicely, so remove the engine from the harness instead of skipping fixtures piecemeal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The WASIX Rust toolchain is not published for every platform we run tests on (notably aarch64-linux and musl hosts), so Rust fixtures can't be compiled there. Since the produced wasm is host-independent, build the fixtures once on linux-x64 and share them with all test jobs: - WASM_TESTS_BUILD_ONLY_DIR=<dir> builds each Rust fixture through its Cranelift trial (artifacts are engine-independent) and exports the wasm to <dir> without running tests, via make build-wasm-tests-fixtures. - WASM_TESTS_PREBUILT_DIR=<dir> makes the suite consume those artifacts instead of invoking cargo wasix build, so test hosts no longer need cargo-wasix or the WASIX Rust toolchain. CI gains a build_wasm_tests_fixtures job that uploads the artifacts; the test-all jobs download them and drop the cargo-wasix install step (which was also broken in containerized jobs due to a hardcoded runner path in the action). C/C++/bash fixtures keep building on each host with wasixcc as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The step only served the old direct-rustc wasm_tests fixture builds, which now use prebuilt artifacts (or cargo-wasix locally). The CLI integration tests install the target themselves in the Makefile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolves the ci-constants.env conflict by taking main's sysroot bump (v2026-07-03.1) and moving the pinned WASIX Rust toolchain to the matching v2026-07-03.1+rust-1.90 release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…only mode - Generate ephemeral fixture manifests with edition 2021: fixtures ported from the old direct-rustc build (edition 2015) use non-unsafe extern blocks, which edition 2024 rejects. - Skip the dynamic_runtime_hooks smoke test in build-only mode; it builds a C fixture with wasixcc and needs nothing prebuilt. - rustfmt fixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fixtures Review feedback (#6698): the wasi_fyi/wasi_wast suites were the only non-spec Singlepass coverage, and switching them to the WASIX toolchain (whose output uses EH opcodes Singlepass lacks) silently dropped it. Instead of a directory carve-out, model the build toolchain as an axis: single-file Rust fixtures default to building with both cargo-wasix (runs on every engine except Singlepass) and rustc --target wasm32-wasip1 (runs on Singlepass only), overridable per fixture with a new Toolchains directive. Singlepass therefore never sees WASIX-built wasm by construction, and no fixture edits are needed. The wasip1 build path restores the old direct-rustc invocation, including nightly selection for #![feature(...)] sources. wasip1 variants build locally on every platform (the rustup target is universal), so the prebuilt-artifact pipeline stays scoped to the WASIX toolchain and existing trial names and artifact keys are unchanged; new trials appear as .../wasip1/singlepass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Local runs of build-wasm-tests-fixtures no longer require exporting the variable: it defaults to target/wasm-tests-prebuilt, and the target prints where the artifacts were exported. An explicit env var (as CI sets) still takes precedence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bump WASIX_RUST_TOOLCHAIN_TAG to v2026-07-07.3+rust-1.96; the lint job requires the pin to match the latest wasix-org/rust release. - Add a rustfmt check for the wasm_tests Rust fixtures to lint-formatting, mirroring the clang-format check for C fixtures: they are not part of any crate, so cargo fmt does not cover them. Format the existing fixtures accordingly (all wasi_wast/wasi_fyi tests re-run green afterwards, quine included). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The io_stdin-* tests compare bytes read from stdin at runtime against a copy of the same fixture baked in via include_bytes! at build time. The wasm fixtures are cross-built on Linux (LF), while the Windows runner checks out *.stdin with core.autocrlf=true, feeding CRLF into stdin and failing the assertion. Mark *.stdin as binary so the fixtures are checked out byte-identically on every platform, matching the existing scene*.txt rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
marxin
left a comment
marxin
left a comment
There was a problem hiding this comment.
To be honest, I am not a fan of the pre-built binary approach for 2 primary reasons:
- Given our
cargo-wasixtest suite has been effectively just created, I would rather test a subset of the platforms natively end-to-end, rather than testing a half of the story . Moreover, if we take cargo-wasix seriously, we should include the platform in a short term. - Complexity increase for the test-harness and the S3 bucket sharing among the CI jobs.
|
@marxin I disagree with your first point. The end goal here is to test the Wasmer codebase E2E as much as possible, not testing the entire world. While I agree cargo-wasix and the rust toolchain could use the extra platform support, that is not something we should concern ourselves with on this specific repository, and cargo-wasix does have its own testing happening elsewhere. On the second point, while I don't like the complexity either, it's the only good way we have to run the tests on every platform without spending considerable time adding builds for the rust toolchain on multiple new platforms. All in all, I think the current approach is the best trade-off, at least in the short term. |
Yes, agree - that's something currently fully achievable by doing a C, C++ or a Rust
If the goal of this effort is to improve test coverage for |
|
The goal of this effort is testing that modules produced by cargo-wasix behave correctly, so it's really a test of the rust stdlib code + rust-specific syscalls, should there be any (I don't think there are today), at the same time making sure the expectations are upheld on the native side. It is not, however, a test of the cargo-wasix toolchain compiling code for WASIX. While both sets of code are produced from the rust toolchain build, we're only interested in code that builds to WASM. Do note that It's also worth noting that putting stdlib tests (rust, but also wasix-libc) in the Wasmer repo is a very conscious, deliberate choice, with the alternative being a full build of Wasmer in the rust/wasix-libc CI pipelines like we had to do for EdgeJS. This is something we decided against. As for how many tests we're cosidering, the answer is "as many as we can reasonably fit in", with immediate priority going to regression tests for any fixes we push to the rust stdlib code. We don't do nearly as many fixes there as we do in wasix-libc, mainly owing to the fact that the rust stdlib routes almost all of its logic through wasix-libc anyway so fixes are propagated automatically. However, I do wish to add more rust tests as soon as I get the chance to. |
|
I see. Still, I’m not convinced the added complexity is worth the benefit, given that the only truly interesting target we’ll test in CI is |
…tests # Conflicts: # .github/ci-constants.env
Revert the prebuild-and-download pipeline for the Rust wasm_tests fixtures (WASM_TESTS_BUILD_ONLY_DIR / WASM_TESTS_PREBUILT_DIR, the build_wasm_tests_fixtures CI job and the build-wasm-tests-fixtures make target). Fixtures build on the test host again, via cargo-wasix for the wasix toolchain and rustc for the wasip1 variants. Instead of shipping artifacts to hosts that cannot build them, only collect Rust fixtures where wasix-org/rust publishes a toolchain: gnu Linux on x86_64 and aarch64, plus macOS on aarch64. musl has no toolchain at all. Windows is left out deliberately even though a toolchain exists for it, since wasixcc does not cover Windows either and the rest of the suite already skips that host. CI installs cargo-wasix and the wasm32-wasip1 targets on exactly those three builds, keyed off a new `wasix_rust_toolchain` matrix flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The release adds an aarch64-unknown-linux-gnu toolchain, which the linux-arm64 test job needs now that Rust wasm_tests fixtures build on the test host again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
There are conflicts related to us dropping Windows support for the Compilers - please rebase. |
|
@marxin at this point, I wonder if this will ever get merged in XD |
|
It will, the surface got much smaller, can review it in the afternoon ;) |
…tests # Conflicts: # lib/wasix/tests/wasm_tests/mod.rs
| # fixtures. | ||
| - name: Install Rust WASI targets | ||
| if: matrix.stage.make == 'test-all' | ||
| if: matrix.stage.make == 'test-all' && matrix.metadata.wasix_rust_toolchain |
There was a problem hiding this comment.
Does not seem to me correct, wasm32-wasip1 as wasm32-wasip1 is available for all platforms we support.
| //! | ||
| //! `Toolchains:{list}` selects which toolchains build a single-file Rust | ||
| //! fixture: `wasix` (cargo-wasix; runs on every engine except Singlepass) and | ||
| //! `wasip1` (rustc with the wasm32-wasip1 target; runs on Singlepass only). |
There was a problem hiding this comment.
I would like to keep the current logic where the wasm32-wasip1 fixtures are run for all Engines.
|
|
||
| /// Whether Rust fixtures are collected on this host. | ||
| /// | ||
| /// Building them needs the WASIX Rust toolchain, which wasix-org/rust only |
There was a problem hiding this comment.
Can we shorted the comment by saying we don't provide the wasix toolchain for all platforms yet.
| /// all. Windows is excluded on purpose even though a toolchain exists for it: | ||
| /// wasixcc does not cover Windows either, so the suite as a whole does not run | ||
| /// there. | ||
| const COLLECT_RUST_FIXTURES: bool = cfg!(any( |
There was a problem hiding this comment.
Shoudn't it be specific to WASIX fixtures (because the wasip1 are always supported). If I see correctly the cfg! will succeed for all the platforms we support (with exception of musl) - or?
| /// platform. | ||
| #[derive(Debug, Clone, Copy, PartialEq, Eq, strum::Display, strum::EnumString)] | ||
| #[strum(ascii_case_insensitive, serialize_all = "lowercase")] | ||
| enum Toolchain { |
| /// exception-handling support, so it only runs the wasip1-built variant; | ||
| /// conversely the other engines only run the WASIX-built variant since | ||
| /// running both toolchains' output there would duplicate coverage. | ||
| fn supports_engine(self, engine: Engine) -> bool { |
There was a problem hiding this comment.
As mentioned it will change the current behavior for wasip1 which can run with any Runtime.
| test_name: String, | ||
| config_name: String, | ||
| engine: Engine, | ||
| toolchain: Toolchain, |
There was a problem hiding this comment.
rust_toolchain and rust_toolchains?
| if *engine == Engine::Singlepass | ||
| && !["wasi_fyi", "wasi_wast"].contains(&test_name.as_str()) | ||
| { | ||
| for toolchain in config.toolchains.as_ref().unwrap_or(&default_toolchains) { |
There was a problem hiding this comment.
This iteration seems weird as it's only a rust_toolchain and makes sense only in context of PrimarySource::RustSourceFile/CargoProject. What about iterating that in the inner-most loop for configs that are of such type?
The lint job requires the pin to be at least the latest wasix-org/rust release, and v2026-08-06.1 landed after the previous bump. Same rust 1.97 base, and it still publishes all four host toolchains including aarch64-unknown-linux-gnu. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review feedback (#6698): - Run the wasip1 fixture variants on every engine again, not just Singlepass. Only the WASIX toolchain's exception-handling output is restricted, so that constraint now lives in engine_runs_wasix_output. - Gate on the WASIX Rust toolchain per toolchain rather than per fixture: wasip1 builds with a plain rustup target and works everywhere, so single-file Rust fixtures keep running on musl and Windows. Only the cargo-wasix variants are skipped, and the constant is renamed WASIX_RUST_TOOLCHAIN_AVAILABLE to say so. - Stop gating the wasm32-wasip1 target install in CI on the platforms that have a WASIX Rust toolchain; the target is available everywhere. - Rename Toolchain to RustToolchain and the config fields to rust_toolchain/rust_toolchains. - Move the toolchain iteration to the innermost loop, where it reads as the per-fixture axis it is. - Shorten the platform-support comment. Also skip the wasix-libc compatibility variants for wasip1: those builds go through rustc and never read the sysroot, so they only duplicated the default variant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace direct rustc invocation with cargo wasix build, using ephemeral Cargo.toml files for single-file fixtures and supporting explicit Cargo projects with ## directives and BuildBin selection.
Since the WASIX Rust toolchain is not published for every platform CI tests on, the Rust fixtures are now prebuilt once on linux-x64 (
make build-wasm-tests-fixtures, exporting viaWASM_TESTS_BUILD_ONLY_DIR) and shared with all test jobs as an artifact consumed throughWASM_TESTS_PREBUILT_DIR; test hosts no longer needcargo-wasix. Singlepass was also dropped from the wasm_tests harness entirely, as the WASIX toolchain emits EH opcodes it doesn't support.