Emulated tests

AGENTS.md

@@ -40,6 +40,14 @@ scripts/update.py    pin updater (nix run .#scripts.update)
   package file.
 - All `WASIXCC_*`/`CC=wasixcc` environment comes from
   `pkgs/toolchain/env.nix`; never write the exports by hand.
+- The wasmer runtime must never be a build input of a package derivation: it is
+  a fast-moving git input, so that would rebuild the whole set on every bump.
+  This holds for the checks too: `pkgs/emulated-check.nix` is build-once /
+  run-many, a wasmer-free build stashing the compiled test tree and a run-only
+  derivation executing it, so a wasmer bump re-runs tests without recompiling.
+  Where an emulator path is baked into a build (cmake, cargo), use
+  `wasixRun.stub`, which resolves the runtime at run time and carries no wasmer;
+  `wasixRun.run` (with the runtime) goes only in the run-only derivation.
 - Patches live next to the file that applies them.
 - Pins: `nix run .#scripts.update` (`docs/updating.md`).
 - "Recheck/drop this on the next version bump" (a vendored patch, a
@@ -93,9 +101,9 @@ builtins.attrNames`. For behaviour-preserving refactors, also diff
   passthru changes don't move drv paths.
 - A CI job name is a build path: `nix build .#librariesByProfile.exnrefEh.zlib`,
   `.#wasmerPackages.git.webc`, `.#pythonWheels.py314.numpy`.
-- Toolchain suites: `.#toolchain.wasixcc.tests` (compile+link+run per
-  profile), `.#toolchain.sysroot.tests`; the Rust suite is
-  `.#checks.x86_64-linux.rust`.
+- Toolchain suites: `.#toolchain.wasixcc.tests.all` (compile+link+run per
+  profile), `.#toolchain.sysroot.tests.all`; individual leaves live beside
+  them under `.tests.<name>`.
 - Touching `pkgs/toolchain/` (except `llvm.nix`) rebuilds everything; use a
   remote builder or the CI cache. To try such a change on one package first,
   `scripts/spot.sh <profile>.<attr>` rebuilds that attr alone against a cached

docs/architecture.md

@@ -76,7 +76,16 @@ webc is Wasmer's package format. CLIs in `shippedCommands`
 (`pkgs/default.nix`) get a webc generated from the package (name from
 `meta.mainProgram`, commands from `bin/*.wasm`); deviations go in
 `passthru.wasmer`. `test-lib.nix` runs tests under Wasmer, usually diffing
-against the native tool.
+against the native tool. `wasix-run.nix` is the emulation trampoline (`.stub`
+carries no wasmer so it may be baked into build artifacts, `.run` pins the
+runtime). There is no crossSystem emulator override: nixpkgs' selectEmulator
+maps the wasi platform to `${pkgs.wasmtime}`, and the overlay shadows
+wasmtime with a wasmer-free wasix-run shim (`overlay/default.nix`), so
+`hostPlatform.emulator` resolves without putting wasmer in build closures.
+The centerpiece of the test architecture is the `check` output: a package's
+own build captures its test tree, and a run-only derivation replays the real
+checkPhase under wasmer, so a wasmer bump re-runs tests without recompiling
+(`docs/packaging.md`, "Emulated build-system checks").
 
 ## 6. Cargo overlay registry (`pkgs/cargo-registry/`)
 
@@ -184,8 +193,12 @@ patch tree, so the two can't drift:
 
 - `packages.<system>`: `wasixcc` (default), `cargo-wasix`, `anybuild`,
   `wasix-rust-toolchain`, `wasmer-bin`, `wasix-{libc,llvm,compiler-rt,libcxx,sysroot}`.
-- `checks.<system>`: every `passthru.tests`: behavioural suites, toolchain
-  suites (`sysroot`, `wasixcc`, `rust`), wheel imports (`wheel-<attr>`),
+- `checks.<system>`: a flat projection of every named `passthru.tests` leaf:
+  behavioural suites, toolchain
+  suites (`sysroot`, `wasixcc`, `rust`), emulated build-system checks
+  (`lib-<profile>-<name>-upstream` for libraries), independent synthetic link
+  checks (`lib-<profile>-<name>-link`), wheel checks (imports plus full
+  upstream suites, `wheel-py<N>-<attr>-upstream`),
   per-profile ABI checks (`abi-<profile>`: built artifacts carry the
   profile's EH feature, PIC relocation flavor, and module kind; see
   `pkgs/toolchain/tests/abi-check.nix`), `treefmt`.
@@ -225,5 +238,7 @@ and the toolchain is measured reproducible.
 
 ## passthru namespaces
 
-`passthru.wasix.*` where it works · `passthru.wasmer.*` webc config ·
-`passthru.tests` standard nixpkgs · `passthru.pkg` the wasmer package · `passthru.webc` the built webc.
+`passthru.wasix.*` where it works (plus `emulatedCheck`, the package's own test
+suite run under wasmer) · `passthru.wasmer.*` webc config ·
+`passthru.tests` named test namespace (`upstream`, `link`, `behavior.*`, `all`)
+· `passthru.pkg` the wasmer package · `passthru.webc` the built webc.

docs/packaging.md

@@ -234,16 +234,103 @@ deps included, is exposed under `pythonRegistry.wheels`.
 
 `pkgs/overlay/packages/<name>/tests/*.nix`, each returning an attrset of
 derivations built with `pkgs/wasmer/test-lib.nix` (a `helpers.nix` is shared
-setup). They attach as `passthru.tests` and appear under `checks.<name>`.
+setup). They attach as named `passthru.tests` leaves and appear as flat
+`checks.<name>-<test>` projections.
 Besides `pkgs`/`testLib`/`wasmerPkgs`, test files can take `crossPkgs` (the
 default-profile cross set) and `makeWasmerPackage` to cross-build and package
 a consumer program (see icu-data's smoke test).
 `mkScriptComparison` diffs against the native tool; `expectFail` marks a
 must-fail test; `broken "reason"` tolerates a known failure and fails loudly
 once it starts passing.
 
+`passthru.tests` is a namespace: each named leaf is independently buildable,
+curated tests live under `tests.behavior`, and `tests.all` is the explicit
+aggregate. Flat `checks.<system>` names are a CI projection of those leaves,
+not the canonical test API.
+
 To run tests against a locally built runtime instead of the pinned one:
-`WASMER_BIN=/path/to/wasmer nix build --impure .#checks.x86_64-linux.<name>`.
+`WASMER_BIN=/path/to/wasmer nix build --impure
+.#wasmerPackages.<name>.tests.<test>`.
+
+### Emulated build-system checks
+
+A package's _own_ suite runs under wasmer, driven by what nixpkgs already
+declares: `doCheck`, `checkPhase`, `checkTarget`, `nativeCheckInputs`, the
+check hooks. Nothing about the suite is restated; there is no per-package
+harness config.
+
+**Build-once / run-many, one compile.** The package's own build captures its
+test tree as a `check` output (`pkgs/lib/check-output.nix`, applied set-wide
+through the stdenv adapter), so the suite is compiled by the same derivation
+that produces the shipped artifact. A run-only derivation
+(`pkgs/emulated-check.nix`) restores that tree, prepends a `#!wasix-run`
+shebang to every executable wasm, and runs the package's real checkPhase with
+the runtime present. wasmer is never a build input, so a wasmer bump re-runs
+tests without recompiling. The restore relies on the sandbox build dir being
+`/build` in both derivations, so baked absolute paths resolve.
+
+The shebang (plus `patches/wasmer-wasm-shebang.patch`, which makes the
+runtime skip it when loading a module) is what lets every harness work
+unmodified: ctest, automake recipes that exec `./prog` directly, meson test.
+harfbuzz runs its meson tests through a per-package `exe_wrapper` in its own
+cross file; the set-wide `mesonEmulatorHook` stays no-op'd deliberately,
+because the stock hook also makes meson execute target binaries during
+configure (`pkgs/overlay/default.nix`).
+
+Opting in and out:
+
+- C libraries: `doCheck` in the package file; `doCheck = false` (with a
+  reason) opts out.
+- Wheels: `passthru.wasix.installCheck = true`/`false`. Read off passthru,
+  never `doInstallCheck`: forcing that on a finalAttrs-style
+  buildPythonPackage recurses (`pkgs/lib/check-output.nix`). The default is
+  what the native nixpkgs package declares.
+- Verdict knobs via `passthru.wasix.emulatedCheck`: `timeout` (seconds,
+  default 1200), `expectFail`, `broken = "reason"`, `profiles` (default: all
+  the package's supported profiles), or `= false` to opt out of a declared
+  suite.
+
+It surfaces as `passthru.tests.upstream`: for example,
+`librariesByProfile.<profile>.<name>.tests.upstream` for a library and
+`pythonWheels.py<N>.<attr>.tests.upstream` for a wheel. The flat CI projections
+are `lib-<profile>-<name>-upstream` and
+`wheel-py<N>-<attr>-upstream` respectively.
+
+Synthetic link probing is independent. A library may expose `tests.link`
+whether or not it has an upstream suite; tune or disable it with
+`passthru.wasix.smokeTest`. `emulatedCheck = false` affects only
+`tests.upstream`, and `smokeTest = false` affects only `tests.link`.
+
+`cargo test` is hand-wired separately (`toolchain/tests/rust-cargo-test.nix`),
+same split: `cargo test --no-run` builds the test binary, a binaryen pass
+translates its legacy EH to exnref, the run step execs it under `wasix-run`.
+Watch the dev-dependency graph: anything pulling `wait-timeout` (assert_cmd,
+proptest via rusty-fork) does not compile for wasi.
+
+`wasix-run` comes in two flavours (`pkgs/wasmer/wasix-run.nix`): `.stub`
+carries no wasmer and is what may be baked into build artifacts, `.run` pins
+the runtime and goes into the run-only derivation.
+
+Measuring: `scripts/check-coverage.py` reports suite pass rates from the
+explicit `*-upstream` CI leaves. Synthetic link, import, and liveness checks
+are excluded.
+
+### Python test suites
+
+The primary path is the same mechanism: the wheel's `check` output captures
+the installed state, and the run step re-runs buildPythonPackage's own
+installCheckPhase (pytestCheckHook / unittestCheckHook) verbatim, with the
+wasix interpreter running the suite. pytest flags, `disabledTests`, plugins:
+all ordinary nixpkgs attributes in the package's own file.
+
+Hand-written `runPytest` specs (`pkgs/python-test-lib.nix`) survive only as
+residual per-package `tests/*.nix` files (numpy, pycryptodome/x).
+
+A structural limit, not a sys.path problem: a suite whose tests live inside
+the package directory (`certifi/tests/...`) imports the source tree as a
+submodule of the package under test, so no sys.path or import-mode setting
+can make an installCheck test the installed copy. Such suites either run from
+the installed site (cd there in preCheck) or stay per-package curated.
 
 ## Pitfalls
 

flake.nix

@@ -37,6 +37,24 @@
         ++ [
           # proc_fork must inherit the parent's signal dispositions; see WASIX-TODO.md
           ./patches/wasmer-signal-inherit-on-fork.patch
+          # skip a leading `#!...` shebang when loading a module, so a wasm file
+          # with a wasix-run shebang is directly executable (autotools `./prog`
+          # suites); see WASIX-TODO.md
+          ./patches/wasmer-wasm-shebang.patch
+          # fdstat reports CharacterDevice for fds 0/1/2 even when redirected,
+          # and wasi-libc's isatty() is that filetype test, so CPython opens the
+          # REPL instead of reading a piped script; see WASIX-TODO.md
+          ./patches/wasmer-stdio-isatty.patch
+          # failed connects lose their errno: a fast loopback RST latches
+          # `Failed` before sock_connect's first status check (blanket
+          # ENOTCONN), and unmapped raw errnos fall through to EIO; port probes
+          # need the real ECONNREFUSED; see WASIX-TODO.md
+          ./patches/wasmer-sock-connect-errno.patch
+          # re-registering a pipe fd in a new epoll after the old one closed
+          # deadlocks the guest: replacing the stale interest handler drops the
+          # dead epoll's join guards under the pipe lock, which they re-take to
+          # detach; python's EpollSelector does this; see WASIX-TODO.md
+          ./patches/wasmer-epoll-stale-handler-deadlock.patch
         ];
       passthru =
         (old.passthru or {})
@@ -82,34 +100,69 @@
       };
     };
 
-    # Collect every package's passthru.tests into the flake checks. tryEval guards
-    # the `pkg ? tests` probe, which forces pkg; a throwing pkg keeps its entry, so
-    # the error surfaces as a failed check instead of aborting the whole output.
+    # Package-local passthru.tests is the canonical test namespace. Project
+    # every named leaf except the explicit `all` aggregate into flat flake
+    # checks; nested namespaces, if any, become hyphen-separated names.
+    flattenTests = prefix:
+      lib.concatMapAttrs (
+        name: value: let
+          key = "${prefix}${name}";
+          kind = builtins.tryEval (
+            if lib.isDerivation value
+            then "drv"
+            else if lib.isAttrs value
+            then "set"
+            else "other"
+          );
+        in
+          if name == "all"
+          then {}
+          else if !kind.success || kind.value == "drv"
+          then {${key} = value;}
+          else if kind.value == "set"
+          then flattenTests "${key}-" value
+          else {}
+      );
+    # tryEval guards the `pkg.tests` probe, which forces pkg. A throwing package
+    # retains one failing eval leaf so it cannot disappear from CI silently.
     collectTestsPrefixed = prefix:
       lib.foldlAttrs (
         acc: name: pkg: let
-          testAttr = {"${prefix}${name}" = pkg.tests;};
-          entry = builtins.tryEval (lib.optionalAttrs (pkg ? tests) testAttr);
+          probe = builtins.tryEval (
+            if pkg ? tests && lib.isAttrs pkg.tests
+            then builtins.seq (builtins.attrNames pkg.tests) pkg.tests
+            else {}
+          );
+          entry =
+            if probe.success
+            then flattenTests "${prefix}${name}-" probe.value
+            else {"${prefix}${name}-eval" = pkg.tests.all;};
         in
-          acc
-          // (
-            if entry.success
-            then entry.value
-            else testAttr
-          )
+          acc // entry
       ) {};
     collectTests = collectTestsPrefixed "";
     flakeChecks =
       collectTests wasix.wasmerPackages
       // collectTests wasix.toolchainTestPkgs
+      # Libraries expose independent upstream and link leaves; their respective
+      # passthru.wasix declarations opt out or tune them independently.
+      // lib.concatMapAttrs
+      (profile: libs: collectTestsPrefixed "lib-${profile}-" libs)
+      wasix.librariesByProfile
       # pythonWheels is nested by version (py313/py314); collect as wheel-py314-<attr>.
       // lib.concatMapAttrs (pv: wheelSet: collectTestsPrefixed "wheel-${pv}-" wheelSet) wasix.pythonWheels
       // collectTests {python-registry = wasix.pythonRegistry;}
       // collectTests {cargo-registry = wasix.cargoRegistry;}
       // lib.mapAttrs' (p: lib.nameValuePair "abi-${p}") wasix.abiChecks
       # non-shipped library packages carrying a tests/ dir
       // collectTests wasix.libraryTestPkgs
-      // {treefmt = treefmtEval.config.build.check self;};
+      # import tests for the python dependency closure (packages that ship
+      # because a wheel pulls them in, which the worklist never names)
+      // lib.mapAttrs' (n: lib.nameValuePair "pyclosure-${n}") wasix.pythonClosureTests
+      // {
+        eval-sanity = wasix.evalSanity;
+        treefmt = treefmtEval.config.build.check self;
+      };
   in {
     formatter.${system} = treefmtEval.config.build.wrapper;
 
@@ -136,14 +189,18 @@
           runtime = wasmerRuntime; # the wasmer runtime (input, patched)
         };
         librariesByProfile = wasix.librariesByProfile; # <profile>.<lib>
-        # <name> = wasm cross build; .pkg = its wasmer package; .webc = the built webc; .tests = its tests
+        # <name> = wasm cross build; .pkg = its wasmer package; .webc = the
+        # built webc; .tests = named leaves plus the explicit .all aggregate
         wasmerPackages = wasix.wasmerPackages;
-        # <attr> = wasm cross build of python3.pkgs.<attr>; .tests = import smoke-test
+        # <attr> = wasm cross build of python3.pkgs.<attr>; .tests = named
+        # import/upstream/contract leaves plus the explicit .all aggregate
         pythonWheels = wasix.pythonWheels;
         # all shipped wheels + transitive deps as a static PEP 503 index
         pythonRegistry = wasix.pythonRegistry;
         # the crate patch tree minted as publishable +wasix.N fork builds
         cargoRegistry = wasix.cargoRegistry;
+        # <dep> = import test for a closure member (not a wheels.nix entry)
+        pythonClosureTests = wasix.pythonClosureTests;
       };
 
       # Flatten nested attrsets of derivations to {"a.b.c" = drv;}, also emitting a
@@ -194,6 +251,9 @@
           inherit lib mkWasix;
           pkgNames = wasix.wasixPkgNames;
         };
+        # the emulation trampoline: .stub (wasmer-free, bakeable into build
+        # artifacts) and .run (stub + the pinned runtime).
+        inherit (wasix) wasixRun;
         pkgsCross.wasix = wasix.pkgsCross;
         allWasmerPackages = wasix.allWasmerPackages;
 
@@ -232,7 +292,7 @@
               '';
             };
         in {
-          ci-build = run "ci-build" [p.jq p.nix-eval-jobs p.nix-fast-build p.findutils] "bash" "ci-build.sh";
+          ci-build = run "ci-build" [p.jq p.nix-eval-jobs p.nix-fast-build p.findutils p.gawk] "bash" "ci-build.sh";
           rebuild-diff = run "rebuild-diff" [p.python3 p.nix-eval-jobs] "bash" "rebuild-diff.sh";
           content-diff = run "content-diff" [] "python3" "content-diff.py";
           ci-report = run "ci-report" [] "python3" "ci-report.py";