Emulated tests - #81
Open
kilyanni wants to merge 25 commits into
Open
Conversation
Build report3 failed · 1850 built · 1331 rebuilt vs base · 1235/2392 outputs identical
❌
|
kilyanni
force-pushed
the
emulated-tests
branch
6 times, most recently
from
August 7, 2026 13:05
fb8984a to
bd46ccc
Compare
A wasm module prefixed with `#!/path/to/wasix-run` becomes a file the kernel can exec AND wasmer can load: the vendored wasmer-wasm-shebang.patch skips a leading shebang line when loading a module. wasix-run is the trampoline that line points at: a wasmer-free stub that resolves the runtime from WASIX_WASMER at exec time (so it can be baked into build artifacts without putting the fast-moving wasmer input into their closures), forwards the environment (whole-environment mode via WASIX_RUN_ENV_ALL for the test harness, a small allowlist otherwise; empty and whitespace-only values are skipped since wasmer's CLI rejects them), and mounts the working tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wasmer's fdstat() said CharacterDevice for fds 0/1/2 unconditionally, and wasi-libc's isatty() is exactly that filetype test, so every redirected stdio looked like a terminal. Worst effect: CPython chooses interactive-vs-script with an isatty call before any user code runs, so `python <<EOF` got a REPL echoing prompts at the heredoc instead of a script run, and nixpkgs' python hooks use exactly that idiom. The vendored wasmer-stdio-isatty.patch reports CharacterDevice only when the host fd really is a terminal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nixpkgs' cross gate computes doCheck && canExecuteHostOnBuild and rebinds the name before deriving inputs, so a cross build silently loses its checkPhase AND its check inputs. Give every package that declares a suite a `check` output from its OWN build: the tree is snapshotted (gzipped, so nix's reference scanner does not see build-host store paths) where checkPhase would have run, with the declared check inputs restored to the build. One compile, shared with the shipped artifact; a runtime bump later re-runs tests without rebuilding anything. Check inputs that cannot evaluate on wasi at all (libredirect and friends throw at eval) are dropped by name. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The emulated check restores a package's `check` output, prepends the wasix-run shebang to every executable wasm in the tree, and runs the package's real nixpkgs check phase (ctest, make check, meson test, pytestCheckHook's installCheckPhase) with the runtime present. Nothing about a suite is restated: the phase, its flags and its inputs are whatever nixpkgs already declares, and per-package tweaks are ordinary nixpkgs attributes in the package's own file. The verdict layer maps expectFail/broken declarations onto pass/fail (XPASS is an error). Wiring: libraries attach the check (or, without a suite, a generic link-smoke) as passthru.tests on every supported profile; wheels re-run their installCheckPhase against the installed wheel, with declared check inputs carried across the cross gate via a passthru stash (wasixDeclaredCheckInputs) that package files can REPLACE when the inherited list cannot run in the guest; shipped CLIs keep curated suites plus a liveness smoke; the python dependency closure gets import tests; rust gets a cargo-test handoff (build once, exnref-translate, run under wasmer). Guests run serialised with a 1200s default timeout and a 64MB output cap, faulthandler neutered (dup() of stderr dies EOVERFLOW), and a hard-exit pytest plugin that os._exit()s after the summary to dodge the shutdown-GC trap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-package enablement of the upstream C/C++ suites: zlib and gmp (autotools, ./prog-direct via the shebang), xz (18/19, the fsync-on- directory failure XFAILed), libtiff/libjpeg/libdeflate/lzo/expat/ brotli/libpng (ctest), harfbuzz (meson, serialised and with raised timeouts), libxcrypt (yescrypt hashes wrong, tracked), zbar (one C test run directly; the C++ unwind failure recorded). Packages whose suites cannot exist on wasix opt out with the reason in place (geos: fenv FE_* macros missing; libsodium: test programs do not link; openssl: perl harness spawns a helper per test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-wheel enablement of the upstream pytest/unittest suites, config co-located in each package's own file: disabledTests/disabledTestPaths/ pytestFlags for genuine wasix gaps (no /proc, no IPv6 interpreter, no subprocess-heavy fixtures), stash REPLACEs where the inherited check inputs cannot run in the guest (build-platform numpy, absent plugins), and reasoned installCheck opt-outs (psutil loops until the output cap; pillow trips the dylib symbol-resolution defect). pandas runs its full 174k-test suite from the installed site. numpy and pycryptodome/x keep hand-written wheel-shipped suites in tests/upstream.nix and opt out of the derived source-tree check explicitly. WASIX-TODO.md records the runtime quirks the suites surfaced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two vendored wasmer patches, found by running dnspython's and pycurl's suites. sock-connect-errno: a failed nonblocking connect returned a blanket ENOTCONN (a fast loopback RST latches Failed before the first status check while the real error stays latched for SO_ERROR), and ENETUNREACH/EHOSTUNREACH fell through the raw-errno fallback to UnknownError -> EIO; NetworkError gains unreachable variants mapped end to end. epoll-stale-handler-deadlock: re-registering a pipe/socketpair fd in a new epoll after the old one closed dropped the stale handler's join guards while the pipe locks were held, deadlocking the guest (python's selectors does exactly this); replaced handlers are dropped only after every lock is released, and EPOLLOUT is primed for socketpair ends, which are always writable but never emit a writable transition. With those fixed (plus whole-env forwarding delivering NO_INTERNET), dnspython runs 1179 tests and pycurl 613 against its loopback fixtures; the TLS-over-loopback remainder is recorded in WASIX-TODO.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ci-build.sh bounds build concurrency by memory (4GB per job, 4x CPU overcommit; an unbounded sweep swapped a 32-core box to death) and ci-build-remote.sh streams the same sweep to the remote builder, refusing to pile onto a run already in flight. check-coverage.py reports upstream-suite coverage from the JUnit result, collapsing nix-fast-build's per-attempt entries; smoke tests are deliberately not counted. docs/packaging.md documents the check mechanism and its opt-in/out vocabulary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kilyanni
force-pushed
the
emulated-tests
branch
from
August 7, 2026 14:41
bd46ccc to
f87f589
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.