rustfs: init

WASIX-TODO.md

@@ -54,6 +54,114 @@ Status: 🔴 needs upstream fix · 🟡 workaround in place · 🟢 fixed.
 - Fix: wasmer's signal delivery must cancel in-flight blocking syscalls
   (EINTR or instance termination).
 
+### tokio's I/O driver panics on an unexpected park errno 🟡
+
+- tokio's I/O driver parks in `mio::Poll::poll` and `panic!`s ("unexpected error
+  when polling the I/O driver") on any errno it does not explicitly tolerate.
+  Upstream tolerates only `Interrupted` and, on wasi, `InvalidInput`, the errno
+  the reference runtime returns for a poll with zero subscriptions
+  (`runtime/io/driver.rs`, arm commented "poll without subscriptions"). Under
+  panic=abort a park panic takes the process down.
+- Workaround (`tokio/1.52.3.patch`): also tolerate `Unsupported` (ENOTSUP,
+  errno 58) in that arm, treating it like the empty-poll case.
+- The ENOTSUP was originally attributed to wasmer's `poll_oneoff` rejecting a
+  zero-length subscription set. That attribution does not hold against the
+  pinned wasmer: upstream fixed the empty-subscription case in January 2026
+  (commit `350a7306`, "handle empty subscription list in poll_oneoff", with a
+  test), and `poll_oneoff` now returns `Errno::Inval` there, via both the early
+  return and `validate_poll_subscriptions_count`. Neither `poll_oneoff` nor
+  `epoll_wait` returns `Errno::Notsup` on any path in the pinned tree, and on
+  wasmer our mio uses the epoll backend (`mio/1.2.2.patch`), not `poll_oneoff`.
+- So the tolerance is kept as cheap insurance, not as a workaround for a known
+  live wasmer bug: it converts a would-be abort into the same graceful park
+  return tokio already performs for the analogous `InvalidInput`. Drop it if the
+  panic never reappears; reopen this entry with a real trace if it does.
+
+### `futex_wake` lost a wakeup when a waiter was mid-registration 🟢
+
+- wasmer's `futex_wake` woke "the first waiter" by taking the first entry of the
+  futex's `wakers` map and calling its stored `Waker`. But a thread that has
+  entered `futex_wait` and inserted its id with `None` (waker not installed yet)
+  sits in that map as `Some(None)`; `futex_wake` consumed that slot, woke nobody,
+  and skipped a truly sleeping waiter that still held its installed waker. A
+  tokio worker parked on the futex then never woke: `tokio::spawn` + a task doing
+  a WASI syscall + `join_all` hung (reproduced minimally; also stalled the rustfs
+  erasure-init metadata-read fanout, blocking `rustfs server` startup entirely).
+- Fix (`patches/wasmer-futex-wake-lost-wakeup.patch`): wake the first waiter that
+  has an installed waker; if none is installed yet, increment a `pending_wakes`
+  counter on the futex that the next `futex_wait` consumes before parking, so the
+  wake survives the register race. Upstreamable to wasmer.
+- A token is scoped to the waiters it was meant for: only recorded while `wakers`
+  is non-empty, only claimable by a poller that still holds its slot, and dropped
+  with the futex once the last waiter leaves. Without those three bounds a token
+  can outlive its waiters, and since a wake then finds no sleeper in the empty
+  map it records another one, so the counter grows unbounded and the futex entry
+  is never reclaimed. The slot check also stops a poller that `futex_wake` has
+  already claimed from swallowing a token owed to a waiter still asleep.
+
+### tokio's I/O driver compiles out the mio `Waker` on wasi 🟡
+
+- tokio gates its `mio::Waker` (the handle that wakes the reactor parked in
+  `epoll_wait` from another thread) behind `#[cfg(not(target_os = "wasi"))]`,
+  because stock mio's wasip1 backend has no `Waker`. Our vendored mio backend
+  (`mio/1.2.2.patch`, the wasix-org wasi backend) does implement an eventfd
+  `Waker`, but tokio still skips it, so any completion that must re-schedule a
+  task through the I/O driver (e.g. `tokio::fs`, which rustfs uses for
+  `access()`) parks the reactor forever with nothing able to wake it.
+- Workaround (`tokio/1.52.3.patch`): widen those four gates to
+  `any(not(target_os = "wasi"), all(target_vendor = "wasmer", tokio_wasix_waker))`,
+  and have the consumer opt in with `RUSTFLAGS = "--cfg tokio_wasix_waker"`
+  (rustfs does). With the futex_wake fix above, a spawned `tokio::fs` task +
+  `join_all` completes and rustfs storage init reaches "store ready".
+- The opt-in is load-bearing, not caution: the Waker is only sound with a mio that
+  has the wasix backend. Enabling it for all of `target_vendor = "wasmer"` broke
+  primp, which pins mio 1.2.0 (`cannot find type Waker in crate mio` -> tokio
+  fails to compile). mio only exports `Waker` off-wasi, and its 1.2.0 wasi arm is
+  the `single_threaded` `AtomicBool` waker, which by its own docs cannot wake an
+  already-blocked thread. Backporting the wasix backend to 1.2.0 is not a fix
+  either: it adds a `wasix` crate dependency, and crate-patches apply post-vendor,
+  so every consumer's lock would need it.
+- Fix: once mio's upstream wasi backend ships a real `Waker` on every version in
+  play, tokio can relax the blanket `not(wasi)` gate and the opt-in can go.
+
+### `fd_datasync`/`fd_sync` deny with EACCES under mapped host dirs 🟢
+
+- wasmer's `fd_datasync`/`fd_sync` return `Errno::Access` unless the fd holds the
+  `FD_DATASYNC`/`FD_SYNC` right. But `path_open` computes a file's rights as
+  `(requested | implied_by_open_mode) & parent_dir.rights_inheriting`
+  (`path_open2.rs`, whose own TODO notes "WASI isn't giving appropriate rights
+  here"): the inheriting mask on a `--mapdir`/`--volume` host preopen strips the
+  implied sync rights, so files opened for writing under it cannot fsync/fdatasync.
+- Symptom: rustfs object writes call `fdatasync` for durability; the EACCES maps
+  (ecstore `to_file_error`: PermissionDenied -> FileAccessDenied) to a 500
+  `InternalError "File access denied"` on every `PutObject`/`UploadPart`, while
+  bucket/metadata ops (no data fsync) succeed. `mc mb` worked, `mc pipe` 500'd.
+- Fix (`patches/wasmer-fd-sync-rights-durability.patch`): drop the rights gate in
+  both syscalls. fsync/fdatasync are durability barriers, not security
+  capabilities (POSIX needs no special permission); sync any regular-file fd.
+  With the fix a full `mc` put/get round-trip against `rustfs server` passes.
+- Cleaner upstream fix would stop masking implied sync rights in path_open, but
+  the syscalls should not hard-deny a durability barrier either way.
+
+### wasix Rust std does not map wasi ENOTEMPTY to `DirectoryNotEmpty` 🟡
+
+- wasix std's error decoding returns an uncategorized `io::ErrorKind` (not
+  `DirectoryNotEmpty`) for wasi `ENOTEMPTY` (errno 55). Rust code that branches on
+  `err.kind() == ErrorKind::DirectoryNotEmpty` (the common "rmdir of a non-empty
+  dir is fine" idiom) then falls through to its error path.
+- Symptom: rustfs object DELETE (`mc rm`) 500'd with "File access denied". Its
+  `delete_file` (ecstore `disk/local.rs`) tolerates NotFound + DirectoryNotEmpty on
+  `remove_dir` and treats anything else as fatal `FileAccessDenied`; the object dir
+  is legitimately non-empty (the xl versioned-delete rename dance stages old data
+  in a sub-dir), so on wasix that ENOTEMPTY aborted + rolled back the delete.
+  Traced: `path_remove_directory -> Errno::notempty` on the object dir.
+- Workaround (`rustfs-code.patch`): `delete_file` also tolerates
+  `err.raw_os_error() == Some(55)` under `cfg(target_os = "wasi")`.
+- Fix: the wasix rust-std fork's `decode_error_kind` should map wasi ENOTEMPTY
+  (and any other missing codes) to the matching `ErrorKind`, so any package keying
+  on `DirectoryNotEmpty` works without per-crate patches. Broader than rustfs and
+  touches the toolchain (expensive rebuild), hence the local workaround for now.
+
 ### spawned commands and PATH 🟢
 
 - Fixed in current wasmer: `posix_spawnp` and fork + `execvp` both resolve the

flake.nix

@@ -37,6 +37,14 @@
         ++ [
           # proc_fork must inherit the parent's signal dispositions; see WASIX-TODO.md
           ./patches/wasmer-signal-inherit-on-fork.patch
+          # futex_wake dropped a wake when the first waiter was still
+          # mid-registration (Some(None)), starving a genuinely-sleeping waiter;
+          # deadlocked tokio multi-thread spawn+blocking (e.g. rustfs server).
+          ./patches/wasmer-futex-wake-lost-wakeup.patch
+          # fd_datasync/fd_sync denied with EACCES when path_open's rights
+          # delegation masked the implied FD_DATASYNC/FD_SYNC off files under
+          # mapped host dirs; rustfs object writes (fdatasync durability) 500'd.
+          ./patches/wasmer-fd-sync-rights-durability.patch
         ];
       passthru =
         (old.passthru or {})
@@ -46,6 +54,8 @@
             noteVersion = "${old.version}-${wasmer.shortRev or "dirty"}";
             updateNotes = [
               {message = "check whether patches/wasmer-signal-inherit-on-fork.patch landed upstream (WASIX-TODO.md)";}
+              {message = "check whether patches/wasmer-futex-wake-lost-wakeup.patch landed upstream (WASIX-TODO.md)";}
+              {message = "check whether patches/wasmer-fd-sync-rights-durability.patch landed upstream (WASIX-TODO.md)";}
             ];
           };
         };

patches/wasmer-fd-sync-rights-durability.patch

@@ -0,0 +1,34 @@
+--- a/lib/wasix/src/syscalls/wasi/fd_datasync.rs
++++ b/lib/wasix/src/syscalls/wasi/fd_datasync.rs
+@@ -14,9 +14,11 @@
+     let env = ctx.data();
+     let (_, state) = unsafe { env.get_memory_and_wasi_state(&ctx, 0) };
+     let fd_entry = wasi_try_ok!(state.fs.get_fd(fd));
+-    if !fd_entry.inner.rights.contains(Rights::FD_DATASYNC) {
+-        return Ok(Errno::Access);
+-    }
++    // fdatasync is a durability barrier, not a security capability (POSIX needs no
++    // special permission). path_open's rights delegation masks the implied
++    // FD_DATASYNC off files opened under mapped host dirs (see path_open2.rs TODO
++    // on rights traversal), so gating on it here fails legitimate syncs (rustfs).
++    // Sync any regular-file fd regardless of the right.
+
+     let file = {
+         let guard = fd_entry.inode.read();
+--- a/lib/wasix/src/syscalls/wasi/fd_sync.rs
++++ b/lib/wasix/src/syscalls/wasi/fd_sync.rs
+@@ -17,9 +17,11 @@
+     let env = ctx.data();
+     let (_, mut state) = unsafe { env.get_memory_and_wasi_state(&ctx, 0) };
+     let fd_entry = wasi_try_ok!(state.fs.get_fd(fd));
+-    if !fd_entry.inner.rights.contains(Rights::FD_SYNC) {
+-        return Ok(Errno::Access);
+-    }
++    // fsync is a durability barrier, not a security capability (POSIX needs no
++    // special permission). path_open's rights delegation masks the implied
++    // FD_SYNC off files opened under mapped host dirs (see path_open2.rs TODO on
++    // rights traversal), so gating on it here fails legitimate syncs (rustfs).
++    // Sync any regular-file fd regardless of the right.
+     let inode = fd_entry.inode;
+
+     // TODO: implement this for more than files

patches/wasmer-futex-wake-lost-wakeup.patch

@@ -0,0 +1,92 @@
+--- a/lib/wasix/src/state/mod.rs
++++ b/lib/wasix/src/state/mod.rs
+@@ -76,6 +76,13 @@
+ #[derive(Debug, Default)]
+ pub struct WasiFutex {
+     pub(crate) wakers: BTreeMap<u64, Option<Waker>>,
++    /// Wakes that arrived while every registered waiter was still
++    /// mid-registration (slot reserved but `Waker` not yet installed by the
++    /// first poll). A waiter's first poll consumes one so a wake landing in the
++    /// registration gap is not lost (see `futex_wake`). Only ever non-zero while
++    /// `wakers` is non-empty: a token outlives neither its waiters nor this
++    /// futex, which is what keeps it from accumulating.
++    pub(crate) pending_wakes: u64,
+ }
+
+ /// Structure that holds the state of BUS calls to this process and from
+--- a/lib/wasix/src/syscalls/wasix/futex_wait.rs
++++ b/lib/wasix/src/syscalls/wasix/futex_wait.rs
+@@ -22,13 +22,29 @@
+             Some(f) => f,
+             None => return Poll::Ready(true),
+         };
+-        let waker = match futex.wakers.get_mut(&self.poller_idx) {
+-            Some(w) => w,
+-            None => return Poll::Ready(true),
+-        };
++        // If our slot is gone then `futex_wake` already claimed us and we are
++        // woken. Checked before `pending_wakes` so an already-woken poller cannot
++        // swallow a token meant for a waiter that is still asleep.
++        if !futex.wakers.contains_key(&self.poller_idx) {
++            return Poll::Ready(true);
++        }
++        // A wake that arrived while we were still mid-registration left a
++        // pending token; consume it and report woken instead of sleeping
++        // through it.
++        if futex.pending_wakes > 0 {
++            futex.pending_wakes -= 1;
++            futex.wakers.remove(&self.poller_idx);
++            // An unclaimed token must not outlive the waiters it was meant for.
++            if futex.wakers.is_empty() {
++                guard.futexes.remove(&self.futex_idx);
++            }
++            return Poll::Ready(true);
++        }
+
+         // Register the waker
+-        waker.replace(cx.waker().clone());
++        if let Some(waker) = futex.wakers.get_mut(&self.poller_idx) {
++            waker.replace(cx.waker().clone());
++        }
+
+         // Check for timeout
+         drop(guard);
+--- a/lib/wasix/src/syscalls/wasix/futex_wake.rs
++++ b/lib/wasix/src/syscalls/wasix/futex_wake.rs
+@@ -27,12 +27,31 @@
+     let woken = {
+         let mut guard = state.futexs.lock().unwrap();
+         if let Some(futex) = guard.futexes.get_mut(&pointer) {
+-            let first = futex.wakers.keys().copied().next();
+-            if let Some(id) = first
+-                && let Some(Some(w)) = futex.wakers.remove(&id)
+-            {
+-                w.wake();
++            // Wake the first waiter that has actually installed its `Waker`
++            // (i.e. is genuinely asleep). A slot mapped to `None` is a waiter
++            // that reserved its place but has not been polled yet to install its
++            // `Waker`; consuming that slot would drop the wake without waking a
++            // real sleeper (the old code did exactly that, because `remove` runs
++            // inside the `&&` even when the pattern fails). When no installed
++            // waker exists, record a pending wake for the imminent first poll to
++            // consume, so the wake is not lost in the registration gap.
++            let sleeper = futex
++                .wakers
++                .iter()
++                .find(|(_, w)| w.is_some())
++                .map(|(id, _)| *id);
++            if let Some(id) = sleeper {
++                if let Some(Some(w)) = futex.wakers.remove(&id) {
++                    w.wake();
++                }
++            } else if !futex.wakers.is_empty() {
++                futex.pending_wakes += 1;
+             }
++            // A pending token only means anything while a registered waiter is
++            // still around to consume it, so drop the futex once the last waiter
++            // leaves. Keeping the entry alive for an unclaimed token would strand
++            // it: every later wake would find no sleeper in the empty map and
++            // increment the counter again, unbounded and never reclaimed.
+             if futex.wakers.is_empty() {
+                 guard.futexes.remove(&pointer);
+             }

pkgs/lib/wasix-crate-patches/README.md

@@ -52,6 +52,16 @@ loudly instead of miscompiling downstream.
 
 A crate with only `<version>.patch` files needs just `{ edited = [...]; }`.
 
+## Patches that add a dependency
+
+The hook runs post-vendor, so a patch that adds a `[dependencies]` entry cannot
+pull the crate down: it must already be vendored, i.e. present in the consumer's
+`Cargo.lock`. `mio/1.2.2.patch` does this (it adds `wasix`), so every package
+locking mio 1.2.2 needs `wasix` in its lock too, or the build fails resolving it.
+rustfs gets it via `rustfs-manifest.patch`; a package that later bumps into the
+patched version has to do the same. Prefer not adding dependencies for this
+reason; when unavoidable, say so here.
+
 ## Rewriters
 
 `rewriters/<name>.nix` is a script derivation run against the crate dir (`$PWD`),

pkgs/lib/wasix-crate-patches/async-rs/0.8.11.patch

@@ -0,0 +1,41 @@
+--- a/src/sys/mod.rs
++++ b/src/sys/mod.rs
+@@ -7,3 +7,8 @@
+ mod windows;
+ #[cfg(windows)]
+ pub(crate) use windows::*;
++
++#[cfg(all(not(unix), not(windows)))]
++mod wasi;
++#[cfg(all(not(unix), not(windows)))]
++pub(crate) use wasi::*;
+--- a/src/sys/wasi.rs
++++ b/src/sys/wasi.rs
+@@ -0,0 +1,27 @@
++// wasix: std::os::unix is the incomplete wasi_ext feature; use stable std::os::fd.
++use std::os::fd::{AsFd, AsRawFd};
++
++/// Abstract trait on top of AsFd + AsRawFd.
++pub trait AsSysFd: AsFd + AsRawFd {}
++impl<H: AsFd + AsRawFd> AsSysFd for H {}
++
++#[cfg(feature = "async-io")]
++mod async_io {
++    use crate::{sys::AsSysFd, util::IOHandle};
++    use std::{
++        io::{Read, Write},
++        os::fd::{AsFd, AsRawFd, BorrowedFd, RawFd},
++    };
++
++    impl<H: Read + Write + AsSysFd + Send + 'static> AsFd for IOHandle<H> {
++        fn as_fd(&self) -> BorrowedFd<'_> {
++            self.0.as_fd()
++        }
++    }
++
++    impl<H: Read + Write + AsSysFd + Send + 'static> AsRawFd for IOHandle<H> {
++        fn as_raw_fd(&self) -> RawFd {
++            self.as_fd().as_raw_fd()
++        }
++    }
++}

pkgs/lib/wasix-crate-patches/async-rs/edits.nix

@@ -0,0 +1,3 @@
+{...}: {
+  edited = [">=0.8.11"];
+}

pkgs/lib/wasix-crate-patches/clocksource/0.8.3.patch

@@ -0,0 +1,67 @@
+--- a/src/sys/mod.rs
++++ b/src/sys/mod.rs
+@@ -1,9 +1,62 @@
+-#[cfg(not(target_os = "windows"))]
++#[cfg(all(not(target_os = "windows"), not(target_os = "wasi")))]
+ mod unix;
+-#[cfg(not(target_os = "windows"))]
++#[cfg(all(not(target_os = "windows"), not(target_os = "wasi")))]
+ pub use unix::*;
+
+ #[cfg(target_os = "windows")]
+ mod windows;
+ #[cfg(target_os = "windows")]
+ pub use windows::*;
++
++// wasix: clockid_t is a non-primitive newtype and CLOCK_*_COARSE are absent, so
++// the unix backend does not compile. Use clockid_t directly with the plain
++// (non-coarse) clocks.
++#[cfg(target_os = "wasi")]
++pub use wasi::*;
++#[cfg(target_os = "wasi")]
++mod wasi {
++    fn read_clock(clock: libc::clockid_t) -> libc::timespec {
++        let mut ts = libc::timespec {
++            tv_sec: 0,
++            tv_nsec: 0,
++        };
++        unsafe {
++            libc::clock_gettime(clock, &mut ts);
++        }
++        ts
++    }
++
++    pub mod monotonic {
++        pub fn coarse() -> crate::coarse::Instant {
++            let ts = super::read_clock(libc::CLOCK_MONOTONIC);
++            crate::coarse::Instant {
++                secs: ts.tv_sec as u32,
++            }
++        }
++
++        pub fn precise() -> crate::precise::Instant {
++            let ts = super::read_clock(libc::CLOCK_MONOTONIC);
++            let now = (ts.tv_sec as u64)
++                .wrapping_mul(1_000_000_000)
++                .wrapping_add(ts.tv_nsec as u64);
++            crate::precise::Instant { ns: now }
++        }
++    }
++
++    pub mod realtime {
++        pub fn coarse() -> crate::coarse::UnixInstant {
++            let ts = super::read_clock(libc::CLOCK_REALTIME);
++            crate::coarse::UnixInstant {
++                secs: ts.tv_sec as u32,
++            }
++        }
++
++        pub fn precise() -> crate::precise::UnixInstant {
++            let ts = super::read_clock(libc::CLOCK_REALTIME);
++            let now = (ts.tv_sec as u64)
++                .wrapping_mul(1_000_000_000)
++                .wrapping_add(ts.tv_nsec as u64);
++            crate::precise::UnixInstant { ns: now }
++        }
++    }
++}