ESPResSo/Open MPI does not survive migration: epoll/eventfd-based progress engine not virtualized by the wormhole

[KrisBuytaert]

Summary

After the fixes in #10, #11 and #13, GROMACS round-trips migration 100% reliably (test-eessi-gromacs.sh, 14/14 passes across repeated runs, both expel and migrate ... home return paths). ESPResSo built against Open MPI still does not survive migration end-to-end, but for a different and narrower reason than any of the above three bugs.

Symptom

Post-migration, an ESPResSo/MPI process's threads intermittently fail wormhole-proxied syscalls with errors that don't correspond to any real local condition:

• epoll_wait: Bad address
• eventfd: Bad file descriptor (also seen from a background monitor loop polling the process, e.g. [warn] Error reading from eventfd: Bad file descriptor)

Separately, a non-leader thread has been observed issuing repeated statx RPCs at high frequency without ever terminating that loop.

No crash, no oops, no coredump — the process just stops making forward progress on its MPI progress loop.

Root cause (hypothesis, not yet fixed)

Open MPI's progress engine uses epoll/eventfd internally to wait on multiple readiness sources efficiently. The wormhole's existing proxy model (kretprobe hook → freeze calling thread → RPC to home node → single reply → resume) works for simple request/reply syscalls (open/read/write/lseek/fsync/statx) where a Ghost file's state is naturally recreated from a single round-trip.

epoll/eventfd are structurally different: they're stateful, long-lived kernel objects that track readiness over time across a set of registered fds, expected to be efficiently poll-able many times without a fresh round-trip each time. A generic single-shot request/reply proxy has no natural way to reconstruct that in-kernel readiness-tracking state after migration — there's no "one RPC" that captures "is any of these N fds ready right now, and wake me when one becomes ready."

What we ruled out

This is not the same bug as #7, #10, #11, or #13 — GROMACS (no epoll/eventfd dependency) is now 100% reliable with those fixes in place; this reproduces specifically and only with MPI's epoll-based progress engine.

Possible directions (not attempted)

1. A real remote-epoll proxy: forward epoll_ctl/epoll_wait registrations to the home node and maintain live readiness state there, pushing wake events back — nontrivial, but the "correct" fix.
2. Explicitly scope MPI/epoll-heavy workloads out of migration support for now (document the limitation) rather than attempt a partial fix.
3. Some MPI-transport-level workaround (e.g. forcing a different Open MPI progress/transport mode that doesn't rely on epoll) — untested, may just move the problem.

Happy to help test any direction you'd want to take here — didn't want to guess at a fix without your input on which of the above (if any) fits where you want this project to go.

[KrisBuytaert]

Follow-up on this: while adding migration test coverage for a few more EESSI workloads (test-eessi-pytorch.sh, test-eessi-bioconductor.sh, test-eessi-tensorflow.sh), I hit what looks like the same underlying issue as ESPResSo above, just via a different mechanism — worth broadening the hypothesis here rather than filing it separately.

All three of these are pure single-process, multi-threaded workloads (no MPI at all — PyTorch's own OpenMP/MKL thread pool, R's own threading, TensorFlow's intra-op/inter-op thread pools). In each case, the migration itself completes cleanly with no errors:

MattX:[IMPORT] All memory transferred! Total pages injected: 416440
MattX:[IMPORT] Commencing full brain transplant on PID 14953...
MattX:[IMPORT] Successfully injected 4 Fake FDs!
MattX:[IMPORT] IT'S ALIVE! Waking 11 threads in Gang PID 14953

— but the very next check (a few seconds later) finds the process not actually running on the destination at all (STAT=T, stopped again). This isn't a "didn't wait long enough" issue: I specifically re-tested the TensorFlow case with the post-migration wait bumped from 8s to 20s to rule that out, and the import still completes successfully and the process still doesn't survive the wake.

Since none of these three touch MPI, epoll, or eventfd directly, this suggests the actual scope of the bug might be broader than MPI-linked binaries specifically — possibly anything where the migrated Gang's worker threads do their own internal futex-based synchronization (which a plain OpenMP/thread-pool runtime does just as much as MPI's progress engine does), rather than something specific to epoll/eventfd as APIs.

I haven't traced this down to a specific thread/stack state the way the wormhole PID/TGID bug was — would need to capture /proc/<pid>/task/*/stack for each thread at the exact moment it re-freezes to say more concretely what's going on. Flagging in case it's useful context for scoping a fix here, since "MPI-heavy processes" in the title may undersell how many workloads this could actually affect.