mongo 8.0.5+ crash-loops with SIGSEGV (exit 139) on arm64 / Raspberry Pi 5; 8.0.4 is fine

[tuffant21]

Summary

On arm64 (Raspberry Pi 5, Cortex-A76 / ARMv8.2-A), mongod from the official image starts, runs for a short time, then dies with SIGSEGV (docker events shows die exitCode=139) and loops forever under a restart policy. No graceful shutdown and no fatal log line is emitted; the next start just reports "Detected unclean shutdown."

Bisecting the published tags gives a clean boundary: 8.0.4 is the last good release; 8.0.5 is the first that crash-loops, and every release after it (8.0.6 → 8.0.26, all 8.2.x, all 8.3.x, latest) is affected.

Reproduction

docker run --rm mongo:8.0.5     # SIGSEGV (exit 139), crash-loops roughly every ~30s
docker run --rm mongo:8.0.4     # runs fine

No app, config, replica set, or volumes required — a bare docker run reproduces it. To watch the crash:

docker events --filter image=mongo:8.0.5 --filter event=die \
  --format '{{.Time}} exit={{.Actor.Attributes.exitCode}}'

Bisect (one standalone container per version)

Version	Result
8.0.0 – 8.0.4	OK
8.0.5	first bad — SIGSEGV 139
8.0.6, 8.0.13, 8.0.26	crash-loop
8.2.1, 8.2.9	crash-loop (139)
8.3.4	crash-loop

There is no published 8.0.2 or 8.0.7, so 8.0.4 → 8.0.5 is the exact consecutive boundary in the official images.

Environment

• Image: official library/mongo, tags 8.0.5+ (build: allocator: tcmalloc-google, distmod: ubuntu2404, distarch: aarch64)
• Host: Raspberry Pi 5, Cortex-A76 (ARMv8.2-A), Ubuntu kernel 7.0.0-1010-raspi, page size 4096
• Docker 29.5.1

Details

• Exit is 139 = SIGSEGV; no backtrace and no shutdown sequence is logged before death.
• GLIBC_TUNABLES=glibc.pthread.rseq=0 (from Enable tcmalloc per-cpu caches #723) is present in the container env and does not prevent it.
• Not OOM (State.OOMKilled=false, memory.events: oom_kill 0). Not the older Pi 4 "Illegal instruction"/SIGILL case (arm64v8 - 27 Illegal instruction #510, raspberryPi ubuntu docker run mongo, failure without error #508, Mongo 5.0.0 crashes but 4.4.6 works #485) — this CPU is ARMv8.2-A and the signal is SIGSEGV, and 8.0.4 on the same host is stable.

[tianon]

This is firmly an upstream bug -- the image is just packaging what mongodb/mongo ships, so further debugging and any fix will need to happen there.

I've bisected the two upstream changes that landed between r8.0.4 and r8.0.5 in mongodb/mongo and traced through the diffs. Two commits in that window touch arm64 in meaningful ways:

• mongodb/mongo@36b257eab87 -- libunwind v1.6.2+patches → v1.8.1 (SERVER-98185)
• mongodb/mongo@087f8eaab2c -- MozJS 115.8 → 115.20 (SERVER-98608)

Here's what I found in each, roughly ordered by how suspicious they look.

---

Most suspicious: libunwind 1.8.1, DW_CFA_AARCH64_negate_ra_state now actually handled

In Gparser.c, opcode 0x2d (DW_CFA_AARCH64_negate_ra_state -- the arm64 pointer-authentication CFI opcode) was previously handled like this in 1.6.2:

case DW_CFA_GNU_window_save:
#ifdef UNW_TARGET_SPARC
    /* ... SPARC handling ... */
    break;
#else
    /* FALL THROUGH */
#endif
case DW_CFA_lo_user:
case DW_CFA_hi_user:
    Debug (1, "Unexpected CFA opcode 0x%x\n", op);
    ret = -UNW_EINVAL;
    break;

In other words: on non-SPARC (including arm64), hitting that opcode immediately returned -UNW_EINVAL and stopped CFI parsing. In 1.8.1, it's now properly handled for arm64:

#elif UNW_TARGET_AARCH64
    /* This is a specific opcode on aarch64, DW_CFA_AARCH64_negate_ra_state */
    aarch64_negate_ra_sign_state(sr);
    break;

This opcode appears in the DWARF CFI of every non-leaf function compiled with -mbranch-protection=pac-ret (common for Ubuntu 24.04 aarch64 toolchains). The behavioral difference is stark: old libunwind silently gave up at those frames; new libunwind continues unwinding further up the stack than it ever did before. If any of those newly-reachable frames have corrupt stack pointers, missing DWARF info (JIT frames, vDSO, tcmalloc internals, etc), or otherwise bad state, libunwind reads from garbage addresses -- which would SIGSEGV inside the signal handler itself, explaining why no backtrace is emitted and the exit code is 139.

The aarch64_strip_pac_local path (which uses hint 0x7 / XPACLRI) is a NOP on Cortex-A76 since it predates ARMv8.3-A PAC support, so the actual address values shouldn't change -- the danger is just the continued unwinding into territory the old libunwind never reached.

---

Also suspicious: libunwind 1.8.1, DlPhdrStore replaces dl_iterate_phdr in DWARF lookups

MONGO_INITIALIZER(LibunwindPrefetch) now snapshots dl_iterate_phdr results into a DlPhdrStore and then calls unw_set_iterate_phdr_function to redirect all future DWARF frame lookups to the cached version (the point being to avoid deadlocking on the dynamic linker's lock inside signal handlers).

The snapshot is taken once, at initializer time. Any library loaded via dlopen after that point is invisible to libunwind -- it simply won't appear in the iteration, dwarf_find_proc_info will return -UNW_ENOINFO for addresses in those libraries, and libunwind may attempt a frame-pointer fallback from a corrupt frame. Whether MongoDB actually dlopens anything after that initializer runs is the question.

---

Less suspicious but worth noting: MozJS 115.20, NonVolatileSingleMask shrinks on arm64

In Architecture-arm64.h, NonVolatileSingleMask went from covering s8-s30 to only s8-s15. This is actually a correctness fix -- the ARM ABI only requires d8-d15 to be preserved by the callee, so the old mask was overly conservative. That said, if any JIT-compiled code (or code that interacts with JIT frames) happened to implicitly rely on s16-s30 surviving across a call, the correction would quietly corrupt those registers in callers. Probably not the cause here, but it's a real behavioral change for arm64 JIT.

---

Things to try to narrow it down:

First, check whether the MongoDB arm64 binary actually has DW_CFA_AARCH64_negate_ra_state in its .eh_frame:

readelf --debug-dump=frames /usr/bin/mongod | grep -c 'negate_ra_state'

If that returns a large number, the libunwind behavior change is almost certainly what's biting you.

Second, if it's possible to rebuild with a different libunwind backend (or with the old libunwind vendored in), that would isolate whether the signal handler is the crash site vs the primary crash being something else entirely.

Third: does mongod --version crash? That would rule out anything that requires actual server startup (storage engine, WiredTiger, replication, etc) and point straight at static/early init or libunwind.

_{(assisted by "claude my eyes right out")}

[tuffant21]

Thanks @tianon, how do I submit an upstream issue? I looked on https://github.com/mongodb/mongo and I do not see a tab for issues.

[yosifkit]

They have a jira instance at https://jira.mongodb.org/ that allows the community to sign up for an account. You can also search issues without logging in https://jira.mongodb.org/issues/?filter=-4.

[tuffant21]

Created https://jira.mongodb.org/browse/SERVER-129295

[tuffant21]

Work was marked as duplicate and is being worked on this ticket: https://jira.mongodb.org/browse/SERVER-125742

I'm not sure the standard proceedure with this open issue, but I'm inclined to leave it open until mongo fixes the issue on their end. Just for visibility in case others come across the issue on google. Happy to track and close this issue once Mongo resolves the issue on their end. Any feedback on that train of thought?

[tianon]

As long as you'll track and close it, sure -- we tend to prefer to close them because we're neither tracking nor directly fixing the issue, so it's not something we'll usually remember to come back to unless/until we're doing a cleaning sweep on open issues. 😅

[TaylorCarpenter]

Note that this affects x86_64 as well, on newer kernels. I see the same crash loop on kernel 7.0.0-22.

[tianon]

From that upstream MongoDB tracking issue:

The RSEQ fix was made available in the 7.0.10 release. The fix for arm64 is still pending at this time.

(so 7.0.0 is not new enough)