Deeply nested expressions trap with "out of bounds memory access" (WASM stack overflow) instead of a catchable error

[paulmanoni]

Summary

Evaluating a deeply‑nested expression traps with wasm error: out of bounds memory access instead of returning a catchable error. The trap is a WASM call‑stack (shadow‑stack) overflow from QuickJS's recursive C parser/evaluator, and it happens at a fairly low nesting depth. Two problems compound it:

1. The depth ceiling is low enough that real‑world code hits it (see "Real‑world impact" below).
2. The overflow surfaces as a Go panic (from (*Runtime).call), not an error, and it leaves the runtime/context unusable for subsequent calls.

Notably, Option.MaxStackSize and Option.MemoryLimit do not prevent it — MaxStackSize bounds QuickJS's JS‑level recursion (which would throw a catchable RangeError), but this overflow is in QuickJS's C‑level recursion compiled into the WASM, which exhausts the module's shadow stack first.

Environment

• github.com/fastschema/qjs v0.0.6
• github.com/tetratelabs/wazero v1.9.0
• Go 1.26.2, darwin/arm64 (also reproduces on linux/amd64)

Minimal repro (no other dependencies)

package main

import (
	"fmt"
	"strings"

	"github.com/fastschema/qjs"
)

func tryDepth(n int) {
	defer func() {
		if r := recover(); r != nil {
			fmt.Printf("depth=%-6d PANIC: %v\n", n, r)
		}
	}()
	// Generous QuickJS limits — these do NOT prevent the trap.
	rt, err := qjs.New(qjs.Option{MaxStackSize: 64 << 20, MemoryLimit: 1 << 30})
	if err != nil {
		panic(err)
	}
	defer rt.Close()

	code := strings.Repeat("(", n) + "1" + strings.Repeat(")", n)
	if _, err := rt.Context().Eval("x.js", qjs.Code(code)); err != nil {
		fmt.Printf("depth=%-6d error: %v\n", n, err)
		return
	}
	fmt.Printf("depth=%-6d ok\n", n)
}

func main() {
	for _, n := range []int{100, 1000, 5000} {
		tryDepth(n)
	}
}

Output:

depth=100    ok
depth=1000   PANIC: failed to call QJS_Eval: wasm error: out of bounds memory access
depth=5000   PANIC: failed to call QJS_Eval: wasm error: out of bounds memory access

(Nested array literals [[[…]]] and nested ternaries behave the same; member chains x?.b?.b… trap around depth ~200.)

Observed vs. expected

• Observed: a Go panic failed to call QJS_Eval: wasm error: out of bounds memory access. The runtime is then unusable — subsequent Evals on it also fail, and Close() panics during teardown.
• Expected: either deeper nesting is supported, or QuickJS's stack‑overflow check fires and Eval returns a normal, catchable error (e.g. a RangeError: Maximum call stack size exceeded) that leaves the runtime usable.

Real‑world impact

I'm using qjs to run @vue/compiler-sfc (which embeds @babel/parser) in‑process as a CGo‑free alternative to a native QuickJS binding. The compiler adds its own recursion on top of the source's nesting, so the effective ceiling drops to only ~11–20 levels of source nesting. Ordinary Vue <script setup> code — an array of config objects with arrow callbacks containing ternaries, optional chaining, and template literals — trips it. Across one real project, 10 of 128 components fail this way, while the equivalent native (CGo) QuickJS binding compiles all 128 fine (it has a large C stack and proper stack‑overflow detection).

Suggestions

1. Grow the WASM shadow/stack (or expose it as an Option) so realistic recursion depths are supported. Wazero lets you configure module memory; the QuickJS‑NG wasm build's stack size is the key lever.
2. Don't panic on wasm traps — have (*Runtime).call / Eval return the trap as an error, and ideally let QuickJS's own stack‑overflow guard (driven by MaxStackSize) fire first so it's a catchable RangeError rather than a hard trap. A single deep Eval shouldn't poison the runtime or crash the host.

Happy to test a patch. Thanks for the library — where it doesn't hit this, output has been byte‑identical to a native QuickJS binding.

[paulmanoni]

Dug into the build — I think this is a linker stack-size setting, and the fix looks small. Sharing in case it helps.

Root cause

qjswasm/qjswasm.cmake sets no -z stack-size, so qjs.wasm links with wasm-ld's default C stack (~1 MiB). QuickJS's parser/evaluator recurse in C, so deep expressions exhaust that fixed shadow stack and Wazero traps it as out of bounds memory access.

Crucially, the wasm stack is not coupled to QuickJS's own overflow guard. In qjswasm/qjs.c:

if (max_stack_size > 0)
    JS_SetMaxStackSize(runtime, max_stack_size);

• With the Go default Option.MaxStackSize == 0, JS_SetMaxStackSize is never called, so QuickJS keeps its built-in default max stack (larger than the ~1 MiB wasm stack). Its guard therefore never fires before the wasm shadow stack runs out → hard trap instead of a catchable RangeError.
• Raising Option.MaxStackSize makes it worse: QuickJS then believes it has even more stack and guards even later, while the real wasm stack is unchanged. (This is why MaxStackSize: 64<<20 had no effect in the repro — and the trap fires at the same depth regardless.)

Suggested fix (two coordinated parts)

1. Grow the wasm C stack in qjswasm.cmake, e.g.:

   target_link_options(qjswasm PRIVATE
       "LINKER:-z,stack-size=16777216"   # 16 MiB
   )

   The current --initial-memory=1310720 (20 pages ≈ 1.25 MiB) would also need to grow to cover the larger stack (the C stack lives in linear memory), plus headroom — e.g. bump --initial-memory and/or allow memory to grow.

2. Couple QuickJS's guard to the wasm stack so it throws a catchable error before the shadow stack overflows — set a sane default max stack (just under the linker stack-size) when the caller passes 0, rather than leaving QuickJS's larger built-in default. Then JS_UpdateStackTop + the guard produce a RangeError: Maximum call stack size exceeded that Eval can return as an error.

3. (Go side, independent) Having (*Runtime).call return wasm traps as an error instead of panic-ing would also stop a single deep Eval from poisoning the runtime / crashing the host, even before the stack fix lands.

I couldn't rebuild qjs.wasm to verify (no WASI SDK on hand), so the exact stack-size value is a starting point — but the missing -z stack-size + the max_stack_size == 0 path look like the levers. Happy to test a build if you push a branch.

[paulmanoni]

Opened a PR with the fix: #48

It moves the stack sizing into the target_link_options(qjswasm …) block (the existing --stack-first/--initial-memory flags were add_link_options placed after add_executable, so they never applied to the target) and bumps the C stack to 16 MiB. Verified: nested-paren Eval now succeeds to depth 20000 (was trapping at 1000), and a real 128-component Vue app compiled via @vue/compiler-sfc goes from 10 failing files to 0.