feat(cranelift): emulate the rwasm stack limits in compiled code

[dmitry123]

Summary

The rwasm VM stops an execution at N_MAX_RECURSION_DEPTH frames or when the frames on the call chain exceed its value-stack window. Wasmtime only ran out of native stack, so a recursion 1024 deep or two nested frames of 5000 slots ran here and trapped StackOverflow on rwasm. This PR makes the Cranelift backend trap exactly where the rwasm VM does, so both engines agree at every boundary.

Stacked on #12 (bulk fuel, 45.0.0-rwasm.3); the rwasm counterpart is fluentlabs-xyz/rwasm#213, which pins 45.0.0-rwasm.3 and cannot build until this is released.

How it works

Config::rwasm_stack_limits(Some(RwasmStackLimits { max_call_depth, max_stack_slots, code_snippets })) makes compiled code keep two counters in VMStoreContext, packed into one u64 after execution_version (rwasm_stack: call depth in the high half, frame base in the low half; the offset test covers the new layout):

• Function prologue: after the entry fuel charge, loads both counters and traps StackOverflow when base + params + height > max_stack_slots, like rwasm's StackCheck. height is the function's entry in the module's rwasm.frames custom section (RWASM_FRAMES_SECTION, one little-endian u32 per function, imports first), written by the rwasm translator, which alone knows the temporaries its lowering pushes. A module without the section, or with the wrong length, fails to compile on such an engine.
• Call sites (call, call_indirect, call_ref): trap when depth >= max_call_depth (one compare of the packed word), then publish the callee's counters with one add and one store: depth plus one, and the callee's frame base (this function's parameters, locals and the operand slots below the arguments, computed from the validator's operand stack). Nothing is restored after the call: every reader runs right after the call site that wrote the counters, and the caller keeps its own in a register. For call_indirect the check sits after the load that traps on a null entry and before the signature compare, for call_ref after an explicit null check, the order rwasm traps in. Tail calls keep both counters, like ReturnCallInternal.
• Snippet operators: with code_snippets, the i64 operators rwasm runs in a hidden CallInternal frame (rwasm_snippet_frames: one frame, two for div/rem which call the shared UDivMod64 core) check that frame's depth and height before the operator runs. The table mirrors rwasm's MSH_* constants; rwasm pins it to its snippet definitions.
• Host functions: the embedder checks the import trampoline's frame itself (rwasm's trampoline has a frame of its own), reading the counters through Store::rwasm_stack_counters / Caller::rwasm_stack_counters, and resets them with Store::set_rwasm_stack_counters before each call into the guest.

Without rwasm_stack_limits nothing changes: the counters stay unused and no code is emitted.

Cost

Measured on Apple Silicon at OptLevel::Speed, best of 5, against the same build with the emulation switched off:

workload	off	on
recursive fib(30) through call (1.6M calls)	4.9-5.1 ms	5.2 ms
recursive fib(30) through call_indirect	5.1 ms	5.4 ms
10M iterations of i64.mul + i64.div_u	17.6 ms	18.2 ms

A call-only recursion pays 3-5% for the depth compare, the add and the store per call plus the load and compare per prologue; arithmetic loops are within noise. The first version, with two stores before and two after every call, cost about 30% on the same recursion.

Verification

• cargo test -p wasmtime-rwasm --features cranelift --lib: green (151 passed, field_offsets included)
• rwasm's full suite, e2e spec suite and clippy against this branch through [patch.crates-io]: green (see fix(wasmtime): emulate the rwasm stack limits on the Wasmtime backend rwasm#213)
• clippy clean on the touched files

[dmitry123]

Applied the same fix to RwasmStackLimits (9648c93), on top of a merge of the updated #12 (6a40ea0):

• RwasmStackLimits now derives Hash and serde. It is hashed into HashedEngineCompileEnv next to rwasm_bulk_fuel, recorded in the precompiled Metadata, and checked on load, including on cache hits.

• New tests in engine/serialization.rs:

  • precompile_compatibility_key_accounts_for_rwasm_stack_limits: varies each field
  • cache_accounts_for_rwasm_stack_limits
  • precompiled_module_requires_the_same_rwasm_stack_limits: a 5×5 matrix using a module that carries rwasm.frames

  All three fail without the fix. One of the failing cases is an artifact compiled without the stack checks deserializing into an engine that enforces them.

Local results (aarch64-apple-darwin), cargo test -p wasmtime-rwasm -p wasmtime-environ-rwasm --lib:

• wasmtime-rwasm: 157 passed, 1 ignored
• wasmtime-environ-rwasm: 44 passed

[hedwig0x]

Reviewed stack-limit emulation and the cache/precompile compatibility fix. Local wasmtime-rwasm test suite passes (173/173).