Skip to content

fix(vm): trap on stack oob at every executor exit - #178

Open
d1r1 wants to merge 1 commit into
claude/module-verification-bounds-2eb26ffrom
fix/trap-stack-oob-on-every-exit
Open

fix(vm): trap on stack oob at every executor exit#178
d1r1 wants to merge 1 commit into
claude/module-verification-bounds-2eb26ffrom
fix/trap-stack-oob-on-every-exit

Conversation

@d1r1

@d1r1 d1r1 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #171, on top of its branch.

Problem

RwasmExecutor::step checks the out-of-bounds flag only after execute returns Ok:

let return_reached = self.execute(instr)?;   // <- any Err short-circuits the check below
if self.sp.is_out_of_bounds() {
    return Err(TrapCode::StackOverflow);
}

Every exit that carries an error out of the instruction loop therefore drops the flag. run then calls value_stack.reset(), which clears it, and for TrapCode::ExecutionHalted returns Ok(()) — the one place in the interpreter that converts an Err into a success. run_with_stack_check and the InterruptionCalled branch lose it the same way.

That is reachable from bytecode new_verified accepts:

StackCheck(16)
Call(_exit)      ; the host function takes one i32, and the stack is empty
Return

Verification accepts the module — the emulated stack height turns Unknown after any call, a documented limitation of the pass added in #171. At runtime the syscall pops its parameter off an empty stack, the pop is suppressed and substitutes a zero, the host handler runs with that fabricated argument, returns ExecutionHalted, and execute hands the caller Ok(()). An invalid module finishes as a successful halt, after the host has already committed a side effect on values the module never pushed.

Fix

step converts the flag before the instruction's own error is propagated. Binding the result first covers every exit at once — both loops, the ExecutionHalted branch and the InterruptionCalled branch — so run and run_with_stack_check need no change:

let result = self.execute(instr);
if self.sp.is_out_of_bounds() {
    return Err(TrapCode::StackOverflow);
}
result

invoke_syscall traps after popping its parameters, before the handler runs. This one cannot be folded into step: step sees the flag only once the instruction returns, and by then the host has already observed the fabricated zeros and acted on them.

The existing check after the result-collection pops in run stays — those pops happen outside step.

Trap-code precedence

An out-of-bounds access now outranks the trap the instruction reported on its own. I32DivU on an underflowed stack reports IntegerDivisionByZero only because the suppressed pop fed it a zero divisor; the underflow is the cause, the division is the symptom.

The blast radius was measured rather than assumed: with ValueStackPtr::mark_out_of_bounds patched to panic!, 120 tests across the lib, compiler, fuel, locals, memory, stack-overflow, snippets and intrinsic pass without a single hit. Nothing RwasmModule::compile emits ever raises the flag, so the rule is observable only through RwasmModule::new_verified. The differential fuzzer is unaffected as well: it treats any trap on both sides as equivalent, and every module it generates goes through the compiler.

Tests

Two tests in tests/value-stack-bounds.rs, each pinning one of the two gates:

  • syscall_with_underflowing_params_traps_before_reaching_the_host — the repro above. Asserts Err(TrapCode::StackOverflow) and that the handler recorded nothing, and documents that new_verified accepts the module. On the parent branch it fails with left: Ok(()). With only the syscall gate removed it fails on the second assertion, with the handler having seen [0].
  • a_trap_caused_by_an_underflow_is_reported_as_the_underflowStackCheck(16); I32Const(0); I32DivU; Return. On the parent branch it fails with left: Err(IntegerDivisionByZero). This is the only bytecode-level way to exercise the precedence rule, and it is the test that fails if the step change alone is reverted.

InterruptionCalled gets no dedicated test on purpose: after this change it is not a separate code path, and once the syscall gate is in place there is no way to raise the flag inside an instruction that goes on to return it, since handlers are the only producers of that trap code.

Verification

  • cargo test — 179 passed, 4 ignored, including interruption, wasmtime, fluentbase and fuzz.
  • cargo test --no-default-features --features std over the targets that build without the wasmtime feature — 130 passed.
  • cargo clippy --all-targets clean.
  • cargo +nightly-2025-09-20 fmt --check reports no diff in either touched file.

- invalid module no longer exits as a successful halt
- host syscall never runs on values from a bad pop
- out-of-bounds now outranks the trap it caused
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d51e303d-875a-486d-be4b-11d67a40a153

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

Criterion results (vs baseline)


running 83 tests
test compiler::compiled_expr::tests::compiledexpr_eval_const_returns_none_for_global_or_funcref ... ignored
test compiler::compiled_expr::tests::compiledexpr_from_const_roundtrips ... ignored
test compiler::compiled_expr::tests::compiledexpr_funcref_and_global_introspection ... ignored
test compiler::compiled_expr::tests::compiledexpr_new_global_get_uses_context ... ignored
test compiler::compiled_expr::tests::compiledexpr_new_i32_add_mixed_const_and_global ... ignored
test compiler::compiled_expr::tests::compiledexpr_new_i32_add_mixed_global_and_funcref ... ignored
test compiler::compiled_expr::tests::compiledexpr_new_i32_add_wraps ... ignored
test compiler::compiled_expr::tests::compiledexpr_new_i32_const ... ignored
test compiler::compiled_expr::tests::compiledexpr_new_i32_sub_wraps ... ignored
test compiler::compiled_expr::tests::compiledexpr_new_i64_const ... ignored
test compiler::compiled_expr::tests::compiledexpr_new_i64_mul_wraps ... ignored
test compiler::compiled_expr::tests::compiledexpr_new_ref_func_uses_context ... ignored
test compiler::compiled_expr::tests::compiledexpr_zero_is_zero ... ignored
test compiler::compiled_expr::tests::constop_eval_returns_value ... ignored
test compiler::compiled_expr::tests::empty_eval_context_always_none ... ignored
test compiler::compiled_expr::tests::eval_with_context_reads_globals_and_funcs ... ignored
test compiler::compiled_expr::tests::expr_op_combines_operands_and_propagates_none ... ignored
test compiler::compiled_expr::tests::funcrefop_reads_from_context ... ignored
test compiler::compiled_expr::tests::globalop_maps_value_kinds_correctly ... ignored
test compiler::compiled_expr::tests::op_clone_panics_for_expr_variant - should panic ... ignored
test compiler::compiled_expr::tests::op_clone_works_for_non_expr_variants ... ignored
test compiler::compiled_expr::tests::op_constant_encodes_f32_f64_bits ... ignored
test compiler::compiled_expr::tests::op_constant_encodes_funcref_externref_ids ... ignored
test compiler::compiled_expr::tests::op_constant_encodes_i32_i64 ... ignored
test compiler::drop_keep::tests::test_drop_keep_translation ... ignored
test compiler::func_type_registry::tests::deduplicates_matching_signatures ... ignored
test compiler::func_type_registry::tests::index_lookup_is_stable ... ignored
test compiler::func_type_registry::tests::resolves_unique_signatures_correctly ... ignored
test compiler::parser::tests::unsupported_component_model_returns_error ... ignored
test module::tests::test_decode_exact_rejects_trailing_garbage ... ignored
test module::tests::test_decode_module_wo_source_pc ... ignored
test module::tests::test_decode_rejects_partial_source_pc ... ignored
test module::tests::test_endianness ... ignored
test module::tests::test_module_encoding ... ignored
test module::verification::tests::accepts_bulk_operands_covered_by_a_stack_reservation ... ignored
test module::verification::tests::accepts_locals_addressed_below_the_function_entry ... ignored
test module::verification::tests::accepts_verified_encoded_module ... ignored
test module::verification::tests::regular_construction_does_not_verify ... ignored
test module::verification::tests::regular_decode_does_not_verify ... ignored
test module::verification::tests::rejects_branch_target_outside_code_section ... ignored
test module::verification::tests::rejects_bulk_operands_beyond_the_value_stack ... ignored
test module::verification::tests::rejects_call_target_outside_code_section ... ignored
test module::verification::tests::rejects_code_running_past_the_code_section ... ignored
test module::verification::tests::rejects_local_depth_beyond_the_value_stack ... ignored
test module::verification::tests::rejects_local_depth_beyond_the_value_stack_after_a_host_call ... ignored
test module::verification::tests::rejects_missing_table_index_payload ... ignored
test module::verification::tests::rejects_popping_below_the_value_stack ... ignored
test module::verification::tests::rejects_pushing_beyond_the_value_stack ... ignored
test module::verification::tests::rejects_section_index_outside_limits ... ignored
test module::verification::tests::rejects_source_pc_outside_code_section ... ignored
test module::verification::tests::rejects_zero_local_depth ... ignored
test strategy::types::tests::checked_memory_range_end_rejects_overflow ... ignored
test types::nan_preserving_float::tests::test_neg_nan_f32 ... ignored
test types::nan_preserving_float::tests::test_neg_nan_f64 ... ignored
test types::nan_preserving_float::tests::test_ops_f32 ... ignored
test types::nan_preserving_float::tests::test_ops_f64 ... ignored
test types::opcode::tests::test_fpu_opcode_encoding_uses_offset ... ignored
test types::opcode::tests::test_opcode_code_values ... ignored
test types::opcode::tests::test_opcode_encoding ... ignored
test types::opcode::tests::test_opcode_encoding_uses_explicit_code ... ignored
test types::opcode::tests::test_opcode_size ... ignored
test types::units::tests::bytes_new16 ... ignored
test types::units::tests::bytes_new32 ... ignored
test types::units::tests::bytes_new64 ... ignored
test types::units::tests::pages_checked_add ... ignored
test types::units::tests::pages_checked_sub ... ignored
test types::units::tests::pages_max ... ignored
test types::units::tests::pages_new ... ignored
test types::units::tests::pages_to_bytes ... ignored
test types::value::copysign_regression_works ... ignored
test types::value::wasm_float_max_regression_works ... ignored
test types::value::wasm_float_min_regression_works ... ignored
test vm::store::tests::clamps_runtime_memory_limit_to_global_maximum ... ignored
test wasmtime::tests::test_call_with_charging_linear_wasmtime ... ignored
test wasmtime::tests::test_call_with_charging_param_overflow_wasmtime ... ignored
test wasmtime::tests::test_call_with_charging_quadratic_wasmtime ... ignored
test wasmtime::tests::test_wasmtime_caller_memory_read_into_vec_checks_bounds_before_allocating ... ignored
test wasmtime::tests::test_wasmtime_caller_missing_memory_returns_trap ... ignored
test wasmtime::tests::test_wasmtime_executor_memory_read_into_vec_checks_bounds_before_allocating ... ignored
test wasmtime::tests::test_wasmtime_executor_missing_entrypoint_returns_trap ... ignored
test wasmtime::tests::test_wasmtime_snapshot_missing_memory_returns_trap ... ignored
test wasmtime::types::tests::maps_unknown_wasmtime_error_to_illegal_opcode ... ignored
test wasmtime::types::tests::maps_wasmtime_traps_to_rwasm_traps ... ignored

test result: ok. 0 passed; 0 failed; 83 ignored; 0 measured; 0 filtered out; finished in 0.00s

Comparisons/bench_native
                        time:   [5.1294 ns 5.1563 ns 5.1852 ns]
Found 110 outliers among 1000 measurements (11.00%)
  28 (2.80%) high mild
  82 (8.20%) high severe
Comparisons/bench_strategy_wasmtime
                        time:   [10.889 µs 11.041 µs 11.221 µs]
Found 131 outliers among 1000 measurements (13.10%)
  23 (2.30%) high mild
  108 (10.80%) high severe
Comparisons/bench_strategy_rwasm
                        time:   [8.5865 µs 8.6405 µs 8.6975 µs]
Found 104 outliers among 1000 measurements (10.40%)
  31 (3.10%) high mild
  73 (7.30%) high severe

Heads-up: runner perf is noisy; treat deltas as a smoke check.

@codecov

codecov Bot commented Aug 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant