how can I reproduce your benchmarks?

[Robbepop]

Hi, I would like to reproduce your CoreMark benchmarks where you used Wasmi v2.0.0-beta.7 since it looks very different from what I'd expect with the comparison to Wasm3.

• What optimization flags have been used?
• What crate feature sets have been enabled?
• Was it really Wasm v2.0.0-beta.7 oder v1.0.9? (looks more like v1.0.9 to me tbh)
• Also: I update Wasmi to v2.0.0-beta.8 today. :)

What I would expect: on Apple Silicon, CoreMark results for both Wasm3 and Wasmi v2.0.0-beta.7 are very close.

[mbbill]

before giving you a how-to for running benchmarks, let me double check to make sure things are running as expected

[mbbill]

the coremark.wasm is from my own build so in order to make sure there is nothing from sf-nano affect the benchmark, I used wasmi from crates.io and coremark.wasm from wasm3 repo. My laptop is Macbook air M4.

[main] ~/code/Silverfir-nano
> cargo +1.91.1 install wasmi_cli --version 2.0.0-beta.7 --locked --root ~/wasmi-beta7
    Updating crates.io index
  Installing wasmi_cli v2.0.0-beta.7
    Updating crates.io index
warning: package `spin v0.9.8` in Cargo.lock is yanked in registry `crates-io`, consider running without --locked
    Updating crates.io index
   Compiling proc-macro2 v1.0.106
   Compiling unicode-ident v1.0.24
   Compiling libc v0.2.186
   Compiling quote v1.0.46
   Compiling bitflags v2.13.0
   Compiling anyhow v1.0.103
   Compiling io-lifetimes v2.0.4
   Compiling rustix v1.1.4
   Compiling io-extras v0.18.4
   Compiling zerocopy v0.8.52
   Compiling leb128fmt v0.1.0
   Compiling cap-primitives v3.4.5
   Compiling ambient-authority v0.0.2
   Compiling thiserror v1.0.69
   Compiling libm v0.2.16
   Compiling heck v0.5.0
   Compiling ipnet v2.12.0
   Compiling cfg-if v1.0.4
   Compiling maybe-owned v0.3.4
   Compiling leb128 v0.2.7
   Compiling wast v35.0.2
   Compiling log v0.4.33
   Compiling bumpalo v3.20.3
   Compiling unicode-width v0.2.2
   Compiling cap-std v3.4.5
   Compiling memchr v2.8.2
   Compiling wasm-encoder v0.252.0
   Compiling rustix v0.38.44
   Compiling once_cell v1.21.4
   Compiling wiggle-macro v36.0.12
   Compiling wasmi_ir v2.0.0-beta.7
   Compiling wasmparser v0.228.0
   Compiling thiserror v2.0.18
   Compiling syn v2.0.118
   Compiling errno v0.3.14
   Compiling getrandom v0.2.17
   Compiling rand_core v0.6.4
   Compiling wast v252.0.0
   Compiling tracing-core v0.1.36
   Compiling wasmi_core v2.0.0-beta.7
   Compiling core-foundation-sys v0.8.7
   Compiling pin-project-lite v0.2.17
   Compiling utf8parse v0.2.2
   Compiling wasm-encoder v0.228.0
   Compiling cap-fs-ext v3.4.5
   Compiling system-interface v0.27.3
   Compiling wasmi v2.0.0-beta.7
   Compiling iana-time-zone v0.1.65
   Compiling anstyle-parse v1.0.0
   Compiling ppv-lite86 v0.2.21
   Compiling fs-set-times v0.20.3
   Compiling rand_chacha v0.3.1
   Compiling rand v0.8.6
   Compiling spin v0.9.8
   Compiling anstyle v1.0.14
   Compiling is_terminal_polyfill v1.70.2
   Compiling colorchoice v1.0.5
   Compiling anstyle-query v1.1.5
   Compiling wasmi_collections v2.0.0-beta.7
   Compiling cap-rand v3.4.5
   Compiling anstream v1.0.0
   Compiling cap-time-ext v3.4.5
   Compiling strsim v0.11.1
   Compiling clap_lex v1.1.0
   Compiling clap_builder v4.6.0
   Compiling thiserror-impl v1.0.69
   Compiling thiserror-impl v2.0.18
   Compiling tracing-attributes v0.1.31
   Compiling async-trait v0.1.89
   Compiling wat v1.252.0
   Compiling wast v228.0.0
   Compiling clap_derive v4.6.1
   Compiling tracing v0.1.44
   Compiling witx v0.9.1
   Compiling clap v4.6.1
   Compiling wiggle-generate v36.0.12
   Compiling wiggle v36.0.12
   Compiling wasi-common v36.0.12
   Compiling wasmi_wast v2.0.0-beta.7
   Compiling wasmi_wasi v2.0.0-beta.7
   Compiling wasmi_cli v2.0.0-beta.7
    Finished `release` profile [optimized] target(s) in 49.33s
  Installing /Users/bytedance/wasmi-beta7/bin/wasmi
   Installed package `wasmi_cli v2.0.0-beta.7` (executable `wasmi`)
warning: be sure to add `/Users/bytedance/wasmi-beta7/bin` to your PATH to be able to run the installed binaries

[main] ~/code/Silverfir-nano
> ~/wasmi-beta7/bin/wasmi --version
wasmi 2.0.0-beta.7

[main] ~/code/Silverfir-nano
> ~/wasmi-beta7/bin/wasmi ../wasm3/bench/coremark.wasm
2K performance run parameters for coremark.
CoreMark Size    : 666
Total ticks      : 12596
Total time (secs): 12.596000
Iterations/Sec   : 3175.611305
Iterations       : 40000
Compiler version : Clang 22.0.0git (https:/github.com/llvm/llvm-project 7f93487862d98bf1c168babba87daf6224d8a46f)
Compiler flags   :
Memory location  : WASM linear memory
seedcrc          : 0xe9f5
[0]crclist       : 0xe714
[0]crcmatrix     : 0x1fd7
[0]crcstate      : 0x8e3a
[0]crcfinal      : 0x25b5
Correct operation validated. See README.md for run and reporting rules.
CoreMark 1.0 : 3175.611305 / Clang 22.0.0git (https:/github.com/llvm/llvm-project 7f93487862d98bf1c168babba87daf6224d8a46f)  / WASM linear memory

[main] ~/code/Silverfir-nano
> ../wasm3/build-release/wasm3 ../wasm3/bench/coremark.wasm
2K performance run parameters for coremark.
CoreMark Size    : 666
Total ticks      : 14107
Total time (secs): 14.107000
Iterations/Sec   : 4253.207627
Iterations       : 60000
Compiler version : Clang 22.0.0git (https:/github.com/llvm/llvm-project 7f93487862d98bf1c168babba87daf6224d8a46f)
Compiler flags   :
Memory location  : WASM linear memory
seedcrc          : 0xe9f5
[0]crclist       : 0xe714
[0]crcmatrix     : 0x1fd7
[0]crcstate      : 0x8e3a
[0]crcfinal      : 0xbd59
Correct operation validated. See README.md for run and reporting rules.
CoreMark 1.0 : 4253.207627 / Clang 22.0.0git (https:/github.com/llvm/llvm-project 7f93487862d98bf1c168babba87daf6224d8a46f)  / WASM linear memory

so the result is: wasmi vs wasm3: 3175 vs 4253, multiple runs doesn't change the result much.
By the way I used the wasmi-benchmarks and got similar results. Feel free to let me know if you want me to try anything differently.

[mbbill]

I did one more test, used taskpolicy -b to force E core on both, and got: wasmi vs wasm3: 1285 - 1249. wasmi do get a lead on E cores. It seems like wasmi is sensitive to arch, maybe the ooo or wider issue on newer P cores helped wasm3 more.
Hope that helps!

[Robbepop]

I did one more test, used taskpolicy -b to force E core on both, and got: wasmi vs wasm3: 1285 - 1249. wasmi do get a lead on E cores. It seems like wasmi is sensitive to arch, maybe the ooo or wider issue on newer P cores helped wasm3 more.
Hope that helps!

I think it might be connected to missing optimizations flags.
In wasmi benchmarks I always use

lto = "fat"
codegen-units = 1

which affects Wasmi significantly whereas Wasm3 is unaffected as it is a C runtime.

So in wasmi benchmarks I always use the predefined --profile bench when running benchmarks or its CoreMark via cargo run --profile bench.

[mbbill]

I think you have the fat LTO on by default

[profile.release]
lto = "fat"
codegen-units = 1

https://crates.io/crates/wasmi_cli/2.0.0-beta.7/code/Cargo.toml

and my test shows it's on during local compilation:

[main] ~/code/Silverfir-nano
> cargo +1.91.1 install wasmi_cli --version 2.0.0-beta.7 --locked --force -v --root /tmp/testwasmi

--error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=103 --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto=fat -C codegen-units=1 --cfg 'feature="auto-dispatch"' --cfg 'feature="default"' --cfg 'feature="memory64"' --cfg 'feature="run"' --cfg 'feature="stable"' --cfg 'feature="validate"' --cfg 'feature="wasi"' --cfg 'feature="wasmi_wasi"' --cfg 'feature="wasmi_wast"' --cfg 'feature="wast"' --cfg 'feature="wat"' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values("auto-dispatch", "debug", "default", "deterministic", "hash-collections", "indirect-dispatch", "memory64", "portable-dispatch", "prefer-btree-collections

Also, if I manually turn off fat lto, I got stack overflow on some of my benchmarks, like bzip2. Coremark is still runnable under lto off, score is slightly lower compared to default fat LTO on (-1%, consistent).

[Robbepop]

@mbbill thanks for all the info!

Is there a reason you are using such an old rustc version?

Current stable is rustc 1.97.1. I have seen quite some performance regressions between compiler versions. For example the makepad-stitch Wasm interpreter regressed dramatically from rustc 1.91 to newer versions on my M2 Pro. Usually it is because of an LLVM update. I have not yet found the root cause of most of those sporadic regressions. Though, Wasmi should be fine within that range IIRC.

The enabled Wasmi crate features should be fine I guess.
Interesting information for me, thanks again!

[mbbill]

upgraded my toolchain to the latest stable and the score stays the same. Let me know if you want me to test more

[Robbepop]

@mbbill thanks a lot but I guess the remaining analysis work is on me now.

[Robbepop]

@mbbill if you have time it would be great if you could benchmark Wasmi v2.0.0-beta.9 (just released) on your Apple M4. I'd be interested if the recent changes since v2.0.0-beta.7 improved the situation.

[mbbill]

yeah no problem.

[mbbill]

results are out:
toolchain with the latest stable 1.97, compared beta.7, beta.9 and wasm3, score average at:
3120, 3143, 4266
slightly improved but not much.

[Robbepop]

thank you! I guess, it will be hard to debug this performance regression for me without owning an M4 myself.

i notice that sometimes high-perf Wasm runtimes such as Wasm3, Wasmi and Stitch are highly affected by low-level things such as placement of execution handlers etc. So maybe this is an artifact of something like that. Kinda hard to get it right everywhere. That's where template interpreter approaches such as Java's Hotspot interpreter or yours show their strengths.

[mbbill]

I can help, but I suspect it might be related to the CPU core arch behaviors, which makes it more difficult to tune. I checked your latest commits on beta 9, it seems like most of the performance gain should come from the improved global access, but the global access is not quite frequent in coremark, so I think the improvement from coremark score alone aligns with the changes. However, when I run it on E cores, wasmi beats wasm3 by a tiny bit, unlike running on P cores. That's why I suspect a larger ooo window or issue width might be the root cause. I'll do more experiments and once I have more findings will let you know

[Robbepop]

beta.8 and beta.9 improve performance of access instance related data in general, e.g. memory, global, func, table, data, elem. this is just very visible in the existing count/globals benchmarks which improved by ~40%. however, all instance related accesses imrproved a lot and thus made the generated machine code denser etc. though, you are right that CoreMark mostly exercises integer compute heavy workloads.

again thank you for the help!

[mbbill]

                 raw medians (0.01% units)      sum
cm-wasmi   rows=10991  [2290, 5233, 1208, 1212]  9943
cm-wasm3   rows=11113  [3960, 2203, 2422, 1345]  9930

Data from xctrace. From left to right: useful, Processing (backend stall), Discarded (predictor), Delivery (frontend)
the actual delivered instruction of wasmi on wide issue processors like M4 is way behind wasm3, and mostly due to the >50% backend stall. still digging why backend stall is this high.