Skip to content

test: demonstrate a HybridFunction performance regression (DO NOT MERGE) - #1588

Draft
mrousavy wants to merge 2 commits into
ci/check-app-version-alignmentfrom
test/performance-regression-canary
Draft

test: demonstrate a HybridFunction performance regression (DO NOT MERGE)#1588
mrousavy wants to merge 2 commits into
ci/check-app-version-alignmentfrom
test/performance-regression-canary

Conversation

@mrousavy

@mrousavy mrousavy commented Sep 4, 2026

Copy link
Copy Markdown
Member

Performance regression canary — DO NOT MERGE

This draft PR intentionally adds 128 rounds of dependent unsigned arithmetic to
each HybridFunction native-state lookup. Volatile local storage keeps the work
in optimized Release builds without shared state, sleeps, allocations, or altered
return values. Closing this PR discards the entire experiment.

The base is the top of the infrastructure stack, including the version-alignment
PR. Both base and head already contain the same benchmark app and definitions.
CI must build both binaries independently and run the existing ABBA sequence.
The suite hash must remain identical; this is not a rebaseline or an A/A run.

Expected observations:

  • C++ and Swift/Kotlin primitive calls and property access get slower.
  • JS and TurboModule controls do not execute this extra work.
  • Expensive operations can show a much smaller relative change.
  • Promise metrics remain advisory. Performance verdicts overall remain advisory;
    a detected regression does not imply that the build check should turn red.

End-to-end validation checks the real GitHub Actions artifacts, confirms base
and head SHAs and matching suite hashes, inspects the paired PR comment, and
verifies Bencher reports for both platform testbeds and this PR branch. Bencher
publishing requires the rotated project key in BENCHER_KEY and
NITRO_BENCHER_ENABLED=true; the paired PR comment is independent of that key.

Reporting fixes belong in the infrastructure PR, not in this canary diff. Keep
this PR draft and never merge its intentional production slowdown.

First paired CI result

Run 33898772711
built both revisions independently and completed six base/head suites per
platform, including the one permitted noise retry. Both sides used Release
Nitro, Hermes, and the same benchmark-suite hash.

  • Android C++ addNumbers: 173.6 ns/op → 621.4 ns/op (+258%, 95% CI
    +247%…+261%). The JS control changed -0.6%; the TurboModule control remained
    inconclusive.
  • iOS C++ addNumbers: 161.5 ns/op → 450.5 ns/op (+179%, 95% CI
    +154%…+211%). The first reporter called every synchronous iOS case noisy
    because its robust CV exceeded 5%, even when the paired interval was
    decisive.

That result exposed a reporting-policy bug rather than a sampling failure. The
infrastructure PR now lets a change whose full paired confidence interval is
beyond the budget keep its regression/improvement verdict. CV still produces a
neutral result when the interval cannot decide. Rebuilding the report from the
same raw samples yields 29 Android and 25 iOS regressions while leaving the JS
and TurboModule controls neutral.

@vercel

vercel Bot commented Sep 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nitro-docs Ready Ready Preview Sep 5, 2026 11:48am UTC

Request Review

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Performance Report

⚠️ Advisory: Results do not fail this PR while the baseline is being calibrated.

iOS

Benchmark Before After Difference
C++ bounce(1 MiB) 471.2 ns 637.2 ns 🔴 +35.23% slower
C++ bounce(4 KiB) 429.9 ns 656.4 ns 🔴 +52.69% slower
C++ synchronousCallback() 407.7 ns 551.2 ns 🔴 +35.21% slower
C++ create() 776.3 ns 1.21 µs 🔴 +56.37% slower
C++ returnExisting() 321.0 ns 808.3 ns 🔴 +151.79% slower
C++ optional trailing string 254.2 ns 531.0 ns 🔴 +108.93% slower
C++ addNumbers() 134.0 ns 386.6 ns 🔴 +188.52% slower
C++ simpleFunc() 128.0 ns 417.2 ns 🔴 +226.04% slower
C++ number property get/set 226.9 ns 808.2 ns 🔴 +256.2% slower
C++ short ASCII string 271.6 ns 465.6 ns 🔴 +71.42% slower
C++ Unicode string 526.1 ns 764.1 ns 🔴 +45.25% slower
C++ number | string variant 236.1 ns 489.9 ns 🔴 +107.45% slower
Swift bounce(1 MiB) 540.6 ns 654.0 ns 🔴 +20.97% slower
Swift large array (1,024) 77.04 µs 63.09 µs 🟢 -18.11% faster
Swift synchronousCallback() 517.7 ns 635.4 ns 🔴 +22.74% slower
Swift create() 45.47 µs 36.45 µs 🟢 -19.84% faster
Swift returnExisting() 505.8 ns 884.1 ns 🔴 +74.79% slower
Swift typed map (8 entries) 4.30 µs 3.58 µs 🟢 -16.7% faster
Swift optional trailing string 299.5 ns 466.6 ns 🔴 +55.76% slower
Swift addNumbers() 156.1 ns 324.5 ns 🔴 +107.94% slower
Swift simpleFunc() 131.2 ns 323.3 ns 🔴 +146.47% slower
Swift number property get/set 293.2 ns 707.0 ns 🔴 +141.16% slower
Swift short ASCII string 262.2 ns 464.8 ns 🔴 +77.25% slower
Swift nested Car struct 4.06 µs 3.36 µs 🟢 -17.19% faster
Swift number | string variant 358.6 ns 531.1 ns 🔴 +48.1% slower
All Benchmarks
Benchmark Before After Difference
JavaScript addNumbers() 38.6 ns 37.6 ns 🟡 -2.54% faster (noisy)
C++ copy(1 MiB) 110.68 µs 67.78 µs 🟡 -38.76% faster (noisy)
C++ copy(4 KiB) 1.75 µs 1.56 µs 🟡 -10.86% faster (noisy)
C++ large array (1,024) 72.86 µs 73.33 µs 🟡 +0.65% slower (noisy)
C++ small array (16) 1.44 µs 1.55 µs 🟡 +7.7% slower (noisy)
C++ typed map (8 entries) 3.02 µs 2.98 µs 🟡 -1.35% faster (noisy)
C++ immediatePromise() 1.52 µs 1.43 µs ℹ️ -6.16% faster (advisory)
C++ nested Car struct 3.66 µs 3.67 µs 🟡 +0.09% slower (noisy)
Swift bounce(4 KiB) 562.5 ns 645.2 ns 🟡 +14.7% slower (noisy)
Swift copy(1 MiB) 98.55 µs 91.78 µs 🟡 -6.87% faster (noisy)
Swift copy(4 KiB) 2.18 µs 1.79 µs 🟡 -17.82% faster (noisy)
Swift small array (16) 1.59 µs 1.50 µs 🟡 -5.6% faster (noisy)
Swift immediatePromise() 2.18 µs 2.26 µs ℹ️ +3.92% slower (advisory)
Swift Unicode string 706.1 ns 791.6 ns 🟡 +12.11% slower (noisy)
TurboModule addNumbers() 1.53 µs 1.51 µs 🟡 -1.62% faster (noisy)

Android

Benchmark Before After Difference
C++ bounce(1 MiB) 638.3 ns 993.0 ns 🔴 +55.57% slower
C++ bounce(4 KiB) 646.2 ns 1.03 µs 🔴 +59.02% slower
C++ copy(4 KiB) 1.57 µs 1.92 µs 🔴 +22.67% slower
C++ small array (16) 1.69 µs 2.11 µs 🔴 +24.61% slower
C++ synchronousCallback() 538.8 ns 922.7 ns 🔴 +71.26% slower
C++ create() 1.09 µs 1.89 µs 🔴 +73.21% slower
C++ returnExisting() 375.9 ns 1.20 µs 🔴 +218.06% slower
C++ typed map (8 entries) 3.37 µs 3.83 µs 🔴 +13.69% slower
C++ optional trailing string 275.3 ns 687.7 ns 🔴 +149.84% slower
C++ addNumbers() 167.8 ns 575.5 ns 🔴 +242.89% slower
C++ simpleFunc() 133.8 ns 546.1 ns 🔴 +308.19% slower
C++ number property get/set 267.5 ns 1.10 µs 🔴 +311.24% slower
C++ short ASCII string 245.8 ns 658.4 ns 🔴 +167.85% slower
C++ Unicode string 536.0 ns 945.7 ns 🔴 +76.45% slower
C++ nested Car struct 4.45 µs 4.91 µs 🔴 +10.46% slower
C++ number | string variant 255.3 ns 660.2 ns 🔴 +158.59% slower
Kotlin bounce(1 MiB) 1.85 µs 2.32 µs 🔴 +25.29% slower
Kotlin bounce(4 KiB) 1.87 µs 2.26 µs 🔴 +20.59% slower
Kotlin copy(1 MiB) 498.81 µs 459.81 µs 🟢 -7.82% faster
Kotlin copy(4 KiB) 8.92 µs 9.59 µs 🔴 +7.6% slower
Kotlin small array (16) 2.31 µs 2.73 µs 🔴 +18.56% slower
Kotlin synchronousCallback() 1.75 µs 2.15 µs 🔴 +22.47% slower
Kotlin create() 3.28 µs 4.00 µs 🔴 +21.87% slower
Kotlin returnExisting() 1.23 µs 2.08 µs 🔴 +68.57% slower
Kotlin typed map (8 entries) 13.23 µs 13.89 µs 🔴 +5.03% slower
Kotlin optional trailing string 687.5 ns 1.12 µs 🔴 +62.64% slower
Kotlin addNumbers() 260.0 ns 676.2 ns 🔴 +160.05% slower
Kotlin simpleFunc() 205.3 ns 624.8 ns 🔴 +204.37% slower
Kotlin number property get/set 544.6 ns 1.38 µs 🔴 +153.24% slower
Kotlin short ASCII string 918.2 ns 1.34 µs 🔴 +46.31% slower
Kotlin Unicode string 1.51 µs 1.92 µs 🔴 +27.42% slower
Kotlin nested Car struct 9.58 µs 10.11 µs 🔴 +5.52% slower
Kotlin number | string variant 866.7 ns 1.27 µs 🔴 +46.25% slower
All Benchmarks
Benchmark Before After Difference
JavaScript addNumbers() 43.9 ns 43.6 ns ⚪ ~0.7% unchanged
C++ copy(1 MiB) 33.25 µs 34.28 µs ⚪ ~3.1% unchanged
C++ large array (1,024) 83.13 µs 83.96 µs ⚪ ~1% unchanged
C++ immediatePromise() 1.72 µs 2.15 µs ℹ️ +24.54% slower (advisory)
Kotlin large array (1,024) 93.22 µs 94.52 µs ⚪ ~1.4% unchanged
Kotlin immediatePromise() 4.64 µs 5.07 µs ℹ️ +9.31% slower (advisory)
TurboModule addNumbers() 542.6 ns 541.6 ns ⚪ ~0.17% unchanged

Benchmarking Code Diff 757b725a...608e0390 (view raw output)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant