Skip to content

test(e2e): user-journey suite with outcome assertions - #227

Merged
starswallow merged 1 commit into
mainfrom
test/user-journeys
Sep 23, 2026
Merged

test(e2e): user-journey suite with outcome assertions#227
starswallow merged 1 commit into
mainfrom
test/user-journeys

Conversation

@starswallow

@starswallow starswallow commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

The journey-shaped suites assert that commands exit 0 and that scaffolds exist. They do not assert that the thing the user asked for happened, and a failed build was allowed to be a passing test. This adds the missing layer and tightens the old one.

New: horus_manager/tests/user_journeys.rs

Seven journeys, #[ignore]d because each compiles a real project, run by a new gating User Journey E2E CI job (wired into Integration Tests Success):

Journey The assertion that was missing
journey_written_test_is_discovered_and_run the test's name appears in the output; deleting the file leaves no stale target
journey_added_dependency_is_resolved_by_a_build horus build after horus add (resolution, not manifest text)
journey_eject_hands_the_build_to_plain_cargo cargo test/cargo build work after eject, and horus test still does
journey_new_cargo_is_plain_cargo_from_the_start plain cargo works on a --cargo project from the first command
journey_workspace_member_tests_run_and_eject_builds a member's tests/ runs; the ejected workspace builds with cargo
journey_cargo_proxy_builds_managed_projects_and_passes_through the proxy builds managed projects and passes through elsewhere
journey_running_node_stops_cleanly_on_sigterm the node starts, acknowledges SIGTERM from its process group, exits, and leaves nothing running

The job sets one CARGO_TARGET_DIR for all seven, so the horus dependency tree compiles once instead of seven times — build_dirs::target_dir honours it for both the build and the artifact lookup, and every journey uses a unique project name so binaries cannot collide (the caveat documented in build_dirs.rs).

Tightened: the seven conditional assertions in e2e_lifecycle.rs

"May fail if cargo/toolchain not available — that's OK" made a broken build indistinguishable from a missing toolchain: the assertions vanished. Now:

  • no toolchain → a loud SKIP line;
  • toolchain present → a failed build/test is a failed test;
  • test_horus_test_workflow writes tests/workflow_smoke.rs and asserts its name appears — the exact assertion whose absence hid the silent-zero-tests bug;
  • horus scripts hello and horus deploy --dry-run must now succeed (neither needs a toolchain or SSH).

Why these are worth the CI minutes

Every bug they now guard was found by hand recently: tests/ compiled to zero targets while horus test reported success; a deleted test left a stale target; workspace member tests compiled to nothing; horus run -p member could exit 0 having run nothing; an ejected workspace was rebuilt into .horus/ instead of the user's manifest.

A finding from writing the SIGTERM journey

horus run spawns the node as a child and does not forward signals, so a supervisor that sends SIGTERM to horus alone kills the parent and orphans the node — the test's first version proved it by leaving one running. A terminal is fine because Ctrl+C signals the whole process group, which is what the test now does (and it asserts the specific node it started, not "anything named journey-run", is gone). Forwarding signals in horus run would be a good follow-up, out of scope here.

Verification

  • All 7 journeys pass locally in one serial run (--ignored --test-threads=1, shared target dir): 73s warm.
  • Tightened e2e_lifecycle tests pass (test_scripts_workflow, test_deploy_dry_run); the build/test ones run in the Test job in CI.
  • cargo check -p horus_manager --tests clean; workflow YAML parses.

Merge Stack
test(e2e): user-journey suite with outcome assertions
ada349e 
The journey-shaped suites assert that commands exit 0 and scaffolds exist.
They do not assert that the thing the user asked for happened — a test file
actually ran, an added dependency actually resolved, an ejected project
actually builds with plain cargo — and a failed build was allowed to be a
passing test ("may fail if cargo/toolchain not available — that's OK").

That gap is where the silent "0 tests, success" bug lived, where workspace
member tests compiled to nothing, and where `horus run -p member` could exit 0
having run nothing. Every one of those was found by hand, not by a test.

New `horus_manager/tests/user_journeys.rs`: seven `#[ignore]`d journeys, each
a sequence a developer actually runs, asserted on its outcome:

- tests discovery: write tests/smoke.rs -> horus test runs it -> delete it ->
  the next run does not fail on the stale target
- add -> build: the build proves the dependency resolves
- eject -> plain cargo test/build, and horus test still works
- new --cargo -> plain cargo from the first command
- workspace: member tests run; eject produces a real cargo workspace that builds
- cargo proxy: builds managed projects; passes through outside them
- run -> SIGTERM (process group, the way Ctrl+C arrives) -> the scheduler
  acknowledges, exits, and leaves nothing running

They run in a new gating `User Journey E2E` CI job wired into the
`Integration Tests Success` aggregator. The job shares one CARGO_TARGET_DIR, so
the horus dependency tree compiles once instead of seven times —
`build_dirs::target_dir` honours it for both the build and the artifact lookup,
and each journey uses a unique project name so no binaries collide.

Also tightens the seven conditional assertions in e2e_lifecycle.rs: a missing
toolchain now skips LOUDLY, and a toolchain with a failed build/test is a
failure. test_horus_test_workflow writes a real tests/smoke.rs and asserts its
name appears — the assertion whose absence hid the silent-zero-tests bug.
@github-actions

github-actions Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Benchmark gate

Baseline: rolling window of 3 run(s) on AMD EPYC 9V74 80-Core Processor. Current arm: median of 6 repetition(s).

Result: PASS (with tail advisories)

Adaptive_CmdVel@16B — 50000 samples
metric baseline current change band (<=) verdict
median 30.0 ns 30.0 ns +0.0% 60.0 ns INCONCLUSIVE
p95 40.0 ns 40.0 ns +0.0% 80.0 ns INCONCLUSIVE
p99 60.0 ns 60.0 ns +0.0% 120.0 ns INCONCLUSIVE
p99.9 80.0 ns 70.0 ns -12.5% 160.0 ns INCONCLUSIVE
p99.99 — 80.0 ns — — N/A (samples)
max 11.50 µs 11.07 µs -3.7% 22.99 µs INCONCLUSIVE
max_jitter 11.49 µs 11.06 µs -3.8% 22.97 µs INCONCLUSIVE
p99/median 2.00x 2.00x +0.0% 4.00x INCONCLUSIVE
p99.9/median 2.67x 2.33x -12.5% 5.33x INCONCLUSIVE
ns/msg 178.8 ns 180.1 ns +0.8% 357.6 ns INCONCLUSIVE
Adaptive_Imu@304B — 50000 samples
metric baseline current change band (<=) verdict
median 40.0 ns 40.0 ns +0.0% 80.0 ns INCONCLUSIVE
p95 60.0 ns 60.0 ns +0.0% 120.0 ns INCONCLUSIVE
p99 90.0 ns 90.0 ns +0.0% 180.0 ns INCONCLUSIVE
p99.9 110.0 ns 110.0 ns +0.0% 220.0 ns INCONCLUSIVE
p99.99 — 300.0 ns — — N/A (samples)
max 11.28 µs 12.55 µs +11.3% 22.55 µs INCONCLUSIVE
max_jitter 11.26 µs 12.53 µs +11.3% 22.51 µs INCONCLUSIVE
p99/median 2.25x 2.25x +0.0% 4.50x INCONCLUSIVE
p99.9/median 2.75x 2.75x +0.0% 5.50x INCONCLUSIVE
ns/msg 314.3 ns 313.2 ns -0.4% 628.7 ns INCONCLUSIVE
Adaptive_JointCommand@928B — 50000 samples
metric baseline current change band (<=) verdict
median 80.0 ns 90.0 ns +12.5% 160.0 ns INCONCLUSIVE
p95 100.0 ns 100.0 ns +0.0% 200.0 ns INCONCLUSIVE
p99 140.0 ns 140.0 ns +0.0% 280.0 ns INCONCLUSIVE
p99.9 160.0 ns 170.0 ns +6.2% 320.0 ns INCONCLUSIVE
p99.99 — 10.29 µs — — N/A (samples)
max 11.30 µs 12.56 µs +11.2% 22.59 µs INCONCLUSIVE
max_jitter 11.25 µs 12.51 µs +11.2% 22.49 µs INCONCLUSIVE
p99/median 1.56x 1.56x +0.0% 3.11x INCONCLUSIVE
p99.9/median 1.78x 1.89x +6.3% 3.56x INCONCLUSIVE
ns/msg 378.6 ns 381.4 ns +0.7% 757.2 ns INCONCLUSIVE
Adaptive_LaserScan@1480B — 10000 samples
metric baseline current change band (<=) verdict
median 90.0 ns 90.0 ns +0.0% 180.0 ns INCONCLUSIVE
p95 110.0 ns 110.0 ns +0.0% 220.0 ns INCONCLUSIVE
p99 140.0 ns 150.0 ns +7.1% 280.0 ns INCONCLUSIVE
p99.9 170.0 ns 180.0 ns +5.9% 340.0 ns INCONCLUSIVE
p99.99 — 11.52 µs — — N/A (samples)
max 460.0 ns 12.09 µs +2527.8% 920.0 ns GROSS REGRESSION
max_jitter 390.0 ns 12.03 µs +2984.1% 780.0 ns GROSS REGRESSION
p99/median 1.56x 1.67x +7.1% 3.11x INCONCLUSIVE
p99.9/median 1.89x 2.00x +5.9% 3.78x INCONCLUSIVE
ns/msg 732.9 ns 733.6 ns +0.1% 1.47 µs INCONCLUSIVE
Topic_cross_process_oneway_paced_stream@1024B — 200000 samples
metric baseline current change band (<=) verdict
median 416.0 ns 416.0 ns +0.0% 832.0 ns INCONCLUSIVE
p95 476.0 ns 476.0 ns +0.0% 952.0 ns INCONCLUSIVE
p99 586.0 ns 596.0 ns +1.7% 1.17 µs INCONCLUSIVE
p99.9 3.37 µs 3.31 µs -1.8% 6.74 µs INCONCLUSIVE
p99.99 27.23 µs 19.74 µs -27.5% 54.47 µs INCONCLUSIVE
max 52.08 µs 83.96 µs +61.2% 104.16 µs INCONCLUSIVE
max_jitter 51.85 µs 83.72 µs +61.5% 103.71 µs INCONCLUSIVE
p99/median 1.49x 1.41x -5.5% 2.98x INCONCLUSIVE
p99.9/median 8.10x 7.77x -4.1% 16.20x INCONCLUSIVE
ns/msg 10.00 µs 10.00 µs +0.0% 20.00 µs INCONCLUSIVE
Topic_cross_process_oneway_paced_stream@64B — 200000 samples
metric baseline current change band (<=) verdict
median 215.0 ns 215.0 ns +0.0% 430.0 ns INCONCLUSIVE
p95 255.0 ns 255.0 ns +0.0% 510.0 ns INCONCLUSIVE
p99 366.0 ns 376.0 ns +2.7% 732.0 ns INCONCLUSIVE
p99.9 2.05 µs 1.78 µs -13.2% 4.10 µs INCONCLUSIVE
p99.99 20.32 µs 13.65 µs -32.8% 40.63 µs INCONCLUSIVE
max 100.16 µs 39.42 µs -60.6% 200.33 µs INCONCLUSIVE
max_jitter 100.04 µs 39.28 µs -60.7% 200.08 µs INCONCLUSIVE
p99/median 1.86x 1.75x -5.8% 3.71x INCONCLUSIVE
p99.9/median 11.70x 8.27x -29.3% 23.41x INCONCLUSIVE
ns/msg 10.00 µs 10.00 µs +0.0% 20.00 µs INCONCLUSIVE
Topic_cross_process_oneway_pingpong@1024B — 1000000 samples
metric baseline current change band (<=) verdict
median 421.0 ns 426.0 ns +1.2% 842.0 ns INCONCLUSIVE
p95 456.0 ns 466.0 ns +2.2% 912.0 ns INCONCLUSIVE
p99 481.0 ns 486.0 ns +1.0% 962.0 ns INCONCLUSIVE
p99.9 4.17 µs 4.15 µs -0.4% 8.33 µs INCONCLUSIVE
p99.99 11.98 µs 12.54 µs +4.7% 23.96 µs INCONCLUSIVE
max 42.54 µs 63.17 µs +48.5% 85.07 µs INCONCLUSIVE
max_jitter 42.26 µs 62.91 µs +48.9% 84.51 µs INCONCLUSIVE
p99/median 1.12x 1.14x +2.1% 2.23x INCONCLUSIVE
p99.9/median 9.56x 9.74x +2.0% 19.11x INCONCLUSIVE
ns/msg 496.4 ns 490.3 ns -1.2% 992.9 ns INCONCLUSIVE
Topic_cross_process_oneway_pingpong@64B — 1000000 samples
metric baseline current change band (<=) verdict
median 225.0 ns 220.0 ns -2.2% 450.0 ns INCONCLUSIVE
p95 255.0 ns 260.0 ns +2.0% 510.0 ns INCONCLUSIVE
p99 275.0 ns 275.0 ns +0.0% 550.0 ns INCONCLUSIVE
p99.9 951.0 ns 956.0 ns +0.5% 1.90 µs INCONCLUSIVE
p99.99 9.86 µs 10.05 µs +1.9% 19.73 µs INCONCLUSIVE
max 22.49 µs 77.63 µs +245.2% 44.98 µs REGRESSED
max_jitter 22.33 µs 77.49 µs +247.0% 44.67 µs REGRESSED
p99/median 1.20x 1.25x +4.2% 2.40x INCONCLUSIVE
p99.9/median 4.26x 4.35x +1.9% 8.53x INCONCLUSIVE
ns/msg 283.5 ns 275.3 ns -2.9% 567.1 ns INCONCLUSIVE
Topic_cross_process_unpaced_queue_delay@64B — 200000 samples
metric baseline current change band (<=) verdict
median 215.0 ns 235.0 ns +9.3% 430.0 ns INCONCLUSIVE
p95 255.0 ns 285.0 ns +11.8% 510.0 ns INCONCLUSIVE
p99 285.0 ns 305.0 ns +7.0% 570.0 ns INCONCLUSIVE
p99.9 1.97 µs 1.66 µs -15.8% 3.94 µs INCONCLUSIVE
p99.99 9.17 µs 8.70 µs -5.1% 18.34 µs INCONCLUSIVE
max 41.08 µs 24.93 µs -39.3% 82.15 µs INCONCLUSIVE
max_jitter 40.97 µs 24.82 µs -39.4% 81.94 µs INCONCLUSIVE
p99/median 1.32x 1.30x -2.0% 2.65x INCONCLUSIVE
p99.9/median 9.15x 7.05x -23.0% 18.31x INCONCLUSIVE
ns/msg 174.9 ns 178.4 ns +2.0% 349.7 ns INCONCLUSIVE

Blocking metrics: median, p99/median. Report-only: p95, p99, p99.9, p99.99, max, max_jitter, p99.9/median, ns/msg. Tail metrics past p99.9 are advisory on a shared runner — one host preemption owns max.

Raw benchmark output 
              1000000     145.2     235.3     295.4     375.6     8467.4     29057.7 |      933       100
  responder turnaround             1000000      40.1      40.1      50.1      50.1      210.3     23444.4 |      791       100
------------------------------------------------------------------------------------------------------------------------------------------
  samples=1000000  warmup=200000  transferred=2000000  lost=0  invalid(TSC-skew)=0  poll cost=13.1 ns  window=0.549 s
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=108 invol=4
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  cross-core TSC offset theta_hat = -15.0 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.

== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
   json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@1024B`)
   payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7663 <-> pid 7673
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way, offset-free             1000000     260.4     420.6     470.7    3410.0    12167.9     63172.8 |     1000        99
  round trip (RTT)                 1000000     560.8     881.3     991.5    8502.5    24566.1    126445.7 |     1000       100
  RTT halved (one-way)             1000000     280.4     440.6     495.7    4251.2    12283.0     63222.9 |     1000       100
  leg A->B (apparent)              1000000     225.3     425.6     485.7     956.4    23229.1    125869.9 |      998       100
  leg B->A (apparent)              1000000     225.3     425.6     495.7     856.3     9559.0     36358.4 |      992        99
  responder turnaround             1000000      40.1      40.1      50.1     110.2      240.4     43884.5 |      954        94
------------------------------------------------------------------------------------------------------------------------------------------
  samples=1000000  warmup=200000  transferred=2000000  lost=0  invalid(TSC-skew)=0  poll cost=37.4 ns  window=0.971 s
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=191 invol=4
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  cross-core TSC offset theta_hat = +0.0 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.

== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
   json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7663 <-> pid 7675
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way (assumes TSC coherence)    200000     145.2     215.3     375.6    1477.2    12503.4     39422.9 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=9.6 ns  window=2.000 s
  measured backlog: 0.02 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=10
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -15.0 ns (add to correct).

== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
   json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
   payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7663 <-> pid 7678
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way (assumes TSC coherence)    200000     215.3     405.6     625.9    7666.3    56278.4    292404.5 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=23.1 ns  window=2.000 s
  measured backlog: 0.04 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=8
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +0.0 ns (add to correct).

== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
   !! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
   !! time spent waiting in the ring. It is a backpressure/throughput figure.
   !! Do not quote it as an IPC latency; that is the exact mistake the retired
   !! `Topic_cross_process` metric shipped for months. Read `measured backlog`
   !! below: ~1 message means the subscriber kept up and no queue formed.
   json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7663 <-> pid 7680
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  publish->receive incl. queue      200000     115.2     235.3     305.4    1537.3     8247.1     48846.7 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=19.3 ns  window=0.036 s
  measured backlog: 1.31 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=7 invol=0
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -15.0 ns (add to correct).

SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
  json metric key                                what it measures                  n    median       p99     p99.9     p99.99         max
------------------------------------------------------------------------------------------------------------------------------------------
  Topic_cross_process_oneway_pingpong@64B        ONE-WAY latency             1000000     220.3     275.4     926.4     9889.5     77634.0
  Topic_cross_process_oneway_pingpong@1024B      ONE-WAY latency             1000000     420.6     470.7    3410.0    12167.9     63172.8
  Topic_cross_process_oneway_paced_stream@64B    ONE-WAY latency              200000     215.3     375.6    1477.2    12503.4     39422.9
  Topic_cross_process_oneway_paced_stream@1024B  ONE-WAY latency              200000     405.6     625.9    7666.3    56278.4    292404.5
  Topic_cross_process_unpaced_queue_delay@64B    QUEUE DELAY (not latency)    200000     235.3     305.4    1537.3     8247.1     48846.7
------------------------------------------------------------------------------------------------------------------------------------------
  Instrumentation is INCLUDED in every figure and never subtracted.
  `Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
  nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
  `RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
  the row to compare against any competitor publishing a halved ping-pong.
  `Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
  however long the message waited in the ring, so it scales with the achieved backlog
  (printed in its scenario block) and is NOT a latency.
  A figure without its payload size, ring depth and topology is not comparable to anything.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!  CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!!  No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!!  pingpong (1024 B) `one-way, offset-free`: p99.99 rests on 99 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  unpaced_queue (64 B): measured backlog is only 1.31 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!!  unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-2.json
  keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
  NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
  will report it as missing and fail the gate until the window is refreshed —
  that is intended, and is why the key was not silently reused.
  Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
  7 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║        HORUS Robotics Message Types Benchmark                    ║
╠══════════════════════════════════════════════════════════════════╣
║  Testing real-world robotics message latency (REP 2014)          ║
╚══════════════════════════════════════════════════════════════════╝

Platform: AMD EPYC 9V74 80-Core Processor (4 cores)
Iterations: 50000
Warmup: 5000

╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type    │ Size (bytes) │ Typical Rate │ Use Case                    ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel          │           16 │ 1000+ Hz     │ Velocity control commands   ║
║ Imu             │          296 │ 500+ Hz      │ IMU sensor fusion           ║
║ LaserScan       │         1480 │ 10-40 Hz     │ 2D lidar navigation         ║
║ JointCommand    │         1032 │ 500+ Hz      │ Manipulator control         ║
╚═════════════════════════════════════════════════════════════════════════════╝


[Topic] Running benchmarks — timing tx.send() enqueue cost only
        (cross-thread consumer drains in the background; no sample
         waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
  CmdVel              16 bytes │ median:      30ns │ p99:      60ns │ CV: 3.0691
  Imu                304 bytes │ median:      40ns │ p99:      90ns │ CV: 2.6306
  LaserScan         1480 bytes │ median:      90ns │ p99:     140ns │ CV: 5.9052
  JointCommand       928 bytes │ median:      80ns │ p99:     140ns │ CV: 1.4626

╔══════════════════════════════════════════════════════════════════════════════════════════╗
║           SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency           ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message      │   Size │ Median (ns) │ p99 (ns) │    CV   │ Misses │ Sustained rate       ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel       │     16 │          30 │       60 │  3.0691 │      0 │      5.55 M msg/s     ║
║ Imu          │    304 │          40 │       90 │  2.6306 │      0 │      3.16 M msg/s     ║
║ LaserScan    │   1480 │          90 │      140 │  5.9052 │      0 │      1.35 M msg/s     ║
║ JointCommand │    928 │          80 │      140 │  1.4626 │      0 │      2.62 M msg/s     ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.

╔══════════════════════════════════════════════════════════════════╗
║              REAL-TIME SUITABILITY ANALYSIS                      ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop       ║
║ guarantees: a real loop also pays the receive side, the node's   ║
║ compute, and the scheduler's jitter, none of which is measured   ║
║ here. The headroom is printed so a pass by four orders of        ║
║ magnitude cannot be read as a tight result.                      ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel    (1kHz  control): ~ PASS by 16667x — gate not discriminating (p99=60ns vs 1000000ns)
║ CmdVel    (10kHz control): ~ PASS by 1667x — gate not discriminating (p99=60ns vs 100000ns)
║ Imu       (500Hz fusion) : ~ PASS by 22222x — gate not discriminating (p99=90ns vs 2000000ns)
║ LaserScan (40Hz  lidar)  : ~ PASS by 178571x — gate not discriminating (p99=140ns vs 25000000ns)
║ JointCmd  (500Hz control): ~ PASS by 14286x — gate not discriminating (p99=140ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝

Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-2.json
===== repetition 3/3 =====

HORUS cross-process benchmark — ONE-WAY LATENCY (headline) + unpaced queue delay
Two OS processes on two distinct physical cores over the real Topic SHM transport.
Every handle holds exactly one role, so the `role == Both` same-thread fast path that
produces the repo's headline send-cost figure is never taken. Round-trip rows are
labelled where they are a halved round trip. See the module header for the clock
decomposition and for why the `Topic_cross_process` JSON key was retired.

MACHINE — every nanosecond below is conditional on this block
------------------------------------------------------------------------------------------------------------------------------------------
  cpu model              : AMD EPYC 9V74 80-Core Processor (2 physical / 4 logical cores)
  kernel                 : 6.17.0-1022-azure (x86_64)
  benchmark cores        : cpu3 = process A (pinger / subscriber), cpu1 = process B (responder / publisher)  [DistinctPhysical]
  governor               : cpu3=n/a cpu1=n/a  driver=n/a
  scaling range          : n/a .. n/a  (hw max n/a)  turbo_disabled=unknown
  clocksource            : tsc   tsc flags: [tsc, rdtscp, constant_tsc, tsc_reliable, nonstop_tsc]
  TSC rate (calibrated)  : 2596.173 MHz (0.3852 ns/cycle) — a WALL clock; NOT the core clock
  clock instrumentation  : 52 cycles = 20.0 ns for serialize()+rdtsc()+rdtscp(), INCLUDED in every sample and never subtracted
  transparent_hugepage   : [always] madvise never
  sched_rt_runtime/period: 950000 / 1000000 us    perf_event_paranoid=4
  isolcpus=(none)  nohz_full=(none)  smt_active=1  mlockall(MCL_CURRENT)=true
------------------------------------------------------------------------------------------------------------------------------------------

running ping-pong (one-way latency), 64 B payload ...
running ping-pong (one-way latency), 1024 B payload ...
running paced stream (one-way latency), 64 B payload ...
running paced stream (one-way latency), 1024 B payload ...
running unpaced stream (QUEUE DELAY, not latency), 64 B payload ...

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! READ THIS BEFORE QUOTING ANY NUMBER BELOW
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!  CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!!  No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!!  paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  unpaced_queue (64 B): measured backlog is only 1.33 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!!  unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
   json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7702 <-> pid 7711
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way, offset-free             1000000     150.2     220.3     275.4     956.4    10054.8     21621.8 |      990       100
  round trip (RTT)                 1000000     340.5     490.7     590.9    1992.9    20440.1     43283.7 |      998       100
  RTT halved (one-way)             1000000     170.3     245.4     295.4     996.5    10220.0     21641.9 |      998       100
  leg A->B (apparent)              1000000     135.2     215.3     285.4    1367.0    13014.2     31200.9 |      989       100
  leg B->A (apparent)              1000000     145.2     235.3     295.4     355.5     8828.0     43088.4 |      989       100
  responder turnaround             1000000      40.1      40.1      50.1      50.1      200.3     26418.9 |      702        93
------------------------------------------------------------------------------------------------------------------------------------------
  samples=1000000  warmup=200000  transferred=2000000  lost=0  invalid(TSC-skew)=0  poll cost=13.5 ns  window=0.555 s
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=109 invol=3
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  cross-core TSC offset theta_hat = -10.0 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.

== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
   json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@1024B`)
   payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7702 <-> pid 7713
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way, offset-free             1000000     245.4     425.6     485.7    4151.1    12543.5     21957.3 |     1000       100
  round trip (RTT)                 1000000     530.8     891.3    1011.5    8512.5    25217.1     43954.7 |     1000        99
  RTT halved (one-way)             1000000     265.4     445.7     505.7    4256.3    12608.6     21977.4 |     1000        99
  leg A->B (apparent)              1000000     225.3     405.6     485.7     836.2    23709.9     43348.8 |      993       100
  leg B->A (apparent)              1000000     235.3     445.7     515.8     836.2    11672.2     40214.2 |      980       100
  responder turnaround             1000000      40.1      40.1      50.1     120.2      240.4     40649.8 |      884        96
------------------------------------------------------------------------------------------------------------------------------------------
  samples=1000000  warmup=200000  transferred=2000000  lost=0  invalid(TSC-skew)=0  poll cost=43.1 ns  window=0.981 s
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=193 invol=4
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  cross-core TSC offset theta_hat = -20.0 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.

== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
   json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7702 <-> pid 7715
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way (assumes TSC coherence)    200000     145.2     215.3     375.6    1777.6    13655.1     29828.9 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=9.6 ns  window=2.000 s
  measured backlog: 0.02 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=11
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -10.0 ns (add to correct).

== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
   json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
   payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7702 <-> pid 7717
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way (assumes TSC coherence)    200000     235.3     425.6     595.9    3309.9    19744.1     83958.6 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=22.7 ns  window=2.000 s
  measured backlog: 0.04 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=13
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -20.0 ns (add to correct).

== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
   !! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
   !! time spent waiting in the ring. It is a backpressure/throughput figure.
   !! Do not quote it as an IPC latency; that is the exact mistake the retired
   !! `Topic_cross_process` metric shipped for months. Read `measured backlog`
   !! below: ~1 message means the subscriber kept up and no queue formed.
   json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7702 <-> pid 7719
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  publish->receive incl. queue      200000     115.2     235.3     315.5    8447.4    15337.6     24931.7 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=19.6 ns  window=0.035 s
  measured backlog: 1.33 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=7 invol=1
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -10.0 ns (add to correct).

SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
  json metric key                                what it measures                  n    median       p99     p99.9     p99.99         max
------------------------------------------------------------------------------------------------------------------------------------------
  Topic_cross_process_oneway_pingpong@64B        ONE-WAY latency             1000000     220.3     275.4     956.4    10054.8     21621.8
  Topic_cross_process_oneway_pingpong@1024B      ONE-WAY latency             1000000     425.6     485.7    4151.1    12543.5     21957.3
  Topic_cross_process_oneway_paced_stream@64B    ONE-WAY latency              200000     215.3     375.6    1777.6    13655.1     29828.9
  Topic_cross_process_oneway_paced_stream@1024B  ONE-WAY latency              200000     425.6     595.9    3309.9    19744.1     83958.6
  Topic_cross_process_unpaced_queue_delay@64B    QUEUE DELAY (not latency)    200000     235.3     315.5    8447.4    15337.6     24931.7
------------------------------------------------------------------------------------------------------------------------------------------
  Instrumentation is INCLUDED in every figure and never subtracted.
  `Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
  nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
  `RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
  the row to compare against any competitor publishing a halved ping-pong.
  `Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
  however long the message waited in the ring, so it scales with the achieved backlog
  (printed in its scenario block) and is NOT a latency.
  A figure without its payload size, ring depth and topology is not comparable to anything.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!  CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!!  No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!!  paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  unpaced_queue (64 B): measured backlog is only 1.33 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!!  unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-3.json
  keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
  NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
  will report it as missing and fail the gate until the window is refreshed —
  that is intended, and is why the key was not silently reused.
  Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
  6 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║        HORUS Robotics Message Types Benchmark                    ║
╠══════════════════════════════════════════════════════════════════╣
║  Testing real-world robotics message latency (REP 2014)          ║
╚══════════════════════════════════════════════════════════════════╝

Platform: AMD EPYC 9V74 80-Core Processor (4 cores)
Iterations: 50000
Warmup: 5000

╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type    │ Size (bytes) │ Typical Rate │ Use Case                    ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel          │           16 │ 1000+ Hz     │ Velocity control commands   ║
║ Imu             │          296 │ 500+ Hz      │ IMU sensor fusion           ║
║ LaserScan       │         1480 │ 10-40 Hz     │ 2D lidar navigation         ║
║ JointCommand    │         1032 │ 500+ Hz      │ Manipulator control         ║
╚═════════════════════════════════════════════════════════════════════════════╝


[Topic] Running benchmarks — timing tx.send() enqueue cost only
        (cross-thread consumer drains in the background; no sample
         waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
  CmdVel              16 bytes │ median:      30ns │ p99:      60ns │ CV: 2.7724
  Imu                304 bytes │ median:      40ns │ p99:      90ns │ CV: 2.1657
  LaserScan         1480 bytes │ median:      90ns │ p99:     150ns │ CV: 1.9965
  JointCommand       928 bytes │ median:      90ns │ p99:     140ns │ CV: 1.3925

╔══════════════════════════════════════════════════════════════════════════════════════════╗
║           SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency           ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message      │   Size │ Median (ns) │ p99 (ns) │    CV   │ Misses │ Sustained rate       ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel       │     16 │          30 │       60 │  2.7724 │      0 │      5.66 M msg/s     ║
║ Imu          │    304 │          40 │       90 │  2.1657 │      0 │      3.19 M msg/s     ║
║ LaserScan    │   1480 │          90 │      150 │  1.9965 │      0 │      1.36 M msg/s     ║
║ JointCommand │    928 │          90 │      140 │  1.3925 │      0 │      2.57 M msg/s     ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.

╔══════════════════════════════════════════════════════════════════╗
║              REAL-TIME SUITABILITY ANALYSIS                      ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop       ║
║ guarantees: a real loop also pays the receive side, the node's   ║
║ compute, and the scheduler's jitter, none of which is measured   ║
║ here. The headroom is printed so a pass by four orders of        ║
║ magnitude cannot be read as a tight result.                      ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel    (1kHz  control): ~ PASS by 16667x — gate not discriminating (p99=60ns vs 1000000ns)
║ CmdVel    (10kHz control): ~ PASS by 1667x — gate not discriminating (p99=60ns vs 100000ns)
║ Imu       (500Hz fusion) : ~ PASS by 22222x — gate not discriminating (p99=90ns vs 2000000ns)
║ LaserScan (40Hz  lidar)  : ~ PASS by 166667x — gate not discriminating (p99=150ns vs 25000000ns)
║ JointCmd  (500Hz control): ~ PASS by 14286x — gate not discriminating (p99=140ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝

Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-3.json

Gated against a rolling window of recent main runs. Each metric's band is max(relative, center + k·sigma_observed, absolute floor), so it is derived from the noise the baseline itself measured — where a band is printed wider than the change you care about, this run did not detect a regression and could not have. Thresholds and their reasoning: RegressionPolicy in benchmarks/src/output.rs.
Commit: a60e5e9

@starswallow
starswallow merged commit 3e78565 into main Sep 23, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@starswallow