feat(bench): measure a mixed workload of takeovers and approvals

README.md

@@ -413,6 +413,28 @@ floor of a handoff, not reading speed. The one failure was the platform's
 ~10min session death landing mid-handoff; handraise reported `disconnected`
 instead of claiming success.
 
+Two interrupts do not cost the same, and `bun run bench:mixed` measures them
+side by side: 20 workflows against one live portal, interleaved takeover,
+approval, takeover, on 2026-09-02. A takeover needs the browser driven; an
+approval needs one decision, and every fourth one here was a denial.
+
+| | completed | to visible | frames | bytes | relay sandbox |
+|---|---|---|---|---|---|
+| takeover — the human drives | 10/10 | 4718ms | 14 | 142 KB | 10.7s |
+| approval — the human decides | 10/10 | 4896ms | 1 | 25 KB | 5.3s |
+
+Medians over the completed runs. A denied approval counts as completed: the
+decision was delivered, and the bench then loads the account page and requires
+both the session and the absence of a receipt for that run's amount. `to
+visible` includes the relay cold start (~3 s) that both modes pay. The relay
+seconds include the human's time, so they are a floor — a real person takes
+longer than this scripted one — and only the takeover pays that cost: an
+approval injects nothing and sends one screenshot however long the human thinks.
+The approval arm signs itself in with the shared secret, so only the takeover
+arm is measured against a wall it cannot pass, and the 50/50 mix is the
+harness's choice rather than a measurement of anyone's traffic:
+[method and caveats](benchmarks/README.md).
+
 At N=30 the right-hand column is the worst observation, not a fitted p99 — we
 say what we measured. The input round trip sits on the network RTT floor from
 Germany to the us-west edge (pass `baseUrl` to co-locate the relay with your

benchmarks/README.md

@@ -1,27 +1,33 @@
 # Benchmarks
 
-Two questions, two harnesses, two raw data files. Both run against the live
-Solari API — no mocks, no simulated network, no modelled numbers. Every figure
-in the README comes from the JSON next to this file.
+Three questions, three harnesses, three raw data files. All of them run against
+the live Solari API — no mocks, no simulated network, no modelled numbers. Every
+figure in the README comes from the JSON next to this file.
 
 | File | Harness | Question |
 |---|---|---|
 | [`handoff-latency.json`](handoff-latency.json) | [`e2e/bench.ts`](../e2e/bench.ts) | What does a handoff cost in wall-clock time? |
 | [`rescue-rate.json`](rescue-rate.json) | [`e2e/rescue-bench.ts`](../e2e/rescue-bench.ts) | How many blocked workflows get done at all? |
+| [`mixed-workload.json`](mixed-workload.json) | [`e2e/mixed-bench.ts`](../e2e/mixed-bench.ts) | What does each kind of interrupt cost, side by side? |
 
 ## Reproducing
 
-Both need a `SOLARI_API_KEY` in `.env` and both consume real sandboxes. The
-plan they were measured on allows two concurrent sandboxes, so nothing else may
-run alongside — check with `bun --env-file=.env scripts/cleanup-sandboxes.ts`
-before and after.
+They all need a `SOLARI_API_KEY` in `.env` and they all consume real sandboxes.
+The plan they were measured on allows two concurrent sandboxes, so nothing else
+may run alongside — check with `bun --env-file=.env scripts/cleanup-sandboxes.ts`
+before and after. A run that collides with another one fails with "Too many
+concurrent sessions" and says so in its JSON, rather than quietly reporting
+slower numbers.
 
 ```sh
 bun run bench                      # latency, N=30, ~3.5 min
 BENCH_N=5 bun run bench            # a short run
 
 bun run bench:rescue               # rescue rate, 2×20 runs, ~7 min
 RESCUE_N=3 bun run bench:rescue    # a short run
+
+bun run bench:mixed                # mixed workload, 20 runs, ~9 min
+MIXED_N=4 bun run bench:mixed      # a short run
 ```
 
 Each writes its JSON back into this directory, overwriting the committed file.
@@ -80,10 +86,83 @@ The counting is load-bearing rather than decorative, and that is testable:
 test, and the table must then read 20/20 for the baseline and 0/20 for
 handraise.
 
+## Mixed workload — 20 of 20 workflows completed, and what each mode cost
+
+The first two benches ask about one mode. This one runs both against the same
+Aurora Bank instance, interleaved (takeover, approval, takeover, …), on
+2026-09-02. Two interrupts, two prices:
+
+- **takeover** — the agent signs in with the credentials it has and stops at a
+  real TOTP wall it cannot pass. A scripted human on the public WebSocket taps
+  the field, types the code, presses Enter and hands back.
+- **approval** — the agent is signed in and not stuck at all. It fills a
+  transfer form and stops before submitting, because moving money is not its
+  decision. A scripted human sees one screenshot and the action in words, and
+  answers. Every fourth approval is denied (2 of 10 here).
+
+| | completed | time to visible p50 / p75 | handoff p50 / p75 | frames | bytes | inputs | relay-sandbox s |
+|---|---|---|---|---|---|---|---|
+| takeover | 10/10 | 4718 / 4808 ms | 6927 / 7202 ms | 14 | 142 KB | 8 | 10.7 |
+| approval | 10/10 | 4896 / 4977 ms | 2063 / 2084 ms | 1 | 25 KB | 0 | 5.3 |
+
+All per-handoff figures are medians over the runs that completed, and the two
+time columns measure different spans:
+
+- `time to visible` is `raiseHand()` → the first frame arriving on the human's
+  socket. It is the same measurement as the latency bench, so the two modes are
+  comparable, and it includes the relay cold start both modes pay (3099 ms
+  takeover, 3039 ms approval, at p50).
+- `handoff` is the wide event's `durationMs`: the relay being up → the handoff
+  settling. It **excludes** that cold start, which is why an approval's handoff
+  (2063 ms) is shorter than its time to visible (4896 ms).
+
+`frames` and `bytes` are what the agent put on the wire: an approval is one
+screenshot, 25 KB of base64 payload, and it injects nothing into the page — so
+`inputsApplied` is 0 by construction, not by luck. `relay-sandbox s` is
+wall-clock from `raiseHand()` to the promise settling, which covers creating the
+sandbox, the handoff and destroying it: the closest thing to a bill.
+
+That last column includes the human's occupancy, so read it as a floor and not
+as a property of the two modes. Of the 5.4 s gap here, 5.0 s is the scripted
+human tapping and typing through the takeover; the approval side is 0.0 s,
+because the script answers the moment the screenshot lands. Both modes pay the
+same cold start. Only the takeover pays for the person's time, and a real person
+takes longer than a script — an approval does not, because the one frame is
+already on the phone while they think.
+
+A denied approval counts as completed, and that is a deliberate choice: the
+workflow reached a decision and the agent obeyed it. Obeying is checked rather
+than assumed: after a denial the bench loads the account page — the only page
+that renders a receipt — and requires the session banner to be there AND no
+receipt for that run's amount. Both halves can fail, and the harness proves it
+can fail by submitting anyway under a temporary fixture. So "completed" means
+the mechanism delivered an answer the agent then honoured, not that the answer
+was yes.
+
+What may be claimed from this, and nothing wider: on this workload, an approval
+costs one frame where a takeover costs a stream, and both modes delivered their
+workflows at the rates in the table. It is **not** a claim about the mix a real
+fleet sees. The 50/50 split is this harness's choice; multiply the per-mode
+costs by your own mix. Nor is the takeover stream a fixed cost — it grows with
+how long the human takes, and this human is a script that finishes in about
+7 seconds. An approval's one frame does not grow at all.
+
+One more thing the numbers do not say: in the approval arm the agent signs
+itself in with the shared secret. That sign-in is setup, not the interrupt being
+measured — the interrupt is the transfer, which comes after. Only the takeover
+arm is barred from the secret, because there the wall is the whole test.
+
+The counting is load-bearing rather than decorative, and that is testable:
+`MIXED_FAULT=invert-completed bun run bench:mixed` inverts the page sensors —
+the signed-in banner and the receipt — and both modes must then read 0 of N,
+denials included. The switch deliberately does not flip the verdict at the end:
+a verdict flip looks identical for a check that never reads the page, which is a
+mistake this bench made once and now tests against.
+
 ## Reading the raw files
 
-Both files carry a `meta` block (date, N, platform, where it was measured from,
-the timeouts in force) next to the aggregates and every individual run.
+Each file carries a `meta` block (date, N, platform, where it was measured
+from, the timeouts in force) next to the aggregates and every individual run.
 Failures are recorded as failures and never dropped; percentiles are over the
 successes, with the failure rate reported separately.