feat(bench): measure a mixed workload of takeovers and approvals - #8
Merged
Conversation
Aurora Bank had one interrupt in it: the TOTP wall, which an agent cannot pass. That covers the capability gap and nothing else. An approval needs the other kind — a step the agent is perfectly able to take and is not allowed to take alone. /transfer sits behind the session, takes an amount and a payee, and posts back to a signed cookie that /account renders as a receipt. No server state, so a cookie-free context starts clean, which is what the benches need. The /totp and /transfer branches move into their own functions to keep route() under the complexity limit.
bench.ts times one handoff and rescue-bench.ts counts workflows a takeover saves. Neither answers what a team asks before adopting this: an agent meets two kinds of interrupt, and they do not cost the same. mixed-bench.ts runs N workflows against one live Aurora Bank instance, interleaved takeover, approval, takeover, and reports each mode's completion, time to visible on the human's socket, handoff duration, frames, bytes, inputs applied and relay-sandbox-seconds. Every fourth approval is denied, and a denied approval counts as completed: the decision was delivered and the bench asserts no money moved. The scripted human gains firstFrameAt() so time-to-visible is the frame's arrival time and not the caller's wake-up time.
A handoff that never got a sandbox — the plan was full and startRelay threw — burned wall clock but no sandbox, and charging it for one would inflate the cost proxy exactly when the platform was busiest. The wide event is the tell: no event, no relay, no seconds. Empty cells print as an em dash instead of NaN.
Two plan-tier facts kept turning a benchable minute into a failed one. A neighbouring job holding both slots made every relay creation throw "Too many concurrent sessions", and five of those in a row aborted the bench. And a sandbox can go away under the run — an expiry, a hiccup, a neighbour's cleanup — after which every workflow fails at the login form, which says nothing about handraise. Neither is a datum, so neither is recorded as one: a full plan is waited out for a minute and retried, up to ten times per workflow, and a run that never reached its interrupt now checks /healthz and rebuilds the test app before trying again. `concurrencyWaits` and `testAppRestarts` in the JSON say how often the bench had to do either.
20 workflows against one live Aurora Bank instance on 2026-09-02, interleaved takeover, approval, takeover, on an uncontended plan: 20 of 20 completed, 10 of 10 in each mode, two of the ten approvals denied. The side-by-side is the point. A takeover put 14 frames and 142 KB on the wire and held a relay sandbox for 11.0 s; an approval put one screenshot and 25 KB on it, applied zero inputs, and held one for 5.5 s. Time to visible is the same for both, because both pay the same relay cold start. Both README sections say what may be claimed and what may not: the 50/50 mix is the harness's choice, not anyone's traffic, and the takeover stream grows with the human while the approval's one frame does not.
…ean something Review round 1 found the sensor had a hole. After a denial the bench asked the transfer form whether a receipt was on it — but `transfer-done` is rendered only by the account page, so the check could not fail, could not notice a transfer that did go through, and would have passed against a blank page. Two published sentences claimed an assertion nobody made. The denial verdict now loads /account and requires the session banner AND the absence of a receipt carrying that run's amount; the approve verdict matches the amount too, so a stale receipt cannot pass either. Proven red-first: with the agent submitting despite the denial, the run reports 0/1 instead of 1/1. MIXED_FAULT=invert-completed now inverts the page sensors instead of the verdict. Flipping the verdict passed identically for a check that never read the page — it could not have caught the bug above. Flipping the sensor does: the proof run reads 0/4 and 0/4 with the denial included. Setup steps keep the raw predicates, and the abort-on-failure-streak guard stands down under the fault, which is supposed to fail everything. Also from the review: a full plan is detected with the typed `concurrency_limit` code rather than a regex over an error message the project says is not a contract, and a replaced attempt is no longer dropped — it stays in `runs` as not-completed with `superseded` and its error, so no retry rule can quietly improve the completion rate.
A second 20-workflow run, this time with a denial check that can fail and a receipt check that matches each run's own amount: 20 of 20 completed, 10 of 10 in each mode, two denials, no concurrency waits, no superseded attempts. A takeover put 14 frames and 142 KB on the wire and held a relay sandbox for 10.7 s; an approval put one screenshot and 25 KB on it, applied zero inputs, and held one for 5.3 s. Three things the review found unstated are now stated: `handoff` is the wide event's durationMs and excludes the cold start, which is why an approval's handoff is shorter than its time to visible; relay-sandbox seconds include the human's occupancy, so 5.0 s of the 5.4 s gap is the scripted human typing and the column is a floor rather than a property of the modes; and the root README now says the approval arm signs itself in with the shared secret, with a link to the method.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One harness, both interrupt kinds, side by side: 20 workflows against one live Aurora Bank instance, interleaved takeover/approval, every fourth approval denied. 20/20 completed. Approval: 1 frame, 25 KB, 0 inputs, 5.5 s of relay sandbox; takeover: 14 frames, 142 KB, 8 inputs, 11.0 s. Time to visible is the same in both (relay cold start dominates).
Adds a transfer page to the test app (the thing an approval decides), a MIXED_FAULT inversion switch (verified: 0/1 in both modes), and two harness fixes learned from contention (wait out a full plan, rebuild a dead test app mid-bench). Numbers in README and benchmarks/README are generated from benchmarks/mixed-workload.json.
🤖 Generated with Claude Code