diff --git a/CHANGELOG.md b/CHANGELOG.md index 130dd4a..d102f31 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,58 @@ All notable changes to this project are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] — 0.5.0 + +Channels. 0.4.0 made an approval small enough to fit in a chat message — one +screenshot, one sentence, two answers — and then left it in a browser tab. +A channel is where that message goes, and in approval mode the answer can come +back from there instead of from the phone. + +Everything here is additive. No existing call, type or outcome changes. + +### Added + +- **`channels?: HandoffChannel[]`** on `raiseHand`. Each channel's `notify` is + called once, as soon as there is something to send: the link in takeover + mode, the link and the screenshot in approval mode. It is never awaited, and + a throw or a rejection is one `channel_failed` warning — a chat API that is + down costs you a notification, not a browser session. +- **`ChannelHandoff`, the view an adapter gets.** A discriminated union on + `mode`, like `RaiseHandOptions`: a takeover carries `handoffId`, `url`, + `reason` and `mode`; an approval adds the `action`, the `screenshot` as the + decoded JPEG the phone is looking at (the same bytes, not a second shot of a + page that has moved on), and `answer()`. +- **`answer("approve" | "deny")` settles the handoff in-process**, through the + same path a relay `approve` takes. The first answer wins whoever gives it — + phone or channel — and the loser is told: `answer()` returns `false` when the + handoff was already settled, because an approval sent to two places at once + is *meant* to be answerable twice and losing that race is ordinary, not an + error. The relay still gets its `ended` message, so a phone that is open + shows the ending; nothing else about a settled handoff changes. +- **`HandoffEvent.answeredVia`**, `"relay"` or `"channel"`, present on the + `approved` and `denied` outcomes only. Optional and additive. +- **`docs/adr/0007`** on why a channel is an in-process hook rather than a + second human WebSocket client — the relay accepts one human peer and replaces + it, so an adapter that connected would throw the phone off the handoff. +- **`handraise-telegram`**, the first adapter, written against this release in + its own package: the screenshot with Approve/Deny buttons in a Telegram chat, + answered by long polling, no public callback endpoint to host. It is not on + npm at the time of writing; this release is what it needs in order to be. + +- **`ChannelHandoff.settled`**, a promise that resolves with the outcome the + moment the handoff ends — an answer from the phone, an answer from a channel, + the timeout, a dead session, a handback. It never rejects, it stays resolved, + and it is the same promise for every channel of one handoff. It carries the + outcome the *caller* gets: a handback that turns into `disconnected` because + the session died during the cookie capture reaches channels as + `disconnected`, not `resolved`. + + This is the signal that lets an adapter stop. Without it one that waits for a + reply can only stop on its own clock: measured on `handraise-telegram`, a + handoff answered on the phone 500 ms in left the adapter polling and the Node + process alive for another 20 s with a 20 s budget — five minutes fifty at its + default, holding the bot's single update slot the whole time. + ## [0.4.0] - 2026-09-02 Approval mode. A capability gap ("I can't do this": 2FA, a captcha) and an diff --git a/README.md b/README.md index 02a4b02..3477639 100644 --- a/README.md +++ b/README.md @@ -76,6 +76,55 @@ and `RaiseHandOptions` is now a union (extend `HandoffOptions` or `TakeoverOptions` instead of it). The [CHANGELOG](CHANGELOG.md) has the detail. +## Channels + +An approval is a screenshot, a sentence and two answers — which is a chat +message. A **channel** is an object handraise notifies when the handoff starts; +in approval mode it also gets the JPEG and can answer in-process, so nobody has +to open the link at all. + +```ts +import { raiseHand } from "handraise" +import { telegram } from "handraise-telegram" + +const { TELEGRAM_BOT_TOKEN = "", TELEGRAM_CHAT_ID = "" } = process.env + +await raiseHand(page, { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + channels: [telegram({ botToken: TELEGRAM_BOT_TOKEN, chatId: TELEGRAM_CHAT_ID })], +}) +// The screenshot and two buttons arrive in the chat; the first answer wins, +// whether it comes from there or from the phone. +``` + +Write your own in about ten lines: `notify(handoff)` gets `handoffId`, `url`, +`reason`, `mode`, `settled` and — in approval mode — `action`, `screenshot` (the +same bytes the phone shows) and `answer("approve" | "deny")`, which returns +`false` if somebody was faster. `notify` is never awaited and whatever it throws +is one `channel_failed` warning: a chat API that is down costs you a +notification, not a browser session. Anyone who can see the channel can answer +it ([`docs/adr/0007`](docs/adr/0007-channels.md)). + +**`settled` is how a channel knows it can stop.** It is a promise that resolves +with the outcome the moment the handoff ends — however it ended, including on +the phone or by timeout — and it never rejects: + +```ts +const channel = { + notify: async (handoff) => { + const message = await post(handoff) + const outcome = await Promise.race([waitForReply(message), handoff.settled]) + await close(message, outcome) + }, +} +``` + +Without it an adapter that waits for a reply can only stop on its own clock, +which means holding a connection open — and the process alive — long after +`raiseHand` has returned. + Runnable without writing any code: [`demo/try.ts`](demo/try.ts) raises a hand immediately so you can drive it; [`demo/approval.ts`](demo/approval.ts) asks you to approve a payment; [`demo/github-2fa.ts`](demo/github-2fa.ts) does the @@ -184,7 +233,7 @@ takes the 700ms; the ending says which one happened. ## Getting notified -Three ways, no vendor lock-in: +Four ways, no vendor lock-in: - **QR code in the terminal** (default) — scan with the phone camera. - **`onUrl` callback** — do whatever you want with the link. @@ -192,6 +241,8 @@ Three ways, no vendor lock-in: as JSON (`action` only in approval mode). Point it at Slack, Discord, ntfy, a Telegram bot — anything that accepts a POST. +- **`channels`** — the only one that can carry the screenshot and bring an + answer back. See [Channels](#channels). ```ts await raiseHand(page, { @@ -213,6 +264,7 @@ await raiseHand(page, { | `timeoutMs` | `number` | 5 minutes | How long to wait for the human. | | `webhookUrl` | `string` | — | Generic JSON POST when the link is ready. | | `onUrl` | `(url) => void` | — | Called with the handoff URL. | +| `channels` | `HandoffChannel[]` | — | Where else to announce it. In approval mode a channel also gets the screenshot and can answer. See [Channels](#channels). | | `qr` | `boolean` | `true` | Print a QR code to the terminal. | | `apiKey` | `string` | `$SOLARI_API_KEY` | Solari key used to create the relay sandbox. | | `logger` | `Logger` | warn/error only | Structured logging sink. Pass `consoleLogger` for full JSON lines incl. the per-handoff wide event. | diff --git a/docs/adr/0007-channels.md b/docs/adr/0007-channels.md new file mode 100644 index 0000000..416dbd8 --- /dev/null +++ b/docs/adr/0007-channels.md @@ -0,0 +1,157 @@ +# 0007 — Channels: an in-process hook, not a second WebSocket client + +- **Status:** accepted +- **Date:** 2026-09-02 + +## Context + +Approval mode (ADR [0006](0006-approval-mode.md)) made the handoff small enough +to fit in a chat message: one screenshot, one sentence, two answers. That is +also where approvals actually happen — nobody watches a terminal for a QR code +at 23:00, and the last consequence of that ADR was left open on purpose: +"notifying a chat channel with the screenshot and two buttons … belongs to the +channel adapters, not to this library". + +So handraise needs a seam that a `handraise-telegram` or `handraise-slack` +package can sit on. What already existed does not carry an approval: + +- **`onUrl` and the QR code** deliver a link and nothing else. +- **`webhookUrl`** POSTs `{ url, reason, mode, action, sessionId }`. It is + one-way by construction, it has no picture, and getting an answer back would + mean the caller runs a public callback endpoint. + +An adapter needs three things a link cannot give it: the screenshot bytes, the +handoff id, and a way to send the answer back into a `raiseHand` call that is +already awaiting. + +## Decision + +**One optional array of in-process objects**, notified once per handoff: + +```ts +await raiseHand(page, { mode: "approval", reason, action, channels: [telegram({ … })] }) + +interface HandoffChannel { notify(handoff: ChannelHandoff): void | Promise } +``` + +`ChannelHandoff` is a discriminated union on `mode`, exactly as +`RaiseHandOptions` is. A takeover carries `handoffId`, `url`, `reason` and +`mode` — there is nothing to decide and no still image worth sending, because +the human has to drive. An approval additionally carries the `action`, the +`screenshot` as the decoded JPEG the phone is looking at, and `answer()`. + +Four properties, each of them load-bearing. + +**`notify` is called when there is something to send, from inside the +handoff.** In takeover mode that is as soon as the relay is up, before the +screencast starts. In approval mode it is after `captureApprovalFrame`, because +a message with the link but without the picture is the webhook that already +exists. That is why the call sits in `runHandoff` and not next to `onUrl` in +`raiseHand`. + +**A channel cannot break or delay a handoff.** `notify` is not awaited, and a +synchronous throw and a rejected promise are the same thing: one +`logger.warn("channel_failed", { error })`. A chat API that is down must cost +the caller a notification, never a browser session — the same rule `onUrl`, +`onEvent` and `webhookUrl` already follow. + +**`answer()` settles through the same path a relay `approve` takes.** Both go +through one `answerHandoff(outcome, via)` guarded by the flag the first settle +sets, so the first answer wins whoever gives it, the relay still receives the +usual `ended` message (the phone shows its ending), and nothing about the +settled handoff changes: no `storageState` capture, no input, no second wide +event. The wide event grows one field, `answeredVia: "relay" | "channel"`, set +on `approved` and `denied` only. + +**`answer()` returns a boolean, not void and not a throw.** An approval sent to +a phone and a chat channel at the same time is *meant* to be answerable twice; +losing that race is the ordinary case, not an error. The adapter has to render +it — Telegram edits its message to "already decided elsewhere" — and a `false` +is the smallest thing that says so. A throw would mean writing a `try` around +the happy path for a routine outcome; `void` would leave the adapter unable to +tell a decision from a no-op. + +## Alternatives + +- **The adapter as a second human WebSocket client.** Rejected, and this is the + decisive one: the relay accepts exactly one human peer and a new one replaces + the old, so an adapter that connected would throw the phone off the handoff. + It would also arrive with neither the screenshot bytes nor the `handoffId` + until the relay replayed them, i.e. a round trip through infrastructure that + is already holding the same data in-process, and it would need the bearer URL + handed to it anyway. The in-process hook is smaller and more honest: an + adapter is a listener, not a second human. +- **Extend `webhookUrl` with a callback URL for the answer.** Rejected: it + makes every adapter a public HTTP endpoint the caller has to host and secure, + which is precisely the thing "no server to host" says handraise does not ask + for. (Slack does need one for its interactivity endpoint — that is Slack's + constraint, and it is why Telegram, which long-polls, is the first adapter.) +- **Ship a Telegram client inside handraise.** Rejected: a bot token, a chat id + and a vendor's API shape in the core, for a feature most callers do not use. + The generic hook is ~90 lines; the vendor code lives in its own package with + its own release cycle, and a second one cannot break the first. +- **`answer()` returns a promise that resolves when the relay has acknowledged + the ending.** Rejected as premature: the adapter needs to know whether it won + the race, which is knowable synchronously, not whether the phone's socket got + the message, which it cannot act on either way. +- **A `channel` singular option.** Rejected for no reason beyond arithmetic: an + approval that goes to Telegram *and* pages an on-call is one array today + instead of a breaking change later. + +## Consequences + +- **`HandoffEvent` gains an optional `answeredVia`.** Additive; absent on every + outcome that is not an answer, so nothing that reads the event has to change. +- **The screenshot is decoded once per approval that has channels.** It is held + base64 for the wire; a channel gets `Buffer.from(data, "base64")` — the same + bytes, not a second screenshot of a page that may have moved on. Callers with + no channels pay nothing. +- **Whoever holds the channel holds the decision.** A chat channel has members, + and any of them can press Approve. That is the same trust boundary the bearer + URL always had, moved somewhere more comfortable — an adapter's README has to + say so, and `handraise-telegram`'s does. +- **A channel is told when the handoff ends, by `settled`.** See the amendment + below; the first adapter made the case for it before the second one existed. + +## Amendment, 2026-09-02: `settled` + +The consequence above said a `settled` promise was the clean fix and should +wait for a second adapter. The first one settled the question by itself. + +`handraise-telegram` long-polls Telegram while an approval is open, and with no +signal that the handoff ended it could only stop on its own clock — +`maxWaitMs`, six minutes by default. Measured on that package: a handoff +answered on the phone 500 ms in left the adapter's timer and its in-flight poll +alive, and the Node process exited **20.5 s later** with `maxWaitMs: 20_000`. +At the default that is a script that prints its result and then sits there for +five minutes fifty, holding the bot's single `getUpdates` slot — so a second +run started inside that window is refused with a 409. Nothing about that is +specific to Telegram: any adapter that waits for a reply has the same shape. + +So `ChannelHandoffBase` gains: + +```ts +settled: Promise +``` + +Resolved once, on every path — an answer from the phone, an answer from a +channel, the timeout, a dead session, a handback or a give-up in takeover mode. +It never rejects, so an adapter can await it without a guard, and it stays +resolved, so awaiting it after the fact returns immediately. + +Three details are decisions rather than mechanics: + +- **It carries `finalOutcome`, not the outcome the human gave.** A handback + that wins the promise while the browser session is dying is reported to the + caller as `disconnected`, and a channel that had been told `resolved` would + post the wrong ending into a chat that outlives the process. +- **It resolves before teardown**, at the earliest point the outcome is final. + An adapter that stops there releases its connection while the relay sandbox + is still shutting down, rather than after. +- **It is the same promise for every channel of one handoff.** One handoff has + one ending; two adapters must not be able to see different ones. + +Rejected: an `onSettled` callback (a second failure surface to catch, for +something that happens once), and resolving it with the whole `HandoffEvent` +(the event is the caller's, and a channel does not need frame counts to decide +whether to stop polling). diff --git a/docs/adr/README.md b/docs/adr/README.md index 54289a2..ea076d9 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -14,6 +14,7 @@ pre-publish security review — they document the history, they do not invent it | [0004](0004-separate-agent-secret-closed-message-set.md) | Agent role via a separate secret, and a closed human message set | accepted | Security review | | [0005](0005-handoff-not-wall-detection.md) | handraise is the handoff mechanism, not the wall detection | accepted | Scope decision | | [0006](0006-approval-mode.md) | Approval mode: one screenshot, a hold on yes | accepted | Scope decision | +| [0007](0007-channels.md) | Channels: an in-process hook, not a second WebSocket client | accepted | Scope decision | ## Format diff --git a/e2e/handoff.e2e.ts b/e2e/handoff.e2e.ts index 2a03125..17d26a6 100644 --- a/e2e/handoff.e2e.ts +++ b/e2e/handoff.e2e.ts @@ -368,6 +368,87 @@ try { await askApproval("approve") await askApproval("deny") + // --- An approval answered by a channel, not by the phone --------------- + // + // The path a Telegram or Slack adapter takes: handraise hands the channel + // the screenshot and an `answer()`, and nobody opens the link at all. The + // in-process channel here stands in for the adapter; what is under test is + // the core's side of it against the real relay. + const channelAt = Date.now() + let channelUrl = "" + let channelEvent: HandoffEvent | undefined + let channelShot = 0 + const channelAnswered = raiseHand(page, { + mode: "approval", + reason: "The agent may not move money without a human", + action: APPROVAL_ACTION, + qr: false, + timeoutMs: 60_000, + onUrl: (url) => { + channelUrl = url + }, + onEvent: (raised) => { + channelEvent = raised + }, + channels: [ + { + notify: (raised) => { + if (raised.mode !== "approval") return + channelShot = raised.screenshot.length + check( + raised.action === APPROVAL_ACTION, + "the channel is handed the action verbatim", + ) + check( + raised.url === channelUrl && channelUrl !== "", + "the channel is handed the same link the phone would open", + ) + check( + raised.answer("approve") === true, + "the channel's first answer settles the handoff", + ) + check( + raised.answer("deny") === false, + "a second answer from the channel is refused", + ) + }, + }, + ], + }) + pending = channelAnswered + const channelResult = await channelAnswered + pending = null + timings.channelApprovalMs = Date.now() - channelAt + log("channel_approval_done", { + outcome: channelResult.outcome, + answeredVia: channelEvent?.answeredVia, + screenshotBytes: channelShot, + ms: timings.channelApprovalMs, + }) + + check( + channelResult.outcome === "approved", + "an approval answered by a channel reports approved", + ) + check( + channelEvent?.answeredVia === "channel", + `the wide event says who answered (${channelEvent?.answeredVia})`, + ) + check( + channelShot > 1000, + `the channel got the real JPEG, not an empty buffer (${channelShot} bytes)`, + ) + check( + channelResult.storageState === undefined, + "a channel-answered approval captures no cookies either", + ) + const channelGone = await fetch(channelUrl, { cache: "no-store" }) + await channelGone.text() + check( + channelGone.status !== 200, + `the channel-answered relay is gone (${channelGone.status})`, + ) + // --- The cheap second case: nobody comes ------------------------------- const timeoutAt = Date.now() let secondUrl = "" diff --git a/package.json b/package.json index 1835c09..b1f128b 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "handraise", - "version": "0.4.0", + "version": "0.5.0", "description": "Human-in-the-loop handoff for Solari cloud browsers. Your agent already knows when it's stuck — handraise lets it raise its hand.", "license": "MIT", "type": "module", diff --git a/src/channels.ts b/src/channels.ts new file mode 100644 index 0000000..426fcc2 --- /dev/null +++ b/src/channels.ts @@ -0,0 +1,104 @@ +/** + * Channels: where else the handoff shows up. + * + * The QR code, `onUrl` and `webhookUrl` all say the same thing — "here is a + * link" — and none of them can carry an answer back. A chat channel can: an + * approval is a screenshot, a sentence and two buttons, which is exactly what + * a Telegram or Slack message already is. + * + * So a channel is an object with one method. handraise calls it once per + * handoff with everything the message needs (the link, the reason, and in + * approval mode the JPEG the phone is looking at), and hands it a way to + * answer in-process. The adapter never speaks the relay's wire protocol, never + * holds the human's WebSocket slot, and cannot make the handoff fail: + * `notify` is not awaited, and a throw or rejection is one `channel_failed` + * warning ([`docs/adr/0007`](../docs/adr/0007-channels.md)). + */ +import type { HandoffMode, HandoffOutcome } from "./types" + +/** The fields every channel gets, whatever the mode. */ +export interface ChannelHandoffBase { + /** Correlation key, the same one the wide event carries. */ + handoffId: string + /** + * The public handoff page. It is a bearer credential: whoever holds it can + * drive the browser (takeover) or answer (approval), so a channel that posts + * it is choosing who may do that. + */ + url: string + /** The `reason` the agent gave, verbatim — the phone shows the same string. */ + reason: string + /** What the human is being asked for. Discriminates this union. */ + mode: HandoffMode + /** + * Resolves with the outcome the moment the handoff ends, whatever ended it — + * an answer from the phone, an answer from this or another channel, the + * timeout, a dead browser session. Never rejects. + * + * This is the only way a channel learns that it is no longer needed. Without + * it an adapter that waits for a reply has nothing to wait on but its own + * clock: it keeps a chat message live and a connection open long after the + * handoff is over, and a script that has already printed its result sits + * there until that clock runs out. + * + * It is the same promise for every channel of one handoff, and it stays + * resolved — awaiting it after the fact returns immediately. + */ + settled: Promise +} + +/** + * A takeover: the human has to drive the browser, which only the handoff page + * can do. All a channel can usefully do here is deliver the link. + */ +export interface TakeoverChannelHandoff extends ChannelHandoffBase { + mode: "takeover" +} + +/** + * An approval: one screenshot and one question, which a chat message can carry + * end to end — including the answer. + */ +export interface ApprovalChannelHandoff extends ChannelHandoffBase { + mode: "approval" + /** The concrete step being decided, verbatim. */ + action: string + /** + * The JPEG the phone is showing, decoded — the same bytes, not a second + * screenshot of a page that may have moved on. + */ + screenshot: Buffer + /** + * Answer the handoff from the channel, without the handoff page. + * + * Returns `true` if this answer settled the handoff and `false` if it was + * already settled — by the phone, by another channel, by the timeout or by a + * dead session. A boolean rather than a throw because "somebody was faster" + * is the ordinary case for an approval that went to two places at once, and + * an adapter has to render it ("already decided elsewhere"), not handle it. + * + * The losing answer changes nothing: the outcome, the wide event and the + * ending the phone sees are the first answer's. + */ + answer(decision: "approve" | "deny"): boolean +} + +/** + * One handoff, as a channel adapter sees it. A discriminated union on `mode`, + * like `RaiseHandOptions`: `answer` and `screenshot` do not exist on a + * takeover, so an adapter cannot reach for them where they would be undefined. + */ +export type ChannelHandoff = TakeoverChannelHandoff | ApprovalChannelHandoff + +/** + * A place a handoff is announced, and possibly answered. + * + * `notify` is called exactly once per handoff, per channel, as soon as there + * is something to send: the URL in takeover mode, the URL and the screenshot + * in approval mode. handraise does not await it — the human may already be + * scanning the QR code while a chat API is still thinking — and catches + * whatever it throws or rejects with. + */ +export interface HandoffChannel { + notify(handoff: ChannelHandoff): void | Promise +} diff --git a/src/core/handoff.test.ts b/src/core/handoff.test.ts index 8f3951e..a69bfc9 100644 --- a/src/core/handoff.test.ts +++ b/src/core/handoff.test.ts @@ -18,8 +18,13 @@ import { fileURLToPath } from "node:url" import type { Browser, BrowserContext, CDPSession, Page } from "playwright-core" import WebSocket, { WebSocketServer } from "ws" +import type { + ChannelHandoff, + HandoffChannel, + TakeoverChannelHandoff, +} from "../channels" import type { HandoffEvent } from "../events" -import { noopLogger } from "../logger" +import { type Logger, noopLogger } from "../logger" import type { RelayMessage } from "../relay/protocol" import type { HandoffMode, RaiseHandOptions, StorageState } from "../types" import { raiseHand, runHandoff } from "./raise-hand" @@ -162,17 +167,44 @@ const VIEWPORT = { width: 1280, height: 800 } /** CDP sessions opened on the fake page since the last reset. */ let cdpSessions = 0 -/** A page whose context yields the fake CDP session and a live fake browser. */ -function fakePage(cdp: CDPSession): Page { +/** Kills the browser session behind the newest `fakePage()`. */ +let killSession: () => void = () => undefined + +/** + * A page whose context yields the fake CDP session and a live fake browser. + * + * `screenshotDelayMs` holds the approval's one screenshot in flight, which is + * the window in which a handoff can settle before the frame ever lands. + */ +function fakePage( + cdp: CDPSession, + screenshotDelayMs = 0, + storageStateDelayMs = 0, +): Page { cdpSessions = 0 let browser: Browser + let connected = true + const gone = new Set<() => void>() + // Solari sessions die on their own about ten minutes in; `killSession()` is + // how a test reproduces that, so the disconnected path is driven rather than + // assumed. + killSession = () => { + connected = false + for (const listener of gone) listener() + } const browserPartial: Partial = { - // SAFETY: registration the test never fires; the returned emitter is only + // SAFETY: only the "disconnected" listener is kept; the returned emitter is // for chaining and is never used, so pointing it back at the fake is safe. - once: (() => browser) as Browser["once"], + once: ((event: string, listener: () => void) => { + if (event === "disconnected") gone.add(listener) + return browser + }) as Browser["once"], // SAFETY: as `once`, above — an unused chaining emitter. - off: (() => browser) as Browser["off"], - isConnected: () => true, + off: ((_event: string, listener: () => void) => { + gone.delete(listener) + return browser + }) as Browser["off"], + isConnected: () => connected, } // SAFETY: runHandoff drives only once/off/isConnected on the browser. browser = browserPartial as Browser @@ -182,7 +214,10 @@ function fakePage(cdp: CDPSession): Page { cdpSessions += 1 return cdp }, - storageState: async () => STORAGE, + storageState: async () => { + if (storageStateDelayMs > 0) await Bun.sleep(storageStateDelayMs) + return STORAGE + }, } // SAFETY: runHandoff drives only browser/newCDPSession/storageState here. const context = contextPartial as BrowserContext @@ -191,7 +226,10 @@ function fakePage(cdp: CDPSession): Page { context: () => context, // SAFETY: approval mode calls screenshot() for its one frame and reads the // viewport for that frame's metadata; neither result is used as anything else. - screenshot: (async () => SAMPLE_JPEG) as Page["screenshot"], + screenshot: (async () => { + if (screenshotDelayMs > 0) await Bun.sleep(screenshotDelayMs) + return SAMPLE_JPEG + }) as Page["screenshot"], viewportSize: () => VIEWPORT, // SAFETY: as the browser's, above — an unused chaining emitter. once: (() => page) as Page["once"], @@ -219,6 +257,7 @@ test("a full handoff emits exactly one wide event with plausible fields", async onEvent: (event) => events.push(event), }, timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", handoffId: "test-handoff", relayColdStartMs: 123, logger: noopLogger, @@ -285,6 +324,7 @@ test("a throwing onEvent callback does not break the handoff", async () => { }, }, timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", handoffId: "throwing", relayColdStartMs: 5, logger: noopLogger, @@ -315,6 +355,7 @@ test("an approval handoff sends one screenshot and settles on approve", async () onEvent: (event) => events.push(event), }, timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", handoffId: "approval-handoff", relayColdStartMs: 42, logger: noopLogger, @@ -376,6 +417,7 @@ test("an approval handoff ignores takeover messages that reach it anyway", async }, timeoutMs: 1500, logger: noopLogger, + url: "https://relay.example/?pt_token=x", handoffId: "approval-mismatch", relayColdStartMs: 7, }) @@ -408,6 +450,7 @@ test("a denied approval reports denied", async () => { logger: noopLogger, }, timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", handoffId: "approval-denied", relayColdStartMs: 9, logger: noopLogger, @@ -493,6 +536,7 @@ test("the approval screenshot is not re-published once the handoff is over", asy // Long enough for a reconnect or two while the "human" decides, short // enough that the ending falls into one of them. timeoutMs: 1200, + url: "https://relay.example/?pt_token=x", handoffId: "approval-reconnect", relayColdStartMs: 3, logger: noopLogger, @@ -553,3 +597,786 @@ test("an unknown mode is refused before anything is created", async () => { const asking = raiseHand(fakePage(fakeCdp().cdp), options) await expect(asking).rejects.toThrow(/unknown mode/) }) + +// --- Channels ------------------------------------------------------------ +// +// A channel is an in-process object handraise notifies when the handoff URL +// exists. In approval mode it also gets the screenshot and can answer without +// the phone, through the same settle path a relay `approve` takes. + +interface ChannelRecorder { + channel: HandoffChannel + /** Every handoff this channel was notified about, in order. */ + seen: ChannelHandoff[] +} + +/** A channel that records what it was handed, and never answers by itself. */ +function recordingChannel(): ChannelRecorder { + const seen: ChannelHandoff[] = [] + return { + channel: { + notify: (handoff) => { + seen.push(handoff) + }, + }, + seen, + } +} + +interface LoggerRecorder { + logger: Logger + /** The event names passed to `warn`, in order. */ + warnings: string[] +} + +/** A logger that keeps its warnings, so a swallowed failure is still provable. */ +function recordingLogger(): LoggerRecorder { + const warnings: string[] = [] + return { + logger: { + debug: () => undefined, + info: () => undefined, + warn: (event) => { + warnings.push(event) + }, + error: () => undefined, + }, + warnings, + } +} + +test("a channel that approves settles the handoff and the phone is told", async () => { + const port = await startRelayProcess("approval") + const human = await connectHuman(port) + const cdp = fakeCdp() + const events: HandoffEvent[] = [] + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + onEvent: (event) => events.push(event), + channels: [ + { + notify: (raised) => { + if (raised.mode === "approval") raised.answer("approve") + }, + }, + ], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "channel-approve", + relayColdStartMs: 11, + logger: noopLogger, + }) + + const end = await handoff + expect(end.outcome).toBe("approved") + + const event = events[0] + if (!event) throw new Error("no event") + expect(event.outcome).toBe("approved") + expect(event.answeredVia).toBe("channel") + + // The phone was never asked, and still sees the handoff end: the relay gets + // the same `ended` message an answer from the phone would have produced. + await until("the phone to be told how it ended", () => + human.inbox.some( + (message) => message.type === "ended" && message.outcome === "approved", + ), + ) +}) + +test("the first answer wins: the phone denies, a later channel approve is refused", async () => { + const port = await startRelayProcess("approval") + const human = await connectHuman(port) + const cdp = fakeCdp() + const events: HandoffEvent[] = [] + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not delete production data", + action: "Delete s3://prod-invoices", + logger: noopLogger, + onEvent: (event) => events.push(event), + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "relay-wins", + relayColdStartMs: 12, + logger: noopLogger, + }) + + await until("the phone to see the screenshot", () => + human.inbox.some((message) => message.type === "frame"), + ) + human.send({ type: "deny" }) + + const end = await handoff + expect(end.outcome).toBe("denied") + expect(events[0]?.answeredVia).toBe("relay") + + // The channel was notified, and its answer arrives too late. + const raised = recorder.seen[0] + if (raised?.mode !== "approval") throw new Error("no approval handoff") + expect(raised.answer("approve")).toBe(false) +}) + +test("the first answer wins the other way round: the channel denies first", async () => { + const port = await startRelayProcess("approval") + const human = await connectHuman(port) + const cdp = fakeCdp() + const events: HandoffEvent[] = [] + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not delete production data", + action: "Delete s3://prod-invoices", + logger: noopLogger, + onEvent: (event) => events.push(event), + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "channel-wins", + relayColdStartMs: 13, + logger: noopLogger, + }) + + await until("the channel to be notified", () => recorder.seen.length === 1) + const raised = recorder.seen[0] + if (raised?.mode !== "approval") throw new Error("no approval handoff") + + // The channel answers first, then the phone tries to overturn it. + expect(raised.answer("deny")).toBe(true) + human.send({ type: "approve" }) + + const end = await handoff + expect(end.outcome).toBe("denied") + expect(events[0]?.answeredVia).toBe("channel") + // And a second answer from the channel itself is refused just the same. + expect(raised.answer("deny")).toBe(false) +}) + +test("a channel that throws is logged and does not touch the handoff", async () => { + const port = await startRelayProcess("approval") + const human = await connectHuman(port) + const cdp = fakeCdp() + const recorder = recordingLogger() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: recorder.logger, + channels: [ + { + notify: () => { + throw new Error("the chat API is down") + }, + }, + // A rejected promise is the same failure one tick later, and the + // channel behind the broken one still has to be notified. + { notify: () => Promise.reject(new Error("and so is the other one")) }, + ], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "channel-throws", + relayColdStartMs: 14, + logger: recorder.logger, + }) + + await until("the phone to see the screenshot", () => + human.inbox.some((message) => message.type === "frame"), + ) + human.send({ type: "approve" }) + + const end = await handoff + expect(end.outcome).toBe("approved") + expect( + recorder.warnings.filter((event) => event === "channel_failed"), + ).toHaveLength(2) +}) + +test("a takeover channel gets the link and nothing to answer with", async () => { + const port = await startRelayProcess() + const human = await connectHuman(port) + const cdp = fakeCdp() + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + reason: "Aurora Bank is asking for a 2FA code", + logger: noopLogger, + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://takeover.example/?pt_token=secret", + handoffId: "takeover-channel", + relayColdStartMs: 15, + logger: noopLogger, + }) + + await until("the channel to be notified", () => recorder.seen.length === 1) + const raised = recorder.seen[0] + if (!raised) throw new Error("the channel was not notified") + expect(raised.mode).toBe("takeover") + expect(raised.url).toBe("https://takeover.example/?pt_token=secret") + expect(raised.reason).toBe("Aurora Bank is asking for a 2FA code") + expect(raised.handoffId).toBe("takeover-channel") + // There is no moment to show and no question to answer in a takeover, so + // neither field exists — the union says so, and the value agrees. + expect("screenshot" in raised).toBe(false) + expect("answer" in raised).toBe(false) + + human.send({ type: "handback" }) + expect((await handoff).outcome).toBe("resolved") +}) + +test("an approval channel gets the same JPEG bytes the phone gets", async () => { + const port = await startRelayProcess("approval") + const human = await connectHuman(port) + const cdp = fakeCdp() + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "channel-bytes", + relayColdStartMs: 16, + logger: noopLogger, + }) + + await until("the phone to see the screenshot", () => + human.inbox.some((message) => message.type === "frame"), + ) + const frame = human.inbox.find((message) => message.type === "frame") + if (frame?.type !== "frame") throw new Error("no frame reached the phone") + + const raised = recorder.seen[0] + if (raised?.mode !== "approval") throw new Error("no approval handoff") + expect(raised.action).toBe("Submit $12,430 vendor payment to Acme GmbH") + // Byte for byte the picture the human on the phone is looking at, so the two + // cannot be shown different things and asked the same question. + expect(raised.screenshot).toEqual(Buffer.from(frame.data, "base64")) + expect(raised.screenshot).toEqual(SAMPLE_JPEG) + + human.send({ type: "deny" }) + expect((await handoff).outcome).toBe("denied") +}) + +test("a handoff that ends before the screenshot lands is never announced", async () => { + // The window between "take the screenshot" and "send it": a round trip to + // the browser, during which the page can close or the wait can run out. + // `sendApprovalFrame` already refuses to put a frame on the wire after that; + // a channel that posted anyway would leave live buttons under a request that + // no longer exists, and the first press would be told "already decided". + const port = await startRelayProcess("approval") + await connectHuman(port) + const cdp = fakeCdp() + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp, 250), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + channels: [recorder.channel], + }, + // Runs out while the screenshot above is still being taken. + timeoutMs: 1, + url: "https://relay.example/?pt_token=x", + handoffId: "settled-before-announce", + relayColdStartMs: 17, + logger: noopLogger, + }) + + const end = await handoff + expect(end.outcome).toBe("timeout") + expect(recorder.seen).toEqual([]) +}) + +test("a takeover handback carries no answeredVia", async () => { + // A handback and a give-up go through the same `answerHandoff` as an + // approval answer, so `answeredVia` is set on them too. The wide event only + // carries it where it means something: "who said yes or no". This is the + // guard that keeps it off every takeover event. + const port = await startRelayProcess() + const human = await connectHuman(port) + const cdp = fakeCdp() + const events: HandoffEvent[] = [] + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + reason: "Aurora Bank is asking for a 2FA code", + logger: noopLogger, + onEvent: (event) => events.push(event), + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "takeover-answered-via", + relayColdStartMs: 18, + logger: noopLogger, + }) + + await until("the phone to connect", () => human.inbox.length >= 0) + human.send({ type: "handback" }) + expect((await handoff).outcome).toBe("resolved") + + const event = events[0] + if (!event) throw new Error("no event") + expect(event.answeredVia).toBeUndefined() + expect(JSON.stringify(event)).not.toContain("answeredVia") +}) + +// --- The boundaries the ADR claims, as failing-first tests --------------- + +test("an answer that arrives after a timeout is refused and emits nothing", async () => { + const port = await startRelayProcess("approval") + await connectHuman(port) + const cdp = fakeCdp() + const events: HandoffEvent[] = [] + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + onEvent: (event) => events.push(event), + channels: [recorder.channel], + }, + timeoutMs: 400, + url: "https://relay.example/?pt_token=x", + handoffId: "late-after-timeout", + relayColdStartMs: 19, + logger: noopLogger, + }) + + const end = await handoff + expect(end.outcome).toBe("timeout") + expect(events).toHaveLength(1) + + // A channel that was notified before the wait ran out still holds a live + // `answer`. It has to be inert: the caller has already been told `timeout` + // and has moved on. + const raised = recorder.seen[0] + if (raised?.mode !== "approval") throw new Error("no approval handoff") + expect(raised.answer("approve")).toBe(false) + await Bun.sleep(50) + expect(events).toHaveLength(1) + expect(events[0]?.outcome).toBe("timeout") + expect(events[0]?.answeredVia).toBeUndefined() +}) + +test("an answer that arrives after the session died is refused and emits nothing", async () => { + const port = await startRelayProcess("approval") + await connectHuman(port) + const cdp = fakeCdp() + const events: HandoffEvent[] = [] + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + onEvent: (event) => events.push(event), + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "late-after-disconnect", + relayColdStartMs: 20, + logger: noopLogger, + }) + + await until("the channel to be notified", () => recorder.seen.length === 1) + killSession() + + const end = await handoff + expect(end.outcome).toBe("disconnected") + expect(events).toHaveLength(1) + + const raised = recorder.seen[0] + if (raised?.mode !== "approval") throw new Error("no approval handoff") + expect(raised.answer("deny")).toBe(false) + await Bun.sleep(50) + expect(events).toHaveLength(1) + expect(events[0]?.outcome).toBe("disconnected") +}) + +test("a session that dies during the screenshot notifies no channel", async () => { + // The timeout half of this window is covered above; this is the other way + // it closes, and the one that actually happened in the field — a Solari + // session hitting its hard lifetime mid-capture. + const port = await startRelayProcess("approval") + await connectHuman(port) + const cdp = fakeCdp() + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp, 250), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "died-mid-screenshot", + relayColdStartMs: 21, + logger: noopLogger, + }) + + // While `page.screenshot()` is still in flight. + await Bun.sleep(40) + killSession() + + expect((await handoff).outcome).toBe("disconnected") + expect(recorder.seen).toEqual([]) +}) + +test("a channel that mutates its screenshot cannot change what the phone got", async () => { + const port = await startRelayProcess("approval") + const human = await connectHuman(port) + const cdp = fakeCdp() + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "buffer-isolation", + relayColdStartMs: 22, + logger: noopLogger, + }) + + await until("the phone to see the screenshot", () => + human.inbox.some((message) => message.type === "frame"), + ) + const raised = recorder.seen[0] + if (raised?.mode !== "approval") throw new Error("no approval handoff") + + // A channel gets a Buffer, and a Buffer is writable. An adapter that + // compresses or watermarks in place must not be able to change the picture + // the human on the phone is deciding on. + raised.screenshot.fill(0) + + const frame = human.inbox.find((message) => message.type === "frame") + if (frame?.type !== "frame") throw new Error("no frame reached the phone") + expect(Buffer.from(frame.data, "base64")).toEqual(SAMPLE_JPEG) + + human.send({ type: "approve" }) + expect((await handoff).outcome).toBe("approved") +}) + +test("a channel whose notify never settles does not hold up the handoff", async () => { + // `notify` is not awaited, and this is what that sentence has to mean: a + // chat API that accepts the request and never answers costs the handoff + // nothing. A regression that awaited it would hang here until the test + // timeout rather than fail an assertion, which is the loudest failure this + // boundary has. + const port = await startRelayProcess("approval") + const human = await connectHuman(port) + const cdp = fakeCdp() + + const startedAt = Date.now() + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + channels: [{ notify: () => new Promise(() => undefined) }], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "never-settles", + relayColdStartMs: 23, + logger: noopLogger, + }) + + await until("the phone to see the screenshot", () => + human.inbox.some((message) => message.type === "frame"), + ) + human.send({ type: "approve" }) + expect((await handoff).outcome).toBe("approved") + expect(Date.now() - startedAt).toBeLessThan(4000) +}) + +test("a takeover ChannelHandoff has no answer and no screenshot, at compile time", () => { + // The runtime shape is asserted elsewhere with `in`. This is the other half: + // the union is what stops an adapter from writing `handoff.answer(...)` on a + // takeover in the first place, and `tsc --noEmit` covers this file, so a + // union that quietly grew those members would fail the typecheck here — + // `@ts-expect-error` is an error of its own when there is no error to expect. + const takeover: TakeoverChannelHandoff = { + mode: "takeover", + handoffId: "compile-negative", + url: "https://relay.example/?pt_token=x", + reason: "Aurora Bank is asking for a 2FA code", + settled: Promise.resolve("resolved"), + } + // @ts-expect-error `answer` exists only on the approval member of the union. + const answer = takeover.answer + // @ts-expect-error `screenshot` exists only on the approval member. + const screenshot = takeover.screenshot + expect(answer).toBeUndefined() + expect(screenshot).toBeUndefined() + + // And through the union itself, which is what an adapter actually receives. + const handoff: ChannelHandoff = takeover + // @ts-expect-error narrow on `mode` before reaching for an approval field. + const unnarrowed = handoff.action + expect(unnarrowed).toBeUndefined() +}) + +// --- `settled`: the signal a channel has to have ------------------------- + +test("settled resolves with the outcome when the phone answers", async () => { + const port = await startRelayProcess("approval") + const human = await connectHuman(port) + const cdp = fakeCdp() + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "settled-relay", + relayColdStartMs: 24, + logger: noopLogger, + }) + + await until("the channel to be notified", () => recorder.seen.length === 1) + const raised = recorder.seen[0] + if (!raised) throw new Error("the channel was not notified") + + human.send({ type: "deny" }) + expect(await handoff).toEqual({ outcome: "denied" }) + // This is the whole point: the channel is told the phone answered, without + // having been the one who was asked. + expect(await raised.settled).toBe("denied") + // And it stays resolved — an adapter may await it long after the fact. + expect(await raised.settled).toBe("denied") +}) + +test("settled resolves when the channel itself answers", async () => { + const port = await startRelayProcess("approval") + await connectHuman(port) + const cdp = fakeCdp() + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "settled-channel", + relayColdStartMs: 25, + logger: noopLogger, + }) + + await until("the channel to be notified", () => recorder.seen.length === 1) + const raised = recorder.seen[0] + if (raised?.mode !== "approval") throw new Error("no approval handoff") + expect(raised.answer("approve")).toBe(true) + + expect((await handoff).outcome).toBe("approved") + expect(await raised.settled).toBe("approved") +}) + +test("settled resolves on a timeout and on a dead session", async () => { + const port = await startRelayProcess("approval") + await connectHuman(port) + const cdp = fakeCdp() + const timedOut = recordingChannel() + + const waiting = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "nobody is going to answer this one", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + channels: [timedOut.channel], + }, + timeoutMs: 400, + url: "https://relay.example/?pt_token=x", + handoffId: "settled-timeout", + relayColdStartMs: 26, + logger: noopLogger, + }) + await until("the channel to be notified", () => timedOut.seen.length === 1) + expect((await waiting).outcome).toBe("timeout") + expect(await timedOut.seen[0]?.settled).toBe("timeout") + + const secondPort = await startRelayProcess("approval") + await connectHuman(secondPort) + const dead = recordingChannel() + const dying = runHandoff({ + page: fakePage(fakeCdp().cdp), + agentWsUrl: `ws://127.0.0.1:${secondPort}/ws?role=agent`, + options: { + mode: "approval", + reason: "the session is about to die", + action: "Submit $12,430 vendor payment to Acme GmbH", + logger: noopLogger, + channels: [dead.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "settled-disconnected", + relayColdStartMs: 27, + logger: noopLogger, + }) + await until("the channel to be notified", () => dead.seen.length === 1) + killSession() + expect((await dying).outcome).toBe("disconnected") + expect(await dead.seen[0]?.settled).toBe("disconnected") +}) + +test("every channel of one handoff gets the same settled promise", async () => { + const port = await startRelayProcess() + const human = await connectHuman(port) + const cdp = fakeCdp() + const first = recordingChannel() + const second = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + reason: "Aurora Bank is asking for a 2FA code", + logger: noopLogger, + channels: [first.channel, second.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "settled-shared", + relayColdStartMs: 28, + logger: noopLogger, + }) + + await until( + "both channels to be notified", + () => first.seen.length === 1 && second.seen.length === 1, + ) + // One handoff, one ending: two adapters must not be able to see different + // ones, and a takeover channel gets it too — it posted a bearer link that + // stops working when this resolves. + expect(first.seen[0]?.settled).toBe(second.seen[0]?.settled) + + human.send({ type: "handback" }) + expect((await handoff).outcome).toBe("resolved") + expect(await first.seen[0]?.settled).toBe("resolved") +}) + +test("settled reports the outcome the caller gets, not the one the human gave", async () => { + // The one path where those differ: a handback wins the promise, and the + // Solari session hits its ~10-minute hard death while the cookies are being + // captured. `raiseHand` reports `disconnected` rather than a dead + // "resolved" — and a channel that had been told "resolved" would post the + // wrong ending into a chat that outlives the process. + const port = await startRelayProcess() + const human = await connectHuman(port) + const cdp = fakeCdp() + const recorder = recordingChannel() + + const handoff = runHandoff({ + page: fakePage(cdp.cdp, 0, 300), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + reason: "Aurora Bank is asking for a 2FA code", + logger: noopLogger, + channels: [recorder.channel], + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "settled-final-outcome", + relayColdStartMs: 29, + logger: noopLogger, + }) + + await until("the channel to be notified", () => recorder.seen.length === 1) + human.send({ type: "handback" }) + // While `storageState()` is in flight: the handback has already settled the + // handoff, so this only changes what `isConnected()` says afterwards. + await Bun.sleep(120) + killSession() + + const end = await handoff + expect(end.outcome).toBe("disconnected") + expect(end.storageState).toBeUndefined() + expect(await recorder.seen[0]?.settled).toBe("disconnected") +}) diff --git a/src/core/raise-hand.ts b/src/core/raise-hand.ts index 36ddda3..03a023e 100644 --- a/src/core/raise-hand.ts +++ b/src/core/raise-hand.ts @@ -16,6 +16,12 @@ * once. */ import type { CDPSession, Page } from "playwright-core" +import type { + ApprovalChannelHandoff, + ChannelHandoff, + HandoffChannel, + TakeoverChannelHandoff, +} from "../channels" import type { HandoffEvent } from "../events" import { type Logger, quietLogger } from "../logger" import { printHandoffQr } from "../qr" @@ -141,6 +147,8 @@ export interface HandoffRun { agentWsUrl: string options: RaiseHandOptions timeoutMs: number + /** The public handoff page, as handed to `onUrl` and to every channel. */ + url: string /** Correlation key; the relay's preview subdomain. */ handoffId: string /** Measured by `startRelay()` and mirrored into the event. */ @@ -185,6 +193,79 @@ function emitHandoffEvent( } } +/** + * The view a channel gets of a takeover: the link, and that is all. There is + * nothing to answer and no one moment worth sending — the human has to drive + * the browser, which only the handoff page can do. + */ +function takeoverChannelHandoff( + run: HandoffRun, + settled: Promise, +): TakeoverChannelHandoff { + return { + mode: "takeover", + handoffId: run.handoffId, + url: run.url, + reason: run.options.reason, + settled, + } +} + +/** + * The view a channel gets of an approval. + * + * `action` and `shot` are parameters rather than fields read off `run`, + * because only the caller has narrowed the options union far enough to know + * they exist. Two builders rather than one with a fallback: an approval + * announced as a takeover would not be a degraded message but the wrong one — + * a bearer link and "drive the browser", in place of a yes or no. + */ +function approvalChannelHandoff( + run: HandoffRun, + action: string, + shot: ApprovalFrame, + answer: (decision: "approve" | "deny") => boolean, + settled: Promise, +): ApprovalChannelHandoff { + return { + mode: "approval", + handoffId: run.handoffId, + url: run.url, + reason: run.options.reason, + settled, + action, + // The same JPEG the phone is looking at. It is held base64 because that is + // what goes on the wire; this is that exact payload decoded back, not a + // second screenshot of a page that may have moved on since. + screenshot: Buffer.from(shot.data, "base64"), + answer, + } +} + +/** + * Announce the handoff to every channel, once each. + * + * Fire-and-forget on purpose: a channel is a side channel. It must not delay + * the handoff (the human may already be scanning the QR code) and it must not + * be able to end it, so a synchronous throw and a rejected promise are the + * same thing here — one warning, and the handoff carries on. + */ +function notifyChannels( + channels: readonly HandoffChannel[], + handoff: ChannelHandoff, + logger: Logger, +): void { + for (const channel of channels) { + try { + void Promise.resolve(channel.notify(handoff)).catch((error) => { + logger.warn("channel_failed", { error: String(error) }) + }) + } catch (error) { + logger.warn("channel_failed", { error: String(error) }) + } + } +} + /** * Run one handoff to its end. Never throws, never leaves a timer, a listener * or a CDP session behind. Emits the wide event exactly once before it settles. @@ -215,6 +296,15 @@ export async function runHandoff(run: HandoffRun): Promise { } }) + // What every channel of this handoff awaits. Resolved once, with the outcome + // the caller is given — after the handback check, so a channel is never told + // "resolved" for a session that turned out to be dead. It never rejects, so + // an adapter can await it without a guard. + let announceSettled: (outcome: HandoffOutcome) => void = () => undefined + const settled = new Promise((resolve) => { + announceSettled = resolve + }) + const browser = page.context().browser() const onGone = (): void => settle("disconnected") browser?.once("disconnected", onGone) @@ -231,6 +321,27 @@ export async function runHandoff(run: HandoffRun): Promise { let terminal = false + // Who ended it, when it was ended by an answer. Only meaningful for + // `approved` and `denied`; the event carries it on those outcomes only. + let answeredVia: "relay" | "channel" | undefined + + /** + * The single settle path for a human answer, from the phone or from a + * channel. First answer wins: `over` is set by the first `settle`, so a + * second answer — the phone tapping Deny while a Telegram button was already + * pressed — changes nothing and is told so. + */ + const answerHandoff = ( + outcome: HandoffOutcome, + via: "relay" | "channel", + ): boolean => { + if (over) return false + terminal = true + answeredVia = via + settle(outcome) + return true + } + // The phone cannot see a caret in a 60-quality JPEG, so the agent tells it // where the typing lands. Strictly off the critical path: a probe is never // awaited by the input it follows, it holds no timer, and the newest result @@ -263,8 +374,7 @@ export async function runHandoff(run: HandoffRun): Promise { const onHuman = (message: HumanToAgent): void => { const ending = endingFor(mode, message.type) if (ending) { - terminal = true - settle(ending) + answerHandoff(ending, "relay") return } // Once a terminal message has arrived the page is being handed back or @@ -319,6 +429,26 @@ export async function runHandoff(run: HandoffRun): Promise { }) link = connection + const answerFromChannel = (decision: "approve" | "deny"): boolean => + answerHandoff(decision === "approve" ? "approved" : "denied", "channel") + + /** + * Tell the channels. Once per handoff, per channel, at the first moment + * there is something worth sending: the link in takeover mode, the link and + * the screenshot in approval mode. + * + * Not once the handoff is over. Taking the screenshot is a round trip to the + * browser, and the page can close or the wait can run out while it is in + * flight — `sendApprovalFrame` guards exactly that window, and a channel + * announced anyway would leave live buttons under a request that no longer + * exists. + */ + const announce = (handoff: ChannelHandoff): void => { + const channels = options.channels ?? [] + if (over || channels.length === 0) return + notifyChannels(channels, handoff, logger) + } + try { if (mode === "approval") { // One screenshot and nothing else: no CDP session, no screencast, no @@ -326,7 +456,24 @@ export async function runHandoff(run: HandoffRun): Promise { // left it, and it is still that page afterwards. approvalFrame = await captureApprovalFrame(page) await sendApprovalFrame() + // `options.mode`, not the `mode` local: this is the check that narrows + // the union, and it is what makes `options.action` readable here. + if (options.mode === "approval") { + announce( + approvalChannelHandoff( + run, + options.action, + approvalFrame, + answerFromChannel, + settled, + ), + ) + } } else { + // The relay is up — `raiseHand` awaited it — so the link in the message + // is already open. Sent before the cast starts, because the cast is not + // what the human needs in order to be told. + announce(takeoverChannelHandoff(run, settled)) cdp = await page.context().newCDPSession(page) input = createInputTarget(cdp) pump = await startTakeoverCast(cdp, connection, (data) => { @@ -384,6 +531,11 @@ export async function runHandoff(run: HandoffRun): Promise { } } + // The earliest point at which the outcome is the one the caller will see. + // Before teardown on purpose: a channel that stops polling here releases the + // chat and the process while the relay is still being shut down. + announceSettled(finalOutcome) + clearTimeout(timer) browser?.off("disconnected", onGone) page.off("close", onGone) @@ -412,6 +564,14 @@ export async function runHandoff(run: HandoffRun): Promise { storageStateCaptured: storageState !== undefined, } if (firstFrameMs !== undefined) event.firstFrameMs = firstFrameMs + // Only an answer has a source. A timeout, a dead session or a handback is + // not "answered via" anything, so the field stays absent there. + if ( + answeredVia !== undefined && + (finalOutcome === "approved" || finalOutcome === "denied") + ) { + event.answeredVia = answeredVia + } if (options.baseUrl !== undefined) event.baseUrl = options.baseUrl if (firstError !== undefined) event.error = firstError emitHandoffEvent(options, logger, event) @@ -523,6 +683,7 @@ export async function raiseHand( agentWsUrl: relay.agentWsUrl, options, timeoutMs, + url: relay.humanUrl, handoffId: handoffId(relay.humanUrl), relayColdStartMs: relay.coldStartMs, logger, diff --git a/src/events.ts b/src/events.ts index 3491b96..724e450 100644 --- a/src/events.ts +++ b/src/events.ts @@ -51,6 +51,13 @@ export interface HandoffEvent { reconnects: number /** Whether cookies + localStorage were captured after a handback. */ storageStateCaptured: boolean + /** + * Who answered, on the `approved` and `denied` outcomes only: `relay` is the + * handoff page (the phone), `channel` is an in-process `HandoffChannel` such + * as a Telegram adapter. Absent on every other outcome, and on an approval + * nobody answered. + */ + answeredVia?: "relay" | "channel" /** Gateway base URL, when the caller overrode the Solari default. */ baseUrl?: string /** Set on a failure path: the first error message that shaped the outcome. */ diff --git a/src/index.ts b/src/index.ts index 8b63192..fedeab6 100644 --- a/src/index.ts +++ b/src/index.ts @@ -22,6 +22,12 @@ * Needs `SOLARI_API_KEY` in the environment: the handoff page is served from a * Solari sandbox that handraise creates and destroys around the call. */ +export type { + ApprovalChannelHandoff, + ChannelHandoff, + HandoffChannel, + TakeoverChannelHandoff, +} from "./channels" export { raiseHand } from "./core/raise-hand" export type { HandoffEvent } from "./events" export { diff --git a/src/types.ts b/src/types.ts index ae4ebf1..3cc4909 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1,4 +1,5 @@ import type { BrowserContext, Page } from "playwright-core" +import type { HandoffChannel } from "./channels" import type { HandoffEvent } from "./events" import type { Logger } from "./logger" @@ -23,6 +24,14 @@ export interface HandoffOptions { webhookUrl?: string /** Called with the public handoff URL as soon as it exists. */ onUrl?: (url: string) => void + /** + * Where else to announce this handoff — a chat channel, a pager, anything + * that implements `HandoffChannel`. Each one is notified once, is never + * awaited, and cannot break the handoff. In approval mode a channel also + * gets the screenshot and can answer in-process, so the human never has to + * open the link (see `handraise-telegram`). + */ + channels?: HandoffChannel[] /** * How long to wait for the human before giving up. Default: 5 minutes. * Keep this short: Solari browser sessions have a hard lifetime of about