diff --git a/CONTEXT.md b/CONTEXT.md index fd4b209f..b85fc322 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -27,6 +27,7 @@ S15/S16 target terms below refine product intent. Definitions explicitly marked **Shared and private computers (S16):** With explicit S16 computers (ADR 0087), each work owns its own workspace lease on a working VM, so different works run concurrently on the shared computer. A *private computer* is a separate working VM the owner gives to exactly one Bot; that Bot's new work uses only it and no other Bot's work is ever placed there. The assignment is permanent: a private computer never becomes shared or another Bot's, and retiring it only stops new work there. New Bots use the shared computer. **Idle release (S16):** Jit stops a working VM that has had no open session for 30 minutes (owner decision 2026-11-07), keeping its disk, and starts it again under a new generation when work is placed there. A VM the owner stopped is never started by work. +**Screen control (S16, DESK16-01):** Who may write to a desktop working computer's screen: work (`agent`), the owner after every earlier work operation settled (`owner`), or nobody after the owner lost control (`fenced`). Every change advances the screen generation, so frames and input from before it are refused. Frames and input go through QEMU's controller-only QMP socket, never the guest network; typed text is sent once and never stored. Sensitive-site login stays in the protected browser. **Protected control plane:** The authority/state/credential/effect boundary that working agents, including a guest administrator, cannot alter or impersonate. Physical placement and enforcement are proposed engineering, not established by a sibling process or a prompt. diff --git a/docs/adr/0090-s16-watch-and-take-over-a-working-computer.md b/docs/adr/0090-s16-watch-and-take-over-a-working-computer.md new file mode 100644 index 00000000..d8d0a8e8 --- /dev/null +++ b/docs/adr/0090-s16-watch-and-take-over-a-working-computer.md @@ -0,0 +1,48 @@ +--- +status: implemented for DESK16-01 (#410), first slice; deterministic evidence; guest desktop session, desktop-browser egress and live host evidence pending (#410 follow-up, #431) +date: 2026-11-08 +scope: DESK16-01 (#410); S16-AC05/AC07; FR-02/10/14/15/18/21/35/36/38/40; NFR-01/06/07/11/12/14 +--- + +# Watch and take over a working computer + +## Context + +S16-AC07 asks the owner to watch the working screen, take exclusive control, complete a protected-input step and hand control back with fresh generations. Agent input must be fenced throughout, and old frames or handles must never regain authority. Sensitive-site credentials stay in the protected browser (S15 BROW15, ADR 0079); trusted sites may keep their session in the working VM browser (S16-D09, ADR 0084). #442 still waits for "a trusted site works in the VM browser after one owner login", which needs this takeover. + +The S15/S16 working VM had no display: QEMU ran with `-display none` and no display device, and work used only shell and file tools over SSH. The protected browser already has an owner takeover (`BrowserOwner.takeover/frame/input/handBack`, field-based, generation-checked) on the S15 Web pages. + +## Options compared + +1. **Rakazo (pinned `abf3c08`)**: the computer image runs Xvfb, fluxbox, Chromium, `x11vnc -viewonly` and websockify/noVNC inside the sandbox; the API seals a screen capability (Bot, computer, screen generation, control lease) and the web client embeds noVNC; human control is a lease that interactive VNC needs. The agent runs as an unprivileged user without escalation, so the in-sandbox VNC server is outside its reach. +2. **Guest VNC in the Jit working VM**: in Jit the agent has root in the VM (RT15-01 option B). A guest VNC server, its input path and its fencing would be under the agent's control; noVNC also adds an MPL-2.0 client and a WebSocket proxy. +3. **Host-side screen through QMP (chosen)**: QEMU owns the virtual display. Frames come from QMP `screendump` (PNG) and input goes through QMP `input-send-event` on the controller-only QMP socket that already exists. No display server, port or guest channel is added; fencing is enforced in the controller, before any event reaches QEMU. + +Option 3 keeps Rakazo's product concept (watch, take control with a lease, generation-fenced capability, hand back) and is simpler and stronger for a root-controlled guest. Frames are polled (about 2 per second while the owner drives, under 1 per second while watching) instead of streamed; that is enough for a login or a confirmation step and needs no new dependency. Streaming can replace polling later behind the same contract. + +## Decision + +1. **Desktop profile** (`s15-runtime/v1` `desktop: true`, requires `computers`). Jit-created VMs start with `-device virtio-vga,xres=1280,yres=800` (`virtio-gpu-pci` on aarch64), `-device virtio-tablet-pci` and `-device virtio-keyboard-pci`; `-display none` stays. The headless argv is unchanged byte for byte, and `/proc` re-verification accepts either exact profile, so a VM that was already running headless keeps working and reports `vm_desktop_restart_required` until its next start (idle sleep makes that natural). The VM15-03 launcher accepts an optional sixth `start` word `desktop` and nothing else; an installed older launcher refuses it, so a separate-identity host needs the installer's launcher refresh before desktop starts (owner-host transition, CUT16-01). +2. **Host screen device** (`QemuVmManagement.screen()`, `src/s15/vm/screen.ts`). `unavailable()` reports `vm_desktop_unavailable`, `vm_not_running` or `vm_desktop_restart_required` from controller facts only. `capture()` runs one QMP `screendump` at a time into the VM's hypervisor directory, reads it once and always removes it before the capture settles (under VM15-03 the file belongs to the QEMU identity, so its launcher's `discard-frame ` removes exactly `/screen.png`; a frame that cannot be removed fails the capture). `stop()` (also the idle stop) and the launcher's `start` remove any frame left behind, so a stopped VM keeps none, and accepts only a bounded PNG whose IHDR is 1280×800. `send()` maps one validated input to QMP events: absolute pointer moves, button press/release pairs, wheel buttons, and key press/release batches with qcodes. Text is limited to characters a US keyboard types; anything else is refused (`screen_text_unsupported`), never approximated. +3. **Exclusive control** (`s16-computer-screen/v1`, `ScreenControl`, protected VM journal tables `s16_screen_control`, `s16_screen_commands`, `s16_screen_events`). States are `agent`, `settling`, `owner` and `fenced`, and every transition (takeover, lease lapse, restart, hand-back) advances the generation: + - Every working-VM operation holds an agent ticket from admission until it settled. `WorkingVmService` checks the ticket wherever it already checks session currency (before the claim, at dispatch, before start, before publish), so after a takeover no work operation is admitted, dispatched or published. + - Takeover moves to `settling` while tickets of the earlier generation are open and to `owner` when the last one ends. Owner input is refused (`takeover_settling`) until then, so work and owner never hold a writable generation together. + - Owner input names the generation and takeover id, is checked against the live row immediately before it is sent, and is serialized per computer. Hand-back first refuses new owner input, waits for the input being sent, then returns the screen to work under a new generation. An input still queued behind a hand-back is refused. + - Owner control lasts 30 s per renewal; the client renews every 10 s. A lapsed lease or a controller restart while the owner held control becomes `fenced`: nobody writes until the owner takes control again or hands back. Work never regains the screen because a tab closed. + - Idle release skips a computer that is not in `agent`. + - Frames are served only for the current generation and are dropped if a transition happened during the capture (`stale_screen`). + - Takeover and hand-back are idempotent per `requestId` (hash of the command, which holds no input). Input request ids are kept in memory for the current owner generation: a retry never sends twice, and a send whose outcome is unknown is refused (`screen_input_unknown`) instead of retried; no typed text, hash of it, or frame is written to the journal, history, memory or logs. +4. **Owner surface.** `GET /api/s16/client/computers//screen` (state), `GET .../screen/frame?generation=N` (PNG, `private, no-store`, `nosniff`, CSP `default-src 'none'`), and `POST /api/s16/client/computers/screen/commands` with exact `s16-screen-command/v1` bodies (`takeover`, `renew`, `input`, `hand_back`), owner-authenticated with CSRF rechecked after the body. The Computers page gains "Watch screen" on running computers: live frame with a text alternative, take over / hand back / cancel, clicks, double and right clicks, wheel, a keyboard-capture region (Tab leaves it), a masked "type on the computer" field that clears after sending, and Enter/Tab/Esc/Backspace buttons. English and Thai copy. +5. **Lineage.** The screen view carries the computer's scope; a private computer's frames are its Bot's private data and the owner's. Frames are never given to a model in this slice: Schway's public tool results are text only, so an agent screenshot tool needs a Schway contract first (not added here). +6. **Protected browser stays separate.** Sensitive-site login keeps using the protected browser takeover (S15 `BrowserOwner`, ADR 0079): its frames and input never pass through the VM screen, and the VM screen never carries a protected-browser session. + +## Consequences + +- With `desktop` on, a Jit-created VM has a virtual GPU and input devices; the guest sees them as ordinary hardware. A text console is what the screen shows until a guest desktop session runs. +- While the owner holds a shared computer, every Bot's work on it pauses (`takeover_active`, which work treats like any refused VM operation). Private computers pause only their Bot. +- Residual: a process the work started earlier inside the guest (for example a detached background job) keeps running during takeover, and guest root can draw on or read its own screen. That is inherent to a root-controlled guest; it is why sensitive login stays in the protected browser. Freezing the work sessions' guest slices during takeover is a possible hardening. + +## Pending (not delivered by this slice) + +- Guest desktop session (X server on the virtual display, window manager, browser) and the desktop browser's egress: the VM proxy admits only per-session leases, so a desktop browser outside a work session needs a reviewed lease design before a trusted-site login can work in the VM. #442's remaining item depends on it. +- Live evidence on `jit-vm` (QEMU `screendump` and input on the real host, VM15-03 file permissions, restart into the desktop profile) belongs to LIVE16-01 (#431). diff --git a/docs/contracts/s15-working-vm-v1.md b/docs/contracts/s15-working-vm-v1.md index fa01fb7c..615ab829 100644 --- a/docs/contracts/s15-working-vm-v1.md +++ b/docs/contracts/s15-working-vm-v1.md @@ -38,7 +38,7 @@ Owner decision 2026-11-03: the default topology is one host running the controll - **Local enrollment.** `VmEnrollment.enrollLocal(owner, {id, managerId, maxSessions?, signal?})` provisions the guest (no pasted bootstrap line), then runs the same inventory, sudo/systemd readiness, `vmIdentityMatches` (controller-chosen MAC and loopback SSH forward) and direct-egress probes as `complete`, and returns the ordinary `VmStatus`. Network isolation is proven only with `network: 'proxy-only'` from the hypervisor plus a refused direct-egress probe; `snapshot-recovery` and `hypervisor-stop` stay `unproven` until an observed protected restore/stop. Since revision 5 the setup job below drives it through `VmEnrollment.provisionLocal`. - **Image.** `localVmImages` pins Ubuntu 26.04 LTS `release-20261016` (`https://cloud-images.ubuntu.com/releases/resolute/release-20261016/…-{amd64,arm64}.img`) with SHA-256 `5645d0318ad21f643ebb1ac4567514620e9893380b07b0201fda163baa5b7eb5` (amd64, not `amd64v3`, because reference hosts may lack AVX2) and `8eb876c2ccc4b6ff89e4a65d511815923b285e04503ba465809bf587f763da24` (arm64), taken from the release's GPG-verified `SHA256SUMS`. Download is HTTPS without redirects or credentials, size-bounded, then hashed; a mismatch is `vm_image_unverified`, unreachable downloads `vm_image_unavailable`. The verified base is mode 0400; each VM uses a qcow2 overlay. - **Keys and seed.** The controller generates an Ed25519 client key and the guest's Ed25519 host key, writes `known_hosts` before first boot and seeds both through NoCloud (`jit` user, key-only with `restrict`, passwordless sudo, no password/root login, sshd forwarding off, no generated extra host keys). The seed also keeps proxy variables through sudo, disables apt pipelining and marks the NIC optional with no DHCP DNS/NTP. A changed host key fails closed (`vm_host_key_changed`). -- **QEMU argv (`qemuArgv`).** KVM, `-cpu host`, `-nodefaults`, `-no-user-config`, no display/monitor, seccomp `-sandbox on,obsolete=deny,resourcecontrol=deny`, a 1 MiB console ring buffer (diagnostics only), a controller-only QMP Unix socket, one virtio disk (overlay, `cache=none`), one read-only virtio seed, and exactly `-netdev user,id=net0,restrict=on,ipv6=off,net=192.168.5.0/24,host=192.168.5.2,hostfwd=tcp:127.0.0.1:-:22,guestfwd=tcp:192.168.5.100:3128-cmd:/usr/bin/nc -U /proxy.sock` with one `virtio-net-pci` NIC. `qemuArgvFacts` parses `/proc//cmdline`; `inspect()` reports `proxy-only` only for a byte-for-byte rebuild of this profile with the persisted MAC and port, otherwise `unproven`; a foreign process behind the unit is `vm_boundary_unproven`. +- **QEMU argv (`qemuArgv`).** KVM, `-cpu host`, `-nodefaults`, `-no-user-config`, no monitor, `-display none` (no VNC, SPICE or window), and no virtual display device unless the owner opted in to DESK16-01 (`desktop: true`: one `virtio-vga` (x86_64) or `virtio-gpu-pci` (aarch64) at 1280×800 with a virtio tablet and keyboard, reached only through the controller's QMP socket and no guest channel, see `s16-computer-screen-v1`; a VM started headless stays headless until its next start), seccomp `-sandbox on,obsolete=deny,resourcecontrol=deny`, a 1 MiB console ring buffer (diagnostics only), a controller-only QMP Unix socket, one virtio disk (overlay, `cache=none`), one read-only virtio seed, and exactly `-netdev user,id=net0,restrict=on,ipv6=off,net=192.168.5.0/24,host=192.168.5.2,hostfwd=tcp:127.0.0.1:-:22,guestfwd=tcp:192.168.5.100:3128-cmd:/usr/bin/nc -U /proxy.sock` with one `virtio-net-pci` NIC. `qemuArgvFacts` parses `/proc//cmdline`; `inspect()` reports `proxy-only` only for a byte-for-byte rebuild of this profile with the persisted MAC and port, otherwise `unproven`; a foreign process behind the unit is `vm_boundary_unproven`. - **Lifecycle.** `systemd-run --user --collect --unit=.service --property=Type=exec --property=TimeoutStopSec=15 -- ` in the service user's lingering user manager. `start()` waits for QMP, the pinned host key and an admitted SSH login. `stop()` sends QMP `quit` (then unit stop/kill) and requires the unit inactive, the QEMU PIDs gone and the overlay write lock released, else `vm_stop_unconfirmed`. Optional `VmManagement.resume?()` and `WorkingVmService.reattach()` re-verify a surviving VM after a controller restart and restart a VM that was running before a host restart (new generation, re-verified before `ready`); stopped or restored VMs stay stopped. - **Snapshots.** Exact `^jit-[a-f0-9]{32}$` qcow2 internal snapshots: live via QMP `blockdev-snapshot-internal-sync` (crash-consistent, siblings keep running) or `qemu-img snapshot -c` when stopped; restore = proven stop plus `qemu-img snapshot -a`, leaving the VM stopped (`vm_snapshot_unknown` for a missing tag); retire deletes only the exact tag. Policy claims, usage, generations and the journal are outside the guest. - **Sizing.** `defaultLocalVmResources`: vCPU = min(⌊CPUs/2⌋, CPUs−2); memory = min(round(0.45 × RAM to whole GiB), ⌊(RAM − 6 GiB)/256 MiB⌋ × 256 MiB); at least 1 vCPU and 2 GiB, otherwise `vm_host_too_small`; disk 60 GiB sparse, refused (`vm_disk_insufficient`) unless free space minus the installer reserve (`localVmDiskReserve`, the `install.sh` rule) holds it. 10 vCPU / 20 GiB gives 5 / 9 GiB; 4 vCPU / 8 GiB gives 2 / 2 GiB. `GET /s15/vm/resources?managerId=` shows current values, defaults and host facts; `POST /s15/vm/resources {managerId,cpus,memoryMiB,diskGiB}` edits them (memory in 256 MiB steps leaving the controller 2 GiB, disks only grow); they apply at the next start. diff --git a/docs/contracts/s16-computer-screen-v1.md b/docs/contracts/s16-computer-screen-v1.md new file mode 100644 index 00000000..a6ebe561 --- /dev/null +++ b/docs/contracts/s16-computer-screen-v1.md @@ -0,0 +1,53 @@ +# s16-computer-screen/v1 + +Owner contract for watching a desktop working computer and taking exclusive control (DESK16-01, [ADR 0090](../adr/0090-s16-watch-and-take-over-a-working-computer.md)). Available when the installation enables `computers` and `desktop`; the bootstrap reports `capabilities.computers.screen: true`. + +## Read the screen state + +`GET /api/s16/client/computers//screen` (owner session, no query) returns: + +```json +{ "contract": "s16-computer-screen/v1", "vmId": "vm", "available": true, "code": null, "width": 1280, "height": 800, + "state": "agent", "generation": 0, "takeoverId": null, "expiresAt": null, "agentBusy": false, "scope": { "kind": "shared" } } +``` + +- `state`: `agent` (work may use the computer), `settling` (takeover requested; waiting for earlier work operations, `agentBusy`), `owner` (only owner input), `fenced` (control lost; nobody writes). +- `generation` advances on takeover, lease lapse, controller restart and hand-back. +- `available`/`code`: `vm_desktop_unavailable`, `vm_not_running` or `vm_desktop_restart_required` when no frame can be read. +- `scope`: lineage of every frame (a private computer's frames are its Bot's private data). +- Reading changes nothing, except that an owner lease that already lapsed is reported (and recorded) as `fenced`. + +## Read one frame + +`GET /api/s16/client/computers//screen/frame?generation=` returns `image/png` (1280×800) with `Cache-Control: private, no-store`. Any other generation, or a transition during the capture, is `409 stale_screen`. The only query parameter is `generation`. + +## Commands + +`POST /api/s16/client/computers/screen/commands` (JSON, CSRF header, body ≤ 4 KiB), exact keys per kind: + +| kind | fields | effect | +| --- | --- | --- | +| `takeover` | `generation` | from `agent` or `fenced`: new generation and takeover id; `owner` at once, or `settling` while work operations admitted earlier are still running | +| `renew` | `generation`, `takeoverId` | extends owner control by the lease (30 s); not journaled | +| `input` | `generation`, `takeoverId`, `input` | one input in `owner` only (`takeover_settling` while settling) | +| `hand_back` | `generation`, `takeoverId` (null when fenced) | waits for the input being sent, then `agent` under a new generation | + +Every command also has `contract: "s16-screen-command/v1"`, `requestId` and `vmId`. The result is `{ "contract": "s16-screen-command-result/v1", "requestId", "view" }`. + +`input` is one of: + +- `{ "kind": "click", "x", "y", "button": "left" | "right" | "middle", "double": boolean }` +- `{ "kind": "move", "x", "y" }` +- `{ "kind": "scroll", "x", "y", "delta": -10..10, not 0 }` +- `{ "kind": "key", "keys": [1..4 distinct names] }`: `ctrl shift alt meta ret esc tab backspace delete insert home end pgup pgdn up down left right spc f1..f12` or one lowercase letter or digit +- `{ "kind": "text", "text": 1..512 characters a US keyboard types }` (`screen_text_unsupported` otherwise) + +Coordinates are display pixels (`0 ≤ x < 1280`, `0 ≤ y < 800`). + +## Idempotency and privacy + +`takeover` and `hand_back` are idempotent per `requestId` (same body returns the current view; a different body is `request_id_conflict`). `input` is sent at most once per `requestId`: a retry joins a running send, returns the view after a finished one, and is refused (`screen_input_unknown`) after a send that failed once events may have reached the computer; only a refusal before sending (for example `takeover_settling`) releases the id. Ids are kept in controller memory for the current owner generation (a retry under an older generation is refused as stale or changed); input commands are never journaled, so no typed text or hash of it is stored. Frames are never stored: each QMP `screendump` file is removed before its capture settles (under VM15-03 by the launcher's `discard-frame `, which removes only `/screen.png`), a frame that cannot be removed fails the capture, and `stop()` and the launcher's `start` remove any frame left behind. + +## Errors + +`screen_input_unknown`, `takeover_active` (work refused while the owner holds or lost control), `takeover_state_changed`, `takeover_settling`, `stale_screen`, `screen_control_lost`, `invalid_screen_input`, `screen_text_unsupported`, `vm_desktop_unavailable`, `vm_desktop_restart_required`, `vm_not_running`, `vm_screen_invalid`, `request_id_conflict`, `screen_unavailable` (not enabled). diff --git a/src/i18n/en.ts b/src/i18n/en.ts index 08a6c796..8fdb6d70 100644 --- a/src/i18n/en.ts +++ b/src/i18n/en.ts @@ -6162,6 +6162,34 @@ export const en: Record = { "s16.computers.unavailable": "Computers are not available on this installation.", "s16.computers.loadFailed": "Could not load computers ({reason}).", "s16.computers.empty": "No computer yet. Jit sets one up during installation.", + "s16.screen.watch": "Watch screen", + "s16.screen.title": "Screen of {computer}", + "s16.screen.alt": "Live screen of {computer}", + "s16.screen.state.agent": "Bot can use it", + "s16.screen.state.settling": "Pausing the Bot", + "s16.screen.state.owner": "You are in control", + "s16.screen.state.fenced": "Paused", + "s16.screen.takeOver": "Take over", + "s16.screen.takeBack": "Take control again", + "s16.screen.handBack": "Hand back to the Bot", + "s16.screen.cancel": "Cancel", + "s16.screen.close": "Close", + "s16.screen.settling": "Waiting for the current step of the Bot to finish. Nothing new starts on this computer.", + "s16.screen.driving": "The Bot starts nothing new on this computer until you hand it back. Programs already on it can still see the screen and what you type, so do not type passwords for sensitive sites here; use the protected owner view for those.", + "s16.screen.fenced": "Control was lost (the page closed or the connection dropped). Nobody can use this computer until you take control again or hand it back.", + "s16.screen.unavailable": "This computer has no screen to show ({reason}).", + "s16.screen.failed": "That did not work ({reason}).", + "s16.screen.private": "This is a private computer. Its screen is private to its Bot and to you.", + "s16.screen.waitingFrame": "Waiting for the screen…", + "s16.screen.keyboard": "Computer screen. Keys you press here go to the computer; Tab leaves it.", + "s16.screen.typeLabel": "Type on the computer", + "s16.screen.show": "Show", + "s16.screen.send": "Type it", + "s16.screen.typeHint": "Typed text goes straight to the computer and is not saved. Only characters on a US keyboard can be typed.", + "s16.screen.key.ret": "Enter", + "s16.screen.key.tab": "Tab", + "s16.screen.key.esc": "Esc", + "s16.screen.key.backspace": "Backspace", "s16.nav.allBots": "All Bots", "s16.bots.title": "Bots", "s16.bots.lead": "Each Bot has its own responsibility, conversations and memory. A new Bot uses the computer, connected apps and rules of the team, so there is nothing else to set up.", diff --git a/src/i18n/th.ts b/src/i18n/th.ts index 5650a0c7..a0660cf8 100644 --- a/src/i18n/th.ts +++ b/src/i18n/th.ts @@ -6166,6 +6166,34 @@ export const th = { "s16.computers.unavailable": "การติดตั้งนี้ยังไม่มีหน้าคอมพิวเตอร์", "s16.computers.loadFailed": "โหลดรายการคอมพิวเตอร์ไม่สำเร็จ ({reason})", "s16.computers.empty": "ยังไม่มีคอมพิวเตอร์ จิตจะตั้งค่าให้ตอนติดตั้ง", + "s16.screen.watch": "ดูหน้าจอ", + "s16.screen.title": "หน้าจอของ{computer}", + "s16.screen.alt": "หน้าจอสดของ{computer}", + "s16.screen.state.agent": "Bot ใช้งานได้", + "s16.screen.state.settling": "กำลังพัก Bot", + "s16.screen.state.owner": "คุณควบคุมอยู่", + "s16.screen.state.fenced": "หยุดชั่วคราว", + "s16.screen.takeOver": "ควบคุมเอง", + "s16.screen.takeBack": "ควบคุมอีกครั้ง", + "s16.screen.handBack": "คืนให้ Bot", + "s16.screen.cancel": "ยกเลิก", + "s16.screen.close": "ปิด", + "s16.screen.settling": "กำลังรอให้ขั้นตอนปัจจุบันของ Bot เสร็จ งานใหม่จะยังไม่เริ่มบนเครื่องนี้", + "s16.screen.driving": "Bot จะไม่เริ่มงานใหม่บนเครื่องนี้จนกว่าคุณจะคืนให้ แต่โปรแกรมที่อยู่ในเครื่องยังเห็นหน้าจอและสิ่งที่คุณพิมพ์ได้ อย่าพิมพ์รหัสผ่านของเว็บที่สำคัญที่นี่ ให้ใช้ protected owner view แทน", + "s16.screen.fenced": "การควบคุมหลุด (ปิดหน้าหรือการเชื่อมต่อขาด) จะไม่มีใครใช้เครื่องนี้ได้จนกว่าคุณจะควบคุมอีกครั้งหรือคืนให้ Bot", + "s16.screen.unavailable": "เครื่องนี้ไม่มีหน้าจอให้แสดง ({reason})", + "s16.screen.failed": "ทำไม่สำเร็จ ({reason})", + "s16.screen.private": "นี่คือเครื่องส่วนตัว หน้าจอนี้เป็นข้อมูลส่วนตัวของ Bot เจ้าของเครื่องและคุณเท่านั้น", + "s16.screen.waitingFrame": "กำลังรอหน้าจอ…", + "s16.screen.keyboard": "หน้าจอของเครื่อง ปุ่มที่กดตรงนี้จะส่งไปที่เครื่อง กด Tab เพื่อออก", + "s16.screen.typeLabel": "พิมพ์บนเครื่อง", + "s16.screen.show": "แสดง", + "s16.screen.send": "พิมพ์", + "s16.screen.typeHint": "ข้อความที่พิมพ์จะส่งตรงไปที่เครื่องและไม่ถูกบันทึก พิมพ์ได้เฉพาะตัวอักษรบนแป้นพิมพ์ภาษาอังกฤษ (US)", + "s16.screen.key.ret": "Enter", + "s16.screen.key.tab": "Tab", + "s16.screen.key.esc": "Esc", + "s16.screen.key.backspace": "Backspace", "s16.nav.allBots": "Bot ทั้งหมด", "s16.bots.title": "Bot", "s16.bots.lead": "Bot แต่ละตัวมีหน้าที่ บทสนทนา และความจำของตัวเอง Bot ใหม่ใช้คอมพิวเตอร์ แอปที่เชื่อมไว้ และกฎเดียวกับทีม จึงไม่ต้องตั้งค่าอะไรเพิ่ม", diff --git a/src/s15/composition.ts b/src/s15/composition.ts index e62abf56..de97fac3 100644 --- a/src/s15/composition.ts +++ b/src/s15/composition.ts @@ -2,7 +2,9 @@ import { fsyncSync, lstatSync, mkdirSync, openSync, closeSync, readFileSync, ren import { isAbsolute, join, resolve } from 'node:path'; import { createS16ClientHandler } from '../s16/http'; import { ComputerScopes } from '../s16/computers/scopes'; +import { ScreenControl } from '../s16/computers/screen'; import { ComputerDirectory } from '../s16/computers/service'; +import { isVmScreenManagement, type VmScreenDevice } from './vm/screen'; import { BotDirectory } from '../s16/bots/service'; import { createBotTools } from '../s16/bots/delegation'; import { ConnectionDirectory } from '../s16/connections/service'; @@ -78,6 +80,10 @@ export interface S15CompositionOptions { * computers for a Bot, and idle release after `idleStopMs` (on-demand start). Opt-in; it changes no installed policy. * `sweepMs` is the idle-check period (0: only explicit `computers.sweep()` calls, for deterministic tests). */ computers?: { idleStopMs: number; sweepMs?: number }; + /** DESK16-01 watch and take over a desktop working computer (`s16-computer-screen/v1`). Requires `computers`; + * the local manager must also start VMs with the desktop profile (`vm.local.desktop`). `leaseMs` is how long + * owner control lasts without a renewal (default 30 s). Opt-in; it changes no installed policy. */ + screen?: { leaseMs?: number }; /** Explicit S16 persistent Bots and delegation (`s16-bots/v1`, BOT16-01/DELEG16-01, ADR 0088). Opt-in; it adds * `bots.sqlite`. A new Bot joins the standing policy only when the owner accepts its exact proposal. */ bots?: Record; @@ -349,7 +355,17 @@ export async function composeS15(options: S15CompositionOptions) { if (localVm) cleanup.push(() => localVm.close()); const managers = options.vm.managementVault ? new VmManagers({ store: vmStore, vault: options.vm.managementVault, requireOwner, ...(localVm ? { local: localVm.host } : {}) }) : new Map(options.vm.managers); for (const [id, manager] of options.vm.managers) managers.set(id, manager); - const vm = new WorkingVmService({ store: vmStore, policy: vmIntegration.bind(policy), managers, requireOwner, ...(disclosure ? { disclosure } : {}), + if (options.screen && !scopes) throw new AppError('invalid_s15_config'); + const screenDevices = new WeakMap(); + const screen = options.screen && scopes ? new ScreenControl({ store: vmStore, requireOwner, scope: id => scopes.scope(id), + ...(options.screen.leaseMs ? { leaseMs: options.screen.leaseMs } : {}), ...(options.clock ? { now: options.clock.now } : {}), + device(id) { + const row = vmStore.registration(id), manager = row.managerId ? managers.get(row.managerId) : undefined; + if (!isVmScreenManagement(manager)) return null; + let device = screenDevices.get(manager); if (!device) { device = manager.screen(); screenDevices.set(manager, device); } + return device; + } }) : null; + const vm = new WorkingVmService({ store: vmStore, policy: vmIntegration.bind(policy), managers, requireOwner, ...(disclosure ? { disclosure } : {}), ...(screen ? { screen } : {}), ...(options.vm.transport ? { transport: options.vm.transport } : {}), ...(options.vm.limits ? { limits: options.vm.limits } : {}), ...(options.clock ? { now: options.clock.now } : {}), ...(localVm ? { proxyLease: localVm.lease } : {}), ...(options.computers ? { idle: { stopAfterMs: options.computers.idleStopMs } } : {}) }); if (localVm) { @@ -715,7 +731,7 @@ export async function composeS15(options: S15CompositionOptions) { return (['owner_named_send', 'github_default'] as const).some(mode => proposed.rules.some(r => r.mode === mode) && !current.rules.some(r => r.mode === mode)); } catch { return false; } }; - const clientHandler = createS16ClientHandler(conversation, { origin: options.origin, cookieName, authentication, memory, policyUpdate, ...(routines ? { routines, wake: () => { void composed.drain().catch(() => {}); } } : {}), ...(notifications ? { notifications } : {}), ...(computers ? { computers } : {}), ...(bots ? { bots } : {}), ...(connections ? { connections } : {}), ...(files ? { files, imageAvailable: () => { const route = options.model.registry.current(options.conversation.selection); return !!route && route.availability === "available" && !!options.files?.imageProbes?.some(probe => probe.verdict === "pass" && probe.route === route.provider && probe.model === route.modelId); } } : {}) }); + const clientHandler = createS16ClientHandler(conversation, { origin: options.origin, cookieName, authentication, memory, policyUpdate, ...(routines ? { routines, wake: () => { void composed.drain().catch(() => {}); } } : {}), ...(notifications ? { notifications } : {}), ...(computers ? { computers } : {}), ...(screen ? { screen } : {}), ...(bots ? { bots } : {}), ...(connections ? { connections } : {}), ...(files ? { files, imageAvailable: () => { const route = options.model.registry.current(options.conversation.selection); return !!route && route.availability === "available" && !!options.files?.imageProbes?.some(probe => probe.verdict === "pass" && probe.route === route.provider && probe.model === route.modelId); } } : {}) }); const s16Web = await createS16WebHandler(clientHandler); const vmHandler = createVmApi(enrollment, async request => authenticate(request), { service: vm, ...(managers instanceof VmManagers ? { managers } : {}) }); const modelHandler = createModelProjectionHandler(models, authenticate), workHandler = createResponsibilityProjectionHandler(work, authenticate); @@ -733,7 +749,7 @@ export async function composeS15(options: S15CompositionOptions) { const dir = openSync(root, 'r'); try { fsyncSync(dir); } finally { closeSync(dir); } added = []; } - const composed = { policy, conversation, team, memory, files, bots, connections, sources: sourceRuntime, disclosure, adoptionStatus, githubConnections, vm, workingVm, vmStore, enrollment, vmManagers: managers, vmPool: pool, provisioning, computers, computerScopes: scopes, models, work, github, browser, authentication, resumes, routines, + const composed = { policy, conversation, team, memory, files, bots, connections, sources: sourceRuntime, disclosure, adoptionStatus, githubConnections, vm, workingVm, vmStore, enrollment, vmManagers: managers, vmPool: pool, provisioning, computers, computerScopes: scopes, screen, models, work, github, browser, authentication, resumes, routines, /** One protected scheduler wake; waiting records alone hold no VM/native session. Stall-reply resumes * settle after CHAT. */ drain(): Promise { if (closing) return Promise.resolve(); return draining ??= (async () => { try { reconcileVm(); try { routines?.tick(); routines?.dispatch(); } catch { /* Committed occurrences retry next wake. */ } await conversation.drain(); try { routines?.observe(); } catch { /* Observed next wake. */ } try { resumes.settle(); } catch { /* Pending bindings stay pending. */ } await github?.drain(); } finally { draining = null; } })(); }, diff --git a/src/s15/main.ts b/src/s15/main.ts index 6bcf241e..663d2b2e 100644 --- a/src/s15/main.ts +++ b/src/s15/main.ts @@ -124,6 +124,8 @@ export async function runS15(args: string[], env: NodeJS.ProcessEnv = process.en ...(config.notifications ? { notifications: {} } : {}), // TEAM16-01: owner decision 2026-11-07, a working computer with no open session sleeps after 30 minutes. ...(config.computers ? { computers: { idleStopMs: s16ComputerIdleStopMs } } : {}), + // DESK16-01: watch and take over a desktop working computer. + ...(config.computers && config.desktop ? { screen: {} } : {}), // BOT16-01/DELEG16-01: persistent Bots the owner accepts and bounded Bot-to-Bot assignments. ...(config.bots ? { bots: {} } : {}), // CONN16-01/MCP16-01/OPEN16-01: apps and tool sources the owner connects; secrets stay in the protected vault. @@ -147,7 +149,7 @@ export async function runS15(args: string[], env: NodeJS.ProcessEnv = process.en ...(config.github ? [{ id: 'github', kind: 'service' as const, label: 'GitHub', state: 'configured' as const, uses: ['github' as const], signInHref: null }] : []), ...(config.google ? [{ id: 'google', kind: 'service' as const, label: 'Google', state: account(config.google.mailConnectionId) || account(config.google.calendarConnectionId) ? 'configured' as const : 'not_connected' as const, uses: ['email_read' as const, 'email_draft' as const, 'calendar' as const], signInHref: '/settings/google' }] : []), ]; } } }, - vm: { id: vmPool.poolId, resource: config.vm.resource, managers, managementVault: vault, upperBound: () => config.vm.upperBound, destination: host => config.disclosure?.acquisitions.find(r => new URL(r.origin).hostname === host)?.recipient ?? null, ...(vmPool.local ? { local: config.vm.vmm ? { vmm: config.vm.vmm, ...(proxySocketFd !== null ? { proxySocketFd } : {}) } : {} } : {}) }, + vm: { id: vmPool.poolId, resource: config.vm.resource, managers, managementVault: vault, upperBound: () => config.vm.upperBound, destination: host => config.disclosure?.acquisitions.find(r => new URL(r.origin).hostname === host)?.recipient ?? null, ...(vmPool.local ? { local: { ...(config.vm.vmm ? { vmm: config.vm.vmm, ...(proxySocketFd !== null ? { proxySocketFd } : {}) } : {}), ...(config.computers && config.desktop ? { desktop: true } : {}) } } : {}) }, ...(config.browser ? { browser: { mode: 'static', resource: config.browser.resource, fences: [], executablePath: config.browser.executablePath, site, upperBound: config.browser.upperBound, ...(config.browser.ownerUi ? { ownerUi: config.browser.ownerUi } : {}) } } : provisioned ? { browser: provisioned.options() } : {}), ...(config.google ? { google: { runtime: 's15', google: new GooglePersonal(), mailAccount: () => account(config.google!.mailConnectionId), calendarAccount: () => account(config.google!.calendarConnectionId), async token(binding, scope) { const current = ownerStore.read(tx => s15GoogleConnection(config.google!.resources, binding, scope, id => ownerApp.personal.connection(tx, id))); diff --git a/src/s15/runtime-config.ts b/src/s15/runtime-config.ts index 7ce0e27d..5a02b0b9 100644 --- a/src/s15/runtime-config.ts +++ b/src/s15/runtime-config.ts @@ -32,6 +32,10 @@ export interface S15RuntimeConfig { /** Explicit S16 working computers (`s16-computers/v1`, TEAM16-01): per-work leases, private computers and 30-minute * idle release with on-demand start. Omission keeps S15 working-computer behavior. */ computers?: boolean; + /** Explicit S16 desktop computers (`s16-computer-screen/v1`, DESK16-01): Jit-created VMs start with one virtual display, + * tablet and keyboard reached only through QMP, and the owner can watch the screen and take exclusive control. + * Requires `computers`. A VM already running headless keeps running until its next start. Omission changes nothing. */ + desktop?: boolean; /** Explicit S16 persistent Bots and delegation (`s16-bots/v1`, BOT16-01/DELEG16-01). Omission keeps the single main Bot. */ bots?: boolean; /** Explicit S16 connections (`s16-connections/v1`, CONN16-01/MCP16-01/OPEN16-01): GitHub and HTTPS MCP/OpenAPI/GraphQL @@ -81,6 +85,7 @@ export function parseS15RuntimeConfig(raw: unknown): S15RuntimeConfig { if (value.routines !== undefined && typeof value.routines !== 'boolean') throw new AppError('invalid_s15_config'); if (value.notifications !== undefined && typeof value.notifications !== 'boolean') throw new AppError('invalid_s15_config'); if (value.computers !== undefined && typeof value.computers !== 'boolean') throw new AppError('invalid_s15_config'); + if (value.desktop !== undefined && (typeof value.desktop !== 'boolean' || value.desktop && value.computers !== true)) throw new AppError('invalid_s15_config'); if (value.bots !== undefined && typeof value.bots !== 'boolean') throw new AppError('invalid_s15_config'); if (value.connections !== undefined && (typeof value.connections !== 'boolean' || value.connections && !value.disclosure)) throw new AppError('invalid_s15_config'); if (value.policy && (!value.disclosure || value.policy.resources.some(r => !value.disclosure!.sources.some(s => equal(s.resource, r))))) throw new AppError('classification_unknown'); diff --git a/src/s15/vm/qemu.ts b/src/s15/vm/qemu.ts index 73b4969f..6e277488 100644 --- a/src/s15/vm/qemu.ts +++ b/src/s15/vm/qemu.ts @@ -7,6 +7,7 @@ import { vmHostCommand } from "./host"; import { detectLocalVmHost, ensureLocalVmImage, localVmDownload, localVmHostProbe, localVmImages, localVmTools, type LocalVmDownload, type LocalVmHostProbe, type LocalVmImage, type LocalVmResources } from "./local"; import type { VmProxyLease } from "./proxy"; import { qmpExchange, type QmpCommand, type QmpExchange } from "./qmp"; +import { vmScreenCommands, vmScreenFrame, vmScreenHeight, vmScreenWidth, type VmScreenDevice, type VmScreenInput } from "./screen"; import { VmSshTransport } from "./transport"; import type { VmEndpoint, VmHostCommand, VmMachineState, VmManagement } from "./types"; import { vmFail, vmInteger } from "./validation"; @@ -25,6 +26,12 @@ export const qemuGuestNetwork = Object.freeze({ net: "192.168.5.0/24", host: "19 export interface QemuMachineSpec { name: string; architecture: "x86_64" | "aarch64"; machineDirectory: string; cpus: number; memoryMiB: number; mac: string; sshPort: number; proxySocket: string; + /** + * DESK16-01 desktop profile: one virtio display at the fixed screen size plus a virtio tablet and keyboard. + * Frames and input travel only through the controller's QMP socket; `-display none` keeps every display + * server (VNC, SPICE, GTK) absent. Omitted, the argv is byte-for-byte the headless profile. + */ + desktop?: boolean; } /** * The complete QEMU argv for a working VM: KVM, one virtio disk (per-VM qcow2 overlay), one read-only @@ -44,10 +51,13 @@ export function qemuArgv(spec: QemuMachineSpec): string[] { "-drive", `if=none,id=disk0,file=${d}/disk.qcow2,format=qcow2,cache=none,discard=unmap`, "-device", "virtio-blk-pci,drive=disk0,bootindex=0", "-drive", `if=none,id=seed,file=${d}/seed.img,format=raw,readonly=on`, "-device", "virtio-blk-pci,drive=seed", "-netdev", `user,id=net0,restrict=on,ipv6=off,net=${qemuGuestNetwork.net},host=${qemuGuestNetwork.host},hostfwd=tcp:127.0.0.1:${spec.sshPort}-:22,guestfwd=tcp:${qemuGuestNetwork.proxy}:${qemuGuestNetwork.proxyPort}-cmd:${localVmTools.nc} -U ${spec.proxySocket}`, - "-device", `virtio-net-pci,netdev=net0,mac=${spec.mac}`]; + "-device", `virtio-net-pci,netdev=net0,mac=${spec.mac}`, + ...(spec.desktop ? ["-device", `${spec.architecture === "x86_64" ? "virtio-vga" : "virtio-gpu-pci"},xres=${vmScreenWidth},yres=${vmScreenHeight}`, "-device", "virtio-tablet-pci", "-device", "virtio-keyboard-pci"] : [])]; } /** Hypervisor facts read from an actual argv. `exact` is true only when it is byte-for-byte the restricted profile. */ -export interface QemuArgvFacts { cpus?: number; memoryMiB?: number; mac?: string; sshPort?: number; exact: boolean } +export interface QemuArgvFacts { cpus?: number; memoryMiB?: number; mac?: string; sshPort?: number; exact: boolean; + /** True only when `exact` holds for the desktop profile; a headless VM keeps working and shows no screen. */ + desktop?: boolean } /** Parses a running QEMU argv (from `/proc//cmdline`) against the expected protected identity. */ export function qemuArgvFacts(argv: readonly string[], expected: Pick): QemuArgvFacts { const value = (flag: string) => argv.filter(a => a === flag).length === 1 ? argv[argv.indexOf(flag) + 1] : undefined; @@ -59,8 +69,11 @@ export function qemuArgvFacts(argv: readonly string[], expected: Pick a === argv[i]); + // Either fixed profile is exact: a VM started before the desktop profile was enabled stays verifiable. + for (const desktop of [false, true]) { + const rebuilt = qemuArgv({ ...expected, cpus: facts.cpus, memoryMiB: facts.memoryMiB, mac: facts.mac, sshPort: facts.sshPort, desktop }); + if (rebuilt.length === argv.length && rebuilt.every((a, i) => a === argv[i])) { facts.exact = true; if (desktop) facts.desktop = true; } + } } return facts; } @@ -92,6 +105,11 @@ export interface QemuHostOptions { * into the shared, controller-owned image directory. Absent, QEMU runs as the controller identity. */ vmm?: QemuVmmOptions; + /** + * DESK16-01: start this controller's VMs with the desktop profile (explicit S16 opt-in). A VM already running + * headless keeps running and reports `vm_desktop_restart_required` until its next start. + */ + desktop?: boolean; } /** Credential-free persisted identity of one Jit-created VM; owner resource edits change only `resources`. */ export interface QemuVmIdentity { name: string; architecture: "x86_64" | "aarch64"; mac: string; sshPort: number; resources: LocalVmResources } @@ -160,7 +178,7 @@ export class QemuVmManagement implements VmLocalManagement { private get stateFile() { return join(this.machineDirectory, "state.json"); } private get image(): LocalVmImage { return (this.host.images ?? localVmImages)[this.identity.architecture]; } private spec(resources: LocalVmResources = this.#resources): QemuMachineSpec { - return { name: this.identity.name, architecture: this.identity.architecture, machineDirectory: this.hypervisorDirectory, cpus: resources.cpus, memoryMiB: resources.memoryMiB, mac: this.identity.mac, sshPort: this.identity.sshPort, proxySocket: this.host.proxySocket }; + return { name: this.identity.name, architecture: this.identity.architecture, machineDirectory: this.hypervisorDirectory, cpus: resources.cpus, memoryMiB: resources.memoryMiB, mac: this.identity.mac, sshPort: this.identity.sshPort, proxySocket: this.host.proxySocket, ...(this.host.desktop ? { desktop: true } : {}) }; } /** Resource values for the next start. Editing them never changes a running guest. */ get resources(): LocalVmResources { return { ...this.#resources }; } @@ -191,14 +209,14 @@ export class QemuVmManagement implements VmLocalManagement { * command in the controller's own user manager; with one it is the fixed `sudo -n -u -- ` * operation, which rebuilds the same command (and the same QEMU argv) as that identity. */ - private hypervisor(operation: ["show" | "stop" | "kill"] | ["info", boolean] | ["resize", number] | ["create", string, number] | ["snapshot", "create" | "apply" | "delete", string] | ["start", LocalVmResources]): string[] { + private hypervisor(operation: ["show" | "stop" | "kill" | "discard-frame"] | ["info", boolean] | ["resize", number] | ["create", string, number] | ["snapshot", "create" | "apply" | "delete", string] | ["start", LocalVmResources]): string[] { const vmm = this.host.vmm, name = this.identity.name, [op] = operation; if (vmm) { if (op === "info") return vmmCommand(vmm, ["info", name, operation[1] ? "shared" : "exclusive"]); if (op === "resize") return vmmCommand(vmm, ["resize", name, String(operation[1])]); if (op === "create") { const sha = /\/([a-f0-9]{64})\.qcow2$/.exec(operation[1])?.[1]; if (!sha || operation[1] !== join(vmmLayout(vmm.directory).images, `${sha}.qcow2`)) vmFail("vm_image_unverified"); return vmmCommand(vmm, ["create", name, sha, String(operation[2])]); } if (op === "snapshot") return vmmCommand(vmm, ["snapshot", name, operation[1], operation[2]]); - if (op === "start") { const r = operation[1]; return vmmCommand(vmm, ["start", name, this.identity.architecture, String(r.cpus), String(r.memoryMiB), this.identity.mac, String(this.identity.sshPort)]); } + if (op === "start") { const r = operation[1]; return vmmCommand(vmm, ["start", name, this.identity.architecture, String(r.cpus), String(r.memoryMiB), this.identity.mac, String(this.identity.sshPort), ...(this.host.desktop ? ["desktop"] : [])]); } return vmmCommand(vmm, [op, name]); } if (op === "show") return ["systemctl", "--user", "show", this.unit, "--property=LoadState", "--property=ActiveState", "--property=MainPID"]; @@ -252,6 +270,42 @@ export class QemuVmManagement implements VmLocalManagement { return { state: "running", identity, boot: `${pid}:${this.#proc.startTicks(pid)}`, cpuCount: facts.cpus ?? 0, memoryBytes: (facts.memoryMiB ?? 0) * 1_048_576, diskBytes, network: exact ? "proxy-only" : "unproven", ...(facts.mac ? { nicMac: facts.mac } : {}), ...(facts.sshPort ? { sshEndpoint: { host: "127.0.0.1", port: facts.sshPort } } : {}) }; } + /** + * DESK16-01 host-side screen of this exact VM: QMP `screendump` (PNG) and `input-send-event` on the + * controller-only socket. Only a running QEMU whose argv is exactly the desktop profile has a screen; frames + * are written into this VM's hypervisor directory, read once and always removed before the capture settles + * (`discardFrame`, also under VM15-03). A frame that cannot be removed fails the capture, so no frame outlives + * it. One capture runs at a time. + */ + screen(): VmScreenDevice { + const self = this; let capturing: Promise = Promise.resolve(); + const running = async (): Promise => { + if (!this.host.desktop) return "vm_desktop_unavailable"; + const unit = await this.unitState(), argv = unit.active ? this.ours(unit.pid) : null; + if (!argv) return "vm_not_running"; + return qemuArgvFacts(argv, this.spec()).desktop ? null : "vm_desktop_restart_required"; + }; + return { + unavailable: running, + async capture() { + const turn = capturing.then(async () => { + const code = await running(); if (code) vmFail(code, 409); + const file = join(self.hypervisorDirectory, "screen.png"); + try { + await self.qmp([{ execute: "screendump", arguments: { filename: file, format: "png" } }], 10_000); + try { return vmScreenFrame(new Uint8Array(readFileSync(file))); } + catch (error) { if (error instanceof AppError) throw error; return vmFail("vm_screen_invalid", 502); } + } finally { await self.discardFrame(); } + }); + capturing = turn.catch(() => {}); + return turn; + }, + async send(input: VmScreenInput) { + const code = await running(); if (code) vmFail(code, 409); + await self.qmp(vmScreenCommands(input), 10_000); + }, + }; + } private async until(test: () => Promise, milliseconds: number, interval: number, signal?: AbortSignal): Promise { const deadline = this.#now() + milliseconds; for (;;) { @@ -302,8 +356,24 @@ export class QemuVmManagement implements VmLocalManagement { const transport = new VmSshTransport(this.endpoint(), this.#run); if ((await this.guest(transport, ["true"], { timeoutMs: 30_000, ...(signal ? { signal } : {}) })) !== "") vmFail("vm_protocol_invalid"); } - /** Hard stop of this exact VM, then independent read-back: unit inactive, QEMU PID gone, image lock released. */ + /** + * DESK16-01: removes this VM's last screen frame. Under VM15-03 the file belongs to the QEMU identity in a + * directory the controller cannot write, so that identity's launcher removes exactly `/screen.png`. + */ + private async discardFrame(): Promise { + if (this.host.vmm) await this.command(this.hypervisor(["discard-frame"]), 15_000, 4096); + else rmSync(join(this.hypervisorDirectory, "screen.png"), { force: true }); + } + /** + * Hard stop of this exact VM, then independent read-back: unit inactive, QEMU PID gone, image lock released. + * A stopped VM keeps no screen frame (DESK16-01), including after an idle stop. Only a desktop controller can + * capture frames, so a headless one never needs the `discard-frame` launcher op (an older launcher keeps working). + */ async stop(): Promise { + await this.halt(); + if (this.host.desktop) await this.discardFrame(); + } + private async halt(): Promise { const before = await this.unitState(), recorded = this.state(); this.saveState({ desired: "stopped" }); const pids = [before.pid, recorded.pid ?? 0].filter(pid => pid > 0); diff --git a/src/s15/vm/screen.ts b/src/s15/vm/screen.ts new file mode 100644 index 00000000..675220ce --- /dev/null +++ b/src/s15/vm/screen.ts @@ -0,0 +1,115 @@ +import type { QmpCommand } from "./qmp"; +import { vmFail } from "./validation"; + +/** Fixed framebuffer of a desktop working VM (`s16-computer-screen/v1`). Owner coordinates are display pixels. */ +export const vmScreenWidth = 1280, vmScreenHeight = 800; +/** Most characters of one typed text. */ +export const vmScreenMaxText = 512; +/** Most keys of one key combination. */ +export const vmScreenMaxKeys = 4; +/** Largest accepted PNG frame. A 1280×800 desktop is far smaller; anything larger is refused, never truncated. */ +export const vmScreenMaxFrameBytes = 8 * 1_048_576; + +/** + * One owner input to the working VM's virtual display. Text is limited to characters a US keyboard types + * (`screen_text_unsupported` otherwise); the value is sent once and never stored, logged or hashed. + */ +export type VmScreenInput = + | { kind: "click"; x: number; y: number; button: "left" | "right" | "middle"; double: boolean } + | { kind: "move"; x: number; y: number } + | { kind: "scroll"; x: number; y: number; delta: number } + | { kind: "key"; keys: string[] } + | { kind: "text"; text: string }; + +/** Structural guard: a manager with a host-side screen (the local QEMU manager). */ +export function isVmScreenManagement(manager: unknown): manager is { screen(): VmScreenDevice } { + return !!manager && typeof (manager as { screen?: unknown }).screen === "function"; +} +/** Host-side screen of one VM. Frames and input go through the controller-only QMP socket, never the guest network. */ +export interface VmScreenDevice { + /** Null when this VM can show a desktop now, else the stable reason (`vm_desktop_unavailable`, `vm_not_running`). */ + unavailable(): Promise; + /** One PNG frame of the current display. */ + capture(signal?: AbortSignal): Promise; + /** Sends one input; resolves after QEMU accepted every event. */ + send(input: VmScreenInput): Promise; +} + +const named = Object.freeze(["ctrl", "shift", "alt", "meta", "ret", "esc", "tab", "backspace", "delete", "insert", "home", "end", "pgup", "pgdn", + "up", "down", "left", "right", "spc", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "f10", "f11", "f12"] as const); +/** Accepted key names: the named keys above plus one lowercase letter or digit. They are QEMU qcodes as is. */ +export function vmScreenKey(key: unknown): key is string { + return typeof key === "string" && ((named as readonly string[]).includes(key) || /^[a-z0-9]$/.test(key)); +} +const plain: Record = { " ": "spc", "\n": "ret", "\t": "tab", "-": "minus", "=": "equal", "[": "bracket_left", "]": "bracket_right", + "\\": "backslash", ";": "semicolon", "'": "apostrophe", "`": "grave_accent", ",": "comma", ".": "dot", "/": "slash" }; +const shifted: Record = { "!": "1", "@": "2", "#": "3", "$": "4", "%": "5", "^": "6", "&": "7", "*": "8", "(": "9", ")": "0", + "_": "minus", "+": "equal", "{": "bracket_left", "}": "bracket_right", "|": "backslash", ":": "semicolon", "\"": "apostrophe", "~": "grave_accent", + "<": "comma", ">": "dot", "?": "slash" }; +/** The qcode and shift state for one character a US keyboard types, or null. */ +export function vmScreenChar(char: string): { key: string; shift: boolean } | null { + if (/^[a-z0-9]$/.test(char)) return { key: char, shift: false }; + if (/^[A-Z]$/.test(char)) return { key: char.toLowerCase(), shift: true }; + if (plain[char]) return { key: plain[char]!, shift: false }; + if (shifted[char]) return { key: shifted[char]!, shift: true }; + return null; +} + +const point = (x: unknown, y: unknown) => Number.isSafeInteger(x) && Number.isSafeInteger(y) && (x as number) >= 0 && (x as number) < vmScreenWidth && (y as number) >= 0 && (y as number) < vmScreenHeight; +const exactKeys = (value: object, keys: string[]) => Object.keys(value).sort().join(",") === [...keys].sort().join(","); +/** Validates one input's exact shape and bounds (`invalid_screen_input`, `screen_text_unsupported`). */ +export function vmScreenInput(value: unknown): VmScreenInput { + if (!value || typeof value !== "object" || Array.isArray(value)) vmFail("invalid_screen_input"); + const v = value as Record; + switch (v.kind) { + case "click": if (exactKeys(v, ["kind", "x", "y", "button", "double"]) && point(v.x, v.y) && ["left", "right", "middle"].includes(v.button as string) && typeof v.double === "boolean") return v as VmScreenInput; break; + case "move": if (exactKeys(v, ["kind", "x", "y"]) && point(v.x, v.y)) return v as VmScreenInput; break; + case "scroll": if (exactKeys(v, ["kind", "x", "y", "delta"]) && point(v.x, v.y) && Number.isSafeInteger(v.delta) && v.delta !== 0 && Math.abs(v.delta as number) <= 10) return v as VmScreenInput; break; + case "key": if (exactKeys(v, ["kind", "keys"]) && Array.isArray(v.keys) && v.keys.length >= 1 && v.keys.length <= vmScreenMaxKeys && v.keys.every(vmScreenKey) && new Set(v.keys).size === v.keys.length) return v as VmScreenInput; break; + case "text": { + if (!exactKeys(v, ["kind", "text"]) || typeof v.text !== "string" || !v.text.length || [...v.text].length > vmScreenMaxText || !v.text.isWellFormed()) break; + if ([...v.text].some(char => !vmScreenChar(char))) vmFail("screen_text_unsupported", 422); + return v as VmScreenInput; + } + } + return vmFail("invalid_screen_input"); +} + +/** QEMU absolute axes run 0..32767 across the display. */ +const axis = (value: number, size: number) => Math.round(value * 32767 / (size - 1)); +const keyEvent = (key: string, down: boolean) => ({ type: "key", data: { down, key: { type: "qcode", data: key } } }); +const button = (name: string, down: boolean) => ({ type: "btn", data: { down, button: name } }); +const move = (x: number, y: number) => [{ type: "abs", data: { axis: "x", value: axis(x, vmScreenWidth) } }, { type: "abs", data: { axis: "y", value: axis(y, vmScreenHeight) } }]; +const send = (events: unknown[]): QmpCommand => ({ execute: "input-send-event", arguments: { events } }); + +/** + * The exact QMP commands for one validated input, in order. Each command is one press/release batch, so a + * key is never left held between commands. No guest or model text is interpreted here; text is mapped to qcodes. + */ +export function vmScreenCommands(input: VmScreenInput): QmpCommand[] { + switch (input.kind) { + case "move": return [send(move(input.x, input.y))]; + case "click": { + const once = [button(input.button, true), button(input.button, false)]; + return [send(move(input.x, input.y)), send(once), ...(input.double ? [send(once)] : [])]; + } + case "scroll": { + const name = input.delta < 0 ? "wheel-up" : "wheel-down"; + return [send(move(input.x, input.y)), ...Array.from({ length: Math.abs(input.delta) }, () => send([button(name, true), button(name, false)]))]; + } + case "key": return [send([...input.keys.map(key => keyEvent(key, true)), ...[...input.keys].reverse().map(key => keyEvent(key, false))])]; + case "text": return [...input.text].map(char => { + const mapped = vmScreenChar(char)!; + return send(mapped.shift ? [keyEvent("shift", true), keyEvent(mapped.key, true), keyEvent(mapped.key, false), keyEvent("shift", false)] : [keyEvent(mapped.key, true), keyEvent(mapped.key, false)]); + }); + } +} + +const pngSignature = [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]; +/** Accepts only a bounded PNG whose IHDR matches the fixed display size (`vm_screen_invalid`). */ +export function vmScreenFrame(bytes: Uint8Array): Uint8Array { + if (bytes.length < 33 || bytes.length > vmScreenMaxFrameBytes || pngSignature.some((byte, index) => bytes[index] !== byte)) vmFail("vm_screen_invalid", 502); + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); + if (String.fromCharCode(...bytes.subarray(12, 16)) !== "IHDR" || view.getUint32(16) !== vmScreenWidth || view.getUint32(20) !== vmScreenHeight) vmFail("vm_screen_invalid", 502); + return bytes; +} diff --git a/src/s15/vm/service.ts b/src/s15/vm/service.ts index 1a2c06e5..0bd0f981 100644 --- a/src/s15/vm/service.ts +++ b/src/s15/vm/service.ts @@ -29,6 +29,13 @@ export interface WorkingVmOptions { * start. Absent, VMs are never stopped or started without the owner (S15 behavior). */ idle?: { stopAfterMs: number }; + /** + * DESK16-01 exclusive owner screen control (explicit S16 desktop computers only). Every work operation holds + * a ticket from admission until it settled; `current()` refuses (`takeover_active`) once the owner asked for + * control, so no work operation is admitted, dispatched or published under the owner's generation. Idle + * release skips a computer the owner holds. Absent, nothing changes. + */ + screen?: { agentInput(vmId: string): { current(): void; end(): void }; idleBlocked(vmId: string): boolean }; } /** Registry reason of a VM that Jit stopped for idleness; only such a VM is started again on demand. */ export const vmIdleReason = "vm_idle"; @@ -214,7 +221,10 @@ export class WorkingVmService implements WorkingVmPort { } private async call(actor: object, s: LiveSession, operation: Record, kind: "read" | "write" | "shell", signal?: AbortSignal): Promise> { return this.enqueue(actor, s, signal, async () => { - await this.health(this.options.store.registration(s.vmId)); this.current(actor, s); + const ticket = this.options.screen?.agentInput(s.vmId); + try { + const current = () => { this.current(actor, s); ticket?.current(); }; + await this.health(this.options.store.registration(s.vmId)); current(); const id = uuid(), request: Record = { v: 1, id, root: s.root, tmp: s.temporary, cap: kind === "shell" ? this.#limits.outputBytes : this.#limits.fileBytes, ...structuredClone(operation) }; const payload = vmCanonical(request), operationId = id, payloadHash = vmHash(payload); if (containsCredential(payload)) vmFail("vm_credentials_denied", 403); @@ -227,30 +237,31 @@ export class WorkingVmService implements WorkingVmPort { outputBytes: null, transport: "pending", execution: "unsettled", published: false }; this.options.store.saveEvidence(evidence); try { - this.current(actor, s); + current(); this.options.disclosure?.vmInput(actor, s.vmId); const transportRequest = request.op === "spawn" ? { ...request, env: { ...(request.env as Record), ...s.proxy?.environment } } : request; const response = await claim.dispatch(() => { - this.current(actor, s); evidence.outcome = "unknown"; this.options.store.saveEvidence(evidence); + current(); evidence.outcome = "unknown"; this.options.store.saveEvidence(evidence); const transport = this.transport(this.options.store.registration(s.vmId)), began = this.#clock(); const invoke = async () => { - if (operation.op === "init") await transport.openScope(s.unit, Math.max(1, s.deadline - this.#clock()), () => this.current(actor, s), s.controller.signal); - this.current(actor, s); + if (operation.op === "init") await transport.openScope(s.unit, Math.max(1, s.deadline - this.#clock()), () => current(), s.controller.signal); + current(); return transport.invoke(transportRequest, { unit: s.unit, timeoutMs: Math.max(1, Math.floor(action.timeoutMs - (this.#clock() - began))), - maxBytes: Math.max(this.#limits.fileBytes, this.#limits.outputBytes) * 3 + 65536, signal: s.controller.signal, beforeStart: () => this.current(actor, s) }); + maxBytes: Math.max(this.#limits.fileBytes, this.#limits.outputBytes) * 3 + 65536, signal: s.controller.signal, beforeStart: () => current() }); }; return invoke(); }); evidence.transport = "closed"; evidence.outcome = "guest_reported"; const bytes = vmCanonical(response); evidence.responseHash = vmHash(bytes); evidence.outputBytes = Buffer.byteLength(bytes); this.options.store.saveEvidence(evidence); claim.observe({ ...evidence }); - this.current(actor, s); - return claim.publish(() => { this.current(actor, s); this.options.disclosure?.vmOutput(actor, s.vmId); evidence.published = true; this.options.store.saveEvidence(evidence); return response; }); + current(); + return claim.publish(() => { current(); this.options.disclosure?.vmOutput(actor, s.vmId); evidence.published = true; this.options.store.saveEvidence(evidence); return response; }); } catch (error) { evidence.transport = "closed"; this.options.store.saveEvidence(evidence); claim.observe({ ...evidence }); if (error instanceof AppError && ["vm_protocol_invalid", "vm_connection_failed", "vm_transport_timeout", "vm_output_limit"].includes(error.code)) this.uncertain(s.vmId, error.code, s.generation); throw error; } } finally { claim.release?.(); } + } finally { ticket?.end(); } }); } private releaseHeld(s: LiveSession): void { @@ -347,6 +358,7 @@ export class WorkingVmService implements WorkingVmPort { const row = this.options.store.registration(id), manager = row.managerId ? this.options.managers.get(row.managerId) : null; if (row.state !== "ready" || !manager?.start || !row.bindingMac || this.#managementBusy.has(id)) continue; if (this.options.store.db.query("SELECT 1 FROM vm_sessions WHERE vm_id=? AND state<>'closed' LIMIT 1").get(id)) continue; + if (this.options.screen?.idleBlocked(id)) continue; const last = this.lastActivity(id); if (last === null) { this.touch(id); continue; } if (this.#now() - last < idle.stopAfterMs) continue; diff --git a/src/s15/vm/vmm-launcher.ts b/src/s15/vm/vmm-launcher.ts index 7c6f43e4..ee4f8e2c 100644 --- a/src/s15/vm/vmm-launcher.ts +++ b/src/s15/vm/vmm-launcher.ts @@ -17,10 +17,10 @@ const tokens = { NAME: "name", CPUS: "cpus", MEMORY: "memory", MAC: "mac", PORT: * The QEMU argv for one architecture as shell words, derived from `qemuArgv` itself with placeholder values, * so the launcher and the controller's `/proc` re-verification can never disagree on the restricted profile. */ -function qemuWords(vmm: QemuVmmOptions, architecture: "x86_64" | "aarch64"): string { +function qemuWords(vmm: QemuVmmOptions, architecture: "x86_64" | "aarch64", desktop = false): string { const layout = vmmLayout(vmm.directory); const argv = qemuArgv({ name: "@@NAME@@", architecture, machineDirectory: `${layout.machines}/@@NAME@@`, cpus: "@@CPUS@@" as unknown as number, - memoryMiB: "@@MEMORY@@" as unknown as number, mac: "@@MAC@@", sshPort: "@@PORT@@" as unknown as number, proxySocket: layout.proxySocket }); + memoryMiB: "@@MEMORY@@" as unknown as number, mac: "@@MAC@@", sshPort: "@@PORT@@" as unknown as number, proxySocket: layout.proxySocket, ...(desktop ? { desktop } : {}) }); return argv.map(arg => arg.split(/(@@[A-Z]+@@)/).map(part => { const token = /^@@([A-Z]+)@@$/.exec(part)?.[1] as keyof typeof tokens | undefined; if (token) { if (!tokens[token]) throw new Error("vmm_launcher_unknown_token"); return `"$${tokens[token]}"`; } @@ -35,7 +35,8 @@ function qemuWords(vmm: QemuVmmOptions, architecture: "x86_64" | "aarch64"): str * refuses any unknown operation or argument (exit 64, nothing run), and runs one fixed command per operation: * `check` (prints `jit-vmm-ready-v1`; the installer's end-to-end sudo probe), `kvm` (device access probe), `show|stop|kill ` (its transient unit), `info shared|exclusive`, * `resize `, `create `, `snapshot create|apply|delete `, - * `seed ` (the NoCloud seed from stdin, at most 16 MiB), `remove ` (that VM's machine directory) and + * `seed ` (the NoCloud seed from stdin, at most 16 MiB), `remove ` (that VM's machine directory), + * `discard-frame ` (only that VM's DESK16-01 `screen.png`) and * `start ` (QEMU in a transient user unit, UMask 0007 so only the * shared group reaches its QMP socket). It reads no environment input and accepts no path, flag or command. */ @@ -125,18 +126,27 @@ case "$op" in remove) [ "$#" -eq 0 ] || refuse rm -rf -- "$dir" ;; + discard-frame) + [ "$#" -eq 0 ] || refuse + rm -f -- "$dir/screen.png" ;; start) - [ "$#" -eq 5 ] || refuse - arch=$1 cpus=$2 memory=$3 mac=$4 port=$5 + # A sixth word selects the DESK16-01 desktop profile; nothing else is accepted there. + [ "$#" -eq 5 ] || { [ "$#" -eq 6 ] && [ "$6" = desktop ]; } || refuse + arch=$1 cpus=$2 memory=$3 mac=$4 port=$5 profile=headless + if [ "$#" -eq 6 ]; then profile=desktop; fi nat "$cpus" 3 && nat "$memory" 7 && nat "$port" 5 || refuse [ "$port" -ge 1024 ] && [ "$port" -le 65535 ] || refuse case "$mac" in 52:54:00:${hex(2)}:${hex(2)}:${hex(2)}) ;; *) refuse ;; esac - rm -f -- "$dir/qmp.sock" - case "$arch" in - x86_64) exec ${unit} \\ + rm -f -- "$dir/qmp.sock" "$dir/screen.png" + case "$arch:$profile" in + x86_64:headless) exec ${unit} \\ ${qemuWords(vmm, "x86_64")} ;; - aarch64) exec ${unit} \\ + aarch64:headless) exec ${unit} \\ ${qemuWords(vmm, "aarch64")} ;; + x86_64:desktop) exec ${unit} \\ + ${qemuWords(vmm, "x86_64", true)} ;; + aarch64:desktop) exec ${unit} \\ + ${qemuWords(vmm, "aarch64", true)} ;; *) refuse ;; esac ;; *) refuse ;; diff --git a/src/s16/client/client.ts b/src/s16/client/client.ts index 0894234c..5d222402 100644 --- a/src/s16/client/client.ts +++ b/src/s16/client/client.ts @@ -1,5 +1,6 @@ import { clientResponseBytes, type ClientConnectionsView, type ClientConnectionCommand, type ClientConnectionReceipt, type ClientSource, type ClientBotsView, type ClientBotDetail, type ClientBot, type ClientBotProposal, type ClientBotDraft, type ClientBotExport, type ClientBotCommand, type ClientBotReceipt, type ClientNotificationDevice, type ClientNotificationSettings, type ClientNotificationSubscribeCommand, type ClientSteer, type ClientSteerReceipt, type ClientRoutineDetail, type ClientRoutineDraft, type ClientRoutineList, type ClientRoutineOptions, type ClientRoutineProposal, type ClientRoutineReceipt, type ClientArtifactDetail, type ClientArtifactInputCommand, type ClientArtifactItem, type ClientArtifactList, type ClientArtifactQuery, type ClientMemoryCommand, type ClientMemoryDetail, type ClientMemoryList, type ClientMemoryQuery, type ClientMemoryResult, type ClientBootstrap, type ClientDecision, type ClientEvents, type ClientMessage, type ClientReceipt, type ClientRequest, type ClientSnapshot } from "./contracts"; import type { FileArtifact, FileReference, FileUpload, FileUploadCommand } from "../files/types"; +import type { ScreenCommand, ScreenCommandResult, ScreenView } from "../computers/screen"; import type { ComputerCommand, ComputerCommandResult, ComputersView } from "../computers/service"; /** Safe transport failure. An unknown mutation outcome is recovered by its original request identity. */ @@ -315,6 +316,49 @@ export class JitClient { if (!record(data) || data.contract !== "s16-computer-command-result/v1" || data.requestId !== input.requestId || !identifier(data.vmId) || !this.computersView(data.view)) throw new JitClientError("client_protocol_invalid", null, "unknown", input.requestId); return data as unknown as ComputerCommandResult; } + /** DESK16-01 screen state of one computer. Reading changes nothing (an expired owner lease is reported fenced). */ + async screen(vmId: string, signal?: AbortSignal): Promise { + if (!identifier(vmId)) throw new JitClientError("invalid_screen_command", null, "not_sent"); + const data = await this.call(`/computers/${vmId}/screen`, "GET", undefined, signal); + if (!this.screenView(data) || data.vmId !== vmId) return invalid(); + return data; + } + /** + * One PNG frame under the generation the owner last saw (`stale_screen` after any takeover, fence or + * hand-back). The bytes stay in memory for display only; nothing is cached or stored. + */ + async screenFrame(vmId: string, generation: number, signal?: AbortSignal): Promise { + if (!identifier(vmId) || !count(generation)) throw new JitClientError("invalid_screen_command", null, "not_sent"); + const session = this.#sessionGeneration; + let response: Response; + try { response = await this.#fetch(new Request(`${this.origin}/api/s16/client/computers/${vmId}/screen/frame?generation=${generation}`, { credentials: "same-origin", mode: "same-origin", cache: "no-store", redirect: "error", referrerPolicy: "no-referrer", ...(signal ? { signal } : {}) })); } + catch { throw new JitClientError(signal?.aborted ? "aborted" : "connection_lost", null, "not_sent"); } + if (session !== this.#sessionGeneration) { await response.body?.cancel().catch(() => {}); throw new JitClientError("client_response_stale", response.status, "response"); } + if (response.status === 401 || response.status === 403) this.clear(); + if (!response.ok) { let code = "screen_unavailable"; try { const data = await this.json(response, "GET", null); code = safeCode(record(data) ? data.error : null); } catch { /* The status alone decides. */ } throw new JitClientError(code, response.status, "response"); } + if (response.redirected || response.headers.get("content-type") !== "image/png") { await response.body?.cancel().catch(() => {}); return invalid(); } + const blob = await response.blob(); + if (session !== this.#sessionGeneration) throw new JitClientError("client_response_stale", response.status, "response"); + if (blob.size < 33 || blob.size > 8 * 1_048_576) return invalid(); + return new Blob([blob], { type: "image/png" }); + } + /** + * One exact screen command. It never retries: after an unknown outcome resend the same command (same + * `requestId`). Typed text is sent once in this body and never kept by the client. + */ + async screenCommand(command: ScreenCommand, signal?: AbortSignal): Promise { + const input = structuredClone(command); + if (input.contract !== "s16-screen-command/v1" || !identifier(input.requestId) || !identifier(input.vmId) || !count(input.generation)) throw new JitClientError("invalid_screen_command", null, "not_sent", input.requestId); + const data = await this.call("/computers/screen/commands", "POST", input, signal, input.requestId); + if (!record(data) || data.contract !== "s16-screen-command-result/v1" || data.requestId !== input.requestId || !this.screenView(data.view) || data.view.vmId !== input.vmId) throw new JitClientError("client_protocol_invalid", null, "unknown", input.requestId); + return data as unknown as ScreenCommandResult; + } + private screenView(data: unknown): data is ScreenView { + return record(data) && data.contract === "s16-computer-screen/v1" && identifier(data.vmId) && typeof data.available === "boolean" && (data.code === null || typeof data.code === "string") + && count(data.width) && count(data.height) && ["agent", "settling", "owner", "fenced"].includes(data.state as string) && count(data.generation) + && (data.takeoverId === null || identifier(data.takeoverId)) && (data.expiresAt === null || count(data.expiresAt)) && typeof data.agentBusy === "boolean" + && record(data.scope) && (data.scope.kind === "shared" || data.scope.kind === "private" && identifier(data.scope.botId)); + } private computersView(data: unknown): data is ComputersView { if (!record(data) || data.contract !== "s16-client-computers/v1" || data.idleStopMs !== null && !count(data.idleStopMs) || !Array.isArray(data.computers) || data.computers.length > 64 || !Array.isArray(data.bots) || data.bots.length > 64 || !record(data.privateCreation)) return false; const states = ["running", "sleeping", "starting", "stopping", "stopped", "setting_up", "setup_failed", "attention"]; diff --git a/src/s16/client/contracts.ts b/src/s16/client/contracts.ts index b0fdf7cb..8853c63b 100644 --- a/src/s16/client/contracts.ts +++ b/src/s16/client/contracts.ts @@ -21,7 +21,9 @@ export interface ClientCapabilities { /** Additive field: Web Push notices (`s16-notifications/v1`) and the fixed service worker/scope that receives them. */ notifications?: { available: false; code: "notifications_unavailable" } | { available: true; contract: "s16-notifications/v1"; transport: "web_push"; serviceWorker: "/s16/sw.js"; scope: "/s16/" }; /** TEAM16-01 working computers; `files` is FILE16-01C staging/collection inside file work. */ - computers?: { available: false; code: "computers_unavailable" } | { available: true; contract: "s16-client-computers/v1"; idleStopMs: number | null; files: boolean }; + computers?: { available: false; code: "computers_unavailable" } | { available: true; contract: "s16-client-computers/v1"; idleStopMs: number | null; files: boolean; + /** DESK16-01: the owner can watch a desktop computer's screen and take exclusive control (`s16-computer-screen/v1`). */ + screen?: boolean }; /** BOT16-01 persistent Bots (`s16-client-bots/v1`); `delegation` is DELEG16-01 Bot-to-Bot assignments. */ bots?: { available: false; code: "bots_unavailable" } | { available: true; contract: "s16-client-bots/v1"; delegation: boolean }; /** CONN16-01/MCP16-01/OPEN16-01 connections (`s16-client-connections/v1`): GitHub plus HTTPS tool sources. */ diff --git a/src/s16/computers/screen.ts b/src/s16/computers/screen.ts new file mode 100644 index 00000000..1c39a7f0 --- /dev/null +++ b/src/s16/computers/screen.ts @@ -0,0 +1,250 @@ +import { AppError } from "../../errors"; +import { canonical, digest, exact, identity, integer } from "../../s15/conversation/validation"; +import { vmScreenHeight, vmScreenInput, vmScreenWidth, type VmScreenDevice, type VmScreenInput } from "../../s15/vm/screen"; +import type { VmStore } from "../../s15/vm/store"; +import type { ComputerScope } from "./scopes"; + +/** + * Who may write to a working computer's screen (`s16-computer-screen/v1`, DESK16-01): + * - `agent`: work may use the computer; the owner can watch. + * - `settling`: the owner asked for control; new work operations are refused and the owner waits until every + * work operation admitted under the previous generation has settled. + * - `owner`: only the owner's input reaches the screen, under the current generation and takeover id. + * - `fenced`: control was lost while the owner held it (lease expiry, closed tab, controller restart). Nobody + * writes until the owner takes control again or hands the computer back. + */ +export type ScreenControlState = "agent" | "settling" | "owner" | "fenced"; +/** Owner projection of one computer's screen. Frames are this computer's private data in `scope`. */ +export interface ScreenView { + contract: "s16-computer-screen/v1"; vmId: string; + /** Whether frames can be read now; `code` is the stable reason when not. */ + available: boolean; code: string | null; width: number; height: number; + state: ScreenControlState; generation: number; takeoverId: string | null; + /** When owner control lapses without a renewal; null outside `settling`/`owner`. */ + expiresAt: number | null; + /** Work operations still running from before the takeover (only meaningful while `settling`). */ + agentBusy: boolean; + /** Lineage of every frame: the computer's scope. A private computer's frames are its Bot's private data. */ + scope: ComputerScope; +} +/** Exact owner command. `generation` is the one the owner last saw; any change makes the command stale. */ +export type ScreenCommand = { contract: "s16-screen-command/v1"; requestId: string; vmId: string } & ( + | { kind: "takeover"; generation: number } + | { kind: "renew"; generation: number; takeoverId: string } + | { kind: "input"; generation: number; takeoverId: string; input: VmScreenInput } + | { kind: "hand_back"; generation: number; takeoverId: string | null }); +export interface ScreenCommandResult { contract: "s16-screen-command-result/v1"; requestId: string; view: ScreenView } + +/** Protected composition only; none of these ports is reachable from a model, guest or conversation. */ +export interface ScreenControlOptions { + store: VmStore; + requireOwner(actor: Owner): void; + /** Host-side screen of an enrolled computer, or null when its manager has no desktop profile. */ + device(vmId: string): VmScreenDevice | null; + scope(vmId: string): ComputerScope; + now?: () => number; + /** Owner control lapses this long after the last renewal (default 30 s). */ + leaseMs?: number; +} +interface Row { vm_id: string; generation: number; state: ScreenControlState; takeover_id: string | null; expires_at: number | null } +/** A work operation admitted under one generation. `current()` refuses once the owner asked for control. */ +export interface ScreenAgentTicket { current(): void; end(): void } + +const uuid = () => crypto.randomUUID().replaceAll("-", ""); +const stale = () => new AppError("stale_screen", 409); + +/** + * DESK16-01 exclusive screen control for working computers. Every transition advances the generation, so + * frames and input from before a takeover, fence or hand-back are refused (`stale_screen`). Work operations + * hold an agent ticket from admission until settlement; owner input starts only after every ticket of the + * previous generation ended, and work starts again only after the owner's last input settled. Owner input is + * sent once and never stored: no typed text, hash of it or frame enters the journal, history or memory. + */ +export class ScreenControl { + readonly #now: () => number; + readonly #leaseMs: number; + readonly #tickets = new Map>(); + readonly #ownerTail = new Map>(); + readonly #closing = new Set(); + /** + * Input request ids attempted under the current owner generation, kept in memory only (never the text). A retry + * joins a running send, returns the view after a finished one, and is refused (`screen_input_unknown`) after a + * send that failed once events may have reached the VM; only a refusal before sending releases the id. Entries of + * earlier generations are dropped on every transition: a retry under an old generation is refused as stale anyway. + */ + readonly #inputs = new Map }>(); + constructor(readonly options: ScreenControlOptions) { + this.#now = options.now ?? Date.now; this.#leaseMs = options.leaseMs ?? 30_000; integer(this.#leaseMs, 1000, 600_000); + options.store.db.exec(`CREATE TABLE IF NOT EXISTS s16_screen_control (vm_id TEXT PRIMARY KEY, generation INTEGER NOT NULL, state TEXT NOT NULL CHECK (state IN ('agent','settling','owner','fenced')), takeover_id TEXT, expires_at INTEGER); + CREATE TABLE IF NOT EXISTS s16_screen_commands (id TEXT PRIMARY KEY, hash TEXT NOT NULL, vm_id TEXT NOT NULL); + CREATE TABLE IF NOT EXISTS s16_screen_events (sequence INTEGER PRIMARY KEY, vm_id TEXT NOT NULL, kind TEXT NOT NULL, generation INTEGER NOT NULL, at INTEGER NOT NULL);`); + // Controller restart: control the owner held (or was acquiring) is fenced, never silently returned to work. + options.store.transaction(() => { + for (const row of options.store.db.query("SELECT * FROM s16_screen_control WHERE state IN ('settling','owner')").all()) this.write(row.vm_id, row.generation + 1, "fenced", null, null, "fenced_restart"); + }); + } + private write(vm: string, generation: number, state: ScreenControlState, takeover: string | null, expires: number | null, event: string): Row { + for (const [id, entry] of this.#inputs) if (entry.vm === vm && entry.generation < generation) this.#inputs.delete(id); + this.options.store.db.query("INSERT INTO s16_screen_control VALUES (?,?,?,?,?) ON CONFLICT(vm_id) DO UPDATE SET generation=excluded.generation, state=excluded.state, takeover_id=excluded.takeover_id, expires_at=excluded.expires_at").run(vm, generation, state, takeover, expires); + this.options.store.db.query("INSERT INTO s16_screen_events(vm_id,kind,generation,at) VALUES (?,?,?,?)").run(vm, event, generation, this.#now()); + return { vm_id: vm, generation, state, takeover_id: takeover, expires_at: expires }; + } + /** Current row; a lapsed owner lease becomes `fenced` under a new generation as it is read. */ + private row(vm: string): Row { + const row = this.options.store.db.query("SELECT * FROM s16_screen_control WHERE vm_id=?").get(vm) ?? { vm_id: vm, generation: 0, state: "agent", takeover_id: null, expires_at: null }; + if ((row.state === "owner" || row.state === "settling") && row.expires_at !== null && row.expires_at <= this.#now()) + return this.options.store.transaction(() => this.write(vm, row.generation + 1, "fenced", null, null, "fenced_lapsed")); + return row; + } + private known(vm: string): void { identity(vm); this.options.store.registration(vm); } + private busy(vm: string): boolean { return (this.#tickets.get(vm)?.size ?? 0) > 0; } + /** `settling` becomes `owner` once the last ticket of the previous generation ended. */ + private settle(vm: string): void { + if (this.busy(vm)) return; + this.options.store.transaction(() => { + const row = this.row(vm); + if (row.state === "settling") this.write(vm, row.generation, "owner", row.takeover_id, row.expires_at, "owner_control"); + }); + } + + /** + * Admits one work operation on `vm` (WorkingVmService). Refused with `takeover_active` unless work holds the + * screen. The ticket must be ended exactly once, after the operation settled (finished, failed or unknown). + */ + agentInput(vm: string): ScreenAgentTicket { + const admitted = this.row(vm); + if (admitted.state !== "agent") throw new AppError("takeover_active", 409); + const token = {}, set = this.#tickets.get(vm) ?? new Set(); set.add(token); this.#tickets.set(vm, set); + let ended = false; + return { + current: () => { const row = this.row(vm); if (ended || row.state !== "agent" || row.generation !== admitted.generation) throw new AppError("takeover_active", 409); }, + end: () => { if (ended) return; ended = true; set.delete(token); if (!set.size) this.#tickets.delete(vm); this.settle(vm); }, + }; + } + /** Idle release never stops a computer the owner holds or lost control of. */ + idleBlocked(vm: string): boolean { return this.row(vm).state !== "agent"; } + + /** Authenticated owner projection. Reading starts, stops or changes nothing except lapsing an expired lease. */ + async view(actor: Owner, vm: string): Promise { + this.options.requireOwner(actor); this.known(vm); + const device = this.options.device(vm), code = device ? await device.unavailable() : "vm_desktop_unavailable"; + this.options.requireOwner(actor); + const row = this.row(vm); + return { contract: "s16-computer-screen/v1", vmId: vm, available: code === null, code, width: vmScreenWidth, height: vmScreenHeight, + state: row.state, generation: row.generation, takeoverId: row.state === "owner" || row.state === "settling" ? row.takeover_id : null, + expiresAt: row.state === "owner" || row.state === "settling" ? row.expires_at : null, agentBusy: row.state === "settling" && this.busy(vm), scope: this.options.scope(vm) }; + } + /** + * One PNG frame for the owner, under the generation the owner last saw. A frame captured across a + * transition is dropped (`stale_screen`), so an old frame never appears as the current screen. + */ + async frame(actor: Owner, vm: string, generation: number): Promise { + this.options.requireOwner(actor); this.known(vm); integer(generation); + if (this.row(vm).generation !== generation) throw stale(); + const device = this.options.device(vm); if (!device) throw new AppError("vm_desktop_unavailable", 409); + const bytes = await device.capture(); + this.options.requireOwner(actor); + if (this.row(vm).generation !== generation) throw stale(); + return bytes; + } + + /** One exact owner command. Takeover and hand-back are idempotent per `requestId`; input is sent at most once per `requestId` in this process. */ + async command(actor: Owner, input: unknown): Promise { + this.options.requireOwner(actor); + const command = parseScreenCommand(input), vm = command.vmId; this.known(vm); + const result = async (): Promise => ({ contract: "s16-screen-command-result/v1", requestId: command.requestId, view: await this.view(actor, vm) }); + if (command.kind === "input") { + const prior = this.#inputs.get(command.requestId); + if (prior) { + if (prior.state === "unknown") throw new AppError("screen_input_unknown", 409); + await prior.run; + return result(); + } + let reached = false; + const entry = { generation: command.generation, vm, state: "sending" as "sending" | "sent" | "unknown", run: Promise.resolve() }; + this.#inputs.set(command.requestId, entry); + entry.run = this.input(actor, command, () => { reached = true; }); + try { await entry.run; entry.state = "sent"; } + catch (error) { if (reached) entry.state = "unknown"; else if (this.#inputs.get(command.requestId) === entry) this.#inputs.delete(command.requestId); throw error; } + return result(); + } + if (command.kind === "renew") { + // Renewal is idempotent by itself, so it is not journaled per request. + this.options.store.transaction(() => { + this.options.requireOwner(actor); + this.held(vm, command.generation, command.takeoverId); + this.options.store.db.query("UPDATE s16_screen_control SET expires_at=? WHERE vm_id=?").run(this.#now() + this.#leaseMs, vm); + }); + return result(); + } + const hash = digest(canonical(command)); + const prior = this.options.store.db.query<{ hash: string }, [string]>("SELECT hash FROM s16_screen_commands WHERE id=?").get(command.requestId); + if (prior) { if (prior.hash !== hash) throw new AppError("request_id_conflict", 409); return result(); } + if (command.kind === "takeover") { + const device = this.options.device(vm), code = device ? await device.unavailable() : "vm_desktop_unavailable"; + if (code) throw new AppError(code, 409); + this.options.store.transaction(() => { + this.options.requireOwner(actor); + const row = this.row(vm); + if (row.state !== "agent" && row.state !== "fenced") throw new AppError("takeover_state_changed", 409); + if (row.generation !== command.generation) throw stale(); + this.write(vm, row.generation + 1, this.busy(vm) ? "settling" : "owner", uuid(), this.#now() + this.#leaseMs, this.busy(vm) ? "takeover_settling" : "takeover"); + this.record(command.requestId, hash, vm); + }); + return result(); + } + // Hand-back: refuse new owner input, wait for the owner's last input to settle, then return the screen to work. + const row = this.row(vm); + if (row.state === "agent") throw new AppError("takeover_state_changed", 409); + if (row.generation !== command.generation || (row.state !== "fenced" && row.takeover_id !== command.takeoverId)) throw stale(); + this.#closing.add(vm); + try { + await (this.#ownerTail.get(vm) ?? Promise.resolve()).catch(() => {}); + this.options.store.transaction(() => { + this.options.requireOwner(actor); + const current = this.row(vm); + if (current.generation !== row.generation || current.state === "agent") throw stale(); + this.write(vm, current.generation + 1, "agent", null, null, "hand_back"); + this.record(command.requestId, hash, vm); + }); + } finally { this.#closing.delete(vm); } + return result(); + } + private record(id: string, hash: string, vm: string): void { this.options.store.db.query("INSERT INTO s16_screen_commands VALUES (?,?,?)").run(id, hash, vm); } + private held(vm: string, generation: number, takeover: string): Row { + const row = this.row(vm); + if (row.state !== "owner" && row.state !== "settling") throw new AppError(row.state === "fenced" ? "screen_control_lost" : "takeover_state_changed", 409); + if (row.generation !== generation || row.takeover_id !== takeover) throw stale(); + return row; + } + /** Owner input, serialized per computer and checked against the live row immediately before it is sent. */ + private async input(actor: Owner, command: Extract, sending: () => void): Promise { + const vm = command.vmId, device = this.options.device(vm); if (!device) throw new AppError("vm_desktop_unavailable", 409); + const check = () => { + this.options.requireOwner(actor); + if (this.#closing.has(vm)) throw stale(); + const row = this.held(vm, command.generation, command.takeoverId); + if (row.state === "settling") throw new AppError("takeover_settling", 409); + }; + check(); + const previous = this.#ownerTail.get(vm) ?? Promise.resolve(), turn = previous.catch(() => {}).then(async () => { check(); sending(); await device.send(command.input); }); + const tail = turn.catch(() => {}); this.#ownerTail.set(vm, tail); + try { await turn; } finally { if (this.#ownerTail.get(vm) === tail) this.#ownerTail.delete(vm); } + } +} + +/** Closed parser for `s16-screen-command/v1`. Input bounds are checked by `vmScreenInput`. */ +export function parseScreenCommand(value: unknown): ScreenCommand { + if (!value || typeof value !== "object" || Array.isArray(value)) throw new AppError("invalid_body"); + const kind = (value as { kind?: unknown }).kind, base = ["contract", "requestId", "kind", "vmId", "generation"]; + if (kind === "takeover") exact(value, base); + else if (kind === "renew" || kind === "hand_back") exact(value, [...base, "takeoverId"]); + else if (kind === "input") exact(value, [...base, "takeoverId", "input"]); + else throw new AppError("invalid_body"); + const v = value as Record; + if (v.contract !== "s16-screen-command/v1") throw new AppError("invalid_contract"); + identity(v.requestId); identity(v.vmId); integer(v.generation); + if (kind === "renew" || kind === "input" || kind === "hand_back" && v.takeoverId !== null) identity(v.takeoverId); + if (kind === "input") return { ...(structuredClone(value) as ScreenCommand), input: vmScreenInput(v.input) } as ScreenCommand; + return structuredClone(value) as ScreenCommand; +} diff --git a/src/s16/http.ts b/src/s16/http.ts index c932e85a..5c40b204 100644 --- a/src/s16/http.ts +++ b/src/s16/http.ts @@ -10,6 +10,7 @@ import { memoryDetail, parseMemoryQuery, searchMemory, type S16MemoryPort } from import { artifactDetail, listArtifacts, parseArtifactQuery } from "./artifacts"; import type { RoutineService } from "./routines/service"; import type { NotificationService } from "./notifications/service"; +import type { ScreenControl } from "./computers/screen"; import type { ComputerDirectory } from "./computers/service"; import type { BotDirectory } from "./bots/service"; import type { ConnectionDirectory } from "./connections/service"; @@ -23,7 +24,7 @@ import { connectionLimits } from "./connections/types"; export const artifactPreviewPolicy = "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src 'none'; form-action 'none'; base-uri 'none'; frame-ancestors 'self'"; /** Protected shared client adapter. It adds no engine, database, approval authority or automatic retry. */ -export function createS16ClientHandler(app: ConversationApplication, options: { origin: string; cookieName: string; /** AUTH16-01 banner: true while a newer default proposal awaits the owner. */ policyUpdate?: (actor: Actor) => boolean; authentication: ConversationHttpAuthentication; files?: FileStore; imageAvailable?: () => boolean; memory?: S16MemoryPort; routines?: RoutineService; wake?: () => void; notifications?: NotificationService; computers?: ComputerDirectory; /** BOT16-01 persistent Bots (`s16-client-bots/v1`). */ bots?: BotDirectory; /** CONN16-01 connections (`s16-client-connections/v1`). */ connections?: ConnectionDirectory }): (request: Request) => Promise { +export function createS16ClientHandler(app: ConversationApplication, options: { origin: string; cookieName: string; /** AUTH16-01 banner: true while a newer default proposal awaits the owner. */ policyUpdate?: (actor: Actor) => boolean; authentication: ConversationHttpAuthentication; files?: FileStore; imageAvailable?: () => boolean; memory?: S16MemoryPort; routines?: RoutineService; wake?: () => void; notifications?: NotificationService; computers?: ComputerDirectory; /** DESK16-01 watch and take over a working computer's screen (`s16-computer-screen/v1`). */ screen?: ScreenControl; /** BOT16-01 persistent Bots (`s16-client-bots/v1`). */ bots?: BotDirectory; /** CONN16-01 connections (`s16-client-connections/v1`). */ connections?: ConnectionDirectory }): (request: Request) => Promise { const origin = new URL(options.origin); if (origin.origin !== options.origin || !/^[A-Za-z0-9_-]+$/.test(options.cookieName)) throw new AppError("invalid_configuration"); const json = (value: unknown, status = 200): Response => { @@ -55,7 +56,7 @@ export function createS16ClientHandler(app: ConversationApplication memory: options.memory ? { available: true, contract: "s16-client-memory/v1" } : { available: false, code: "memory_unavailable" }, routines: options.routines ? { available: true, contract: "s16-client-routines/v1", reactions: { signed: true, github: true, artifacts: !!options.routines.options.files }, feed: true } : { available: false, code: "routines_unavailable" }, notifications: options.notifications ? { available: true, contract: "s16-notifications/v1", transport: "web_push", serviceWorker: "/s16/sw.js", scope: "/s16/" } : { available: false, code: "notifications_unavailable" }, - computers: options.computers ? { available: true, contract: "s16-client-computers/v1", idleStopMs: options.computers.options.vm.idleStopAfterMs, files: !!(files && app.options.files) } : { available: false, code: "computers_unavailable" }, + computers: options.computers ? { available: true, contract: "s16-client-computers/v1", idleStopMs: options.computers.options.vm.idleStopAfterMs, files: !!(files && app.options.files), ...(options.screen ? { screen: true } : {}) } : { available: false, code: "computers_unavailable" }, bots: options.bots ? { available: true, contract: "s16-client-bots/v1", delegation: true } : { available: false, code: "bots_unavailable" }, connections: options.connections ? { available: true, contract: "s16-client-connections/v1", kinds: ["mcp", "openapi", "graphql"], github: !!options.connections.options.github, oauth: true } : { available: false, code: "connections_unavailable" } }; }; @@ -163,6 +164,22 @@ export function createS16ClientHandler(app: ConversationApplication if (request.method === "POST" && notify[1] === "unsubscribe") { const input = await body(request); return json(options.notifications.unsubscribe(currentOwner(), input)); } throw new AppError("method_rejected", 405); } + const screen = /^\/api\/s16\/client\/computers\/(?:screen\/(commands)|([A-Za-z0-9_-]{1,128})\/screen(?:\/(frame))?)$/.exec(url.pathname); + if (screen) { + if (!options.screen) throw new AppError("screen_unavailable", 503); + if (request.method === "GET" && screen[3]) { + // The only query is the generation the owner last saw; a frame from any other generation is refused. + const generation = url.searchParams.get("generation"); + if ([...url.searchParams.keys()].join() !== "generation" || !/^(0|[1-9][0-9]{0,9})$/.test(generation ?? "")) throw new AppError("invalid_body"); + const bytes = await options.screen.frame(currentOwner(), screen[2]!, Number(generation)); + return new Response(new Uint8Array(bytes), { headers: { "Content-Type": "image/png", "Cache-Control": "private, no-store", "X-Content-Type-Options": "nosniff", "Content-Security-Policy": "default-src 'none'; frame-ancestors 'none'", "Referrer-Policy": "no-referrer", "Cross-Origin-Resource-Policy": "same-origin", Vary: "Cookie" } }); + } + if (url.search) throw new AppError("invalid_body"); + if (request.method === "GET" && screen[2]) return json(await options.screen.view(currentOwner(), screen[2])); + // Body streaming is an async boundary: owner/session and CSRF are checked again after it. + if (request.method === "POST" && screen[1]) { const input = await body(request, 4096); return json(await options.screen.command(currentOwner(), input)); } + throw new AppError("method_rejected", 405); + } const computer = /^\/api\/s16\/client\/computers(?:\/(commands))?$/.exec(url.pathname); if (computer) { if (!options.computers) throw new AppError("computers_unavailable", 503); diff --git a/src/s16/web/ComputerScreen.tsx b/src/s16/web/ComputerScreen.tsx new file mode 100644 index 00000000..25679501 --- /dev/null +++ b/src/s16/web/ComputerScreen.tsx @@ -0,0 +1,186 @@ +import { useEffect, useRef, useState, type KeyboardEvent, type MouseEvent, type WheelEvent } from "react"; +import { JitClientError, type JitClient } from "../client"; +import type { ScreenCommand, ScreenView } from "../computers/screen"; +import type { VmScreenInput } from "../../s15/vm/screen"; +import type { MessageKey } from "../../i18n"; +import type { Copy } from "./copy"; +import { editDraft, emptyDraft, liveDraft, screenInputQueue, screenPin, type ScreenDraft, type ScreenInputQueue, type ScreenPin } from "./screenInput"; + +/** Props for one computer's live screen (DESK16-01). Every fact comes from `s16-computer-screen/v1`. */ +export interface ComputerScreenProps { + client: JitClient; t: Copy; vmId: string; title: string; + onClose(): void; + /** Shared private-content withholding for authentication/authorization failures. */ + onFailure(error: unknown): void; +} +type Command = { kind: "takeover" } | { kind: "hand_back" } | { kind: "renew" } | { kind: "input"; input: VmScreenInput }; +const uid = () => crypto.randomUUID().replaceAll("-", ""); +const code = (error: unknown) => error instanceof JitClientError ? error.code : "client_unavailable"; +const privateFailure = (error: unknown) => error instanceof JitClientError && ([401, 403].includes(error.status ?? 0) || ["authentication_required", "authorization_required", "csrf_rejected"].includes(error.code)); +/** How long a left click waits for a second click before it is sent as a single click (the common desktop default). */ +const doubleClickMs = 500; +/** Browser key names the screen forwards as one named key (everything printable is typed as text). */ +const named: Record = { Enter: "ret", Escape: "esc", Backspace: "backspace", Delete: "delete", Insert: "insert", Home: "home", End: "end", PageUp: "pgup", PageDown: "pgdn", + ArrowUp: "up", ArrowDown: "down", ArrowLeft: "left", ArrowRight: "right", ...Object.fromEntries(Array.from({ length: 12 }, (_, i) => [`F${i + 1}`, `f${i + 1}`])) }; + +/** + * Watch a working computer's screen and, when needed, take exclusive control: the Bot's work pauses, the Bot + * cannot send screen input, and handing back gives the Bot a fresh generation. Programs on the computer still see + * the screen and what the owner types, so this is not a protected place for site passwords. A closed tab or lost + * connection keeps the computer fenced (nobody writes) until the owner takes over again or hands it back. + * Frames stay in memory only; typed text is sent once and cleared from the field. + */ +export function ComputerScreen({ client, t, vmId, title, onClose, onFailure }: ComputerScreenProps) { + const [view, setView] = useState(null), [frame, setFrame] = useState<{ url: string; generation: number } | null>(null); + const [error, setError] = useState(null), [busy, setBusy] = useState(false), [refresh, setRefresh] = useState(0); + const [draft, setDraft] = useState(emptyDraft), [reveal, setReveal] = useState(false); + // Bumped by every command: a poll that started before a command never overwrites the command's fresh view. + const commands = useRef(0); + // A left click waits one double-click interval, so a double click reaches the computer as one `double` click + // (both presses in one QMP batch) instead of two separate round trips the guest may not pair. + const pendingClick = useRef<{ timer: ReturnType; send(): void } | undefined>(undefined); + // Screen input goes out one at a time, in the order the owner made it, each under the takeover it was made in. + const inputQueue = useRef(undefined); + const viewRef = useRef(null), screenRef = useRef(null), headingRef = useRef(null); + viewRef.current = view; + const fail = (failure: unknown) => { if (privateFailure(failure)) { onFailure(failure); return; } const reason = code(failure); if (reason === "stale_screen") setRefresh(value => value + 1); else setError(reason); }; + + // Screen state: quick while control is changing hands, otherwise every few seconds. + useEffect(() => { + const stop = new AbortController(); let timer: ReturnType | undefined; + const load = async () => { + try { const turn = commands.current, next = await client.screen(vmId, stop.signal); if (turn === commands.current) { viewRef.current = next; setView(next); setError(null); } timer = setTimeout(() => void load(), next.state === "settling" ? 1000 : 3000); } + catch (failure) { if (stop.signal.aborted) return; fail(failure); timer = setTimeout(() => void load(), 5000); } + }; + void load(); + return () => { stop.abort(); clearTimeout(timer); }; + }, [vmId, refresh]); + useEffect(() => { headingRef.current?.focus(); }, []); + + // Frames under the generation last seen; a stale frame reloads the state instead of showing an old screen. + const generation = view?.available ? view.generation : null, driving = view?.state === "owner", takeoverId = view?.takeoverId ?? null; + useEffect(() => { + if (generation === null) { setFrame(null); return; } + const stop = new AbortController(); let timer: ReturnType | undefined, url: string | null = null; + const next = async () => { + if (document.visibilityState === "visible") { + try { + const blob = await client.screenFrame(vmId, generation, stop.signal), fresh = URL.createObjectURL(blob); + if (stop.signal.aborted) { URL.revokeObjectURL(fresh); return; } + if (url) URL.revokeObjectURL(url); url = fresh; setFrame({ url: fresh, generation }); + } catch (failure) { if (stop.signal.aborted) return; if (code(failure) === "stale_screen") { setRefresh(value => value + 1); return; } fail(failure); } + } + timer = setTimeout(() => void next(), driving ? 500 : 1500); + }; + void next(); + // The frame is dropped with its URL, so a frame of an earlier generation is never shown as the current screen. + return () => { stop.abort(); clearTimeout(timer); if (url) URL.revokeObjectURL(url); setFrame(null); }; + }, [vmId, generation, driving]); + + // Owner control lasts only while this page renews it. + useEffect(() => { + if (view?.state !== "owner" && view?.state !== "settling") return; + const timer = setInterval(() => void send({ kind: "renew" }, true), 10_000); + return () => clearInterval(timer); + }, [view?.state, view?.generation, view?.takeoverId]); + + /** Sends one command under the control last seen; input is sent under the control it was made in (`pin`). */ + async function send(command: Command, quiet = false, pin?: ScreenPin): Promise { + const seen = viewRef.current; if (!seen) return false; + const current = pin ?? seen; + const base = { contract: "s16-screen-command/v1" as const, requestId: uid(), vmId, generation: current.generation }; + const body: ScreenCommand = command.kind === "takeover" ? { ...base, kind: "takeover" } : command.kind === "hand_back" ? { ...base, kind: "hand_back", takeoverId: current.takeoverId } + : command.kind === "renew" ? { ...base, kind: "renew", takeoverId: current.takeoverId ?? "" } : { ...base, kind: "input", takeoverId: current.takeoverId ?? "", input: command.input }; + if (!quiet) setBusy(true); + commands.current++; + try { const result = await client.screenCommand(body); commands.current++; viewRef.current = result.view; setView(result.view); setError(null); return true; } + catch (failure) { fail(failure); return false; } + finally { if (!quiet) setBusy(false); } + } + const point = (event: MouseEvent | WheelEvent) => { + const box = event.currentTarget.getBoundingClientRect(), width = view?.width ?? 1280, height = view?.height ?? 800; + return { x: Math.min(width - 1, Math.max(0, Math.floor((event.clientX - box.left) * width / box.width))), y: Math.min(height - 1, Math.max(0, Math.floor((event.clientY - box.top) * height / box.height))) }; + }; + function key(event: KeyboardEvent) { + if (!driving || event.key === "Tab") return; + const mods = [event.ctrlKey && "ctrl", event.altKey && "alt", event.metaKey && "meta"].filter(Boolean) as string[]; + const name = named[event.key] ?? (mods.length && /^[a-zA-Z0-9]$/.test(event.key) ? event.key.toLowerCase() : null); + // Shift is part of every forwarded combination (Ctrl+Shift+Z is not Ctrl+Z); alone with a character it is just typing. + if (name) { event.preventDefault(); input({ kind: "key", keys: [...mods, ...(event.shiftKey ? ["shift"] : []), name] }); return; } + if (event.key.length === 1 && !mods.length) { event.preventDefault(); input({ kind: "text", text: event.key }); } + } + // One queue per open panel: closing the panel (or switching computer) drops whatever is still waiting. + // It always sends through this render's `send`, so later props (client, onFailure) are the ones used. + const latestSend = useRef(send); latestSend.current = send; + useEffect(() => { + const queue = screenInputQueue((next, pin) => latestSend.current({ kind: "input", input: next }, false, pin), () => viewRef.current); + inputQueue.current = queue; + return () => { queue.close(); if (inputQueue.current === queue) inputQueue.current = undefined; }; + }, [vmId]); + /** Queues one input after everything queued before it, under the takeover it was made in (`pin`, default: now). */ + function enqueue(next: VmScreenInput, pin = screenPin(viewRef.current)) { inputQueue.current?.push(next, pin); } + /** Sends a waiting single click now (where it was made), so it is never overtaken by later input. */ + function flushClick() { const pending = pendingClick.current; if (!pending) return; clearTimeout(pending.timer); pendingClick.current = undefined; pending.send(); } + /** Every input other than a left click: a waiting click goes first, so typing lands where the owner clicked. */ + function input(next: VmScreenInput) { flushClick(); enqueue(next); } + + // A change of control (hand-back, fencing, a new takeover from any tab) ends what the owner was composing: typed + // text, the "show" choice and a waiting click never carry over to another takeover. + useEffect(() => () => { clearTimeout(pendingClick.current?.timer); pendingClick.current = undefined; setDraft(emptyDraft); setReveal(false); }, [driving, generation, takeoverId]); + // What the field shows and Send sends: only text typed under the control in force now. + const typed = liveDraft(draft, view); + function submitTyped() { + const value = liveDraft(draft, viewRef.current); setDraft(emptyDraft); + // The text goes out under the takeover it was typed in; the queue drops it if that control has ended. + if (value.text) { flushClick(); enqueue({ kind: "text", text: value.text }, value.pin); } + } + function leftClick(event: MouseEvent) { + if (!driving) return; + // The click belongs to the takeover in force now, even if it is sent after the double-click wait. + const at = point(event), pending = pendingClick.current, pin = screenPin(viewRef.current); + if (pending) { clearTimeout(pending.timer); pendingClick.current = undefined; } + // The second press of a browser double click replaces the pending single with one `double` click. + if (event.detail === 2 && pending) { enqueue({ kind: "click", ...at, button: "left", double: true }, pin); return; } + // Any other click first sends the pending single where it was made, so quick clicks at different places all arrive. + pending?.send(); + // The third press of a triple click is not a further press on the computer. + if (event.detail > 2) return; + const single = () => enqueue({ kind: "click", ...at, button: "left", double: false }, pin); + pendingClick.current = { send: single, timer: setTimeout(() => { pendingClick.current = undefined; single(); }, doubleClickMs) }; + } + + const state = view?.state; + return
+
+

{t("s16.screen.title", { computer: title })}

+ {view ? t(`s16.screen.state.${view.state}` as MessageKey) : t("s16.common.loading")} +
+ {view?.available && (state === "agent" || state === "fenced") && } + {(state === "owner" || state === "settling" || state === "fenced") && } + +
+
+
+ {state === "settling" &&

{t("s16.screen.settling")}

} + {state === "owner" &&

{t("s16.screen.driving")}

} + {state === "fenced" &&

{t("s16.screen.fenced")}

} + {view && !view.available &&

{t("s16.screen.unavailable", { reason: view.code ?? "vm_desktop_unavailable" })}

} + {error &&

{t("s16.screen.failed", { reason: error })}

} +
+ {view?.scope.kind === "private" &&

{t("s16.screen.private")}

} +
+ {frame && view?.available && frame.generation === view.generation ? {t("s16.screen.alt", { if (!driving) return; event.preventDefault(); input({ kind: "click", ...point(event), button: "right", double: false }); }} + onWheel={event => { if (driving && event.deltaY) input({ kind: "scroll", ...point(event), delta: Math.max(-5, Math.min(5, Math.round(event.deltaY / 100) || Math.sign(event.deltaY))) }); }} /> + : view?.available &&

{t("s16.screen.waitingFrame")}

} +
+ {driving &&
{ event.preventDefault(); submitTyped(); }}> + + + + {(["ret", "tab", "esc", "backspace"] as const).map(name => )} +

{t("s16.screen.typeHint")}

+
} +
; +} diff --git a/src/s16/web/ComputersView.tsx b/src/s16/web/ComputersView.tsx index c9f753fd..37630610 100644 --- a/src/s16/web/ComputersView.tsx +++ b/src/s16/web/ComputersView.tsx @@ -2,6 +2,7 @@ import { useEffect, useRef, useState } from "react"; import type { Locale, MessageKey } from "../../i18n"; import { JitClientError, type JitClient } from "../client"; import type { ComputerBotView, ComputerCommand, ComputersView as ComputersData, ComputerView } from "../computers/service"; +import { ComputerScreen } from "./ComputerScreen"; import type { Copy } from "./copy"; /** Props for the owner computers surface (TEAM16-01). Data comes only from the protected client. */ @@ -15,6 +16,8 @@ export interface ComputersViewProps { onFailure(error: unknown): void; /** BOT16-01: owner-written names of persistent Bots, keyed by Bot ID. */ names?: Record; + /** DESK16-01: the owner can watch a running computer's screen and take control. */ + screen?: boolean; } type Pending = { command: ComputerCommand; state: "pending" | "unknown" | "error"; code?: string } | { command: ComputerCommand; state: "done" }; type Confirm = { kind: "create"; bot: ComputerBotView } | { kind: "retire"; bot: ComputerBotView; vmId: string }; @@ -28,7 +31,8 @@ const tones: Record = { running: "ok", sleeping: * private-computer creation/retirement and "sleep now". Every change is one explicit command that is never * retried silently; an unknown outcome is rechecked with the same command identity. */ -export function ComputersView({ client, t, locale, available, files, onFailure, names }: ComputersViewProps) { +export function ComputersView({ client, t, locale, available, files, onFailure, names, screen }: ComputersViewProps) { + const [watching, setWatching] = useState(null), watchButton = useRef(null); const [data, setData] = useState(null), [loadError, setLoadError] = useState(null); const [pending, setPending] = useState(null), [confirm, setConfirm] = useState(null), [reload, setReload] = useState(0); // Bumped by every command: a poll that started before a command never overwrites the command's fresh view. @@ -115,9 +119,11 @@ export function ComputersView({ client, t, locale, available, files, onFailure, {status(item) &&

{status(item)}

} {item.reason &&
{t("s16.common.details")}{item.reason}
}
- {item.state === "running" && item.activeSessions === 0 && data.idleStopMs !== null && } + {item.state === "running" && item.activeSessions === 0 && data.idleStopMs !== null && } + {screen && item.state === "running" && watching !== item.id && } {item.state === "setup_failed" && !item.retired && }
+ {screen && watching === item.id && { setWatching(null); requestAnimationFrame(() => { if (watchButton.current?.isConnected) watchButton.current.focus(); }); }} />} )} } ; diff --git a/src/s16/web/app.tsx b/src/s16/web/app.tsx index 9bd21584..e81d1f66 100644 --- a/src/s16/web/app.tsx +++ b/src/s16/web/app.tsx @@ -289,7 +289,7 @@ function App() { {boot ? { void bootstrap(); }} onFailure={hidePrivate} /> : !error &&

{t("s16.common.loading")}

} : computers ?
{error &&

{t(authHidden ? "s16.common.authenticationRequired" : "s16.common.error")}

{t("s16.common.details")}{error}
} - {boot ? !bot.main).map(bot => [bot.id, bot.name]))} available={!!boot.capabilities.computers?.available} files={!!(boot.capabilities.computers?.available && boot.capabilities.computers.files)} onFailure={hidePrivate} /> : !error &&

{t("s16.common.loading")}

} + {boot ? !bot.main).map(bot => [bot.id, bot.name]))} available={!!boot.capabilities.computers?.available} files={!!(boot.capabilities.computers?.available && boot.capabilities.computers.files)} screen={!!(boot.capabilities.computers?.available && boot.capabilities.computers.screen)} onFailure={hidePrivate} /> : !error &&

{t("s16.common.loading")}

}
: routines ?
{error &&

{t(authHidden ? "s16.common.authenticationRequired" : "s16.common.error")}

{t("s16.common.details")}{error}
} {boot ? { if (routinesPath()) history.replaceState(null, "", `/s16/routines${id ? '/' + id : ''}`); }} /> : !error &&

{t("s16.common.loading")}

} diff --git a/src/s16/web/handler.ts b/src/s16/web/handler.ts index 45b78ccf..bf00f33b 100644 --- a/src/s16/web/handler.ts +++ b/src/s16/web/handler.ts @@ -10,7 +10,7 @@ export async function createS16WebHandler(authenticate: (request: Request) => Pr const built = await Bun.build({ entrypoints: [new URL("./app.tsx", import.meta.url).pathname], target: "browser", minify: true, metafile: true, define: { "process.env.NODE_ENV": JSON.stringify("production") } }); if (!built.success || built.outputs.length !== 1 || !built.metafile) throw new AppError("client_build_unavailable", 503); const script = browserLicenseHeader(resolve(import.meta.dir, "../../.."), built.metafile.inputs) + await built.outputs[0]!.text(), style = readFileSync(new URL("./style.css", import.meta.url), "utf8"); - const headers = { "Cache-Control": "private, no-store", "X-Content-Type-Options": "nosniff", "Referrer-Policy": "no-referrer", "Content-Security-Policy": "default-src 'none'; script-src 'self'; style-src 'self'; connect-src 'self'; frame-src 'self'; base-uri 'none'; form-action 'self'; frame-ancestors 'none'", Vary: "Cookie, Accept-Language" }; + const headers = { "Cache-Control": "private, no-store", "X-Content-Type-Options": "nosniff", "Referrer-Policy": "no-referrer", "Content-Security-Policy": "default-src 'none'; script-src 'self'; style-src 'self'; connect-src 'self'; img-src blob:; frame-src 'self'; base-uri 'none'; form-action 'self'; frame-ancestors 'none'", Vary: "Cookie, Accept-Language" }; const worker = s16WorkerScript(); return async request => { const url = new URL(request.url); diff --git a/src/s16/web/screenInput.ts b/src/s16/web/screenInput.ts new file mode 100644 index 00000000..d112b561 --- /dev/null +++ b/src/s16/web/screenInput.ts @@ -0,0 +1,58 @@ +import type { ScreenView } from "../computers/screen"; +import type { VmScreenInput } from "../../s15/vm/screen"; + +/** The owner control one screen input belongs to: the takeover in force when the owner made it. */ +export interface ScreenPin { generation: number; takeoverId: string } + +/** The control the owner holds in `view`, or null when the owner is not driving. */ +export function screenPin(view: ScreenView | null): ScreenPin | null { + return view?.state === "owner" && view.takeoverId ? { generation: view.generation, takeoverId: view.takeoverId } : null; +} + +/** Whether `view` still shows the owner holding the control `pin` was made under. */ +export function holds(view: ScreenView | null, pin: ScreenPin): boolean { + return view?.state === "owner" && view.generation === pin.generation && view.takeoverId === pin.takeoverId; +} + +/** Text the owner is composing in the "type on the computer" field, tied to the control it was typed under. */ +export interface ScreenDraft { text: string; pin: ScreenPin | null } +export const emptyDraft: ScreenDraft = { text: "", pin: null }; + +/** Editing the field: the text belongs to the control in force now (none when the owner is not driving). */ +export function editDraft(view: ScreenView | null, text: string): ScreenDraft { + const pin = screenPin(view); + return pin && text ? { text, pin } : emptyDraft; +} + +/** + * The draft still usable under `view`. Text typed under one takeover is gone after a hand-back, fencing or a + * new takeover (even by another tab), so a later Send can never put it under different control. + */ +export function liveDraft(draft: ScreenDraft, view: ScreenView | null): ScreenDraft { + return draft.pin && holds(view, draft.pin) ? draft : emptyDraft; +} + +/** An ordered queue of the owner's screen input; see `screenInputQueue`. */ +export interface ScreenInputQueue { + /** Queues one input after everything queued before it, under the control it was made in (null: not driving, dropped). */ + push(input: VmScreenInput, pin: ScreenPin | null): void; + /** Drops everything still waiting (the panel closed). */ + close(): void; +} + +/** + * Sends screen input one at a time, in the order the owner made it. Each input carries the generation and + * takeover id it was made under, never the ones current when it leaves the queue, and an input still waiting + * when control changes (hand-back, a new takeover, fencing) or the panel closes is dropped instead of sent. + * So keys typed under one takeover can never act under a later one. + */ +export function screenInputQueue(send: (input: VmScreenInput, pin: ScreenPin) => Promise, current: () => ScreenView | null): ScreenInputQueue { + let tail: Promise = Promise.resolve(), closed = false; + return { + push(input, pin) { + if (!pin || closed) return; + tail = tail.then(() => closed || !holds(current(), pin) ? undefined : send(input, pin)).catch(() => undefined); + }, + close() { closed = true; }, + }; +} diff --git a/src/s16/web/style.css b/src/s16/web/style.css index 7b4750ac..48c9733b 100644 --- a/src/s16/web/style.css +++ b/src/s16/web/style.css @@ -51,3 +51,4 @@ .bot-form input:not([type=checkbox]),.bot-form textarea,.bot-form select{border:1px solid var(--line);border-radius:8px;padding:8px 10px;background:var(--bg)}.bot-form label:not(.check){font-weight:600;margin-top:8px}.bot-form input:focus-visible,.bot-form textarea:focus-visible,.bot-form select:focus-visible{outline:2px solid var(--focus);outline-offset:1px} .connections .tabs{display:flex;gap:4px;border-bottom:1px solid var(--line);margin:20px 0 16px}.connections .tabs button{border:0;border-bottom:3px solid transparent;border-radius:0;background:none}.connections .tabs button[aria-selected=true]{border-bottom-color:var(--accent);font-weight:600}.connection-card h2{margin:0;flex:1;min-width:180px;font-size:18px}.connection-card fieldset{border:1px solid var(--line);border-radius:8px;padding:8px 12px;margin:12px 0;min-width:0}.connection-card .check,.connection-form .check{display:flex;align-items:center;gap:8px;padding:4px 0;font-weight:400}.connection-form{display:flex;flex-direction:column;gap:6px;max-width:640px;margin:12px 0}.connection-form label:not(.check){display:flex;flex-direction:column;gap:4px;font-weight:600;font-size:14px}.connection-form input:not([type=checkbox]):not([type=radio]):not([type=file]),.connection-form textarea,.connection-form select,.connection-review select{width:100%;min-width:0;border:1px solid var(--line);border-radius:8px;padding:8px 10px;background:var(--bg)}.connection-form textarea{min-height:140px;resize:vertical;font:13px/1.5 ui-monospace,monospace}.connection-form fieldset{border:1px solid var(--line);border-radius:8px;padding:8px 12px;min-width:0}.connection-review{border-top:1px solid var(--line);margin-top:12px;padding-top:12px}.connection-review h4{margin:0 0 8px}.tool-list{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:10px}.tool-list li{border:1px solid var(--line);border-radius:10px;padding:10px 12px;display:flex;flex-direction:column;gap:4px}.connections .mono{font:13px ui-monospace,monospace;overflow-wrap:anywhere}.connections .primary{margin-left:0}.connections .actions{display:flex;flex-wrap:wrap;gap:8px;margin-top:8px}.connections .notice.warn{border-left-color:var(--warning)}.connection-request .card-actions{display:flex;flex-wrap:wrap;gap:8px} .primary:hover:not(:disabled){background:var(--accent);box-shadow:inset 0 0 0 999px rgb(0 0 0 / 14%)} +.computer-screen{margin-top:12px;border-top:1px solid var(--line);padding-top:12px}.computer-screen .screen-bar{display:flex;flex-wrap:wrap;align-items:center;gap:8px;margin-bottom:8px}.computer-screen .screen-bar h3{margin:0;flex:1;min-width:180px;font-size:16px}.computer-screen .screen-bar h3:focus{outline:none}.computer-screen .screen-bar .actions{display:flex;flex-wrap:wrap;gap:8px}.computer-screen .screen-bar[data-driver=owner]{border-left:4px solid var(--accent);padding-left:10px}.computer-screen .screen-frame{border:1px solid var(--line);border-radius:8px;overflow:hidden;background:var(--card);min-height:120px}.computer-screen .screen-frame:focus-visible{outline:3px solid var(--accent);outline-offset:2px}.computer-screen .screen-frame img{display:block;width:100%;height:auto;user-select:none}.computer-screen .screen-frame [role=status]{padding:16px;margin:0}.computer-screen .screen-type{display:flex;flex-wrap:wrap;align-items:center;gap:8px;margin-top:10px}.computer-screen .screen-type input[type=password],.computer-screen .screen-type input[type=text]{min-width:200px;max-width:100%}.computer-screen .screen-type .limits{flex-basis:100%;margin:0;font-size:13px;color:var(--muted)} diff --git a/tests/fixtures/s15-qemu.ts b/tests/fixtures/s15-qemu.ts index c1d4fa25..c3d72846 100644 --- a/tests/fixtures/s15-qemu.ts +++ b/tests/fixtures/s15-qemu.ts @@ -41,6 +41,8 @@ export class FakeQemuHost { guestReachable = false; guestMac: string | null = null; qmpDown = false; + /** DESK16-01: bytes QMP `screendump` writes (a 1280×800 PNG header by default). */ + screenBytes: Uint8Array = (() => { const b = new Uint8Array(64); b.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]); const v = new DataView(b.buffer); v.setUint32(8, 13); b.set([73, 72, 68, 82], 12); v.setUint32(16, 1280); v.setUint32(20, 800); return b; })(); /** Guest logins refused (pam_nologin) for this many SSH attempts after each boot. */ bootingLogins = 0; #refused = 0; @@ -63,6 +65,9 @@ export class FakeQemuHost { if (command.execute === 'blockdev-snapshot-internal-sync') { disk!.snapshots.push(String(command.arguments!.name)); return {}; } if (command.execute === 'blockdev-snapshot-delete-internal-sync') { disk!.snapshots = disk!.snapshots.filter(n => n !== command.arguments!.name); return {}; } if (command.execute === 'quit') { if (!this.stuck) this.kill(unit[0]); return {}; } + // QEMU writes the frame itself; a VM without a display device has no console surface to dump. + if (command.execute === 'screendump') { if (!argv.some(a => a.startsWith('virtio-vga') || a.startsWith('virtio-gpu-pci'))) throw new Error('vm_management_unavailable'); writeFileSync(String(command.arguments!.filename), this.screenBytes); return {}; } + if (command.execute === 'input-send-event') return {}; throw new Error('unsupported'); }); }; diff --git a/tests/s15-vm-vmm.test.ts b/tests/s15-vm-vmm.test.ts index 66b4f746..10638f8a 100644 --- a/tests/s15-vm-vmm.test.ts +++ b/tests/s15-vm-vmm.test.ts @@ -75,14 +75,23 @@ describe('the argv-fixed launcher', () => { test.each(['x86_64', 'aarch64'] as const)('start builds the restricted QEMU argv byte-for-byte as qemuArgv for %s', async architecture => { const { l, run } = launcher(), dir = machine(l); - mkdirSync(dir); writeFileSync(join(dir, 'qmp.sock'), 'stale'); + mkdirSync(dir); writeFileSync(join(dir, 'qmp.sock'), 'stale'); writeFileSync(join(dir, 'screen.png'), 'stale frame'); const result = await run(['start', identity.name, architecture, '5', '9216', identity.mac, '22022']); const spec = { name: identity.name, architecture, machineDirectory: dir, cpus: 5, memoryMiB: 9216, mac: identity.mac, sshPort: 22022, proxySocket: l.proxySocket }; expect(result.code).toBe(0); expect(result.recorded).toEqual(['systemd-run', '--user', '--quiet', '--collect', `--unit=${identity.name}.service`, '--property=Type=exec', '--property=TimeoutStopSec=15', '--property=UMask=0007', '--', ...qemuArgv(spec)]); expect(qemuArgvFacts(result.recorded!.slice(9), spec).exact).toBe(true); - // The stale QMP socket of the previous boot is removed by the QEMU identity itself. - expect(existsSync(join(dir, 'qmp.sock'))).toBe(false); + // The stale QMP socket and any screen frame of the previous boot are removed by the QEMU identity itself. + expect(existsSync(join(dir, 'qmp.sock'))).toBe(false); expect(existsSync(join(dir, 'screen.png'))).toBe(false); + }); + + test.each(['x86_64', 'aarch64'] as const)('start with the desktop word builds the desktop QEMU argv byte-for-byte as qemuArgv for %s', async architecture => { + const { l, run } = launcher(), dir = machine(l); + mkdirSync(dir); + const result = await run(['start', identity.name, architecture, '5', '9216', identity.mac, '22022', 'desktop']); + const spec = { name: identity.name, architecture, machineDirectory: dir, cpus: 5, memoryMiB: 9216, mac: identity.mac, sshPort: 22022, proxySocket: l.proxySocket, desktop: true }; + expect(result.code).toBe(0); + expect(result.recorded).toEqual(['systemd-run', '--user', '--quiet', '--collect', `--unit=${identity.name}.service`, '--property=Type=exec', '--property=TimeoutStopSec=15', '--property=UMask=0007', '--', ...qemuArgv(spec)]); }); test('seed stores stdin as the VM seed; remove deletes only that VM directory', async () => { @@ -97,6 +106,17 @@ describe('the argv-fixed launcher', () => { expect(existsSync(dir)).toBe(false); expect(readFileSync(join(other, 'disk.qcow2'), 'utf8')).toBe('other VM'); }); + test('discard-frame removes only that VM\'s screen frame and runs no tool', async () => { + const { l, run } = launcher(), dir = machine(l), other = join(l.machines, 'jit-work-ffffffffffff'); + mkdirSync(dir); mkdirSync(other); + for (const d of [dir, other]) { writeFileSync(join(d, 'screen.png'), 'frame'); writeFileSync(join(d, 'disk.qcow2'), 'disk'); } + expect(await run(['discard-frame', identity.name])).toMatchObject({ code: 0, recorded: null }); + expect(readdirSync(dir)).toEqual(['disk.qcow2']); + expect(readdirSync(other).sort()).toEqual(['disk.qcow2', 'screen.png']); + // Idempotent: nothing to remove is still success. + expect(await run(['discard-frame', identity.name])).toMatchObject({ code: 0, recorded: null }); + }); + test('every malformed operation, name, value or extra argument is refused before any command runs', async () => { const { run } = launcher(), n = identity.name; const refused = [[], ['bogus', n], ['show'], ['show', n, 'extra'], ['show', 'jit-work-XYZ'], ['show', '../etc'], ['show', `${n}/..`], ['show', 'jit-work-0123456789abc'], @@ -104,7 +124,7 @@ describe('the argv-fixed launcher', () => { ['create', n, 'AB'.repeat(32), '60'], ['create', n, sha], ['create', n, `${sha}/../x`, '60'], ['snapshot', n, 'create', 'other'], ['snapshot', n, 'revert', tag], ['start', n, 'riscv64', '1', '2048', identity.mac, '22022'], ['start', n, 'x86_64', '0', '2048', identity.mac, '22022'], ['start', n, 'x86_64', '1000', '2048', identity.mac, '22022'], ['start', n, 'x86_64', '1', '2048', 'aa:bb:cc:dd:ee:ff', '22022'], ['start', n, 'x86_64', '1', '2048', identity.mac, '80'], ['start', n, 'x86_64', '1', '2048', identity.mac, '70000'], - ['start', n, 'x86_64', '1', '2048 -device foo', identity.mac, '22022'], ['start', n, 'x86_64', '1', '2048', identity.mac], ['kvm', n], ['check', n], ['remove', n, '/']]; + ['start', n, 'x86_64', '1', '2048 -device foo', identity.mac, '22022'], ['start', n, 'x86_64', '1', '2048', identity.mac], ['start', n, 'x86_64', '1', '2048', identity.mac, '22022', 'gui'], ['start', n, 'x86_64', '1', '2048', identity.mac, '22022', 'desktop', 'vnc'], ['kvm', n], ['check', n], ['remove', n, '/'], ['discard-frame', n, 'screen.png'], ['discard-frame', '../etc']]; for (const args of refused) { const result = await run(args); expect({ args, code: result.code, recorded: result.recorded }).toEqual({ args, code: 64, recorded: null }); @@ -128,7 +148,7 @@ describe('the argv-fixed launcher', () => { }); describe('QEMU as the separate identity', () => { - function vmmHost(options: { wrongUid?: boolean } = {}) { + function vmmHost(options: { wrongUid?: boolean; desktop?: boolean; discardFails?: boolean } = {}) { const h = new FakeQemuHost(); hosts.push(h); const l = layout(h.root), vmm: QemuVmmOptions = { user: 'jit-test-vmm', uid: 4242, directory: l.directory, launcher: '/opt/jit-s15/libexec/jit-s15-vmm' }; const prefix = vmmCommand(vmm, []), launched: string[][] = [], direct: string[][] = []; @@ -149,14 +169,15 @@ describe('QEMU as the separate identity', () => { if (op === 'snapshot') return h.run(['/usr/bin/qemu-img', 'snapshot', rest[0] === 'create' ? '-c' : rest[0] === 'apply' ? '-a' : '-d', rest[1]!, disk], settings); if (op === 'seed') { mkdirSync(dir, { recursive: true }); writeFileSync(join(dir, 'seed.img'), settings.input!); return { exitCode: 0, stdout: Buffer.alloc(0) }; } if (op === 'remove') { rmSync(dir, { recursive: true, force: true }); return { exitCode: 0, stdout: Buffer.alloc(0) }; } + if (op === 'discard-frame') { if (options.discardFails) return { exitCode: 1, stdout: Buffer.alloc(0) }; rmSync(join(dir, 'screen.png'), { force: true }); return { exitCode: 0, stdout: Buffer.alloc(0) }; } if (op === 'start') { - const spec = { name, architecture: rest[0] as 'x86_64', machineDirectory: dir, cpus: Number(rest[1]), memoryMiB: Number(rest[2]), mac: rest[3]!, sshPort: Number(rest[4]), proxySocket: l.proxySocket }; + const spec = { name, architecture: rest[0] as 'x86_64', machineDirectory: dir, cpus: Number(rest[1]), memoryMiB: Number(rest[2]), mac: rest[3]!, sshPort: Number(rest[4]), proxySocket: l.proxySocket, ...(rest[5] === 'desktop' ? { desktop: true } : {}) }; return h.run(['systemd-run', '--user', '--quiet', '--collect', `--unit=${unit}`, '--property=Type=exec', '--property=TimeoutStopSec=15', '--property=UMask=0007', '--', ...qemuArgv(spec)], settings); } return { exitCode: 64, stdout: Buffer.alloc(0) }; }; const proc = { ...h.proc, uid: (pid: number) => h.processes.has(pid) ? options.wrongUid ? process.getuid!() : vmm.uid : null }; - const host = { ...h.options(l.proxySocket), run, proc, vmm }; + const host = { ...h.options(l.proxySocket), run, proc, vmm, ...(options.desktop ? { desktop: true } : {}) }; return { h, l, vmm, host, launched, direct, m: new QemuVmManagement(identity, host) }; } @@ -209,6 +230,39 @@ describe('QEMU as the separate identity', () => { expect(launched.filter(c => c[0] === 'snapshot').map(c => c[2])).toEqual(['create', 'apply', 'delete']); }); + test('DESK16-01: no screen frame stays in the QEMU identity directory after a capture or a stop', async () => { + const { h, m, launched } = vmmHost({ desktop: true }); + await m.provision(); + const frame = join(m.hypervisorDirectory, 'screen.png'), screen = m.screen(); + expect(await screen.capture()).toEqual(h.screenBytes); + expect(existsSync(frame)).toBe(false); + expect(launched.filter(c => c[0] === 'discard-frame')).toEqual([['discard-frame', identity.name]]); + // A capture that fails after QEMU wrote the file still removes it. + h.screenBytes = new Uint8Array(64).fill(1); + await expect(screen.capture()).rejects.toMatchObject({ code: 'vm_screen_invalid' }); + expect(existsSync(frame)).toBe(false); + // A stop (also the idle stop) removes a frame left by anything else, e.g. a controller crash mid-capture. + writeFileSync(frame, 'left behind'); + await m.stop(); expect(await m.inspect()).toMatchObject({ state: 'stopped' }); + expect(existsSync(frame)).toBe(false); + expect(readdirSync(m.hypervisorDirectory).sort()).toEqual(['disk.qcow2', 'seed.img']); + }); + + test('DESK16-01: a frame that cannot be removed fails the capture instead of being shown', async () => { + const { m } = vmmHost({ desktop: true, discardFails: true }); + await m.provision(); + await expect(m.screen().capture()).rejects.toMatchObject({ code: 'vm_management_unavailable' }); + }); + + test('DESK16-01: a headless controller never asks for discard-frame, so a launcher without it still stops', async () => { + // discardFails stands in for an older launcher that refuses the op. + const { m, launched } = vmmHost({ discardFails: true }); + await m.provision(); + expect(await m.inspect()).toMatchObject({ state: 'running' }); + await m.stop(); expect(await m.inspect()).toMatchObject({ state: 'stopped' }); + expect(launched.some(c => c[0] === 'discard-frame')).toBe(false); + }); + test('a QEMU process that is not owned by the separate identity is never this VM', async () => { const { m } = vmmHost({ wrongUid: true }); await expect(m.provision()).rejects.toMatchObject({ code: 'vm_management_unavailable' }); diff --git a/tests/s16-computer-screen.browser.test.ts b/tests/s16-computer-screen.browser.test.ts new file mode 100644 index 00000000..48175e0f --- /dev/null +++ b/tests/s16-computer-screen.browser.test.ts @@ -0,0 +1,99 @@ +import { afterAll, beforeAll, expect, test } from "bun:test"; +import { chromium, type Browser } from "playwright-core"; +import { composedFixture } from "./fixtures/s15-composition"; +import type { VmScreenDevice, VmScreenInput } from "../src/s15/vm/screen"; +import { en } from "../src/i18n/en"; +import { th } from "../src/i18n/th"; + +// DESK16-01 real React bundle: watch a computer's screen, take over, click and type (the field clears), send a key +// from the keyboard-capture region and hand back, in English (desktop) and Thai (mobile). Opt-in like the other S16 +// browser suites. +const browserTest = process.env.JIT_S16_BROWSER_TESTS === "1" ? test : test.skip; +let browser: Browser; +beforeAll(async () => { if (process.env.JIT_S16_BROWSER_TESTS === "1") browser = await chromium.launch({ executablePath: process.env.JIT_BROWSER_EXECUTABLE || "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", headless: true }); }); +afterAll(async () => { await browser?.close(); }, 30000); +/** Waits (bounded) until the scripted device recorded `count` inputs. */ +async function inputs(sent: unknown[], count: number): Promise { for (let i = 0; i < 100 && sent.length < count; i++) await Bun.sleep(50); expect(sent).toHaveLength(count); } + +/** A real 1280×800 PNG (solid colour) so the browser decodes and lays out an actual frame. */ +async function frame(): Promise { + const page = await browser.newPage(); + try { + const data = await page.evaluate(() => { const c = document.createElement("canvas"); c.width = 1280; c.height = 800; const g = c.getContext("2d")!; g.fillStyle = "#2b6cb0"; g.fillRect(0, 0, 1280, 800); return c.toDataURL("image/png"); }); + return new Uint8Array(Buffer.from(data.split(",")[1]!, "base64")); + } finally { await page.close(); } +} + +browserTest.each([1440, 390])("actual S16 computer screen: watch, take over, click, type and hand back at width %i", async width => { + const png = await frame(), sent: VmScreenInput[] = []; + const device: VmScreenDevice = { async unavailable() { return null; }, async capture() { return png; }, async send(input) { sent.push(input); } }; + const f = await composedFixture("default", false, undefined, o => ({ ...o, computers: { idleStopMs: 30 * 60_000, sweepMs: 0 }, screen: {}, + vm: { ...o.vm, managers: new Map([...o.vm.managers].map(([id, manager]) => [id, Object.assign(manager, { screen: () => device })])) } })); + const copy = width === 1440 ? en : th, locale = width === 1440 ? "en-US" : "th-TH"; + const context = await browser.newContext({ viewport: { width, height: 900 }, locale, timezoneId: "Asia/Bangkok" }); + const errors: string[] = [], external: string[] = []; + await context.route("**/*", async route => { + const request = route.request(), url = new URL(request.url()); if (url.origin !== "http://localhost:3099") { if (!url.protocol.startsWith("blob")) external.push(request.url()); await route.abort(); return; } + const headers = { ...request.headers(), host: "localhost:3099" }; + const response = request.method() === "POST" ? await f.request(url.pathname + url.search, "POST", request.postDataJSON(), headers) : await f.request(url.pathname + url.search, request.method(), undefined, headers); + await route.fulfill({ status: response.status, headers: Object.fromEntries(response.headers), body: Buffer.from(await response.arrayBuffer()) }); + }); + const page = await context.newPage(); page.on("pageerror", error => errors.push(error.message)); + const noOverflow = () => page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth); + const title = copy["s16.screen.title"].replace("{computer}", copy["s16.computers.shared"]); + try { + await page.goto("http://localhost:3099/s16/computers"); + await page.getByRole("button", { name: copy["s16.screen.watch"] }).click(); + const panel = page.getByRole("region", { name: title }); + await panel.getByRole("heading", { name: title }).waitFor(); + await panel.getByText(copy["s16.screen.state.agent"], { exact: true }).waitFor(); + const image = panel.getByRole("img", { name: copy["s16.screen.alt"].replace("{computer}", copy["s16.computers.shared"]) }); + await image.waitFor(); + expect(await image.evaluate(element => (element as HTMLImageElement).naturalWidth)).toBe(1280); + // Watching sends nothing to the computer. + await image.click(); expect(sent).toEqual([]); + await panel.getByRole("button", { name: copy["s16.screen.takeOver"] }).click(); + await panel.getByText(copy["s16.screen.driving"], { exact: true }).waitFor(); + await panel.getByText(copy["s16.screen.state.owner"], { exact: true }).waitFor(); + const box = (await image.boundingBox())!; + await image.click({ position: { x: box.width / 2, y: box.height / 4 } }); + await inputs(sent, 1); + expect(sent[0]).toMatchObject({ kind: "click", button: "left", double: false }); + const click = sent[0] as Extract; + expect(Math.abs(click.x - 640)).toBeLessThanOrEqual(2); expect(Math.abs(click.y - 200)).toBeLessThanOrEqual(2); + // A double click is one double click on the computer, never two singles or three presses. + await image.dblclick({ position: { x: box.width / 2, y: box.height / 4 } }); + await inputs(sent, 2); await Bun.sleep(700); + expect(sent).toHaveLength(2); expect(sent[1]).toMatchObject({ kind: "click", button: "left", double: true }); + // Two quick single clicks at different places both arrive, each where it was made. + await image.click({ position: { x: box.width / 4, y: box.height / 2 } }); + await image.click({ position: { x: (box.width * 3) / 4, y: box.height / 2 } }); + await inputs(sent, 4); + const [first, second] = sent.slice(2, 4) as Extract[]; + expect(first).toMatchObject({ kind: "click", button: "left", double: false }); expect(second).toMatchObject({ kind: "click", button: "left", double: false }); + expect(Math.abs(first!.x - 320)).toBeLessThanOrEqual(2); expect(Math.abs(second!.x - 960)).toBeLessThanOrEqual(2); + const field = panel.getByLabel(copy["s16.screen.typeLabel"]); + expect(await field.getAttribute("type")).toBe("password"); + await field.fill("Owner-Pass1"); await panel.getByRole("button", { name: copy["s16.screen.send"], exact: true }).click(); + await inputs(sent, 5); + expect(sent[4]).toEqual({ kind: "text", text: "Owner-Pass1" }); expect(await field.inputValue()).toBe(""); + // Keyboard capture: Enter on the focused screen region reaches the computer as one key. + await panel.locator(".screen-frame").focus(); await page.keyboard.press("Enter"); + await inputs(sent, 6); + expect(sent[5]).toEqual({ kind: "key", keys: ["ret"] }); + // A key pressed right after a click arrives after that click, so typing lands where the owner clicked. + await image.click({ position: { x: box.width / 2, y: box.height / 2 } }); + await panel.locator(".screen-frame").focus(); await page.keyboard.press("Escape"); + await inputs(sent, 8); + expect(sent[6]).toMatchObject({ kind: "click", button: "left", double: false }); expect(sent[7]).toEqual({ kind: "key", keys: ["esc"] }); + expect(await noOverflow()).toBe(true); + await panel.getByRole("button", { name: copy["s16.screen.handBack"] }).click(); + await panel.getByText(copy["s16.screen.state.agent"], { exact: true }).waitFor(); + await panel.getByRole("button", { name: copy["s16.screen.takeOver"] }).waitFor(); + await panel.getByRole("button", { name: copy["s16.screen.close"] }).click(); + await page.getByRole("button", { name: copy["s16.screen.watch"] }).waitFor(); + expect(await page.evaluate(() => document.activeElement?.textContent)).toBe(copy["s16.screen.watch"]); + expect(sent).toHaveLength(8); + expect(errors).toEqual([]); expect(external).toEqual([]); + } finally { await context.close(); await f.close(); } +}, 60000); diff --git a/tests/s16-computer-screen.test.ts b/tests/s16-computer-screen.test.ts new file mode 100644 index 00000000..8743805c --- /dev/null +++ b/tests/s16-computer-screen.test.ts @@ -0,0 +1,346 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { existsSync } from "node:fs"; +import { join } from "node:path"; +import { QemuVmManagement, qemuArgv, qemuArgvFacts } from "../src/s15/vm/qemu"; +import { FakeQemuHost } from "./fixtures/s15-qemu"; +import { composedFixture } from "./fixtures/s15-composition"; +import { dictionaries } from "../src/i18n"; +import { vmScreenCommands, vmScreenFrame, vmScreenInput, type VmScreenDevice, type VmScreenInput } from "../src/s15/vm/screen"; +import { WorkingVmService } from "../src/s15/vm/service"; +import { VmStore } from "../src/s15/vm/store"; +import { VmSshTransport } from "../src/s15/vm/transport"; +import type { VmManagement, VmPolicyPort } from "../src/s15/vm/types"; +import { ScreenControl, parseScreenCommand, type ScreenCommand } from "../src/s16/computers/screen"; + +const stores: VmStore[] = [], hosts: FakeQemuHost[] = []; +afterEach(() => { for (const store of stores.splice(0)) store.close(); for (const h of hosts.splice(0)) h.cleanup(); }); +const endpoint = { host: "vm.example.test", port: 22, user: "jit", identityFile: "/protected/key", knownHostsFile: "/protected/known" }; +const enroll = (store: VmStore, id = "vm1") => store.save({ id, generation: 1, endpoint, managerId: "mgr", state: "ready", networkCheckedAt: 1, networkIsolated: true, maxSessions: 2, reason: null, machineIdentity: "fixture-vm", boot: "boot1", bindingMac: "52:54:00:12:34:56" }); + +/** A PNG header of the given size; the screen contract checks the signature and IHDR dimensions. */ +function png(width = 1280, height = 800): Uint8Array { + const bytes = new Uint8Array(64); bytes.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]); + const view = new DataView(bytes.buffer); view.setUint32(8, 13); bytes.set([..."IHDR"].map(c => c.charCodeAt(0)), 12); view.setUint32(16, width); view.setUint32(20, height); + return bytes; +} +/** Scripted QMP-side screen: records every input; capture and send can be held at a barrier. */ +function device() { + const sent: VmScreenInput[] = []; let unavailable: string | null = null; + let captureGate: Promise | null = null, sendGate: Promise | null = null; + const value: VmScreenDevice = { + async unavailable() { return unavailable; }, + async capture() { await captureGate; return png(); }, + async send(input) { await sendGate; sent.push(input); }, + }; + return { value, sent, setUnavailable: (code: string | null) => { unavailable = code; }, holdCapture: (gate: Promise | null) => { captureGate = gate; }, holdSend: (gate: Promise | null) => { sendGate = gate; } }; +} +function control(store = new VmStore(":memory:"), screen = device(), clock = { now: 1000 }) { + if (!stores.includes(store)) { stores.push(store); enroll(store); } + const owner = { owner: true }; + const value = new ScreenControl({ store, requireOwner: actor => { if (actor !== owner) throw new Error("owner required"); }, device: () => screen.value, scope: () => ({ kind: "shared" }), now: () => clock.now, leaseMs: 30_000 }); + let n = 0; const id = () => `req_${++n}_${Math.random().toString(36).slice(2, 8)}`; + const command = (body: Record) => value.command(owner, { contract: "s16-screen-command/v1", requestId: id(), vmId: "vm1", ...body }); + return { store, screen, clock, owner, control: value, command }; +} +const text = (value: string): VmScreenInput => ({ kind: "text", text: value }); + +describe("desktop VM profile", () => { + const spec = { name: "jit-work-0123456789ab", architecture: "x86_64" as const, machineDirectory: "/srv/jit/machines/jit-work-0123456789ab", cpus: 4, memoryMiB: 8192, mac: "52:54:00:12:34:56", sshPort: 22022, proxySocket: "/srv/jit/proxy.sock" }; + test("the headless argv is unchanged and the desktop profile adds only a virtio display, tablet and keyboard", () => { + const headless = qemuArgv(spec), desktop = qemuArgv({ ...spec, desktop: true }); + expect(headless.some(arg => /vga|gpu|tablet|keyboard|vnc|spice/.test(arg))).toBe(false); + expect(desktop.slice(0, headless.length)).toEqual(headless); + expect(desktop.slice(headless.length)).toEqual(["-device", "virtio-vga,xres=1280,yres=800", "-device", "virtio-tablet-pci", "-device", "virtio-keyboard-pci"]); + expect(desktop[desktop.indexOf("-display") + 1]).toBe("none"); + expect(qemuArgv({ ...spec, architecture: "aarch64", desktop: true })).toContain("virtio-gpu-pci,xres=1280,yres=800"); + }); + test("both profiles re-verify as exact; only the desktop profile reports a screen; anything else is not exact", () => { + expect(qemuArgvFacts(qemuArgv(spec), spec)).toMatchObject({ exact: true }); expect(qemuArgvFacts(qemuArgv(spec), spec).desktop).toBeUndefined(); + expect(qemuArgvFacts(qemuArgv({ ...spec, desktop: true }), spec)).toMatchObject({ exact: true, desktop: true }); + expect(qemuArgvFacts([...qemuArgv(spec), "-vnc", ":0"], spec).exact).toBe(false); + expect(qemuArgvFacts([...qemuArgv({ ...spec, desktop: true }), "-device", "virtio-tablet-pci"], spec)).toMatchObject({ exact: false }); + }); +}); + +describe("host-side screen of a local QEMU VM", () => { + const identity = { name: "jit-work-0123456789ab", architecture: "x86_64" as const, mac: "52:54:00:12:34:56", sshPort: 22022, resources: { cpus: 5, memoryMiB: 9216, diskGiB: 60 } }; + test("a desktop VM serves PNG frames and input through QMP only; the frame file does not stay behind", async () => { + const h = new FakeQemuHost(); hosts.push(h); + const m = new QemuVmManagement(identity, { ...h.options(), desktop: true }); await m.provision(); + expect(h.launches.at(-1)).toContain("virtio-tablet-pci"); + const screen = m.screen(); + expect(await screen.unavailable()).toBeNull(); + expect(await screen.capture()).toEqual(h.screenBytes); + expect(existsSync(join(m.hypervisorDirectory, "screen.png"))).toBe(false); + const before = h.qmpLog.length; await screen.send({ kind: "key", keys: ["ctrl", "l"] }); + expect(h.qmpLog.slice(before).map(c => c.execute)).toEqual(["input-send-event"]); + h.screenBytes = new Uint8Array(64).fill(1); + await expect(screen.capture()).rejects.toMatchObject({ code: "vm_screen_invalid" }); + await m.stop(); expect(await screen.unavailable()).toBe("vm_not_running"); + await expect(screen.send({ kind: "text", text: "x" })).rejects.toMatchObject({ code: "vm_not_running" }); + }); + test("a VM still running headless needs a restart before it has a screen; without the opt-in there is none", async () => { + const h = new FakeQemuHost(); hosts.push(h); + const headless = new QemuVmManagement(identity, h.options()); await headless.provision(); + expect(await headless.screen().unavailable()).toBe("vm_desktop_unavailable"); + const enabled = new QemuVmManagement(identity, { ...h.options(), desktop: true }); + expect((await enabled.inspect()).network).toBe("proxy-only"); + expect(await enabled.screen().unavailable()).toBe("vm_desktop_restart_required"); + await expect(enabled.screen().capture()).rejects.toMatchObject({ code: "vm_desktop_restart_required" }); + }); +}); + +describe("screen input and frames", () => { + test("input shapes are exact and bounded; text outside a US keyboard is refused, not approximated", () => { + expect(vmScreenInput({ kind: "click", x: 0, y: 799, button: "left", double: false })).toBeTruthy(); + for (const bad of [{ kind: "click", x: 1280, y: 0, button: "left", double: false }, { kind: "click", x: 1, y: 1, button: "left" }, { kind: "key", keys: ["ctrl", "ctrl"] }, + { kind: "key", keys: ["0xff0d"] }, { kind: "scroll", x: 1, y: 1, delta: 0 }, { kind: "text", text: "" }, { kind: "text", text: "a".repeat(513) }, { kind: "drag" }]) + expect(() => vmScreenInput(bad)).toThrow(expect.objectContaining({ code: "invalid_screen_input" })); + expect(() => vmScreenInput(text("สวัสดี"))).toThrow(expect.objectContaining({ code: "screen_text_unsupported" })); + }); + test("text maps to press/release qcodes with shift; a combination releases in reverse; a click moves first", () => { + const events = vmScreenCommands(text("A!")).map(c => (c.arguments as { events: { data: { down: boolean; key: { data: string } } }[] }).events.map(e => `${e.data.down ? "+" : "-"}${e.data.key.data}`)); + expect(events).toEqual([["+shift", "+a", "-a", "-shift"], ["+shift", "+1", "-1", "-shift"]]); + const combo = vmScreenCommands({ kind: "key", keys: ["ctrl", "l"] })[0]!.arguments as { events: { data: { down: boolean; key: { data: string } } }[] }; + expect(combo.events.map(e => `${e.data.down ? "+" : "-"}${e.data.key.data}`)).toEqual(["+ctrl", "+l", "-l", "-ctrl"]); + const click = vmScreenCommands({ kind: "click", x: 1279, y: 0, button: "left", double: true }); + expect(click).toHaveLength(3); expect(click.every(c => c.execute === "input-send-event")).toBe(true); + expect((click[0]!.arguments as { events: { data: { value: number } }[] }).events.map(e => e.data.value)).toEqual([32767, 0]); + }); + test("a frame must be a bounded PNG of the fixed display size", () => { + expect(vmScreenFrame(png())).toBeTruthy(); + expect(() => vmScreenFrame(png(1024, 768))).toThrow(expect.objectContaining({ code: "vm_screen_invalid" })); + expect(() => vmScreenFrame(new TextEncoder().encode("".padEnd(64)))).toThrow(expect.objectContaining({ code: "vm_screen_invalid" })); + }); +}); + +describe("exclusive owner control", () => { + test("takeover waits for work admitted before it; work and owner never hold the same writable generation", async () => { + const f = control(), ticket = f.control.agentInput("vm1"); + let view = (await f.command({ kind: "takeover", generation: 0 })).view; + expect(view).toMatchObject({ state: "settling", generation: 1, agentBusy: true }); + // The admitted operation can no longer dispatch or publish, and no new work is admitted. + expect(() => ticket.current()).toThrow(expect.objectContaining({ code: "takeover_active" })); + expect(() => f.control.agentInput("vm1")).toThrow(expect.objectContaining({ code: "takeover_active" })); + await expect(f.command({ kind: "input", generation: 1, takeoverId: view.takeoverId, input: text("a") })).rejects.toMatchObject({ code: "takeover_settling" }); + expect(f.screen.sent).toEqual([]); + ticket.end(); ticket.end(); + view = await f.control.view(f.owner, "vm1"); + expect(view).toMatchObject({ state: "owner", generation: 1, agentBusy: false }); + await f.command({ kind: "input", generation: 1, takeoverId: view.takeoverId, input: text("a") }); + expect(f.screen.sent).toEqual([text("a")]); + }); + + test("old frames and input after hand-back or a second takeover are refused", async () => { + const f = control(); + const taken = (await f.command({ kind: "takeover", generation: 0 })).view; + await expect(f.control.frame(f.owner, "vm1", 0)).rejects.toMatchObject({ code: "stale_screen" }); + expect(await f.control.frame(f.owner, "vm1", 1)).toEqual(png()); + const back = (await f.command({ kind: "hand_back", generation: 1, takeoverId: taken.takeoverId })).view; + expect(back).toMatchObject({ state: "agent", generation: 2, takeoverId: null }); + await expect(f.command({ kind: "input", generation: 1, takeoverId: taken.takeoverId, input: text("x") })).rejects.toMatchObject({ code: "takeover_state_changed" }); + const again = (await f.command({ kind: "takeover", generation: 2 })).view; + await expect(f.command({ kind: "input", generation: 3, takeoverId: taken.takeoverId, input: text("x") })).rejects.toMatchObject({ code: "stale_screen" }); + await expect(f.command({ kind: "takeover", generation: 2 })).rejects.toMatchObject({ code: "takeover_state_changed" }); + expect(again.generation).toBe(3); expect(f.screen.sent).toEqual([]); + }); + + test("a frame captured across a transition is dropped instead of shown as current", async () => { + const f = control(), gate = Promise.withResolvers(); f.screen.holdCapture(gate.promise); + const frame = f.control.frame(f.owner, "vm1", 0); + await f.command({ kind: "takeover", generation: 0 }); gate.resolve(); + await expect(frame).rejects.toMatchObject({ code: "stale_screen" }); + }); + + test("hand-back waits for the owner's input being sent before work may write again", async () => { + const f = control(), gate = Promise.withResolvers(); + const taken = (await f.command({ kind: "takeover", generation: 0 })).view; + f.screen.holdSend(gate.promise); + const typing = f.command({ kind: "input", generation: 1, takeoverId: taken.takeoverId, input: text("pw") }); + await Bun.sleep(1); // The input is now being sent. + const handing = f.command({ kind: "hand_back", generation: 1, takeoverId: taken.takeoverId }); + await Bun.sleep(5); + // Still the owner's generation: work is refused and further owner input is refused as closing. + expect(() => f.control.agentInput("vm1")).toThrow(expect.objectContaining({ code: "takeover_active" })); + await expect(f.command({ kind: "input", generation: 1, takeoverId: taken.takeoverId, input: text("z") })).rejects.toMatchObject({ code: "stale_screen" }); + gate.resolve(); await typing; expect((await handing).view.state).toBe("agent"); + // An input queued behind a hand-back is refused rather than sent after it. + const late = f.command({ kind: "input", generation: 1, takeoverId: taken.takeoverId, input: text("late") }); + await expect(late).rejects.toMatchObject({ code: "takeover_state_changed" }); + f.control.agentInput("vm1").end(); + expect(f.screen.sent).toEqual([text("pw")]); + }); + + test("an input still queued when hand-back starts is never sent", async () => { + const f = control(), taken = (await f.command({ kind: "takeover", generation: 0 })).view; + const queued = f.command({ kind: "input", generation: 1, takeoverId: taken.takeoverId, input: text("q") }); + const handing = f.command({ kind: "hand_back", generation: 1, takeoverId: taken.takeoverId }); + await expect(queued).rejects.toMatchObject({ code: "stale_screen" }); + expect((await handing).view.state).toBe("agent"); expect(f.screen.sent).toEqual([]); + }); + + test("a closed tab during login keeps the computer fenced until the owner decides", async () => { + const f = control(); + const taken = (await f.command({ kind: "takeover", generation: 0 })).view; + f.clock.now += 29_000; await f.command({ kind: "renew", generation: 1, takeoverId: taken.takeoverId }); + f.clock.now += 29_000; expect((await f.control.view(f.owner, "vm1")).state).toBe("owner"); + f.clock.now += 2_000; + const lapsed = await f.control.view(f.owner, "vm1"); + expect(lapsed).toMatchObject({ state: "fenced", generation: 2, takeoverId: null, expiresAt: null }); + expect(() => f.control.agentInput("vm1")).toThrow(expect.objectContaining({ code: "takeover_active" })); + expect(f.control.idleBlocked("vm1")).toBe(true); + await expect(f.command({ kind: "renew", generation: 1, takeoverId: taken.takeoverId })).rejects.toMatchObject({ code: "screen_control_lost" }); + await expect(f.command({ kind: "input", generation: 2, takeoverId: taken.takeoverId, input: text("x") })).rejects.toMatchObject({ code: "screen_control_lost" }); + const resumed = (await f.command({ kind: "takeover", generation: 2 })).view; + expect(resumed).toMatchObject({ state: "owner", generation: 3 }); + f.clock.now += 31_000; await f.control.view(f.owner, "vm1"); + expect((await f.command({ kind: "hand_back", generation: 4, takeoverId: null })).view).toMatchObject({ state: "agent", generation: 5 }); + expect(f.control.idleBlocked("vm1")).toBe(false); + }); + + test("a controller restart fences control the owner held, under a new generation", async () => { + const f = control(); + const taken = (await f.command({ kind: "takeover", generation: 0 })).view; + const restarted = control(f.store, f.screen, f.clock); + expect(await restarted.control.view(restarted.owner, "vm1")).toMatchObject({ state: "fenced", generation: 2 }); + await expect(restarted.command({ kind: "input", generation: 1, takeoverId: taken.takeoverId, input: text("x") })).rejects.toMatchObject({ code: "screen_control_lost" }); + expect(() => restarted.control.agentInput("vm1")).toThrow(expect.objectContaining({ code: "takeover_active" })); + }); + + test("typed text never reaches the journal: no value or hash of it is stored", async () => { + const f = control(), secret = "Hunter2-Correct-Horse"; + const taken = (await f.command({ kind: "takeover", generation: 0 })).view; + await f.command({ kind: "input", generation: 1, takeoverId: taken.takeoverId, input: text(secret) }); + await f.command({ kind: "hand_back", generation: 1, takeoverId: taken.takeoverId }); + const hash = new Bun.CryptoHasher("sha256").update(secret).digest("hex"); + const dump = f.store.db.query<{ name: string }, []>("SELECT name FROM sqlite_master WHERE type='table'").all() + .flatMap(({ name }) => f.store.db.query, []>(`SELECT * FROM "${name}"`).all()).map(row => JSON.stringify(row)).join("\n"); + expect(dump).not.toContain(secret); expect(dump).not.toContain(hash); + expect(f.store.db.query<{ kind: string }, []>("SELECT kind FROM s16_screen_events ORDER BY sequence").all().map(row => row.kind)).toEqual(["takeover", "hand_back"]); + }); + + test("takeover and hand-back are idempotent per request id; input is sent at most once per request id", async () => { + const f = control(), requestId = "req_takeover"; + const body: ScreenCommand = { contract: "s16-screen-command/v1", requestId, vmId: "vm1", kind: "takeover", generation: 0 }; + const first = await f.control.command(f.owner, body), retried = await f.control.command(f.owner, body); + expect(retried.view.generation).toBe(first.view.generation); + await expect(f.control.command(f.owner, { ...body, generation: 1 })).rejects.toMatchObject({ code: "request_id_conflict" }); + const input = { contract: "s16-screen-command/v1", requestId: "req_input", vmId: "vm1", kind: "input", generation: 1, takeoverId: first.view.takeoverId, input: text("once") }; + await f.control.command(f.owner, input); await f.control.command(f.owner, input); + expect(f.screen.sent).toEqual([text("once")]); + }); + + test("an input id is never sent twice: not after many later inputs, and not after a send whose outcome is unknown", async () => { + const f = control(), ticket = f.control.agentInput("vm1"); + const settling = (await f.command({ kind: "takeover", generation: 0 })).view; + const body = (requestId: string, value: string) => ({ contract: "s16-screen-command/v1", requestId, vmId: "vm1", kind: "input", generation: 1, takeoverId: settling.takeoverId, input: text(value) }); + // Refused before anything was sent: the same id may be used again once the owner holds the screen. + await expect(f.control.command(f.owner, body("early", "e"))).rejects.toMatchObject({ code: "takeover_settling" }); + ticket.end(); await f.control.command(f.owner, body("early", "e")); + for (let i = 0; i < 300; i++) await f.control.command(f.owner, body("many_" + i, "m")); + await f.control.command(f.owner, body("early", "e")); await f.control.command(f.owner, body("many_0", "m")); + expect(f.screen.sent).toHaveLength(301); + const failing = f.screen.value.send; f.screen.value.send = async () => { throw Object.assign(new Error("vm_management_unavailable"), { code: "vm_management_unavailable" }); }; + await expect(f.control.command(f.owner, body("lost", "x"))).rejects.toThrow("vm_management_unavailable"); + f.screen.value.send = failing; + await expect(f.control.command(f.owner, body("lost", "x"))).rejects.toMatchObject({ code: "screen_input_unknown" }); + expect(f.screen.sent).toHaveLength(301); + }); + + test("a computer without a desktop cannot be taken over, and only the owner reaches the screen", async () => { + const f = control(); f.screen.setUnavailable("vm_desktop_restart_required"); + expect(await f.control.view(f.owner, "vm1")).toMatchObject({ available: false, code: "vm_desktop_restart_required", state: "agent" }); + await expect(f.command({ kind: "takeover", generation: 0 })).rejects.toMatchObject({ code: "vm_desktop_restart_required" }); + await expect(f.control.view({}, "vm1")).rejects.toThrow("owner required"); + await expect(f.control.frame({}, "vm1", 0)).rejects.toThrow("owner required"); + await expect(f.control.view(f.owner, "vm2")).rejects.toMatchObject({ code: "vm_not_enrolled" }); + expect(() => parseScreenCommand({ contract: "s16-screen-command/v1", requestId: "r", vmId: "vm1", kind: "takeover", generation: 0, extra: 1 })).toThrow(); + }); +}); + +describe("working VM operations under screen control", () => { + function vm(screen: ScreenControl, store: VmStore) { + const actor = {}; let afterSend: (() => Promise) | undefined; const sends: Record[] = []; + const policy: VmPolicyPort = { + assertCurrent() {}, + async claim() { return { async dispatch(start) { return start(); }, publish(consume) { return consume(); }, observe() {} }; }, + }; + const manager: VmManagement = { async inspect() { return { state: "running", identity: "fixture-vm", boot: "boot1", cpuCount: 4, memoryBytes: 8388608, diskBytes: 1048576, network: "proxy-only", nicMac: "52:54:00:12:34:56" }; }, async stop() {}, async snapshot() {}, async restore() {}, async retireSnapshot() {} }; + class Transport extends VmSshTransport { + override async control(argv: readonly string[]) { if (argv.some(a => a.includes("nicMacs"))) return '{"nicMacs":["52:54:00:12:34:56"]}'; if (argv.includes("-c")) return '{"reachable":[false,false,false,false,false,false]}'; return ""; } + override async openScope(_unit: string, _ms: number, guard: () => void) { guard(); } + override async closeScope() { return true; } + override async invoke(request: Record, settings: Parameters[1]) { + settings.beforeStart(); sends.push(request); await afterSend?.(); + if (request.op === "spawn") return { stdout: "", stderr: "", exitCode: 0, timedOut: false, truncated: false, totalBytes: 0 }; + if (request.op === "realpath") return { path: request.path }; + if (request.op === "stat") return { type: "directory" }; + return {}; + } + } + const service = new WorkingVmService({ store, policy, managers: new Map([["mgr", manager]]), transport: () => new Transport(endpoint), now: () => 100, screen }); + return { service, actor, sends, setAfterSend: (f: () => Promise) => { afterSend = f; } }; + } + test("a running command delays owner control; nothing new is admitted until hand-back", async () => { + const f = control(), w = vm(f.control, f.store); + const session = await w.service.openAttempt(w.actor, { vmId: "vm1", workId: "work1", attemptId: "attempt1" }); + const entered = Promise.withResolvers(), release = Promise.withResolvers(); + w.setAfterSend(async () => { entered.resolve(); await release.promise; }); + const running = session.backend.spawn({ command: "sleep 1", cwd: session.backend.workspaceRoot }); + await entered.promise; + const taken = (await f.command({ kind: "takeover", generation: 0 })).view; + expect(taken).toMatchObject({ state: "settling", agentBusy: true }); + release.resolve(); + // The response arrived after the takeover: it is not published to the work. + await expect(running).rejects.toMatchObject({ code: "takeover_active" }); + expect((await f.control.view(f.owner, "vm1")).state).toBe("owner"); + w.setAfterSend(async () => {}); const before = w.sends.length; + await expect(session.backend.spawn({ command: "true", cwd: session.backend.workspaceRoot })).rejects.toMatchObject({ code: "takeover_active" }); + expect(w.sends.length).toBe(before); + await f.command({ kind: "hand_back", generation: 1, takeoverId: taken.takeoverId }); + await session.backend.spawn({ command: "true", cwd: session.backend.workspaceRoot }); + expect(w.sends.length).toBe(before + 1); + await session.close(); + }); +}); + +describe("owner screen API under the real composition", () => { + const closers: (() => Promise)[] = []; + afterEach(async () => { for (const close of closers.splice(0)) await close(); }); + test("the owner watches, takes over (work is refused), types once and hands back; nothing without the opt-in", async () => { + const screen = device(); + const f = await composedFixture("default", false, undefined, o => ({ ...o, computers: { idleStopMs: 30 * 60_000, sweepMs: 0 }, screen: {}, + vm: { ...o.vm, managers: new Map([...o.vm.managers].map(([id, manager]) => [id, Object.assign(manager, { screen: () => screen.value })])) } })); + closers.push(() => f.close()); + expect((await (await f.request("/api/s16/client/bootstrap")).json()).capabilities.computers).toMatchObject({ available: true, screen: true }); + // The page shows frames only from in-memory blob URLs; no remote or data image source is allowed. + expect((await f.request("/s16/computers")).headers.get("content-security-policy")).toContain("img-src blob:;"); + const view = await (await f.request("/api/s16/client/computers/vm/screen")).json(); + expect(view).toMatchObject({ contract: "s16-computer-screen/v1", vmId: "vm", available: true, state: "agent", generation: 0, scope: { kind: "shared" } }); + const frame = await f.request("/api/s16/client/computers/vm/screen/frame?generation=0"); + expect(frame.status).toBe(200); expect(frame.headers.get("content-type")).toBe("image/png"); expect(frame.headers.get("cache-control")).toBe("private, no-store"); + expect([...new Uint8Array(await frame.arrayBuffer())]).toEqual([...png()]); + expect((await f.request("/api/s16/client/computers/vm/screen/frame?generation=1")).status).toBe(409); + expect((await f.request("/api/s16/client/computers/vm/screen/frame?generation=0&x=1")).status).toBe(400); + expect((await f.request("/api/s16/client/computers/vm/screen?x=1")).status).toBe(400); + const takeover = { contract: "s16-screen-command/v1", requestId: "take_1", vmId: "vm", kind: "takeover", generation: 0 }; + expect((await f.request("/api/s16/client/computers/screen/commands", "POST", takeover, { "x-csrf-token": "forged" })).status).toBe(403); + const taken = await (await f.request("/api/s16/client/computers/screen/commands", "POST", takeover)).json(); + expect(taken.view).toMatchObject({ state: "owner", generation: 1 }); + // Work cannot start on the computer while the owner holds it. + const work = f.service.policy.owner.admit(f.owner, { requestId: "screen_work", intent: "execute", access: "reversible_write", purpose: f.policyInput.purposes[0]!, bot: f.policyInput.bots[0]!, resources: [f.policyInput.resources[0]!], recipients: [], selection: f.choice, selectionPin: null, expiresAt: 800000 }).work!; + const actor = f.service.policy.control.bindWork(work.id), input = { vmId: "vm", workId: work.id, attemptId: "chat_" + work.id }; + await expect(f.service.workingVm.openAttempt(actor, input)).rejects.toMatchObject({ code: "takeover_active" }); + const typed = { contract: "s16-screen-command/v1", requestId: "type_1", vmId: "vm", kind: "input", generation: 1, takeoverId: taken.view.takeoverId, input: { kind: "text", text: "owner-secret" } }; + expect((await f.request("/api/s16/client/computers/screen/commands", "POST", typed)).status).toBe(200); + expect(screen.sent).toEqual([{ kind: "text", text: "owner-secret" }]); + const back = await (await f.request("/api/s16/client/computers/screen/commands", "POST", { contract: "s16-screen-command/v1", requestId: "back_1", vmId: "vm", kind: "hand_back", generation: 1, takeoverId: taken.view.takeoverId })).json(); + expect(back.view).toMatchObject({ state: "agent", generation: 2 }); + expect((await f.request("/api/s16/client/computers/screen/commands", "POST", typed)).status).toBe(409); // Retried after hand-back: refused, not sent again. + expect(screen.sent).toHaveLength(1); + const session = await f.service.workingVm.openAttempt(actor, input); await session.close(); + const plain = await composedFixture(); closers.push(() => plain.close()); + expect((await plain.request("/api/s16/client/computers/vm/screen")).status).toBe(503); + for (const locale of ["en", "th"] as const) for (const key of ["s16.screen.watch", "s16.screen.takeOver", "s16.screen.handBack", "s16.screen.fenced", "s16.screen.typeHint"]) expect((dictionaries[locale] as Record)[key]).toBeTruthy(); + }); +}); diff --git a/tests/s16-screen-input.test.ts b/tests/s16-screen-input.test.ts new file mode 100644 index 00000000..928be7d6 --- /dev/null +++ b/tests/s16-screen-input.test.ts @@ -0,0 +1,105 @@ +import { describe, expect, test } from "bun:test"; +import type { ScreenView } from "../src/s16/computers/screen"; +import type { VmScreenInput } from "../src/s15/vm/screen"; +import { editDraft, emptyDraft, liveDraft, screenInputQueue, screenPin, type ScreenPin } from "../src/s16/web/screenInput"; + +const view = (state: ScreenView["state"], generation: number, takeoverId: string | null): ScreenView => ({ + contract: "s16-computer-screen/v1", vmId: "vm1", available: true, code: null, width: 1280, height: 800, + state, generation, takeoverId, expiresAt: null, agentBusy: false, scope: { kind: "shared" }, +}); +const key = (name: string): VmScreenInput => ({ kind: "key", keys: [name] }); +const settle = () => new Promise(resolve => setTimeout(resolve, 0)); + +/** A send that stays open until released, so tests can change control while inputs wait. */ +function harness(start: ScreenView) { + let current: ScreenView | null = start; + const sent: { input: VmScreenInput; pin: ScreenPin }[] = [], open: (() => void)[] = []; + const queue = screenInputQueue((input, pin) => { sent.push({ input, pin }); return new Promise(resolve => open.push(resolve)); }, () => current); + return { queue, sent, set(next: ScreenView | null) { current = next; }, async release() { open.shift()?.(); await settle(); } }; +} + +describe("screen input queue (DESK16-01)", () => { + test("screenPin is the owner's control only", () => { + expect(screenPin(view("owner", 4, "t1"))).toEqual({ generation: 4, takeoverId: "t1" }); + for (const state of ["agent", "settling", "fenced"] as const) expect(screenPin(view(state, 4, "t1"))).toBeNull(); + expect(screenPin(null)).toBeNull(); + }); + + test("inputs go out one at a time, in order, under the takeover they were made in", async () => { + const h = harness(view("owner", 4, "t1")), pin = screenPin(view("owner", 4, "t1")); + h.queue.push(key("a"), pin); h.queue.push(key("b"), pin); await settle(); + expect(h.sent.map(s => s.input)).toEqual([key("a")]); + await h.release(); + expect(h.sent).toEqual([{ input: key("a"), pin: { generation: 4, takeoverId: "t1" } }, { input: key("b"), pin: { generation: 4, takeoverId: "t1" } }]); + }); + + test("waiting inputs from an earlier takeover never go out under a later one", async () => { + const h = harness(view("owner", 4, "t1")), old = screenPin(view("owner", 4, "t1")); + h.queue.push(key("a"), old); h.queue.push(key("b"), old); h.queue.push(key("c"), old); await settle(); + // Another tab hands back and takes control again while "a" is still on its way. + h.set(view("owner", 5, "t2")); + await h.release(); await h.release(); + expect(h.sent.map(s => s.input)).toEqual([key("a")]); + // New input made under the new takeover still goes out, with the new ids. + h.queue.push(key("d"), screenPin(view("owner", 5, "t2"))); await settle(); + expect(h.sent.at(-1)).toEqual({ input: key("d"), pin: { generation: 5, takeoverId: "t2" } }); + }); + + test("input waiting through a hand-back or fencing is dropped", async () => { + for (const after of [view("agent", 5, null), view("fenced", 4, null)]) { + const h = harness(view("owner", 4, "t1")), pin = screenPin(view("owner", 4, "t1")); + h.queue.push(key("a"), pin); h.queue.push(key("b"), pin); await settle(); + h.set(after); await h.release(); + expect(h.sent.map(s => s.input)).toEqual([key("a")]); + } + }); + + test("closing the panel drops waiting input; input made while not driving is never queued", async () => { + const h = harness(view("owner", 4, "t1")), pin = screenPin(view("owner", 4, "t1")); + h.queue.push(key("a"), pin); h.queue.push(key("b"), pin); await settle(); + h.queue.close(); await h.release(); + h.queue.push(key("c"), pin); await settle(); + expect(h.sent.map(s => s.input)).toEqual([key("a")]); + const idle = harness(view("agent", 4, null)); + idle.queue.push(key("a"), null); await settle(); + expect(idle.sent).toEqual([]); + }); + + test("a failed send does not stop later input", async () => { + let calls = 0; + const current = view("owner", 4, "t1"), pin = screenPin(current); + const queue = screenInputQueue(async () => { calls++; if (calls === 1) throw new Error("network"); }, () => current); + queue.push(key("a"), pin); queue.push(key("b"), pin); await settle(); await settle(); + expect(calls).toBe(2); + }); +}); + +describe("typed text stays with its takeover (DESK16-01)", () => { + /** The panel's Send: the field's live draft goes out under the pin it was typed with. */ + const submit = (h: ReturnType, draft: ReturnType, now: ScreenView) => { + const value = liveDraft(draft, now); if (value.text) h.queue.push({ kind: "text", text: value.text }, value.pin); + }; + + test("text typed under takeover A is not sent after takeover B", async () => { + const a = view("owner", 4, "t1"), b = view("owner", 5, "t2"), h = harness(a); + const draft = editDraft(a, "Owner-Pass1"); + expect(draft).toEqual({ text: "Owner-Pass1", pin: { generation: 4, takeoverId: "t1" } }); + // Hand-back and a new takeover (this tab or another) while the text sits in the field. + for (const between of [view("agent", 5, null), view("fenced", 4, null), b]) expect(liveDraft(draft, between)).toEqual(emptyDraft); + h.set(b); submit(h, draft, b); await settle(); + expect(h.sent).toEqual([]); + // Even if a stale draft reached the queue, it keeps takeover A's pin and is dropped under B. + h.queue.push({ kind: "text", text: draft.text }, draft.pin); await settle(); + expect(h.sent).toEqual([]); + // Text typed under B goes out under B. + submit(h, editDraft(b, "new"), b); await settle(); + expect(h.sent).toEqual([{ input: { kind: "text", text: "new" }, pin: { generation: 5, takeoverId: "t2" } }]); + }); + + test("the draft lasts while the same takeover is in force; nothing is composed while not driving", () => { + const a = view("owner", 4, "t1"), draft = editDraft(a, "abc"); + expect(liveDraft(draft, view("owner", 4, "t1"))).toBe(draft); + expect(editDraft(view("agent", 4, null), "abc")).toEqual(emptyDraft); + expect(editDraft(a, "")).toEqual(emptyDraft); + }); +});