Computer use lands in the language: computer.observe, typed actions, replay with drift checks

[NekomyaDev]

This one has been in the works for a while, and it is now merged into main.

Nudge programs can now use the computer as a typed effect. computer.observe(app) returns an Observation — the accessibility tree as text, the element list, and optionally a screenshot. Then computer.click, computer.type, computer.key, computer.scroll, computer.set_value, computer.drag, computer.perform, and computer.paste return ActionResult. Every call goes through the effect system like everything else in the language: inferred, checked, and recorded one NTF trace per step.

The division of labor is deliberate: the language owns control and safety, the model owns perception and judgment. App allowlists (allow: [...]), deadlines, the NUDGE_COMPUTER_KILL=1 kill switch, and replay are compiler/runtime guarantees. What the screen means and what to do next — that is the model's job, fed obs.tree or obs.drift through a plain llm block or decide{}.

A few details worth knowing:

• Replay never re-fires actions. They run as dry-run audits, and with NUDGE_COMPUTER_DRIFT=1 the runtime re-observes live and attaches a mechanical diff (tree diff + screenshot hash) for the model to interpret.
• The reference bridge (tools/cu_bridge.py) speaks a documented JSONL contract and enforces hardening invariants: one controller at a time, stale element indices are refused instead of blind-clicked, and every action returns a dispatch receipt so retries are provably safe.
• It ships with two backends: a deterministic virtual desktop, so the whole loop runs in nudgec test at $0 — and a live X11 backend (AT-SPI + xdotool) behind the same contract.
• llm blocks can now carry images, so screenshots can go straight to multimodal models.

The full RFC is in docs/computer-use.md. There is a working example under examples/computer-automation — record, replay, and drift-check it with the fake provider, no API key needed.

Feedback and bug reports welcome here or in issues. What would you automate with this?

— The Nudge maintainers