Skip to content

Commit f34c9f3

Browse files
committed
feat: document act observe changes
1 parent bb1977c commit f34c9f3

3 files changed

Lines changed: 41 additions & 9 deletions

File tree

box/overall/browser/ai-actions.mdx

Lines changed: 36 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,38 +5,40 @@ title: "AI Actions"
55
Beyond reading pages, a tab can act. A DOM-aware browser agent runs inside the box and resolves natural-language instructions against the live page. It can find elements, execute single actions, or complete multi-step tasks on its own.
66

77
<Note>
8-
AI actions use an LLM and are metered. They need an API key for the model's
8+
LLM-resolved AI actions are metered and need an API key for the model's
99
provider (Anthropic, OpenAI, OpenRouter, Vercel, or OpenCode) on the box or
1010
your account. Every method accepts a provider-prefixed `model` override such
1111
as `"openai/gpt-4o"`. Without an override, the call uses the model the box
1212
was configured with. If the box has no model, it falls back to
13-
`anthropic/claude-sonnet-4-5`.
13+
`anthropic/claude-sonnet-4-5`. Replaying a pre-resolved action with
14+
`act(action)` is the exception: it uses no LLM and needs no key (see
15+
[Replay an action without an LLM](#replay-an-action-without-an-llm)).
1416
</Note>
1517

1618
## Observe
1719

18-
`observe()` finds actionable elements matching an instruction. Use it to check the page before acting, or to build your own action loop:
20+
`observe()` finds actionable elements matching an instruction. Use it to check the page before acting, or to build your own action loop. Each element carries a `selector` plus a suggested `method` and `arguments`, so you can replay it directly with `act(el)` (see [below](#replay-an-action-without-an-llm)):
1921

2022
<CodeGroup>
2123
```typescript box.ts
2224
const { elements } = await tab.observe("find the login and signup buttons")
2325

2426
for (const el of elements) {
25-
console.log(el.description, el.selector)
27+
console.log(el.description, el.selector, el.method)
2628
}
2729
```
2830

2931
```python box.py
3032
result = tab.observe("find the login and signup buttons")
3133

3234
for el in result.elements:
33-
print(el.description, el.selector)
35+
print(el.description, el.selector, el.method)
3436
```
3537
</CodeGroup>
3638

3739
## Act
3840

39-
`act()` resolves and executes exactly one action described in natural language:
41+
`act()` resolves and executes exactly one action described in natural language. It also accepts a pre-resolved action from `observe()` to replay without an LLM (see [Replay an action without an LLM](#replay-an-action-without-an-llm)):
4042

4143
<CodeGroup>
4244
```typescript box.ts
@@ -56,6 +58,32 @@ print(action.input_tokens, action.output_tokens)
5658

5759
The result reports what was done (`actions` with the resolved selectors), whether it succeeded, and the token usage of the call.
5860

61+
### Replay an action without an LLM
62+
63+
`observe()` returns each element's resolved `selector` plus a suggested `method` and `arguments`. Pass that element straight back into `act()` to replay it deterministically: no LLM call, no tokens, and no model provider key required. Resolve once with the model, then reuse the action as many times as you like.
64+
65+
<CodeGroup>
66+
```typescript box.ts
67+
// Resolve once (metered, needs a model key)
68+
const { elements } = await tab.observe("the primary call-to-action")
69+
const action = elements[0]
70+
71+
// Replay as many times as you like: no LLM, no key
72+
await tab.act(action)
73+
```
74+
75+
```python box.py
76+
# Resolve once (metered, needs a model key)
77+
result = tab.observe("the primary call-to-action")
78+
action = result.elements[0]
79+
80+
# Replay as many times as you like: no LLM, no key
81+
tab.act(action)
82+
```
83+
</CodeGroup>
84+
85+
Observe narrowly (or check the element) before relying on a fixed index like `elements[0]`. Cache the returned action (in your own store or on the box filesystem) and replay it across pages or runs. This is the built-in path for turning an AI-discovered step into a fast, repeatable one. If the page changes and the selector no longer matches, `observe()` again to re-resolve.
86+
5987
## Run
6088

6189
`run()` is the autonomous mode. The agent reads the page, acts, and repeats until the task is complete or it hits the step limit. Pass a schema to get structured data back at the end:
@@ -107,7 +135,7 @@ for step in result.steps:
107135
| Method | Does | Best for |
108136
|---|---|---|
109137
| `observe` | Finds elements, executes nothing | Inspecting a page, building custom loops |
110-
| `act` | Executes one action | Flows where your code decides each step |
138+
| `act` | Executes one action (natural language, metered; or a pre-resolved action, no LLM) | Flows where your code decides each step, or replaying a resolved action |
111139
| `run` | Executes a whole task autonomously | Open-ended or navigation-heavy tasks |
112140

113-
For fully scripted control with no LLM in the loop, [connect over CDP](/box/overall/browser/connect) with Playwright or Puppeteer instead. Both drive the same tabs, so you can mix scripted steps with AI steps. To watch or replay what the agent did, see [Live View](/box/overall/browser/live-view) and [Recordings](/box/overall/browser/recordings).
141+
To turn a single AI-resolved step into a no-LLM one, replay an `observe()` result through `act()` (see [Replay an action without an LLM](#replay-an-action-without-an-llm)). For fully scripted control with no LLM anywhere in the loop, [connect over CDP](/box/overall/browser/connect) with Playwright or Puppeteer instead. Both drive the same tabs, so you can mix scripted steps with AI steps. To watch or replay what the agent did, see [Live View](/box/overall/browser/live-view) and [Recordings](/box/overall/browser/recordings).

box/overall/browser/connect.mdx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ title: "Connect over CDP"
44

55
The box browser is a real Chromium, and you can drive it with the tools you already use. `cdpUrl()` returns an authenticated Chrome DevTools Protocol WebSocket URL that Playwright, Puppeteer, or Stagehand can connect to directly. There is no browser to install and nothing to manage.
66

7+
For a single no-LLM step without wiring up a CDP client, replaying an observed action with [`act(action)`](/box/overall/browser/ai-actions#replay-an-action-without-an-llm) is often enough. Reach for CDP when you want fully scripted, multi-step control.
8+
79
<CodeGroup>
810
```typescript box.ts
911
const cdpUrl = await box.browser.cdpUrl()

box/overall/browser/overview.mdx

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,9 @@ print(page.title)
8282
[`extract`](/box/overall/browser/reading-pages) and [`observe`, `act`,
8383
`run`](/box/overall/browser/ai-actions). They need an API key for the model's
8484
provider (Anthropic, OpenAI, OpenRouter, Vercel, or OpenCode) on the box or
85-
your account.
85+
your account. The exception is replaying a resolved action with
86+
[`act(action)`](/box/overall/browser/ai-actions#replay-an-action-without-an-llm),
87+
which uses no LLM and needs no key.
8688
</Note>
8789

8890
You can also watch and control the browser from the **Browser** tab on your box's page in the [Upstash Console](https://console.upstash.com). It shows the live view, runs AI tasks, and includes the SDK snippet for everything you do there.

0 commit comments

Comments
 (0)