You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: box/overall/browser/ai-actions.mdx
+36-8Lines changed: 36 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,38 +5,40 @@ title: "AI Actions"
5
5
Beyond reading pages, a tab can act. A DOM-aware browser agent runs inside the box and resolves natural-language instructions against the live page. It can find elements, execute single actions, or complete multi-step tasks on its own.
6
6
7
7
<Note>
8
-
AI actions use an LLM and are metered. They need an API key for the model's
8
+
LLM-resolved AI actions are metered and need an API key for the model's
9
9
provider (Anthropic, OpenAI, OpenRouter, Vercel, or OpenCode) on the box or
10
10
your account. Every method accepts a provider-prefixed `model` override such
11
11
as `"openai/gpt-4o"`. Without an override, the call uses the model the box
12
12
was configured with. If the box has no model, it falls back to
13
-
`anthropic/claude-sonnet-4-5`.
13
+
`anthropic/claude-sonnet-4-5`. Replaying a pre-resolved action with
14
+
`act(action)` is the exception: it uses no LLM and needs no key (see
15
+
[Replay an action without an LLM](#replay-an-action-without-an-llm)).
14
16
</Note>
15
17
16
18
## Observe
17
19
18
-
`observe()` finds actionable elements matching an instruction. Use it to check the page before acting, or to build your own action loop:
20
+
`observe()` finds actionable elements matching an instruction. Use it to check the page before acting, or to build your own action loop. Each element carries a `selector` plus a suggested `method` and `arguments`, so you can replay it directly with `act(el)` (see [below](#replay-an-action-without-an-llm)):
19
21
20
22
<CodeGroup>
21
23
```typescript box.ts
22
24
const { elements } =awaittab.observe("find the login and signup buttons")
result = tab.observe("find the login and signup buttons")
31
33
32
34
for el in result.elements:
33
-
print(el.description, el.selector)
35
+
print(el.description, el.selector, el.method)
34
36
```
35
37
</CodeGroup>
36
38
37
39
## Act
38
40
39
-
`act()` resolves and executes exactly one action described in natural language:
41
+
`act()` resolves and executes exactly one action described in natural language. It also accepts a pre-resolved action from `observe()` to replay without an LLM (see [Replay an action without an LLM](#replay-an-action-without-an-llm)):
The result reports what was done (`actions` with the resolved selectors), whether it succeeded, and the token usage of the call.
58
60
61
+
### Replay an action without an LLM
62
+
63
+
`observe()` returns each element's resolved `selector` plus a suggested `method` and `arguments`. Pass that element straight back into `act()` to replay it deterministically: no LLM call, no tokens, and no model provider key required. Resolve once with the model, then reuse the action as many times as you like.
64
+
65
+
<CodeGroup>
66
+
```typescript box.ts
67
+
// Resolve once (metered, needs a model key)
68
+
const { elements } =awaittab.observe("the primary call-to-action")
69
+
const action =elements[0]
70
+
71
+
// Replay as many times as you like: no LLM, no key
72
+
awaittab.act(action)
73
+
```
74
+
75
+
```python box.py
76
+
# Resolve once (metered, needs a model key)
77
+
result = tab.observe("the primary call-to-action")
78
+
action = result.elements[0]
79
+
80
+
# Replay as many times as you like: no LLM, no key
81
+
tab.act(action)
82
+
```
83
+
</CodeGroup>
84
+
85
+
Observe narrowly (or check the element) before relying on a fixed index like `elements[0]`. Cache the returned action (in your own store or on the box filesystem) and replay it across pages or runs. This is the built-in path for turning an AI-discovered step into a fast, repeatable one. If the page changes and the selector no longer matches, `observe()` again to re-resolve.
86
+
59
87
## Run
60
88
61
89
`run()` is the autonomous mode. The agent reads the page, acts, and repeats until the task is complete or it hits the step limit. Pass a schema to get structured data back at the end:
@@ -107,7 +135,7 @@ for step in result.steps:
107
135
| Method | Does | Best for |
108
136
|---|---|---|
109
137
|`observe`| Finds elements, executes nothing | Inspecting a page, building custom loops |
110
-
|`act`| Executes one action | Flows where your code decides each step |
138
+
|`act`| Executes one action (natural language, metered; or a pre-resolved action, no LLM) | Flows where your code decides each step, or replaying a resolved action|
111
139
|`run`| Executes a whole task autonomously | Open-ended or navigation-heavy tasks |
112
140
113
-
For fully scripted control with no LLM in the loop, [connect over CDP](/box/overall/browser/connect) with Playwright or Puppeteer instead. Both drive the same tabs, so you can mix scripted steps with AI steps. To watch or replay what the agent did, see [Live View](/box/overall/browser/live-view) and [Recordings](/box/overall/browser/recordings).
141
+
To turn a single AI-resolved step into a no-LLM one, replay an `observe()` result through `act()` (see [Replay an action without an LLM](#replay-an-action-without-an-llm)). For fully scripted control with no LLM anywhere in the loop, [connect over CDP](/box/overall/browser/connect) with Playwright or Puppeteer instead. Both drive the same tabs, so you can mix scripted steps with AI steps. To watch or replay what the agent did, see [Live View](/box/overall/browser/live-view) and [Recordings](/box/overall/browser/recordings).
Copy file name to clipboardExpand all lines: box/overall/browser/connect.mdx
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,6 +4,8 @@ title: "Connect over CDP"
4
4
5
5
The box browser is a real Chromium, and you can drive it with the tools you already use. `cdpUrl()` returns an authenticated Chrome DevTools Protocol WebSocket URL that Playwright, Puppeteer, or Stagehand can connect to directly. There is no browser to install and nothing to manage.
6
6
7
+
For a single no-LLM step without wiring up a CDP client, replaying an observed action with [`act(action)`](/box/overall/browser/ai-actions#replay-an-action-without-an-llm) is often enough. Reach for CDP when you want fully scripted, multi-step control.
You can also watch and control the browser from the **Browser** tab on your box's page in the [Upstash Console](https://console.upstash.com). It shows the live view, runs AI tasks, and includes the SDK snippet for everything you do there.
0 commit comments