|
18 | 18 |
|
19 | 19 | --- |
20 | 20 |
|
21 | | -AI agents are getting better at terminals, but real work still spans browsers, desktop apps, files, and messaging tools — all with different interfaces, states, and habits. |
| 21 | +Open-source local AI agent that operates your entire computer — GUI, browser, shell, messaging — from a single instruction. Teach it once, it learns. Use it daily, it gets faster. Bring your own model. |
22 | 22 |
|
23 | | -**Understudy is a general-purpose local agent for your computer.** Give it one instruction and it can research, browse, click through apps, run commands, manage files, and reply through your existing channels. |
| 23 | +## Showcase |
| 24 | + |
| 25 | +> **Demo environment:** macOS + GPT-5.4 via Codex (OpenAI). All demos also work with Claude, Gemini, and other providers. See [Supported Models](#supported-models) for the full list. |
| 26 | +
|
| 27 | +The demos below map to the product story in order: general agent first, computer use next, then teach, and finally a full autonomous pipeline that combines everything. |
| 28 | + |
| 29 | +### General Agent — One Message, Done |
| 30 | + |
| 31 | +[](https://youtube.com/shorts/KObeVm7MK1Y) |
| 32 | + |
| 33 | +This is the starting point: Understudy is first a general-purpose agent. It researches the web, controls your browser, invokes skills, and delivers a polished result — all from a single instruction. No staging, no multi-step prompting. Just say what you need. |
| 34 | + |
| 35 | +> *Example prompt: "Research Cowork and build a tech-style landing page in my downloads folder."* |
| 36 | +
|
| 37 | +### Computer Use + Remote Dispatch — Agent on Desktop, You on Phone |
| 38 | + |
| 39 | +[](https://youtu.be/HlTD6Jvm3gk) |
| 40 | + |
| 41 | +This is computer use in practice: send a message from your phone via Telegram, and Understudy receives it on your Mac, converts a file to PDF, opens desktop Telegram, finds the right contact, and sends it — all through GUI automation. The demo shows phone and desktop views side by side. |
| 42 | + |
| 43 | +> *Example prompt: "Convert the Cowork webpage to PDF and send it to Alex on Telegram."* |
| 44 | +
|
| 45 | +Understudy works with messaging apps people already use: Telegram, Discord, Slack, WhatsApp, Signal, LINE, iMessage, and Web. |
24 | 46 |
|
25 | | -**It also brings modern computer-use capability without locking you into a subscription product.** Understudy can see the screen and operate software through grounded GUI actions, while still letting you use your own model/API key. |
| 47 | +### Teach — Show Once, Refine, Replay with Generalization |
| 48 | + |
| 49 | +[](https://youtube.com/shorts/ZOZU6vb4rRs) |
| 50 | + |
| 51 | +Teach a task by demonstrating it once. Understudy learns the **intent**, not the coordinates — so the skill survives UI redesigns, window resizing, even switching to a different app. Interactively refine the generated skill, then invoke it with natural language. On replay, the agent automatically generalizes: Google Image search becomes browser automation, downloads become shell commands, while native app control (Pixelmator Pro) stays GUI-driven. |
| 52 | + |
| 53 | +> *Demo flow: `/teach start` → search Google Images for Sam Altman → download photo → remove background in Pixelmator Pro → export → send via Telegram to Alex. Then interactively refine the skill. Finally, invoke with natural language: "Find a photo of [person], remove the background, and send it to [contact] on Telegram" — the agent discovers the taught skill and replays it with automatic upgrades.* |
| 54 | +
|
| 55 | +See the [published skill from this demo](./examples/published-skills/taught-create-a-background-removed-portrait-for-a-requested-person-and-send-it-in-telegram-cd861a/SKILL.md) for a real example of what teach produces. [Full unedited recording](https://drive.google.com/file/d/1vTMpYaCOIO8IVmciI1DpvEBC6x5MaJ4f/view?usp=sharing). |
26 | 56 |
|
27 | | -**What makes it distinctive after that is the learning loop.** You can teach by demonstration today, and the same system already carries early crystallization and route-upgrade machinery for repeated work over time. |
| 57 | +### AI App Critic — One Prompt to a Published iPhone App Review |
28 | 58 |
|
29 | | -- **General agent first** — one runtime across GUI, browser, shell, web, files, memory, messaging, scheduling, and subagents. |
30 | | -- **Computer use built in** — grounded desktop operation inside the same local runtime, with your own model/API key. |
31 | | -- **Teach + crystallize + upgrade** — explicit teaching today, plus early workflow crystallization and route-aware replay in one system. |
| 59 | +This is everything combined. One prompt triggers a six-stage pipeline: the agent browses the real App Store in Chrome, installs Snapseed on a real iPhone through iPhone Mirroring, explores the app autonomously — discovering background removal and filters it's never seen — composes a narrated vertical video locally with FFmpeg, uploads it to YouTube, and cleans up the device. About one hour, zero human intervention. |
| 60 | + |
| 61 | +The pipeline introduces **workspace artifact composition**: a playbook orchestrates workers (deterministic browser/device automation) and skills (agentic subagents that make their own decisions). Each stage runs as a separate child session with its own context. The middle stage — app exploration — is genuinely agentic: 51 quality-gate rules guide the agent, but it navigates freely through an app it has never seen. |
| 62 | + |
| 63 | +| The published review | How it was made | |
| 64 | +|:---:|:---:| |
| 65 | +| [](https://youtu.be/jliTvpTnsKY) | [](https://youtu.be/gYMYI0bxkJs) | |
| 66 | + |
| 67 | +> *Example prompt: "Make a Snapseed iPhone app review video from scratch: use the real App Store and iPhone Mirroring, capture proof-first clips focusing on background removal and filters (like black & white), add English narration and subtitles, export a vertical video, upload it unlisted to YouTube, clean up the device, and share the result."* |
| 68 | +
|
| 69 | +[Full unedited recording](https://drive.google.com/file/d/1Ap5hGWWemU04UkRm495waHjB1-3nq3g5/view?usp=sharing). |
32 | 70 |
|
33 | 71 | ### Why Understudy? |
34 | 72 |
|
@@ -83,54 +121,6 @@ Current status: Layers 1-2 are implemented and usable today. Layers 3-4 are part |
83 | 121 |
|
84 | 122 | Every layer depends on the one below it. No shortcuts — the system earns its way up. Read the full story: **[Overview →](https://understudy-ai.github.io/understudy/)** | **[Chinese Overview →](https://understudy-ai.github.io/understudy/zh-CN/index.html)** | **[Product Design →](./docs/Product_Design.md)** |
85 | 123 |
|
86 | | -## Showcase |
87 | | - |
88 | | -> **Demo environment:** macOS + GPT-5.4 via Codex (OpenAI). All demos also work with Claude, Gemini, and other providers. See [Supported Models](#supported-models) for the full list. |
89 | | -
|
90 | | -The demos below map to the product story in order: general agent first, computer use next, then teach, and finally a full autonomous pipeline that combines everything. |
91 | | - |
92 | | -### General Agent — One Message, Done |
93 | | - |
94 | | -[](https://youtube.com/shorts/KObeVm7MK1Y) |
95 | | - |
96 | | -This is the starting point: Understudy is first a general-purpose agent. It researches the web, controls your browser, invokes skills, and delivers a polished result — all from a single instruction. No staging, no multi-step prompting. Just say what you need. |
97 | | - |
98 | | -> *Example prompt: "Research Cowork and build a tech-style landing page in my downloads folder."* |
99 | | -
|
100 | | -### Computer Use + Remote Dispatch — Agent on Desktop, You on Phone |
101 | | - |
102 | | -[](https://youtu.be/HlTD6Jvm3gk) |
103 | | - |
104 | | -This is computer use in practice: send a message from your phone via Telegram, and Understudy receives it on your Mac, converts a file to PDF, opens desktop Telegram, finds the right contact, and sends it — all through GUI automation. The demo shows phone and desktop views side by side. |
105 | | - |
106 | | -> *Example prompt: "Convert the Cowork webpage to PDF and send it to Alex on Telegram."* |
107 | | -
|
108 | | -Understudy works with messaging apps people already use: Telegram, Discord, Slack, WhatsApp, Signal, LINE, iMessage, and Web. |
109 | | - |
110 | | -### Teach — Show Once, Refine, Replay with Generalization |
111 | | - |
112 | | -[](https://youtube.com/shorts/ZOZU6vb4rRs) |
113 | | - |
114 | | -Teach a task by demonstrating it once. Understudy learns the **intent**, not the coordinates — so the skill survives UI redesigns, window resizing, even switching to a different app. Interactively refine the generated skill, then invoke it with natural language. On replay, the agent automatically generalizes: Google Image search becomes browser automation, downloads become shell commands, while native app control (Pixelmator Pro) stays GUI-driven. |
115 | | - |
116 | | -> *Demo flow: `/teach start` → search Google Images for Sam Altman → download photo → remove background in Pixelmator Pro → export → send via Telegram to Alex. Then interactively refine the skill. Finally, invoke with natural language: "Find a photo of [person], remove the background, and send it to [contact] on Telegram" — the agent discovers the taught skill and replays it with automatic upgrades.* |
117 | | -
|
118 | | -See the [published skill from this demo](./examples/published-skills/taught-create-a-background-removed-portrait-for-a-requested-person-and-send-it-in-telegram-cd861a/SKILL.md) for a real example of what teach produces. [Full unedited recording](https://drive.google.com/file/d/1vTMpYaCOIO8IVmciI1DpvEBC6x5MaJ4f/view?usp=sharing). |
119 | | - |
120 | | -### AI App Critic — One Prompt to a Published iPhone App Review |
121 | | - |
122 | | -This is everything combined. One prompt triggers a six-stage pipeline: the agent browses the real App Store in Chrome, installs Snapseed on a real iPhone through iPhone Mirroring, explores the app autonomously — discovering background removal and filters it's never seen — composes a narrated vertical video locally with FFmpeg, uploads it to YouTube, and cleans up the device. About one hour, zero human intervention. |
123 | | - |
124 | | -The pipeline introduces **workspace artifact composition**: a playbook orchestrates workers (deterministic browser/device automation) and skills (agentic subagents that make their own decisions). Each stage runs as a separate child session with its own context. The middle stage — app exploration — is genuinely agentic: 51 quality-gate rules guide the agent, but it navigates freely through an app it has never seen. |
125 | | - |
126 | | -| The published review | How it was made | |
127 | | -|:---:|:---:| |
128 | | -| [](https://youtu.be/jliTvpTnsKY) | [](https://youtu.be/gYMYI0bxkJs) | |
129 | | - |
130 | | -> *Example prompt: "Make a Snapseed iPhone app review video from scratch: use the real App Store and iPhone Mirroring, capture proof-first clips focusing on background removal and filters (like black & white), add English narration and subtitles, export a vertical video, upload it unlisted to YouTube, clean up the device, and share the result."* |
131 | | -
|
132 | | -[Full unedited recording](https://drive.google.com/file/d/1Ap5hGWWemU04UkRm495waHjB1-3nq3g5/view?usp=sharing). |
133 | | - |
134 | 124 | ## Workspace Artifacts — Playbook, Worker, Skill |
135 | 125 |
|
136 | 126 | Understudy's teach and crystallization pipelines can produce three types of workspace artifacts that compose into larger automation: |
|
0 commit comments