Repository navigation
feat(computer-use): open an application by name with a launch action - #2533
Conversation
Finding an icon in a screenshot and clicking it was the slowest and least reliable way for an agent to start a program. `computer_act` gains a `launch` action that takes an application's name: `open -a` on a Mac, and on Linux the menu entry whose name or id matches (exact, then prefix, then substring), opened with `exo-open` or `gtk-launch`. An ambiguous or unknown name is answered with the installed names. The name is validated to a name, so it can never be a path, an option or a command line. In the computerd container the application is started by a launcher running as `desk`, reached over a socket only computerd can connect to, so it never runs as the account holding the token. The desktop app's own-screen backend refuses the action with a sentence pointing at the computer's own launcher. The instructions now tell the model to wait or look after a launch or a shortcut that opens a window, before typing into it. Verified live on the computerd container on macOS (Colima) and NixOS, and through a web chat turn that launched a terminal and Firefox.
The first cut matched names in three tiers, parsed desktop entries three different ways, fell back to a gtk-launch nothing ships, and copied the launcher's stderr through a temp file. Now a name or file name matches exactly, else the one entry containing it, else the model is given the names; the entry is opened with exo-open, what the desktop runtime and the computerd image have, with no pipes for the application to hold. The action union is named once (`AnyAction`) instead of repeated. The computerd image runs tini as PID 1: an application launch opened outlives the launcher that started it, and when it quit it was left a zombie under computerd, which reaps nothing. Verified on the computerd container (HTTP and a web chat turn that launched Xfce Terminal and Thunar with no clicks), the sandbox scripts run in the same image, and `open -a` through computerd on macOS.
A launch left its application open for good: the TextEdit, Firefox and terminal windows of a test stayed on the desktop after the run that opened them had finished. The run now quits what it launched when it ends, unsaved work included, so the desktop is left as it was found. Not quit: an application that was already running, which is never the run's to close, and one launched with `keep_open`, which the model sets only when the person asked for it to stay open. A run parked for approval has not ended, so `after_run` marks it parked and the handles travel to the resumed run in the answer's metadata; that run quits them. Each backend's `launch` returns what `quit` closes it by. In a sandbox or computerd on Linux that is the session the launch started, so the quit ends exactly what it started; on a Mac it is `quit saving no`. computerd only quits what it launched, by names it gave out. An E2B desktop is killed with the run and returns none. Verified on the Linux container on this Mac and on NixOS, on the macOS screen, with the sandbox scripts in the container, and in a chat run in which Firefox was quit at the run's end without the model doing it.
Quitting at the end of the run closed an application between two messages of one task: "open the browser and go to X", then "log in" found the browser gone, because each message is a run of its own. Whether a launch should be closed, and when - a conversation has no end to hook - is left to a separate change. This reverts commit f86be39.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c987cbcf39
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Both branches restamped the capability catalogue's translations. The merged pl, de and es pages carry both changes and are stamped against the merged English page.
On Linux, launch looked in two hard-coded directories, read only the untranslated Name, opened the first of two entries sharing a name and refused real names with punctuation. In the sandbox and in computerd: - search $XDG_DATA_HOME and every $XDG_DATA_DIRS entry, so /usr/local/share and Flatpak exports are found - match the file name, the untranslated Name and the locale's own Name[...] in the desktop entry spec's order, from the main group only; other languages' names are not matched, so a German "Terminal" does not beat a Polish desktop's names - refuse every match that is not unique, exact ones included, naming each entry with its file name so the model can pick one - let a hidden entry shadow the system one it hides - accept any punctuation but a slash, a backslash or a control character: the name is one argument to open -a, never a command line Refs: #2533
The instructions told every run to open applications with launch, but the desktop app refuses it, so a run on the person's own computer spent a refused computer_act before falling back to its launcher. The instructions are now a per-run callable: a run that drives the person's computer is told to use cmd+space or the Start menu, every other run is told to use launch. Refs: #2533
A launch matched names loosely in two copies of the lookup, one in the backend's sandbox script and one in computerd, and returned before the window existed. Now: - `_desktop_apps.py` is the one lookup. A sandbox runs its source with `python3 -c`; the computerd image copies it in for the launcher running as `desk`. The image is built from the repository root, with Dockerfile.dockerignore letting in only the three files it copies. - `computer_apps`, a read-only tool, lists what the menu shows by name and id. `launch` takes either exactly (the id first), and refuses a shared name with the ids rather than guessing. - `launch` waits for a new window or a focus change, up to ten seconds, so the screenshot after it shows the application. - Launching is Linux-only: a sandbox, E2B and the computerd container. The Mac's `open -a` and computerd's native fallback are gone; computerd without its container's launcher refuses `launch` and `/apps`, as the desktop app already did. Self-hosted desktops are separate work. Verified in a container built from the new Dockerfile: `/apps` lists desk's 30 entries, Firefox launches as desk and is on the next screenshot, unknown and malformed names are refused, and a computerd without the launcher refuses both.
The instructions for the person's own computer now say `computer_apps` does not work there either, not only `launch`, so the model does not spend a call on it. The rest is comments that the rework left behind: the backends answer three questions, not two, the launch docstring names its time limit, the README says why listing is a read-only tool of its own, and two comments are rewrapped.
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 501d4311a3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The second round of review on the launch action: - Subdirectories of each applications directory are read, their names prefixing the id (kde/konsole.desktop is kde-konsole), as the desktop entry specification derives desktop file ids. - Entries the menu would not show are left out and still shadow later ones: Type other than Application, OnlyShowIn/NotShowIn for another desktop (XFCE when XDG_CURRENT_DESKTOP is unset, as it is outside the session), and a TryExec program that is not installed. - Names are unescaped (\s, \n, \t, \r, \\) before they are listed and matched. - A launch with no new window and no focus change within ten seconds fails, so the rest of its batch does not type into the window that had the focus before. - A daemon may or may not launch, so its instructions say to call computer_apps first and use the computer's own launcher if refused. Against the computerd container's Debian/XFCE menu the new lookup lists the same 30 applications as before.
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c82268b53
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The third round of review on the launch action: - The action that stops a batch counts against max_actions: a launch whose window never came has still started the application, so retries are not free. - An id is matched with its case before ignoring it, so two ids that differ only in case can each be launched. - computer_apps updates the live view's outcome, as computer_screenshot does, both when it is refused and when it succeeds. - computerd's /health answers 503 while the container's launcher cannot be connected to, and the entrypoint waits for the launcher's socket and exits if it never appears. The launcher ignores the empty request a health probe's connection makes. Rebuilt on the NixOS host: healthy, 30 applications listed, a terminal launched; with the launcher killed, /health answers 503.
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 46dae31b32
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
A desktop file id includes its subdirectories, so one can pass the 64 characters Launch.app allowed, and the ambiguity refusal would point at an id the action then refused. The limit is 255, a file name's, in the schema and in computerd. The launcher reads up to 1100 bytes for the request: 255 characters of up to four bytes each, and the verb; the old 256-byte read could already cut a non-ASCII name short. On the NixOS host, a 255-character name of two-byte letters reaches the launcher whole, 256 characters are refused, and Firefox ESR launches.
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
What this changes
computer_actgets alaunchaction that opens an installed application ({"action": "launch", "app": "Firefox ESR"}) instead of making the model find and click its icon, and a new read-only tool,computer_apps, lists what it can open.computer_apps: the applications the desktop's menu shows, one line each: the name in the desktop's language, and the desktop file id (the entry's path underapplications/,/becoming-). Entries the menu would hide are left out:Hidden/NoDisplay, aTypeother thanApplication,OnlyShowIn/NotShowInfor another desktop (XFCE whenXDG_CURRENT_DESKTOPis unset, as it is outside the session), and a missingTryExecprogram. Escaped names (\s,\\, ...) are decoded.launch: takes an id or a name exactly as listed, ignoring case, the id first. A name two entries share (a system and a Flatpak Firefox) is refused with their ids; an unknown name is refused with a pointer tocomputer_apps. Nothing is guessed.exo-openand returns once a new window appears or the focus moves. If neither happens within ten seconds the launch fails and the rest of the batch stops, so nothing is typed into the window that had the focus before. The failed launch still counts againstmax_actions, since the application was started.launchandcomputer_appswith a pointer to that computer's own launcher (cmd+space, Start menu). Supporting self-hosted desktops is separate work.The model's instructions recommend
launchandcomputer_appsonly where the desktop could have them. On the person's own computer they say up front that neither works there, so the model goes straight to that computer's launcher. Through a daemon, which may or may not have the container's launcher, they say to callcomputer_appsfirst and fall back to the computer's own launcher if it is refused. They still tell it to wait after a shortcut that opens a window before typing into it.Why this way
Hunting for an icon in a screenshot was the slowest and least reliable step in every desktop task we tried. Giving the agent a shell instead was considered and rejected: a shell reaches much more than opening an app.
launchtakes a name and nothing else, validated so it can never be a path, an option or a second line, and it reachesexo-openonly as the matched entry's path, an argument.One lookup.
backend/app/agents/capabilities/computer_use/_desktop_apps.pyis standard library only. A sandbox runs its source withpython3 -c, and the computerd image copies it in beside the daemon. So the computerd image is now built from the repository root (docker build -f deploy/computer/Dockerfile -t agenticos-computer .), withdeploy/computer/Dockerfile.dockerignoreletting through only the three files it copies (a 453-byte context).Listing and exact match follow OpenClaw's
list_apps/launch_app. It also waits for the window before returning. Codex's desktop plugin is closed, and Hermes has no launch at all.computerd changes:
desk. A launched app must not run as the token's account, so a small launcher runs asdeskand computerd asks it over a unix socket in/run/launcher, which only those two accounts can reach. It answerslistandlaunch <app>with a JSON line, and computerd serves the list asGET /apps.tinias PID 1: it reaps the apps a launch orphans. Without it, zombies piled up under computerd.Launched apps get no pipes and a session of their own: an app inherits pipes and would hold the call open until it quits.
Checks
TryExecentries, escaped names), exact, ambiguous and unknown names, the window wait and its failure, the script's entry point, the sandbox, daemon, E2B and local backends, and the toolmake checkpasses. Locally,make lint-backendpasses andmake testran in full: 17,793 passed and 4 failed. The coverage gate is reached (100%). All 4 failures also fail in the latest CI run on the base branch,feat/desktop-native(listed below). CI on6c82268b5matches the base branch's latest run (f2ed3c111) exactly:testfails on the same 4 tests with coverage at 100%,test-frontendon the same 2 files (spend-tab.test.tsx,admin/settings/page.test.tsx), ande2eon the same 6mobile-surfacesspecs. This PR changes none of them.make test)README.mdsays why;docs/reference/capabilities.mdis updated in en, pl, de and esLive verification:
/appslistsdesk's 30 entries. LaunchingFirefox ESRreturned after 1.2 s, and the screenshot taken right after shows its window.xfce4-terminalopens asdesk. Unknown and malformed names are refused. A computerd in the same container withoutCOMPUTERD_LAUNCHERrefuses bothlaunchand/apps. In the container,/healthanswers 503 while the launcher is down, and the entrypoint exits if the launcher's socket never appears./appslists 30 entries. Firefox ESR launched in 1.1 s and the terminal in 0.27 s, both asdeskand both on the next screenshot. Refusals match. After the test apps were closed, no zombies remained undertini.listandlaunchrun asdeskin the same image withpython3 -c, on both hosts./healthanswer 503, and the health probes leave nothing in the launcher's log.deskin the NixOS container, lists the same 30 entries as the deployed one, so the visibility rules drop nothing the XFCE menu shows.Notes for the reviewer
Launched apps are never closed. A launch opens the app and nothing quits it afterwards, so it stays open when the reply ends, when the conversation goes quiet, and indefinitely on a persistent desktop. This branch had a version that quit everything a run launched when the run ended. It was reverted, because each chat message is its own run: "open the browser and go to X" followed by "now log in" found the browser closed. A conversation has no end event to hook, so the cleanup probably needs a policy of its own, such as quitting after the conversation goes idle or when it is deleted, or an explicit
quitaction. It is left out of this PR on purpose. The reverted commit (f86be3922) stays in the history as a starting point.Other gaps:
computer_appsfirst; a Mac or self-hosted computerd refuses it, and the model falls back to the computer's own launcher. That costs one call.exo-open. That fits the images we ship. Other desktops need agtk-launch/xdg-openpath later.feat/desktop-native, without this PR:tests/test_computer_use.py::TestTheTools::test_every_run_of_one_built_agent_has_a_session_of_its_owntests/test_capability_registry.py::TestCapabilitySettingsCopy::test_the_english_catalog_says_what_the_schema_says(browser_use settings copy)tests/test_migrations.py::TestLegacyCookieSessionRevocation::test_upgrade_revokes_old_refresh_chains_without_restoring_them_on_downgradetests/test_workspace_capacity_conflict.py::test_a_failed_recovery_is_visible_instead_of_reported_as_saved