Skip to content

Add working Linux interaction engine (AT-SPI2 perception, X11 input, X11 capture) - #4

Open
devin-ai-integration[bot] wants to merge 5 commits into
devin/1784740678-linux-supportfrom
devin/1784750227-linux-engine
Open

Add working Linux interaction engine (AT-SPI2 perception, X11 input, X11 capture)#4
devin-ai-integration[bot] wants to merge 5 commits into
devin/1784740678-linux-supportfrom
devin/1784750227-linux-engine

Conversation

@devin-ai-integration

Copy link
Copy Markdown

What changed

Builds on #3 (Linux compile/test support) and replaces the fail-closed Linux stubs with a working interaction engine, so the server can actually perceive and drive apps on X11 Linux:

  • Perception via AT-SPI2: new CAtSpi system-library target (pkg-config atspi-2). The Linux AXUIElement shim now wraps AtspiAccessible*, implementing the AX attribute surface (role/title/value/children/parent/extents/enabled/focused/selected, actions) with AT-SPI role→AX-role mapping (push buttonAXButton, entryAXTextField, …). TreeBuilder/snapshots/locators work unchanged. NSWorkspace.runningApplications is backed by AT-SPI desktop children, so list_apps/app targeting resolve real apps.
  • Input via X11/XTest: new CX11 system-library target (x11, xtst, png). deliverClick/deliverScroll (wheel buttons 4–7)/deliverDrag/deliverKey/typeUnicodeText are real; Unicode text uses temporary spare-keycode remapping (keyboard-map lock + settle delays). Tiers are honest: XTest is global, so outcomes report tier4-global-cursor/tier4-global-xtest with an x11-global-input fallback reason — no pretense of macOS-style background-safety.
  • Capture via X11: captureWindow does an XGetImage root-window crop of the window frame, RGBA-converted and encoded with libpng, honoring ScreenshotDetail scaling — get_app_state now returns real screenshots on Linux.
  • System tools: list_windows (AT-SPI frames/titles/focus), clipboard read/write via checked xclip/xsel subprocess with verify-and-restore, wait_for via AT-SPI tree polling, open_app (direct exec or gtk-launch), open_url (xdg-open).
  • health_report --json now reports AT-SPI2 accessibility granted, X11/XTest input availability, and X11 capture responsiveness; README's Linux boundary updated (Wayland, overlay, OCR, skills recorder remain unsupported/macOS-only).
  • Package.swift: CAtSpi/CX11 dependencies and linker settings are .when(platforms: [.linux])-gated; macOS build is untouched.

Type of change

  • New feature / capability
  • Bug fix
  • Docs only
  • Refactor / internal (no behavior change)
  • Safety-policy change

Verification

All on Ubuntu 22.04 (X11, Swift 6.0.3); macOS lane via hosted CI:

  • swift build
  • swift test — 398 tests passed
  • CLI smoke: version, help, health_report --json
  • scripts/preflight.py --bin .build/debug/computer-use-mcp — passed
  • Local live verification (Linux, self-launched GTK apps on DISPLAY=:0):
    • call list_apps shows real AT-SPI apps; get_app_state returns a real element tree and a rendered 420x209 PNG screenshot (validated with ImageMagick)
    • call click on a GTK button → follow-up state shows the "clicked" label; type_text round trips ASCII and Unicode (hé☃) via read_text; ctrl+a + replacement verified modifier delivery
    • clipboard round trip (28 chars incl. non-ASCII), wait_for observed a live state change, open_app launched gtk3-demo, list_windows returned real frames

Checklist

  • One logical change; commits are granular and messages explain the why.
  • Tool schemas stay backwards-compatible (or the break is called out).
  • Safety-policy changes include tests for both the gated and the confirmed/recovery path. (N/A — safety gates unchanged; X11 paths fail closed when DISPLAY/XTest unavailable.)
  • No default safety gate, app lease, or daemon isolation was disabled to make a test pass.
  • No live GUI/TCC/input checks added to the default suite or hosted CI.
  • Relevant docs updated (README / CHANGELOG / contracts under docs/).

Link to Devin session: https://app.devin.ai/sessions/5f22c5a8af704f368b963a71f94eb46f
Requested by: @minghinmatthewlam

devin-ai-integration Bot and others added 3 commits July 22, 2026 20:07
Co-Authored-By: Matthew Lam <minghinmatthew.lam@gmail.com>
Co-Authored-By: Matthew Lam <minghinmatthew.lam@gmail.com>
Co-Authored-By: Matthew Lam <minghinmatthew.lam@gmail.com>
@minghinmatthewlam minghinmatthewlam self-assigned this Jul 22, 2026
@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

devin-ai-integration Bot and others added 2 commits July 22, 2026 20:41
Co-Authored-By: Matthew Lam <minghinmatthew.lam@gmail.com>
Co-Authored-By: Matthew Lam <minghinmatthew.lam@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

Recorded a live Ubuntu desktop demo of the agent driving a real GTK app end-to-end through the MCP CLI (video in the Devin session).

Works on camera

  • open_app launched gtk3-widget-factory; list_apps lists it
  • get_app_state returns a real AT-SPI element tree and a genuine X11 screenshot
  • click visibly toggled a checkbutton; batch click+ctrl+a+type_text visibly replaced the entry text with "REPLACED ✓ 日本語" (Unicode incl. CJK)
  • ✅ Clipboard write→read round trip; health_report shows accessibility granted / capture responsive
🔴 Before click 🟢 After call click
before after

batch ctrl+a + Unicode replacement visible in the app

Bugs found during testing — both fixed in 868a39b and re-verified live:

  1. BadMatch capture crash on partially-offscreen windows → capture rect is now clamped to screen bounds with a CX11 X error handler; offscreen windows return a clamped screenshot or a structured error instead of killing the daemon.
  2. Unicode type_text false success (typed into whichever window held X focus) → the engine now finds the target app's X11 window via _NET_WM_PID, activates it, verifies X input focus reached the target pid, and clicks the target element before typing; if focus can't be established it refuses with X11 focus did not reach the target app (pid …); refusing to send global keyboard input.
Minor notes
  • Linux health wording clarified: X11 display access provides screenshots; macOS Screen Recording permission is not applicable
  • press_key uses combo strings ("ctrl+a"), not a modifiers array
  • repeated get_app_state prints "tree unchanged" without the tree — use find to re-locate elements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant