Skip to content

windows: add targeted UIA lookup and fix MCP schemas#3

Merged
voctory merged 6 commits into
mainfrom
feature/windows-cua-automation-integration
May 8, 2026
Merged

windows: add targeted UIA lookup and fix MCP schemas#3
voctory merged 6 commits into
mainfrom
feature/windows-cua-automation-integration

Conversation

@voctory

@voctory voctory commented May 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a targeted find_element MCP tool for locating UIA elements by query, label, automation id, identifier, control type, and browser/page search zones.

Improves Windows action routing by preferring supported UIA action patterns before MSAA fallback, treating UIA link controls as actionable, and keeping the named-pipe daemon alive when clients disconnect during slow requests.

Makes MCP array schemas declare items and adds schema coverage for OpenAI-compatible tool definitions.

Updates the Windows build helper to honor global.json SDK roll-forward by accepting the SDK that dotnet resolves from the repo root.

Related Issue

Closes #2

Area

  • Windows driver
  • macOS driver
  • MCP tools
  • Input routing
  • Browser automation
  • Capture or screenshots
  • Visual agent cursor
  • Recording or replay
  • Configuration
  • Packaging or install
  • Documentation
  • Tests

Safety Contract

  • Mutating actions do not move the user's real cursor.
  • Mutating actions do not steal foreground focus.
  • Unsafe routes are explicit opt-ins or refusals, not hidden fallbacks.
  • Receipts report the actual route and lane.
  • Receipts preserve background_safe, cursor_moved, and foreground_changed.
  • Browser routes avoid unsafe UIA/provider fallbacks unless explicitly configured.

Validation

  • dotnet build trope-cua.sln
  • scripts\run-tests.ps1
  • cd native/macos/trope-cua && swift build
  • cd native/macos/trope-cua && ./scripts/test.sh
  • git diff --check
  • Installed or restarted the affected daemon for runtime/install changes.
  • Added or updated targeted regression coverage.
  • Docs-only change; runtime tests are not applicable.

Evidence

scripts\run-tests.ps1: 78 passed in 30.54s after the SDK roll-forward helper fix.

git diff --check: passed.

Risks and Follow-Up

An earlier local full-suite run had one cursor-position mismatch in test_visual_move_cursor_does_not_move_real_cursor; the isolated test passed immediately afterward and subsequent full-suite runs passed. macOS validation was not run locally because this PR changes the Windows driver/MCP/build-script path only; GitHub macOS CI passed on the first PR run.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@voctory

voctory commented May 8, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@voctory
voctory merged commit a99c612 into main May 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI API MCP JSON Schema compatibility issue

1 participant