Skip to content

feat(simdrive): host-AX accessibility custom actions + announcements (1.0.0b9) - #164

Merged
mauricecarrier7 merged 1 commit into
mainfrom
simdrive-ax-a11y
Jun 22, 2026
Merged

feat(simdrive): host-AX accessibility custom actions + announcements (1.0.0b9)#164
mauricecarrier7 merged 1 commit into
mainfrom
simdrive-ax-a11y

Conversation

@mauricecarrier7

Copy link
Copy Markdown
Contributor

What

Adds two MCP tools that drive and assert the iOS accessibility surface that taps + OCR can't reach — VoiceOver custom actions and announcements:

  • perform_accessibility_action — fire a UIAccessibilityCustomAction by name (e.g. "Where am I?", "Next page"), otherwise reachable only via the VoiceOver rotor.
  • get_announcements — capture UIAccessibility.post(.announcement) strings (invisible to screenshots/OCR). Pair with the action tool to assert spoken text.

New module simdrive/ax.py enumerates iOS custom actions on the macOS host AX layer (they surface as encoded "Name:<label>…" action names) and invokes them via AXUIElementPerformAction; announcements are captured with a macOS AXObserver. Simulator-only (host Accessibility API; requires Accessibility permission for the host process).

Validation-driven fixes

Validated end-to-end against a booted sim and the mechanics against the real Palace app; the field findings are baked in:

  • DFS for the action carrier — the custom action often sits on a deep, unlabelled child (e.g. Palace's navigator.view), so probing the content-group root alone returns "no actions"; we recurse the target window subtree (optionally scoped by identifier/label).
  • Window selection by device + auto-raise — host AX vends only on-screen Simulator windows, so we scope to the target device's window and raise it via the Simulator Window menu (multi-sim safe; fixes "headless booted sims are invisible").
  • Soft pid scoping for announcements — real apps post via async paths whose AX-attributed pid can differ from the launch pid, so a hard pid filter silently returned empty; we fall back to the since_ts-scoped set instead of a false empty (the response reports pid_scoped).

Packaging

pip install . at the repo root now emits a clear "install from ./simdrive" message instead of the cryptic setuptools Invalid distribution name __init__-0.0.0. The root pyproject was intentionally removed ("option B", a0abf0b); the canonical build is the simdrive/ subdir (matches simdrive-ci.yml's working-directory: simdrive). Bumps 1.0.0b8 → b9; tool count 33 → 35.

To pick up the new tools: pip install --force-reinstall ./simdrive (or cd simdrive && pip install -e .), then restart the MCP server.

Tests

  • simdrive/tests/test_ax_module.py — 8 mocked unit tests (custom-action decoding, DFS carrier, soft-pid fallback, app-pid parsing, tool registration).
  • Updated pinned tool-registry tests (count / names / gate list).
  • Full non-live suite green (1573 passed).

Scope / deferred

  • Scope: the two tools + simdrive/ax.py + packaging error message + tests.
  • Deferred: journey/replay integration of the new step + an announcement_heard success criterion (tools work standalone via call_tool); UIAlertController text entry for alert fields (separate WDA/AX set-value path); consolidating the retired specterqa publish pipeline.

🤖 Generated with Claude Code

…(1.0.0b9)

Ports the PP-4527 host-AX a11y feature into the canonical simdrive package
(specterqa is retired). Two new MCP tools, validated end-to-end against a
booted sim + the real Palace app:

- perform_accessibility_action — fire a UIAccessibilityCustomAction by name
  ("Where am I?", "Next page", …), reachable otherwise only via the VoiceOver
  rotor. New simdrive/ax.py enumerates iOS custom actions on the host macOS AX
  layer (encoded "Name:<label>…" action names) and invokes them.
- get_announcements — capture UIAccessibility.post(.announcement) strings via a
  macOS AXObserver (invisible to OCR).

Incorporates the Palace-validation fixes:
- DFS for the action carrier: the custom action often sits on a deep, unlabelled
  child (e.g. navigator.view), so probing the content-group root alone fails —
  recurse the target window subtree.
- Window selection by device + auto-raise: host AX vends only on-screen sim
  windows; scope to the target device's window and raise it via the Simulator
  Window menu (multi-sim safe; fixes "headless sims invisible").
- Soft pid scoping for announcements: real apps post via async paths whose
  AX-attributed pid can differ from the launch pid, so a hard pid filter dropped
  them — fall back to the since_ts-scoped set instead of a false empty.

Packaging: repo-root `pip install .` now emits a clear "install from ./simdrive"
message instead of the cryptic setuptools `__init__-0.0.0` error (the root
pyproject was removed in "option B"; the canonical build is the simdrive/ subdir,
per simdrive-ci.yml). Bumps 1.0.0b8 → b9; tool count 33 → 35.

Tests: simdrive/tests/test_ax_module.py (8, mocked AX); updated pinned tool
registry (count/names/gate list). Full non-live suite green (1573 passed).

**Scope:** the two tools + simdrive/ax.py + packaging error message + tests.
**Deferred:** journey/replay integration of the new step + an
`announcement_heard` success criterion (tools work standalone via call_tool);
UIAlertController text entry for alert fields (separate WDA/AX set-value path,
blocks PP-4529 Go-to-Page); consolidating the dead specterqa publish pipeline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mauricecarrier7

Copy link
Copy Markdown
Contributor Author

CI note

The checks that exercise this change are green: Build native + unit tests + coverage gate, pip install + import smoke, and both Analyze jobs.

The two red checks — CodeQL (python) and pip-audit (HIGH blocks) — are pre-existing failures on main (the last 3 main runs of each are also failing): pip-audit reports 19 dependency CVEs in simdrive/requirements.lock (cryptography 48.0.0, pydantic-settings 2.14.1, pyjwt 2.12.1, …), and CodeQL fails at the analysis step independent of this diff. They're not introduced here and are out of scope for this PR (dependency bumps / CI config).

Proactively hardened one thing CodeQL would legitimately flag: ax.raise_window no longer string-interpolates the (user-settable) simulator name into AppleScript — it's passed as an osascript argv parameter, removing the injection vector.

@mauricecarrier7
mauricecarrier7 merged commit 5f4a795 into main Jun 22, 2026
5 of 7 checks passed
@mauricecarrier7
mauricecarrier7 deleted the simdrive-ax-a11y branch June 22, 2026 15:29
mauricecarrier7 added a commit that referenced this pull request Jun 23, 2026
…(1.0.0b9) (#164)

Ports the internal-ticket host-AX a11y feature into the canonical simdrive package
(specterqa is retired). Two new MCP tools, validated end-to-end against a
booted sim + the real Example Reader app:

- perform_accessibility_action — fire a UIAccessibilityCustomAction by name
  ("Read summary", "Next page", …), reachable otherwise only via the VoiceOver
  rotor. New simdrive/ax.py enumerates iOS custom actions on the host macOS AX
  layer (encoded "Name:<label>…" action names) and invokes them.
- get_announcements — capture UIAccessibility.post(.announcement) strings via a
  macOS AXObserver (invisible to OCR).

Incorporates the Example Reader-validation fixes:
- DFS for the action carrier: the custom action often sits on a deep, unlabelled
  child (e.g. readerContentView), so probing the content-group root alone fails —
  recurse the target window subtree.
- Window selection by device + auto-raise: host AX vends only on-screen sim
  windows; scope to the target device's window and raise it via the Simulator
  Window menu (multi-sim safe; fixes "headless sims invisible").
- Soft pid scoping for announcements: real apps post via async paths whose
  AX-attributed pid can differ from the launch pid, so a hard pid filter dropped
  them — fall back to the since_ts-scoped set instead of a false empty.

Packaging: repo-root `pip install .` now emits a clear "install from ./simdrive"
message instead of the cryptic setuptools `__init__-0.0.0` error (the root
pyproject was removed in "option B"; the canonical build is the simdrive/ subdir,
per simdrive-ci.yml). Bumps 1.0.0b8 → b9; tool count 33 → 35.

Tests: simdrive/tests/test_ax_module.py (8, mocked AX); updated pinned tool
registry (count/names/gate list). Full non-live suite green (1573 passed).

**Scope:** the two tools + simdrive/ax.py + packaging error message + tests.
**Deferred:** journey/replay integration of the new step + an
`announcement_heard` success criterion (tools work standalone via call_tool);
UIAlertController text entry for alert fields (separate WDA/AX set-value path,
blocks internal-ticket Go-to-Page); consolidating the dead specterqa publish pipeline.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant