feat(simdrive): host-AX accessibility custom actions + announcements (1.0.0b9) - #164
Conversation
…(1.0.0b9)
Ports the PP-4527 host-AX a11y feature into the canonical simdrive package
(specterqa is retired). Two new MCP tools, validated end-to-end against a
booted sim + the real Palace app:
- perform_accessibility_action — fire a UIAccessibilityCustomAction by name
("Where am I?", "Next page", …), reachable otherwise only via the VoiceOver
rotor. New simdrive/ax.py enumerates iOS custom actions on the host macOS AX
layer (encoded "Name:<label>…" action names) and invokes them.
- get_announcements — capture UIAccessibility.post(.announcement) strings via a
macOS AXObserver (invisible to OCR).
Incorporates the Palace-validation fixes:
- DFS for the action carrier: the custom action often sits on a deep, unlabelled
child (e.g. navigator.view), so probing the content-group root alone fails —
recurse the target window subtree.
- Window selection by device + auto-raise: host AX vends only on-screen sim
windows; scope to the target device's window and raise it via the Simulator
Window menu (multi-sim safe; fixes "headless sims invisible").
- Soft pid scoping for announcements: real apps post via async paths whose
AX-attributed pid can differ from the launch pid, so a hard pid filter dropped
them — fall back to the since_ts-scoped set instead of a false empty.
Packaging: repo-root `pip install .` now emits a clear "install from ./simdrive"
message instead of the cryptic setuptools `__init__-0.0.0` error (the root
pyproject was removed in "option B"; the canonical build is the simdrive/ subdir,
per simdrive-ci.yml). Bumps 1.0.0b8 → b9; tool count 33 → 35.
Tests: simdrive/tests/test_ax_module.py (8, mocked AX); updated pinned tool
registry (count/names/gate list). Full non-live suite green (1573 passed).
**Scope:** the two tools + simdrive/ax.py + packaging error message + tests.
**Deferred:** journey/replay integration of the new step + an
`announcement_heard` success criterion (tools work standalone via call_tool);
UIAlertController text entry for alert fields (separate WDA/AX set-value path,
blocks PP-4529 Go-to-Page); consolidating the dead specterqa publish pipeline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
e858021 to
5f075cf
Compare
CI noteThe checks that exercise this change are green: Build native + unit tests + coverage gate, pip install + import smoke, and both Analyze jobs. The two red checks — CodeQL (python) and pip-audit (HIGH blocks) — are pre-existing failures on Proactively hardened one thing CodeQL would legitimately flag: |
…(1.0.0b9) (#164) Ports the internal-ticket host-AX a11y feature into the canonical simdrive package (specterqa is retired). Two new MCP tools, validated end-to-end against a booted sim + the real Example Reader app: - perform_accessibility_action — fire a UIAccessibilityCustomAction by name ("Read summary", "Next page", …), reachable otherwise only via the VoiceOver rotor. New simdrive/ax.py enumerates iOS custom actions on the host macOS AX layer (encoded "Name:<label>…" action names) and invokes them. - get_announcements — capture UIAccessibility.post(.announcement) strings via a macOS AXObserver (invisible to OCR). Incorporates the Example Reader-validation fixes: - DFS for the action carrier: the custom action often sits on a deep, unlabelled child (e.g. readerContentView), so probing the content-group root alone fails — recurse the target window subtree. - Window selection by device + auto-raise: host AX vends only on-screen sim windows; scope to the target device's window and raise it via the Simulator Window menu (multi-sim safe; fixes "headless sims invisible"). - Soft pid scoping for announcements: real apps post via async paths whose AX-attributed pid can differ from the launch pid, so a hard pid filter dropped them — fall back to the since_ts-scoped set instead of a false empty. Packaging: repo-root `pip install .` now emits a clear "install from ./simdrive" message instead of the cryptic setuptools `__init__-0.0.0` error (the root pyproject was removed in "option B"; the canonical build is the simdrive/ subdir, per simdrive-ci.yml). Bumps 1.0.0b8 → b9; tool count 33 → 35. Tests: simdrive/tests/test_ax_module.py (8, mocked AX); updated pinned tool registry (count/names/gate list). Full non-live suite green (1573 passed). **Scope:** the two tools + simdrive/ax.py + packaging error message + tests. **Deferred:** journey/replay integration of the new step + an `announcement_heard` success criterion (tools work standalone via call_tool); UIAlertController text entry for alert fields (separate WDA/AX set-value path, blocks internal-ticket Go-to-Page); consolidating the dead specterqa publish pipeline. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What
Adds two MCP tools that drive and assert the iOS accessibility surface that taps + OCR can't reach — VoiceOver custom actions and announcements:
perform_accessibility_action— fire aUIAccessibilityCustomActionby name (e.g. "Where am I?", "Next page"), otherwise reachable only via the VoiceOver rotor.get_announcements— captureUIAccessibility.post(.announcement)strings (invisible to screenshots/OCR). Pair with the action tool to assert spoken text.New module
simdrive/ax.pyenumerates iOS custom actions on the macOS host AX layer (they surface as encoded"Name:<label>…"action names) and invokes them viaAXUIElementPerformAction; announcements are captured with a macOSAXObserver. Simulator-only (host Accessibility API; requires Accessibility permission for the host process).Validation-driven fixes
Validated end-to-end against a booted sim and the mechanics against the real Palace app; the field findings are baked in:
navigator.view), so probing the content-group root alone returns "no actions"; we recurse the target window subtree (optionally scoped byidentifier/label).since_ts-scoped set instead of a false empty (the response reportspid_scoped).Packaging
pip install .at the repo root now emits a clear "install from./simdrive" message instead of the cryptic setuptoolsInvalid distribution name __init__-0.0.0. The root pyproject was intentionally removed ("option B", a0abf0b); the canonical build is thesimdrive/subdir (matchessimdrive-ci.yml'sworking-directory: simdrive). Bumps1.0.0b8 → b9; tool count33 → 35.To pick up the new tools:
pip install --force-reinstall ./simdrive(orcd simdrive && pip install -e .), then restart the MCP server.Tests
simdrive/tests/test_ax_module.py— 8 mocked unit tests (custom-action decoding, DFS carrier, soft-pid fallback, app-pid parsing, tool registration).Scope / deferred
simdrive/ax.py+ packaging error message + tests.announcement_heardsuccess criterion (tools work standalone viacall_tool);UIAlertControllertext entry for alert fields (separate WDA/AX set-value path); consolidating the retiredspecterqapublish pipeline.🤖 Generated with Claude Code