feat(simdrive): a11y journeys + set_text for alert fields (1.0.0b10) - #165
Merged
Conversation
…ents Wires the host-AX tools into the LLM journey runner: - New step tool `perform_accessibility_action` — added to StepDecision's tool set, the runner dispatch map, and the system prompt (so the agent can invoke a VoiceOver custom action by name during a journey). - New success criterion `announcement_heard: "<text>"` — the runner injects the announcements captured so far into the observation (simulator-only, best-effort), and a pure evaluator checks the spoken text (case-insensitive substring). This makes a full accessibility journey assertable end-to-end: drive to the reader → perform "Where am I?" → succeed on announcement_heard. Tests: tests/test_journey_ax.py (7); full non-live suite green (1580 passed). **Scope:** journey step + criterion + prompt + tests. **Deferred:** UIAlertController text entry (next). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HID type_text can't land in a UIAlertController prompt (e.g. a "Go to Page" dialog) — the field never receives synthesized keystrokes. Add a host-AX set-value path that the app actually receives: - ax.set_text() + tool `set_text`: set a field's AXValue directly. Verified the value propagates to the field's binding (the app reads it), against a real SwiftUI .alert text field. Targets by identifier/label, else the first editable field in the window (the alert's field when a prompt is up). - Journey wiring: `set_text` added to the runner's tool set + dispatch + prompt (alongside the perform_accessibility_action step from the previous commit). - TestKitApp FormTab gains a "Go to Page" alert fixture to exercise it. Tool count 35 → 36; bumps 1.0.0b9 → b10. Tests: ax set_text field-find + tool registration + journey dispatch. Full non-live suite green (1583 passed). **Scope:** set_text + alert fixture + journey wiring + tests. **Deferred:** label-scoping by placeholder text (AXPlaceholderValue) — first-field default covers alerts; device-target alert entry (WDA) unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mauricecarrier7
added a commit
that referenced
this pull request
Jun 23, 2026
…165) * feat(simdrive): journey support for accessibility actions + announcements Wires the host-AX tools into the LLM journey runner: - New step tool `perform_accessibility_action` — added to StepDecision's tool set, the runner dispatch map, and the system prompt (so the agent can invoke a VoiceOver custom action by name during a journey). - New success criterion `announcement_heard: "<text>"` — the runner injects the announcements captured so far into the observation (simulator-only, best-effort), and a pure evaluator checks the spoken text (case-insensitive substring). This makes a full accessibility journey assertable end-to-end: drive to the reader → perform "Read summary" → succeed on announcement_heard. Tests: tests/test_journey_ax.py (7); full non-live suite green (1580 passed). **Scope:** journey step + criterion + prompt + tests. **Deferred:** UIAlertController text entry (next). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(simdrive): set_text for alert fields via host-AX (1.0.0b10) HID type_text can't land in a UIAlertController prompt (e.g. a "Go to Page" dialog) — the field never receives synthesized keystrokes. Add a host-AX set-value path that the app actually receives: - ax.set_text() + tool `set_text`: set a field's AXValue directly. Verified the value propagates to the field's binding (the app reads it), against a real SwiftUI .alert text field. Targets by identifier/label, else the first editable field in the window (the alert's field when a prompt is up). - Journey wiring: `set_text` added to the runner's tool set + dispatch + prompt (alongside the perform_accessibility_action step from the previous commit). - TestKitApp FormTab gains a "Go to Page" alert fixture to exercise it. Tool count 35 → 36; bumps 1.0.0b9 → b10. Tests: ax set_text field-find + tool registration + journey dispatch. Full non-live suite green (1583 passed). **Scope:** set_text + alert fixture + journey wiring + tests. **Deferred:** label-scoping by placeholder text (AXPlaceholderValue) — first-field default covers alerts; device-target alert entry (WDA) unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-on to #164. Two pieces, both validated end-to-end against a booted sim.
1. Accessibility journeys
The host-AX tools are now usable from the LLM journey runner:
perform_accessibility_actionstep — added to the runner's tool set, dispatch map, and system prompt, so a journey can invoke a VoiceOver custom action by name.announcement_heardsuccess criterion — the runner injects the announcements captured so far into the observation (simulator-only, best-effort) and a pure evaluator asserts the spoken text (case-insensitive substring).A full VoiceOver flow is now expressible as a journey: drive to the reader →
perform_accessibility_action "Where am I?"→ succeed onannouncement_heard.2.
set_text— text entry into fields HID can't reachHID
type_text(synthesized keystrokes) doesn't land in aUIAlertControllerprompt (e.g. a "Go to Page" dialog) — the field never receives the keys.set_textsets the field'sAXValuedirectly via the host Accessibility API.Verified the value propagates to the field's binding (the app actually reads it), against a real SwiftUI
.alerttext field —set_text "42"→ tap Go → app shows "Went to page 42". Targets byidentifier/label, or defaults to the first editable field in the window (the alert's field when a prompt is up). Simulator-only;TestKitAppFormTab gains a "Go to Page" alert fixture to exercise it.This unblocks the PP-4529 Go-to-Page found/not-found branches that HID couldn't drive.
Packaging / tests
34 → 36(perform_accessibility_action/get_announcementsfrom feat(simdrive): host-AX accessibility custom actions + announcements (1.0.0b9) #164, nowset_text); bumps1.0.0b9 → b10.ax.set_textfield-find, tool registration; updated pinned tool registry. Full non-live suite green (1583 passed).CI note
As on #164 and
main, CodeQL and pip-audit fail pre-existing (dependency CVEs inrequirements.lock; CodeQL config) — unrelated to this diff.Scope / deferred
set_text+ alert fixture, tests.AXPlaceholderValue) — the first-editable-field default covers alerts; device-target (WDA) alert entry unchanged.🤖 Generated with Claude Code