Skip to content

Commit 6df46c4

Browse files
authored
fix(ios): Simulator AX bridge reports enabled from the NotEnabled trait (#2644)
* fix(ios): Simulator AX bridge reports enabled from the NotEnabled trait The bridge requested no state attribute, so a disabled control read as a plain button and clients could not observe it; the XCTest runner answered the same screen with `enabled: false`. The guest now reads `XC_kAXXCAttributeTraits` and the decoder derives `enabled` from `UIAccessibilityTraitNotEnabled`. The word is masked as a BigInt because `UIAccessibilityTraitToggleButton` is 2^53, so every switch carries a value past the safe integer range; a missing word leaves `enabled` unknown and a non-integer or negative one is a malformed tree. Source version v1.6.0 rebuilds the cached bridge. Verified on an iPhone 17 Pro simulator (iOS 26.5) against a React Native app with `Pressable disabled` and `accessibilityState={{ disabled: true }}`: the bridge-served raw snapshot now carries `enabled: false` on both controls, and a client `toBeDisabled` assertion that failed on 0.21.1 passes. * fix(ios): a source-declared disabled node folds to non-hittable without hittability evidence Review found the new `enabled` fact never reached presentation on the bridge path: hittability evidence is unavailable there, so the fold returned before consulting `enabled`, and a disabled control kept an open `hittable`. The fold now marks a node the source declared disabled as `hittable: false` regardless of evidence availability, which is what the runner path already presents. The tree test asserts the typed `failureCode` instead of matching error text. * fix(ios): a promoted navigation title affordance retracts the field's hittability The navigation title rule presents a disabled title field as an enabled Button for the whole row, but the fold had already marked the field `hittable: false` from its disabled state, so the Button carried a claim it contradicted; the cloud iOS provider scenario caught it. A replacement patch may now retract a fact by setting it `undefined`, and the rule retracts `hittable`, leaving the row's actionability to the evidence the acquisition actually has. * fix(ios): keep retracted facts through later patches and carry the traits word exactly Review (thymikee): `currentReplacement` rebuilt a node from its source under the stored replacement, so a later patch on the same node — a scroll clip, a row label — brought back a fact an earlier rule had retracted: the web Heading's `value`, and the promoted navigation title's `hittable`. The stored replacement is now authoritative, as every other reader of the map already treats it; a regression applies two patches in sequence. The guest sends the accessibility traits word as a decimal string and the decoder parses it with BigInt, so a bit set past 2^53 cannot round into or out of the NotEnabled bit on the way through JSON; the test covers a word past 2^60. The navigation title test moves to transitions.test.ts, keeping presentation.test.ts under the size ratchet. The CHANGELOG names the two presentation effects: bridge disabled controls leave the interactive and Maestro atomic-dispatch counts, and the runner-path title Button joins them.
1 parent ef62c7f commit 6df46c4

13 files changed

Lines changed: 197 additions & 8 deletions

File tree

CHANGELOG.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,18 @@
77
`session.name` (`default`) instead of the store address (`cwd:<hash>:default`), so the record
88
read as missing, the `log stream` child leaked, and the next `logs start` on that device failed
99
with "has not reached a confirmed terminal state" (#2647).
10+
- Fixed (ios): Simulator AX bridge snapshots report `enabled`. The bridge requested no state
11+
attribute, so a disabled control — a React Native `Pressable` with `disabled`, for example — read
12+
as a plain button while the XCTest runner answered the same screen with `enabled: false`. The
13+
bridge now reads the element's accessibility traits (sent as a decimal string, since the word has
14+
bits past 2^53) and derives `enabled` from `UIAccessibilityTraitNotEnabled`. Source version
15+
`agent-device-simulator-ax-v1.6.0` rebuilds the cached bridge on first use.
16+
- Changed (ios): a node the source declares disabled is presented `hittable: false` even when the
17+
capture has no hittability evidence, so on the Simulator bridge a disabled control stops counting
18+
as an interactive node and as a Maestro atomic-dispatch candidate. The navigation title affordance
19+
(a disabled title field presented as an enabled Button for the whole row) no longer carries the
20+
field's `hittable: false`; on the XCTest runner path that Button now counts as interactive and
21+
becomes a Maestro atomic-dispatch candidate where it was excluded before.
1022
- Added (limrun): `longpress` on Limrun iOS direct sessions. The interactor refused it as
1123
unsupported although the SDK exposes the HID primitives; it now holds one touch as a
1224
`performActions` batch of `touchDown`, `wait`, `touchUp`, defaulting to the 800 ms the Android

apple/snapshot-bridge/SnapshotBridgeRuntime.m

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
NSString *const kProtocolVersionKey = @"protocolVersion";
1919
NSString *const kSourceVersionKey = @"sourceVersion";
2020
NSString *const kRequestIdKey = @"requestId";
21-
NSString *const kSourceVersion = @"agent-device-simulator-ax-v1.5.5";
21+
NSString *const kSourceVersion = @"agent-device-simulator-ax-v1.6.0";
2222
const NSUInteger kProtocolVersion = 1;
2323
const uint32_t kMaximumFrameBytes = 16 * 1024 * 1024;
2424
const NSUInteger kMaximumDepth = 128;
@@ -32,6 +32,7 @@
3232
static NSString *const kAttributeIdentifier = @"XC_kAXXCAttributeIdentifier";
3333
static NSString *const kAttributeFrame = @"XC_kAXXCAttributeFrame";
3434
static NSString *const kAttributeAutomationType = @"XC_kAXXCAttributeAutomationType";
35+
static NSString *const kAttributeTraits = @"XC_kAXXCAttributeTraits";
3536
static NSString *const kAttributeChildren = @"XC_kAXXCAttributeChildren";
3637
static NSString *const kSnapshotAttributes = @"UIAccessibilitySnapshotKeyAttributes";
3738
static NSString *const kSnapshotChildren = @"UIAccessibilitySnapshotKeyChildren";
@@ -185,6 +186,11 @@ - (BOOL)isPrimaryForegroundProcess:(pid_t)pid
185186
- (nullable id)jsonValue:(id)value name:(NSString *)name
186187
{
187188
if (!value || value == [NSNull null]) return nil;
189+
// The traits word is a uint64 bit set; JSON numbers lose its high bits past 2^53, a decimal
190+
// string keeps every bit for the host to parse exactly.
191+
if ([name isEqualToString:kAttributeTraits] && [value isKindOfClass:NSNumber.class]) {
192+
return ((NSNumber *)value).stringValue;
193+
}
188194
if ([value isKindOfClass:NSString.class] || [value isKindOfClass:NSNumber.class]) return value;
189195

190196
const void *raw = (__bridge const void *)value;
@@ -291,6 +297,7 @@ - (nullable NSDictionary *)snapshotForProcess:(pid_t)pid
291297
kAttributeIdentifier,
292298
kAttributeFrame,
293299
kAttributeAutomationType,
300+
kAttributeTraits,
294301
kAttributeChildren,
295302
];
296303
NSArray<NSNumber *> *numbers = _attributeNumbersForNames(names);

packages/capture-kit/src/ios-snapshot-engine/engine.test.ts

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -324,6 +324,22 @@ test('unavailable hittability never becomes regular actionability', () => {
324324
);
325325
});
326326

327+
test('a source-declared disabled node is not actionable without hittability evidence', () => {
328+
const request = createIosSnapshotRequest();
329+
const nodes = nestedNodes().map((entry) =>
330+
entry.label === 'Partially visible' ? { ...entry, enabled: false } : entry,
331+
);
332+
const unavailable = {
333+
...acquisition(request, nodes),
334+
residue: [{ kind: 'unavailable-fact' as const, fact: 'hittability' as const }],
335+
} satisfies IosSnapshotAcquisition;
336+
const acquired = publishIosSnapshot({ stage: 'acquired', acquisition: unavailable }, request);
337+
const disabled = acquired.payload.nodes.find((node) => node.label === 'Partially visible');
338+
assert.ok(disabled);
339+
assert.equal(disabled.enabled, false);
340+
assert.equal(disabled.hittable, false);
341+
});
342+
327343
test('interactive compaction stays available through the engine boundary', () => {
328344
const rowRect = { x: 16, y: 80, width: 288, height: 52 };
329345
const compacted = presentIosInteractiveSnapshot([

packages/capture-kit/src/ios-snapshot-engine/geometry.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ function foldedHittability(
139139
options: IosSnapshotFoldOptions,
140140
): Partial<Pick<RawSnapshotNode, 'hittable'>> {
141141
if (options.hittabilityAvailable === false) {
142-
return sourceHittable === false ? { hittable: false } : {};
142+
return sourceHittable === false || !enabled ? { hittable: false } : {};
143143
}
144144
return {
145145
hittable:
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
import { expect, test } from 'vitest';
2+
import type { RawSnapshotNode } from '@agent-device/kernel/snapshot';
3+
import { presentIosInteractiveSnapshot } from '@agent-device/capture-kit/ios-snapshot-engine';
4+
5+
test('a disabled navigation title field is promoted to a Button without its hittability', () => {
6+
const nodes: RawSnapshotNode[] = [
7+
{
8+
index: 0,
9+
depth: 0,
10+
type: 'Application',
11+
label: 'Demo',
12+
rect: { x: 0, y: 0, width: 390, height: 844 },
13+
},
14+
{
15+
index: 1,
16+
depth: 1,
17+
parentIndex: 0,
18+
type: 'NavigationBar',
19+
label: 'Team Standup',
20+
rect: { x: 0, y: 56, width: 390, height: 44 },
21+
},
22+
{
23+
index: 2,
24+
depth: 2,
25+
parentIndex: 1,
26+
type: 'Image',
27+
identifier: 'RoomDetailsIconImageView',
28+
rect: { x: 81, y: 80, width: 14, height: 14 },
29+
},
30+
{
31+
index: 3,
32+
depth: 2,
33+
parentIndex: 1,
34+
type: 'TextField',
35+
label: 'Team Standup',
36+
value: 'Team Standup',
37+
identifier: 'DisplayNameTextField',
38+
enabled: false,
39+
hittable: false,
40+
rect: { x: 100, y: 67, width: 113, height: 22 },
41+
},
42+
{
43+
index: 4,
44+
depth: 2,
45+
parentIndex: 1,
46+
type: 'StaticText',
47+
label: 'Team Standup',
48+
rect: { x: 219, y: 58, width: 85, height: 40 },
49+
},
50+
];
51+
52+
const presented = presentIosInteractiveSnapshot(nodes);
53+
const affordance = presented.find((node) => node.identifier === 'DisplayNameTextField');
54+
55+
expect(affordance).toMatchObject({ type: 'Button', label: 'Team Standup', enabled: true });
56+
expect(affordance && 'hittable' in affordance).toBe(false);
57+
});

packages/capture-kit/src/ios-snapshot-engine/transitions.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,12 @@ function collectNavigationTitleAffordances(
2828
);
2929
if (candidates.length !== 1) continue;
3030
const { field, title, image, label } = candidates[0]!;
31+
// The affordance is the whole row, so the disabled field's own actionability does not carry.
3132
mergeReplacement(context.replacements, field, {
3233
type: 'Button',
3334
label,
3435
enabled: true,
36+
hittable: undefined,
3537
rect: unionRects([image.rect!, field.rect!, title.rect!]),
3638
});
3739
context.semanticRepresentativeIndexes.add(field.index);

packages/capture-kit/src/ios-snapshot-engine/tree.test.ts

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,3 +18,23 @@ test('replacement updates derive patches from the composed node', () => {
1818
hiddenContentBelow: true,
1919
});
2020
});
21+
22+
test('a retracted fact stays retracted when a later rule patches the same node', () => {
23+
const heading: RawSnapshotNode = {
24+
index: 3,
25+
depth: 3,
26+
parentIndex: 2,
27+
type: 'Other',
28+
label: 'Welcome',
29+
value: '1',
30+
rect: { x: 0, y: 700, width: 390, height: 300 },
31+
};
32+
const replacements = new Map<number, RawSnapshotNode>();
33+
mergeReplacement(replacements, heading, { type: 'Heading', value: undefined });
34+
mergeReplacement(replacements, heading, { rect: { x: 0, y: 700, width: 390, height: 144 } });
35+
updateReplacement(replacements, heading, () => ({ label: 'Welcome!' }));
36+
37+
const presented = replacements.get(heading.index);
38+
expect(presented).toMatchObject({ type: 'Heading', label: 'Welcome!' });
39+
expect(presented && 'value' in presented).toBe(false);
40+
});

packages/capture-kit/src/ios-snapshot-engine/tree.ts

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -187,12 +187,13 @@ export function isRepeatedStaticNode(node: RawSnapshotNode, parentLabel: string)
187187
return type === 'other' || type === 'statictext' || type === 'link';
188188
}
189189

190+
/** A patch key set to `undefined` retracts that fact from the presented node. */
190191
export function mergeReplacement(
191192
replacements: Map<number, RawSnapshotNode>,
192193
node: RawSnapshotNode,
193194
patch: Partial<RawSnapshotNode>,
194195
): void {
195-
replacements.set(node.index, { ...currentReplacement(replacements, node), ...patch });
196+
replacements.set(node.index, patched(currentReplacement(replacements, node), patch));
196197
}
197198

198199
export function updateReplacement(
@@ -201,14 +202,22 @@ export function updateReplacement(
201202
update: (current: RawSnapshotNode) => Partial<RawSnapshotNode>,
202203
): void {
203204
const current = currentReplacement(replacements, node);
204-
replacements.set(node.index, { ...current, ...update(current) });
205+
replacements.set(node.index, patched(current, update(current)));
206+
}
207+
208+
function patched(current: RawSnapshotNode, patch: Partial<RawSnapshotNode>): RawSnapshotNode {
209+
const next: Record<string, unknown> = { ...current, ...patch };
210+
for (const [key, value] of Object.entries(patch)) {
211+
if (value === undefined) delete next[key];
212+
}
213+
return next as RawSnapshotNode;
205214
}
206215

207216
function currentReplacement(
208217
replacements: ReadonlyMap<number, RawSnapshotNode>,
209218
node: RawSnapshotNode,
210219
): RawSnapshotNode {
211-
return { ...node, ...replacements.get(node.index) };
220+
return replacements.get(node.index) ?? node;
212221
}
213222

214223
export function findLargestViewportRect(nodes: Iterable<RawSnapshotNode>): RawSnapshotNode['rect'] {

packages/platform-apple/src/snapshot-source/fixtures/wire-vocabulary.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"protocolVersion": 1,
3-
"sourceVersion": "agent-device-simulator-ax-v1.5.5",
3+
"sourceVersion": "agent-device-simulator-ax-v1.6.0",
44
"requestKeys": [
55
"verb",
66
"requestId",
@@ -37,6 +37,7 @@
3737
"XC_kAXXCAttributeIdentifier",
3838
"XC_kAXXCAttributeFrame",
3939
"XC_kAXXCAttributeAutomationType",
40+
"XC_kAXXCAttributeTraits",
4041
"XC_kAXXCAttributeChildren"
4142
]
4243
}

packages/platform-apple/src/snapshot-source/protocol.test.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,7 @@ test('wire vocabulary guard keeps TS and Objective-C literals aligned', async ()
142142
assert.deepEqual(wireVocabulary.responseKeys, SNAPSHOT_SOURCE_RESPONSE_KEYS);
143143
assert.deepEqual(wireVocabulary.attributeKeys, SNAPSHOT_SOURCE_ATTRIBUTE_KEYS);
144144
assert.match(nativeSource, /kProtocolVersion = 1/);
145-
assert.match(nativeSource, /kSourceVersion = @"agent-device-simulator-ax-v1\.5\.5"/);
145+
assert.match(nativeSource, /kSourceVersion = @"agent-device-simulator-ax-v1\.6\.0"/);
146146
for (const key of [
147147
...wireVocabulary.requestKeys,
148148
...wireVocabulary.responseKeys,

0 commit comments

Comments
 (0)