Skip to content

Commit 9c25bc6

Browse files
authored
docs(cli): advertise open --foreground and snapshot --actions in the workflow card (#1682)
* docs(cli): advertise open --foreground and snapshot --actions in the workflow card open --foreground (#1670/#1671) and snapshot -i --actions (#1665) shipped with no mention in the compact `help workflow` card, so a planning model never discovers either. Add one terse line each: the foreground fast-path in Bootstrap, and the merged-element custom-action guidance in Validation and evidence. Stays under the 9,000-byte compact-card budget (8493 -> 8908 bytes). Adds two help-conformance bench cases per the repo's changed-guidance rule: foreground-attach-single-sim (correct plan starts with `open --foreground` in an unambiguous single-sim scenario, fail-closed alternative forbidden) and merged-card-actions-not-directly-invokable (a merged Bluesky-style feed card's actions list is evidence, not a selector). Both use a real pinned sample rebuilt through the production snapshot renderer. * fix(scripts): accept flag order in the foreground-attach conformance matcher Flag order after `open` isn't semantically meaningful (`open --platform ios --foreground` is exactly as correct as `open --foreground --platform ios`), but startsWithForegroundOpen required --foreground to be the literal next token after `open`. Rescoring the completed repeat=3 bench report shows this docked codex:gpt-5.4-mini on all 3 trials even though its plan was config-order noise, not a real deviation -- the no-positional/no-device guarantee already comes from the forbidden checks. Loosened to require --foreground anywhere on the open line; foreground-attach-single-sim now scores 54/54 across both runners. * fix: close workflow help conformance gaps
1 parent ac9e4d0 commit 9c25bc6

10 files changed

Lines changed: 267 additions & 5 deletions

scripts/__tests__/help-conformance-bench.test.ts

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -496,6 +496,35 @@ test('case matchers score parsed tokens so shell quoting does not change results
496496
});
497497
});
498498

499+
test('foreground attach scoring rejects an explicit app positional after flags', async () => {
500+
const validCommand = 'agent-device open --foreground --platform ios';
501+
const explicitTargetCommand = 'agent-device open --foreground --platform ios com.example.app';
502+
const [validAttach, explicitTarget] = await validatePlanCommands([
503+
validCommand,
504+
explicitTargetCommand,
505+
]);
506+
507+
assert.deepEqual(explicitTarget.agentCommand, {
508+
command: 'open',
509+
positionals: ['com.example.app'],
510+
});
511+
assert.equal(
512+
scoreExpectations({ expectations: ['noExplicitForegroundTarget'] }, [validCommand], '', [
513+
validAttach,
514+
]).noExplicitForegroundTarget,
515+
true,
516+
);
517+
assert.equal(
518+
scoreExpectations(
519+
{ expectations: ['noExplicitForegroundTarget'] },
520+
[explicitTargetCommand],
521+
'',
522+
[explicitTarget],
523+
).noExplicitForegroundTarget,
524+
false,
525+
);
526+
});
527+
499528
test('plan validator applies narrow grammar to permitted external commands', async () => {
500529
const results = await validatePlanCommands(
501530
[

scripts/__tests__/help-conformance-sample-producers.ts

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ import {
44
APP_NOT_INSTALLED_SAMPLE,
55
BROWSERSTACK_CONNECT_SAMPLE,
66
DEVICE_IN_USE_SAMPLE,
7+
FOREGROUND_SNAPSHOT_FAILURE_SAMPLE,
8+
MERGED_CARD_ACTIONS_SAMPLE,
79
NOT_SETTLED_SAMPLE,
810
OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
911
PRIVATE_AX_RECOVERY_SAMPLE,
@@ -13,6 +15,8 @@ import {
1315
STALE_REF_SAMPLE,
1416
} from '../help-conformance-sample-outputs.mjs';
1517
import { interactionCliOutputFormatters } from '../../src/commands/interaction/output.ts';
18+
import { snapshotCliOutput } from '../../src/commands/capture/output.ts';
19+
import { openCliOutput } from '../../src/commands/management/output.ts';
1620
import { NEVER_SETTLED_HINT } from '../../src/commands/interaction/runtime/settle.ts';
1721
import { buildAmbiguousMatchError } from '../../src/daemon/handlers/find.ts';
1822
import { refMutationAdmissionResponse } from '../../src/daemon/handlers/interaction-ref-policy.ts';
@@ -267,6 +271,80 @@ export const SAMPLE_PRODUCERS: SampleProducer[] = [
267271
).trimEnd();
268272
},
269273
},
274+
{
275+
name: 'FOREGROUND_SNAPSHOT_FAILURE_SAMPLE',
276+
producer: 'the open success renderer with a failed composed snapshot',
277+
sample: FOREGROUND_SNAPSHOT_FAILURE_SAMPLE,
278+
render: () => {
279+
const warning =
280+
'The session is open, but the initial interactive snapshot failed (COMMAND_FAILED: capture failed). Run: agent-device snapshot -i';
281+
return (
282+
openCliOutput({
283+
session: 'default',
284+
warnings: [warning],
285+
initialSnapshotError: {
286+
code: 'COMMAND_FAILED',
287+
message: 'capture failed',
288+
},
289+
identifiers: { session: 'default' },
290+
}).text ?? ''
291+
);
292+
},
293+
},
294+
{
295+
name: 'MERGED_CARD_ACTIONS_SAMPLE',
296+
producer: "the snapshot renderer with --actions naming a merged element's custom actions",
297+
sample: MERGED_CARD_ACTIONS_SAMPLE,
298+
render: () => {
299+
// A Bluesky-style feed item merged into one Link node: its Reply/Repost/
300+
// menu controls are AX custom actions, not child nodes, so they only
301+
// surface when --actions is passed through to the renderer.
302+
const nodes = [
303+
{
304+
index: 0,
305+
ref: 'e1',
306+
depth: 0,
307+
type: 'Application',
308+
label: 'Bluesky',
309+
rect: { x: 0, y: 0, width: 390, height: 844 },
310+
},
311+
{
312+
index: 1,
313+
ref: 'e2',
314+
parentIndex: 0,
315+
depth: 1,
316+
type: 'Window',
317+
rect: { x: 0, y: 0, width: 390, height: 844 },
318+
},
319+
{
320+
index: 2,
321+
ref: 'e3',
322+
parentIndex: 1,
323+
depth: 2,
324+
type: 'CollectionView',
325+
interactive: true,
326+
rect: { x: 0, y: 60, width: 390, height: 700 },
327+
},
328+
{
329+
index: 3,
330+
ref: 'e72',
331+
parentIndex: 2,
332+
depth: 3,
333+
type: 'Link',
334+
label: 'feedItem-by-whiskers.test',
335+
interactive: true,
336+
rect: { x: 0, y: 60, width: 390, height: 140 },
337+
actions: ['Reply', 'Repost', 'Open post options menu'],
338+
},
339+
];
340+
return (
341+
snapshotCliOutput({
342+
result: { nodes, backend: 'xctest', truncated: false },
343+
interactiveOnly: true,
344+
}).text ?? ''
345+
).trimEnd();
346+
},
347+
},
270348
{
271349
name: 'DEVICE_IN_USE_SAMPLE',
272350
producer: 'the real session-open by-session conflict producer',

scripts/help-conformance-case-checks.mjs

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,13 @@ const EXPECTATION_SCORERS = {
1616
// validator splits and validates each segment), so it is the only place
1717
// that can tell whether the model actually chained.
1818
usesConfidentChaining: ({ commands }) => commands.some((command) => /&&/.test(command)),
19+
noExplicitForegroundTarget: ({ commandValidation }) =>
20+
commandValidation
21+
.filter(
22+
({ tokens, agentCommand }) =>
23+
agentCommand?.command === 'open' && tokens.includes('--foreground'),
24+
)
25+
.every(({ agentCommand }) => agentCommand.positionals.length === 0),
1926
noWaitStable: ({ joined }) => !joined.includes('wait stable'),
2027
verifiesNamedExpectation: ({ joined }) => /\b(wait|is|get|find)\b/.test(joined),
2128
usesDogfoodEvidence: ({ joined }) =>

scripts/help-conformance-cases.mjs

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ import {
33
APP_NOT_INSTALLED_SAMPLE,
44
BROWSERSTACK_CONNECT_SAMPLE,
55
DEVICE_IN_USE_SAMPLE,
6+
FOREGROUND_SNAPSHOT_FAILURE_SAMPLE,
7+
MERGED_CARD_ACTIONS_SAMPLE,
68
NOT_SETTLED_SAMPLE,
79
OFFSCREEN_TARGET_SNAPSHOT_SAMPLE,
810
SETTLE_DIFF_SAMPLE,
@@ -642,4 +644,79 @@ Use the output already shown to determine whether the feed-search UI is present,
642644
{ id: 'noRedundantInstall', pattern: /(?:^|\n)agent-device\s+install\b/i },
643645
],
644646
},
647+
{
648+
id: 'foreground-attach-single-sim',
649+
docs: ['--help:first30', 'workflow'],
650+
task: 'You are starting fresh with no active session. The environment guarantees exactly one booted iOS simulator with exactly one app running on it -- the app you want to keep testing. Plan the command to attach to it and get its initial interactive snapshot in a single call (this only resolves unambiguously because of that guarantee, and it rejects an explicit app or device selector), then press the visible Continue control and close the session.',
651+
expectations: [
652+
'validPlanCommands',
653+
'fullPrefix',
654+
'usesSettleOnMutations',
655+
'opensAndCloses',
656+
'noExplicitForegroundTarget',
657+
],
658+
matchers: [
659+
{
660+
id: 'startsWithForegroundOpen',
661+
// Flag order is not semantically meaningful (`open --platform ios
662+
// --foreground` is exactly as correct as `open --foreground
663+
// --platform ios`); this only checks that --foreground is on the
664+
// first command and there is no app positional before it. The
665+
// no-positional guarantee comes from the forbidden checks below.
666+
pattern: /^agent-device\s+open\b[^\n]*--foreground\b/i,
667+
},
668+
{
669+
id: 'pressesContinueAfterAttach',
670+
pattern: /agent-device\s+press\s+[^\n]*continue[^\n]*--settle\b/i,
671+
},
672+
],
673+
forbidden: [
674+
{
675+
id: 'noDeviceSelectorWithForeground',
676+
pattern: /--foreground\b[^\n]*--(?:udid|device)\b|--(?:udid|device)\b[^\n]*--foreground\b/i,
677+
},
678+
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
679+
],
680+
},
681+
{
682+
id: 'merged-card-actions-not-directly-invokable',
683+
docs: ['--help:first30', 'workflow'],
684+
task: quiz(
685+
MERGED_CARD_ACTIONS_SAMPLE,
686+
'The goal is to reply to this post. The actions list names "Reply" as a hidden affordance on @e72, but that name is not a pressable selector. What command should run next?',
687+
),
688+
expectations: ['validPlanCommands', 'fullPrefix'],
689+
matchers: [
690+
{
691+
id: 'opensCardToReachReply',
692+
pattern: /(?:^|\n)agent-device\s+(?:press|click)\s+@e72\b[^\n]*--settle\b/i,
693+
},
694+
],
695+
forbidden: [
696+
{
697+
id: 'noPressingActionNameAsSelector',
698+
pattern: /(?:^|\n)agent-device\s+(?:press|click|find)\b[^\n]*(?:label|text)="?reply"?/i,
699+
},
700+
{ id: 'noRawCoordinateTarget', pattern: RAW_COORDINATE_TARGET },
701+
],
702+
},
703+
{
704+
id: 'foreground-attach-snapshot-recovery',
705+
docs: ['--help:first30', 'workflow'],
706+
task: quiz(
707+
FOREGROUND_SNAPSHOT_FAILURE_SAMPLE,
708+
'The foreground attach succeeded and the session is still open, but its initial snapshot failed. What command should run next to get interactive refs?',
709+
),
710+
expectations: ['validPlanCommands', 'fullPrefix', 'usesSnapshotI'],
711+
matchers: [
712+
{
713+
id: 'retriesSnapshotInOpenSession',
714+
pattern: /(?:^|\n)agent-device\s+snapshot\s+-i\b/i,
715+
},
716+
],
717+
forbidden: [
718+
{ id: 'noSecondOpen', pattern: /(?:^|\n)agent-device\s+open\b/i },
719+
{ id: 'noPrematureClose', pattern: /(?:^|\n)agent-device\s+close\b/i },
720+
],
721+
},
645722
];

scripts/help-conformance-command-validator.ts

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,13 @@ import { isCommandName } from '../src/commands/command-metadata.ts';
55

66
type ValidationKind = 'agent-device-grammar' | 'pseudo-ref';
77

8+
type ParsedAgentCommand = {
9+
command: string;
10+
positionals: string[];
11+
};
12+
813
type ValidationResult = { valid: true } | { valid: false; kind: ValidationKind; error: string };
14+
type PlanValidationResult = ValidationResult & { agentCommand?: ParsedAgentCommand };
915

1016
const TARGET_POSITION_BY_COMMAND = new Map<string, number>([
1117
['click', 0],
@@ -71,10 +77,23 @@ function isStringArray(value: unknown): value is string[] {
7177
return Array.isArray(value) && value.every((entry) => typeof entry === 'string');
7278
}
7379

74-
function validateInput(value: unknown): ValidationResult | ValidationResult[] {
75-
if (Array.isArray(value) && value.every(isStringArray))
76-
return value.map(validateAgentDeviceCommand);
77-
return validateAgentDeviceCommand(value);
80+
function validateInput(value: unknown): PlanValidationResult | PlanValidationResult[] {
81+
if (Array.isArray(value) && value.every(isStringArray)) return value.map(validatePlanCommand);
82+
return validatePlanCommand(value);
83+
}
84+
85+
function validatePlanCommand(value: unknown): PlanValidationResult {
86+
const validation = validateAgentDeviceCommand(value);
87+
if (!validation.valid || !isStringArray(value)) return validation;
88+
const parsed = parseArgs(value, { strictFlags: true });
89+
if (!parsed.command) return validation;
90+
return {
91+
...validation,
92+
agentCommand: {
93+
command: parsed.command,
94+
positionals: [...parsed.positionals],
95+
},
96+
};
7897
}
7998

8099
function runCli(): void {

scripts/help-conformance-plan-validator.mjs

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,14 +30,18 @@ function applyCommandPolicy(parsed, agentResultState, allowedExternalCommands) {
3030
}
3131
const result = agentResultState.results[agentResultState.index++];
3232
return result?.valid
33-
? parsed
33+
? attachAgentCommand(parsed, result.agentCommand)
3434
: withIssue(
3535
parsed,
3636
result?.kind ?? 'agent-device-grammar',
3737
result?.error ?? 'Command validation returned no result.',
3838
);
3939
}
4040

41+
function attachAgentCommand(parsed, agentCommand) {
42+
return agentCommand ? { ...parsed, agentCommand } : parsed;
43+
}
44+
4145
// The compact workflow card teaches chaining confident consecutive steps with
4246
// an unquoted `&&` (`press ... --settle && fill ... --settle`). Split on it
4347
// before tokenizing a line so each chained segment is validated as its own

scripts/help-conformance-sample-outputs.mjs

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,3 +147,27 @@ Next:
147147
Use the installed package or bundle identifier in open, not the app artifact name.
148148
After close, run agent-device artifacts --json --session adc-browserstack for provider video and logs.`,
149149
};
150+
151+
// open --foreground succeeded, but its composed snapshot failed. The session
152+
// remains usable, so recovery is snapshot -i rather than a second open.
153+
export const FOREGROUND_SNAPSHOT_FAILURE_SAMPLE = {
154+
command: 'agent-device open --foreground',
155+
output: `Opened: default
156+
Warning: The session is open, but the initial interactive snapshot failed (COMMAND_FAILED: capture failed). Run: agent-device snapshot -i`,
157+
};
158+
159+
// Merged feed-item card on iOS (#1665): the row itself is the only ref — its
160+
// Reply/Repost/menu controls have no separate child nodes in the tree, so
161+
// snapshot -i alone would show a plain link with no way to act on it.
162+
// snapshot -i --actions names the hidden affordances instead of hiding them
163+
// silently; the names are evidence only, never directly invokable (help
164+
// workflow: "reach via its detail screen, labeled children elsewhere, or
165+
// coordinates").
166+
export const MERGED_CARD_ACTIONS_SAMPLE = {
167+
command: 'agent-device snapshot -i --actions',
168+
output: `Snapshot: 4 nodes
169+
@e1 [application] "Bluesky"
170+
@e2 [window]
171+
@e3 [collection]
172+
@e72 [link] "feedItem-by-whiskers.test" actions: ["Reply", "Repost", "Open post options menu"]`,
173+
};

src/__tests__/cli-help.test.ts

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,20 @@ test('help workflow documents open/close/relaunch runner guarantees as lifecycle
135135
assert.match(result.stdout, /Env vars: help physical-device/);
136136
});
137137

138+
test('help workflow advertises open --foreground and snapshot -i --actions', async () => {
139+
const result = await runCliCapture(['help', 'workflow']);
140+
assert.equal(result.code, 0);
141+
assert.equal(result.calls.length, 0);
142+
assert.match(
143+
result.stdout,
144+
/One iOS sim, app running, no session: open --foreground: attach \+ snapshot\. Capture fails; stays open: snapshot -i\. App\/device\/ambiguity fail/,
145+
);
146+
assert.match(
147+
result.stdout,
148+
/iOS sim: snapshot -i --actions shows merged actions; use detail\/coords, not names/,
149+
);
150+
});
151+
138152
test('help physical-device documents the runner/daemon lifecycle detail moved out of workflow (#1051)', async () => {
139153
const result = await runCliCapture(['help', 'physical-device']);
140154
assert.equal(result.code, 0);

src/cli/parser/__tests__/cli-help-topics.test.ts

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -253,6 +253,10 @@ test('usageForCommand resolves workflow help topic', async () => {
253253
);
254254
assert.match(help, /Known flow: batch \.\/steps\.json \(help scripting\)/);
255255
assert.match(help, /Shapes and platform quirks: help gestures/);
256+
assert.match(
257+
help,
258+
/One iOS sim, app running, no session: open --foreground: attach \+ snapshot\. Capture fails; stays open: snapshot -i\. App\/device\/ambiguity fail/,
259+
);
256260
assert.match(help, /Never open artifact paths or invent package ids/);
257261
assert.match(
258262
help,
@@ -302,6 +306,10 @@ test('usageForCommand resolves workflow help topic', async () => {
302306
/confirm the requested end state is actually visible on the current screen, scrolling it into view if needed/,
303307
);
304308
assert.match(help, /get text alone, or stopping one screen early, is not enough/);
309+
assert.match(
310+
help,
311+
/iOS sim: snapshot -i --actions shows merged actions; use detail\/coords, not names/,
312+
);
305313
assert.match(help, /Perf\/memory\/log\/network\/trace\/crash: help debugging/);
306314
assert.match(help, /Recording, save-script, batch, replay repair: help scripting/);
307315
assert.match(help, /help react-native for Metro\/Re\.Pack reload/);

src/cli/parser/cli-help.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,7 @@ Command shape:
226226
Bootstrap:
227227
agent-device devices --platform ios
228228
agent-device open MyApp --platform ios --device "iPhone 17 Pro"
229+
One iOS sim, app running, no session: open --foreground: attach + snapshot. Capture fails; stays open: snapshot -i. App/device/ambiguity fail.
229230
Install arguments are app/package id then artifact path: agent-device install com.example.app ./dist/app.apk --platform android, then open <id> --relaunch for fresh state. Use reinstall only when explicitly requested.
230231
Unknown app id: devices, then apps, then open <discovered-app-id>. Never open artifact paths or invent package ids; ask if lookup misses the target.
231232
Apple CI: prepare ios-runner after boot/install, before replay/test (help prepare). Remote/cloud: connect -> open -> commands -> close -> disconnect (help remote). Reusable scripts, secret-safe fills, replay repair: help scripting.
@@ -264,6 +265,7 @@ Validation and evidence:
264265
Nearby mutation diff: diff snapshot -i; with no prior snapshot it initializes the baseline (zero changes) instead of failing.
265266
Named expectations need the exact text/selector via wait/is/get/find -- a bare screenshot/snapshot is not verification. Before declaring a task done, confirm the requested end state is actually visible on the current screen, scrolling it into view if needed; get text alone, or stopping one screen early, is not enough.
266267
When an action only reveals or reaches a target, verify the exact target named, not just the action. Prefer testIDs/ids/selectors over visible text. Icon/tappable proof: screenshot --overlay-refs; if snapshot is sparse/AX-unavailable, use plain screenshot and coordinates, then retry snapshot -i on another screen.
268+
iOS sim: snapshot -i --actions shows merged actions; use detail/coords, not names.
267269
Perf/memory/log/network/trace/crash: help debugging. Recording, save-script, batch, replay repair: help scripting.
268270
269271
React Native: help react-native for Metro/Re.Pack reload, DevTools, RN overlays. JS-only change: metro reload, find "Home"; open --relaunch for native reset.

0 commit comments

Comments
 (0)