Skip to content

Commit 999920f

Browse files
thymikeethymikee
committed
docs(ios): correct simulator bridge verdict
1 parent eac2c7f commit 999920f

17 files changed

Lines changed: 1007 additions & 35 deletions

‎docs/evidence/ios-simulator-ax-bridge-2026-09-02-corrected.json.gz‎

6.7 KB
Binary file not shown.

‎docs/evidence/ios-simulator-ax-bridge-2026-09-02-corrected.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# iOS Simulator AX bridge corrected evidence
2+
3+
- Decision: **NO-GO**
4+
- Interpretation: **maintainer-corrected**
5+
- Revision: eac2c7f409f4148bbeb1af87a55ad74eef54e8fc (codex/2192-guest-bridge-evidence)
6+
- Target: bench-golden-v2 (7E76ECA9-D40C-4833-A711-F870F8CE9363, com.apple.CoreSimulator.SimRuntime.iOS-27-0)
7+
- Generated: 2026-09-02T15:37:43.300Z
8+
- Immutable broad raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-01-final.json.gz` (original NO-GO; interpretation superseded to stretch-only)
9+
- Narrow targeted raw artifact: `docs/evidence/ios-simulator-ax-bridge-2026-09-02-targeted.json.gz`
10+
11+
The broad run is preserved unchanged. Its old NO-GO was caused by readiness-inclusive first-look and stretch thresholds; this report evaluates the corrected hard contract.
12+
13+
## Hard gates
14+
15+
| Gate | Status | Target | Evidence |
16+
|---|---|---|---|
17+
| warm | **PASS** | p50 <300 ms and p95 <500 ms per screen | 6/6 warm screen cells passed; quiet p50/p95=8.6 ms/9.3 ms ready=20/20; list p50/p95=118.2 ms/120.9 ms ready=20/20; nested-scroll p50/p95=15.1 ms/15.8 ms ready=20/20; alert p50/p95=41.6 ms/43.3 ms ready=20/20; system-surface p50/p95=37.2 ms/39.6 ms ready=20/20; xctest-stress p50/p95=39.6 ms/41.1 ms ready=20/20 |
18+
| relaunch | **PASS** | p95 <500 ms per screen after observed new-generation app readiness | 6/6 relaunch screen cells passed; quiet p50/p95=8.9 ms/9.6 ms ready=20/20; list p50/p95=119.2 ms/121.1 ms ready=20/20; nested-scroll p50/p95=15.4 ms/16.5 ms ready=20/20; alert p50/p95=41.1 ms/42.7 ms ready=20/20; system-surface p50/p95=37.3 ms/39.5 ms ready=20/20; xctest-stress p50/p95=40.4 ms/42.6 ms ready=20/20 |
19+
| nonresidentBootstrap | **FAIL** | nonresident companion + reader bootstrap and first usable tree p95 <2,000 ms | 1/5 usable trees; p95=5768.8 ms; timer covered adapter acquireBatch only after app readiness, with no xcodebuild, XCTest, or agent-device runner in the timed path |
20+
| liveRecovery | **FAIL** | live crash, timeout, cancellation, and honest target-generation handling | 0/4 probes returned a typed failure or typed unavailable-generation residue and a usable recovered response |
21+
| hierarchyResidue | **PASS** | missing hierarchy represented as typed provider-pruned depth residue | provider-pruned/depth observed; traversal depth is not treated as complete |
22+
23+
## Readiness boundary and candidate-owned latency
24+
25+
Warm and relaunch timing starts at the bridge acquisition after fixture/app readiness admission. Relaunch readiness is recorded separately; the old first-look value includes Simulator, app, daemon, and runner costs.
26+
27+
| State | Screen | Samples | Readable | Ready generation | Candidate p50/p95 ms | Readiness p95 ms | Old first-look p95 ms | Generations |
28+
|---|---|---:|---:|---:|---:|---:|---:|---:|
29+
| warm | quiet | 20 | 20 | 20 | 8.6/9.3 | 0.0 | 9.3 | 1 |
30+
| warm | list | 20 | 20 | 20 | 118.2/120.9 | 0.0 | 120.9 | 1 |
31+
| warm | nested-scroll | 20 | 20 | 20 | 15.1/15.8 | 0.0 | 15.8 | 1 |
32+
| warm | alert | 20 | 20 | 20 | 41.6/43.3 | 0.0 | 43.3 | 1 |
33+
| warm | system-surface | 20 | 20 | 20 | 37.2/39.6 | 0.0 | 39.6 | 1 |
34+
| warm | xctest-stress | 20 | 20 | 20 | 39.6/41.1 | 0.0 | 41.1 | 1 |
35+
| relaunch | quiet | 20 | 20 | 20 | 8.9/9.6 | 4390.1 | 4399.3 | 1 |
36+
| relaunch | list | 20 | 20 | 20 | 119.2/121.1 | 5061.4 | 5177.9 | 1 |
37+
| relaunch | nested-scroll | 20 | 20 | 20 | 15.4/16.5 | 5078.9 | 5093.8 | 1 |
38+
| relaunch | alert | 20 | 20 | 20 | 41.1/42.7 | 4418.9 | 4461.2 | 1 |
39+
| relaunch | system-surface | 20 | 20 | 20 | 37.3/39.5 | 4976.2 | 5013.4 | 1 |
40+
| relaunch | xctest-stress | 20 | 20 | 20 | 40.4/42.6 | 4463.5 | 4503.0 | 1 |
41+
42+
## Cold diagnostics
43+
44+
Cold and cold-cold first-look measurements remain visible for diagnosis, but are excluded from the candidate-owned hard verdict because they combine environment and readiness boundaries with bridge work.
45+
46+
| State | Screen | Preparation p95 ms | First-look p95 ms | Interpretation |
47+
|---|---|---:|---:|---|
48+
| cold-cold | quiet | 16151.2 | 16575.5 | excluded runner/app readiness costs |
49+
| cold-cold | list | 19475.9 | 20062.0 | excluded runner/app readiness costs |
50+
| cold-cold | nested-scroll | 18561.5 | 19089.0 | excluded runner/app readiness costs |
51+
| cold-cold | alert | 17112.3 | 17549.4 | excluded runner/app readiness costs |
52+
| cold-cold | system-surface | 18355.9 | 18866.3 | excluded runner/app readiness costs |
53+
| cold-cold | xctest-stress | 16930.2 | 17368.5 | excluded runner/app readiness costs |
54+
| cold | quiet | 7138.6 | 7147.0 | excluded runner/app readiness costs |
55+
| cold | list | 6985.7 | 7104.5 | excluded runner/app readiness costs |
56+
| cold | nested-scroll | 6930.9 | 6947.1 | excluded runner/app readiness costs |
57+
| cold | alert | 6855.4 | 6895.5 | excluded runner/app readiness costs |
58+
| cold | system-surface | 7145.6 | 7181.9 | excluded runner/app readiness costs |
59+
| cold | xctest-stress | 6918.1 | 6959.1 | excluded runner/app readiness costs |
60+
61+
## Nonresident bootstrap
62+
63+
- 1/5 usable trees; p95=5768.8 ms; timer covered adapter acquireBatch only after app readiness, with no xcodebuild, XCTest, or agent-device runner in the timed path.
64+
- The timed boundary begins with a nonresident adapter and ends at the first usable guest tree; Simulator/app readiness was established before the timer.
65+
66+
| Sample | Duration ms | Usable tree | Failure | Nodes | Generation |
67+
|---:|---:|---|---|---:|---|
68+
| 1 | 1990.6 | true | none/none | 159 | – |
69+
| 2 | 5718.8 | false | timeout/batch-duration-limit | 0 | – |
70+
| 3 | 5702.4 | false | timeout/batch-duration-limit | 0 | – |
71+
| 4 | 5740.5 | false | timeout/batch-duration-limit | 0 | – |
72+
| 5 | 5768.8 | false | timeout/batch-duration-limit | 0 | – |
73+
74+
## Live candidate recovery
75+
76+
- 0/4 probes returned a typed failure or typed unavailable-generation residue and a usable recovered response.
77+
78+
| Operation | Observed failure | Recovery response | Recovered tree |
79+
|---|---|---|---|
80+
| process-crash | process-crash/persistent-process-exited | failed | 0 nodes |
81+
| timeout | timeout/guest-read-timeout | failed | 0 nodes |
82+
| cancelled | cancelled/abort-signal | failed | 0 nodes |
83+
| stale-generation | timeout/batch-duration-limit | failed | 0 nodes |
84+
85+
## Hierarchy residue
86+
87+
- provider-pruned/depth observed; traversal depth is not treated as complete.
88+
- Observed traversal depth: 0; depth complete: **false**. The guest response is flat and carries typed `provider-pruned/depth` residue.
89+
90+
## Stretch findings
91+
92+
- Original broad-run finding: guest-simulator-framework-bridge cold-cold first look missed the 5 second target.
93+
- Original broad-run finding: guest-simulator-framework-bridge cold prepared first look missed the 1.5 second target.
94+
- Original broad-run finding: guest-simulator-framework-bridge warm/list acquisition missed the 75/150 ms target.
95+
- Original broad-run finding: guest-simulator-framework-bridge relaunch first look missed the 250 ms target.
96+
- Cold and cold-cold first-look measurements include Simulator, app, daemon, and runner readiness costs; they are diagnostics, not candidate-owned hard gates.
97+
- The former warm 75/150 ms and relaunch 250 ms thresholds are stretch findings under the corrected contract.
98+
99+
## Production boundary
100+
101+
- No production backend selection, fallback, runner-demand, open/relaunch, proxy, XCTest interaction, or public CLI changes were made.
102+
- The corrected result is evidence for the #2192 decision boundary only; it does not start production routing.

‎docs/evidence/ios-simulator-ax-bridge-2026-09-02-targeted.json.gz‎

5.31 KB
Binary file not shown.

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,7 @@
124124
"bench:ios-snapshot": "node --experimental-strip-types scripts/ios-snapshot-benchmark/run.ts",
125125
"bench:ios-snapshot:deep-button": "node --experimental-strip-types scripts/ios-snapshot-benchmark/deep-button.ts",
126126
"bench:ios-ax-bridge": "node --experimental-strip-types scripts/ios-ax-bridge-spike/run.ts",
127+
"bench:ios-ax-bridge:targeted": "node --experimental-strip-types scripts/ios-ax-bridge-spike/targeted-run.ts",
127128
"mutation:run": "node --experimental-strip-types scripts/mutation/run.ts",
128129
"mutation:check": "node --experimental-strip-types scripts/mutation/run.ts --no-run",
129130
"mutation:affected": "node --experimental-strip-types scripts/mutation/run.ts --affected",

‎scripts/ios-ax-bridge-spike/README.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,21 @@ The default candidate set, state set, screen set, and sample minimums come from
3030
`--apply-preferences` is the only way the experiment edits Simulator preference plists. The Simulator must be shutdown; the harness records exact plist hashes and targeted key changes, then restores the original bytes before reporting. The keys are not production defaults.
3131

3232
The harness fails closed. It reports `NO-GO` when the guest candidate is unsupported, unavailable, unreadable, stale, over a bound, below the sample minimum, or when crash/timeout/cancellation recovery is not typed and recovered. XCTest is a control result and cannot turn a passing guest corpus into a failure. It stops before reporting timings if the fixture app cannot be prepared deterministically. The adjacent gzipped JSON and readable Markdown report are the decision artifact; no production route should be implemented from a `NO-GO` run.
33+
34+
The checked-in broad corpus predates the corrected hard-latency contract. Reproduce only the
35+
missing live bootstrap and lifecycle evidence with:
36+
37+
```sh
38+
pnpm bench:ios-ax-bridge:targeted -- \
39+
--udid SIMULATOR_UDID \
40+
--apply-preferences \
41+
--guest-companion /path/to/idb_companion \
42+
--guest-python python3 \
43+
--guest-site-packages /path/to/idb-cli/libexec/lib/python3.14/site-packages
44+
```
45+
46+
This preserves the broad raw artifact and writes a narrow raw artifact plus the superseding
47+
corrected report. Each nonresident bootstrap sample re-establishes a booted Simulator and ready app
48+
before its timer, so helper teardown contention and Simulator/app readiness are outside the measured
49+
candidate boundary. Missing provider generation is emitted as typed residue; the reader never echoes
50+
an expected generation it did not observe.

‎scripts/ios-ax-bridge-spike/adapter.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ export type AcquisitionAdapter = Readonly<{
2222
options?: Readonly<{ signal?: AbortSignal }>,
2323
): Promise<AcquisitionBatchResult>;
2424
close?: () => Promise<void>;
25+
evidence?: Readonly<{ terminateReaderOnNextBatch?: () => void }>;
2526
}>;
2627

2728
export type AcquisitionBatchResult = Readonly<{

‎scripts/ios-ax-bridge-spike/corrected-markdown.ts‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
import type { CorrectedReport, GateResult, LatencySummary } from './corrected-types.ts';
2+
3+
export function renderCorrectedMarkdown(report: CorrectedReport): string {
4+
const lines = [
5+
'# iOS Simulator AX bridge corrected evidence',
6+
'',
7+
`- Decision: **${report.decision}**`,
8+
'- Interpretation: **maintainer-corrected**',
9+
`- Revision: ${report.revision.commit} (${report.revision.branch})`,
10+
`- Target: ${report.target.name} (${report.target.udid}, ${report.target.runtime})`,
11+
`- Generated: ${report.generatedAt}`,
12+
`- Immutable broad raw artifact: \`${report.sourceArtifact.path}\` (original ${report.sourceArtifact.originalDecision}; interpretation superseded to stretch-only)`,
13+
`- Narrow targeted raw artifact: \`${report.targetedArtifact.path}\``,
14+
'',
15+
'The broad run is preserved unchanged. Its old NO-GO was caused by readiness-inclusive first-look and stretch thresholds; this report evaluates the corrected hard contract.',
16+
'',
17+
'## Hard gates',
18+
'',
19+
'| Gate | Status | Target | Evidence |',
20+
'|---|---|---|---|',
21+
...Object.entries(report.hardGates).map(([name, gate]) => gateLine(name, gate)),
22+
'',
23+
'## Readiness boundary and candidate-owned latency',
24+
'',
25+
'Warm and relaunch timing starts at the bridge acquisition after fixture/app readiness admission. Relaunch readiness is recorded separately; the old first-look value includes Simulator, app, daemon, and runner costs.',
26+
'',
27+
'| State | Screen | Samples | Readable | Ready generation | Candidate p50/p95 ms | Readiness p95 ms | Old first-look p95 ms | Generations |',
28+
'|---|---|---:|---:|---:|---:|---:|---:|---:|',
29+
...report.readiness.map(readinessLine),
30+
'',
31+
'## Cold diagnostics',
32+
'',
33+
'Cold and cold-cold first-look measurements remain visible for diagnosis, but are excluded from the candidate-owned hard verdict because they combine environment and readiness boundaries with bridge work.',
34+
'',
35+
'| State | Screen | Preparation p95 ms | First-look p95 ms | Interpretation |',
36+
'|---|---|---:|---:|---|',
37+
...report.coldDiagnostics.map(coldDiagnosticLine),
38+
'',
39+
'## Nonresident bootstrap',
40+
'',
41+
`- ${report.hardGates.nonresidentBootstrap.evidence}.`,
42+
'- The timed boundary begins with a nonresident adapter and ends at the first usable guest tree; Simulator/app readiness was established before the timer.',
43+
'',
44+
'| Sample | Duration ms | Usable tree | Failure | Nodes | Generation |',
45+
'|---:|---:|---|---|---:|---|',
46+
...report.bootstrap.map(bootstrapLine),
47+
'',
48+
'## Live candidate recovery',
49+
'',
50+
`- ${report.hardGates.liveRecovery.evidence}.`,
51+
'',
52+
'| Operation | Observed failure | Recovery response | Recovered tree |',
53+
'|---|---|---|---|',
54+
...report.liveRecovery.map(recoveryLine),
55+
'',
56+
'## Hierarchy residue',
57+
'',
58+
`- ${report.hardGates.hierarchyResidue.evidence}.`,
59+
`- Observed traversal depth: ${report.hierarchy.observedTraversalDepth}; depth complete: **${report.hierarchy.depthComplete}**. The guest response is flat and carries typed \`${report.hierarchy.residue.kind}/${report.hierarchy.residue.fields.join(',')}\` residue.`,
60+
'',
61+
'## Stretch findings',
62+
'',
63+
...report.stretchFindings.map((finding) => `- ${finding}`),
64+
'',
65+
'## Production boundary',
66+
'',
67+
'- No production backend selection, fallback, runner-demand, open/relaunch, proxy, XCTest interaction, or public CLI changes were made.',
68+
'- The corrected result is evidence for the #2192 decision boundary only; it does not start production routing.',
69+
];
70+
return `${lines.join('\n')}\n`;
71+
}
72+
73+
function gateLine(name: string, gate: GateResult): string {
74+
return `| ${name} | **${gate.status}** | ${gate.target} | ${gate.evidence} |`;
75+
}
76+
77+
function readinessLine(summary: LatencySummary): string {
78+
return `| ${summary.state} | ${summary.screen} | ${summary.samples} | ${summary.readableSamples} | ${summary.readinessObservedSamples} | ${formatMs(summary.candidateP50Ms)}/${formatMs(summary.candidateP95Ms)} | ${formatMs(summary.preparationP95Ms)} | ${formatMs(summary.firstLookP95Ms)} | ${summary.generationCount} |`;
79+
}
80+
81+
function coldDiagnosticLine(diagnostic: CorrectedReport['coldDiagnostics'][number]): string {
82+
return `| ${diagnostic.state} | ${diagnostic.screen} | ${formatMs(diagnostic.preparationP95Ms)} | ${formatMs(diagnostic.firstLookP95Ms)} | excluded runner/app readiness costs |`;
83+
}
84+
85+
function bootstrapLine(sample: CorrectedReport['bootstrap'][number]): string {
86+
const response = sample.response;
87+
return `| ${sample.index} | ${sample.durationMs.toFixed(1)} | ${sample.usableTree} | ${failureText(response.failure)} | ${response.metrics.nodeCount} | ${response.acquisition?.targetGeneration ?? '–'} |`;
88+
}
89+
90+
function recoveryLine(probe: CorrectedReport['liveRecovery'][number]): string {
91+
const recovery = probe.recoveredResponse;
92+
const status = recovery.ok ? 'ok' : 'failed';
93+
const nodes = recovery.acquisition?.nodes.length ?? 0;
94+
return `| ${probe.operation} | ${failureText(probe.response.failure)} | ${status} | ${nodes} nodes |`;
95+
}
96+
97+
function failureText(failure: CorrectedReport['bootstrap'][number]['response']['failure']): string {
98+
if (!failure) return 'none/none';
99+
return `${failure.kind}/${failure.code ?? 'none'}`;
100+
}
101+
102+
function formatMs(value: number | null): string {
103+
return value === null ? '–' : value.toFixed(1);
104+
}

‎scripts/ios-ax-bridge-spike/corrected-report.test.ts‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
import path from 'node:path';
2+
import { describe, expect, test } from 'vitest';
3+
import { buildCorrectedReport, readSpikeReport, readTargetedArtifact } from './corrected-report.ts';
4+
import { renderCorrectedMarkdown } from './corrected-markdown.ts';
5+
6+
const SOURCE = 'docs/evidence/ios-simulator-ax-bridge-2026-09-01-final.json.gz';
7+
const TARGETED = 'docs/evidence/ios-simulator-ax-bridge-2026-09-02-targeted.json.gz';
8+
9+
describe('corrected Simulator AX bridge report', () => {
10+
test('keeps fast resident acquisition separate from failed cold bootstrap and recovery', () => {
11+
const report = buildCorrectedReport({
12+
sourcePath: SOURCE,
13+
source: readSpikeReport(path.resolve(SOURCE)),
14+
targetedPath: TARGETED,
15+
targeted: readTargetedArtifact(path.resolve(TARGETED)),
16+
});
17+
18+
expect(report.decision).toBe('NO-GO');
19+
expect(report.hardGates.warm.status).toBe('PASS');
20+
expect(report.hardGates.relaunch.status).toBe('PASS');
21+
expect(report.hardGates.nonresidentBootstrap.status).toBe('FAIL');
22+
expect(report.hardGates.liveRecovery.status).toBe('FAIL');
23+
expect(renderCorrectedMarkdown(report)).toContain('Decision: **NO-GO**');
24+
});
25+
});

0 commit comments

Comments
 (0)