XCTest Nightly #8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: XCTest Nightly | |
| # The full iOS runner XCTest suite (#1781 A7). The PR lane (ios.yml) names a hand-written | |
| # subset of the target's methods in an `-only-testing:` list; everything outside that list ran | |
| # nowhere at all. This lane drops the filter and runs the test plan whole, skipping only | |
| # `testCommand` — which is the runner's server entry point rather than a test (see the step | |
| # below). | |
| # | |
| # No count is quoted here on purpose. `pnpm check:xctest-selection` prints the live split | |
| # (declared / PR-selected / skipped / nightly-only) and is the only place those numbers are | |
| # derived rather than remembered. A hand-copied figure in a comment is the same class of | |
| # defect this lane exists to close: between this file being written and first review, the | |
| # suite gained three methods and the PR list gained five entries, so every number originally | |
| # written here was already wrong. | |
| # | |
| # Under the lane rule in #1781: | |
| # | |
| # - Catches: regressions in the simulator-only tests (guard `… && os(iOS)`: they launch | |
| # the host app, route through SpringBoard, or assert an iOS-only branch) that ios.yml's | |
| # PR list leaves out. Since the A7 classification the pure-decision tests also run on | |
| # the macOS host lane on every PR (ci.yml "Swift Runner Host XCTests"), so this lane's | |
| # unique reach is the simulator-only remainder — `pnpm check:xctest-selection` prints | |
| # both numbers. `testCommand` is skipped, and the tests guarded to macOS/tvOS only | |
| # cannot execute on this lane's iOS Simulator destination (the host lane runs them). | |
| # - Evidence: the first execution found #1812 in the dark set (2 of 155 red, both real); | |
| # the other ~114 never-run tests all passed. | |
| # - Cost: this is a public repository, so nothing here is billed. The real cost is one macOS | |
| # concurrency slot per night. The Apple runner build shares ios.yml's cache (identical | |
| # inputs and build variant), but scheduled runs only see DEFAULT-BRANCH caches, so the hit | |
| # depends on ios.yml's `push: main` run having populated the cache at the same source | |
| # hash — on a miss this lane also pays a full runner build. Adds nothing to any PR. | |
| # - Kill criterion: what ONLY this lane reaches is 11 tests / ~42 s of simulator time | |
| # (measured from the first night's xcresult) — the app-launching, SpringBoard-probing | |
| # class. Once ios.yml runs the whole iOS-compiled bundle instead of its hand-written | |
| # list — the follow-up stacked on the classification — every test reaches a PR lane and | |
| # this lane goes. That trade is not free: those 11 are the flakiest tests in the suite, | |
| # and they would land on a blocking 15-minute gate that is already cancelled ~19% of the | |
| # time, so measure their flake rate here before moving them. | |
| # | |
| # Scheduled + manual only. It is deliberately NOT a PR gate: ios.yml is already over half of | |
| # this repo's CI spend (#1781 A9), and moving the unrun set onto it would make that worse for | |
| # tests whose flakiness on a shared runner is unknown. | |
| # | |
| # Note on `concurrency` below: it is keyed on `github.ref`, so a manual dispatch on the same | |
| # ref cancels an in-flight scheduled run. That is the intended trade — an operator asking for | |
| # a run wants the fresh one — but it means "dispatch while the nightly is running" loses the | |
| # nightly's result rather than queueing behind it. | |
| on: | |
| schedule: | |
| - cron: '30 4 * * *' | |
| workflow_dispatch: | |
| permissions: | |
| contents: read | |
| actions: read | |
| concurrency: | |
| group: ci-${{ github.workflow }}-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| full-suite: | |
| name: iOS Runner Full XCTest Suite | |
| runs-on: macos-26 | |
| # Measured, not guessed. A full local run of this exact command on matching toolchain | |
| # (Xcode 26.2, iOS 26.2 Simulator) executed 154 tests in 120s wall — the suite itself is | |
| # ~2 minutes, not the bulk of the job. What dominates the budget is the runner build on a | |
| # derived-data cache MISS, plus checkout and simulator boot. ios.yml carries 80 minutes | |
| # while doing strictly more than this lane (same build and boot, plus replays, prepare, | |
| # and the fixture E2E), so half of that is generous headroom for a cache-miss night and | |
| # still caps a wedged run at 45 minutes instead of two hours of held macOS slot. | |
| timeout-minutes: 45 | |
| env: | |
| # Must match ios.yml. The runtime version rides in the derived-data cache key, and | |
| # AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS is BOTH an input to the cache's build-variant | |
| # hash and the `-D AGENT_DEVICE_RUNNER_UNIT_TESTS` compile flag that makes the RunnerTests | |
| # methods exist at all. Dropping it would build a different variant (cache miss) whose | |
| # bundle contains no tests, and `xcodebuild` reports running zero tests as success — the | |
| # "silent 0-test green" this lane would then be. The test-count assertion below is the | |
| # backstop for exactly that. AGENT_DEVICE_XCUITEST_ARCHS rides the cache key too: dropping | |
| # it silently reverts the runner to Xcode's x86_64 default under Rosetta. | |
| IOS_RUNTIME_VERSION: '26.2' | |
| AGENT_DEVICE_XCUITEST_ARCHS: 'arm64' | |
| AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS: '1' | |
| AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived | |
| RESULT_BUNDLE_PATH: ${{ github.workspace }}/.tmp/xctest-nightly/RunnerTests.xcresult | |
| RESULT_SUMMARY_PATH: ${{ github.workspace }}/.tmp/xctest-nightly/test-results-summary.json | |
| XCTEST_LANE: nightly | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | |
| - name: Setup toolchain | |
| uses: ./.github/actions/setup-node-pnpm | |
| with: | |
| cache-dependency-path: | | |
| pnpm-lock.yaml | |
| examples/test-app/pnpm-lock.yaml | |
| # Byte-identical inputs to ios.yml's step, so both lanes restore the same cache entry | |
| # and only the first one to run on a commit pays for the build. | |
| - name: Restore and build iOS XCTest runner | |
| uses: ./.github/actions/setup-apple-runner-build | |
| with: | |
| derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }} | |
| cache-key-prefix: ios-runner-prebuilt | |
| cache-key-suffix: -ios-${{ env.IOS_RUNTIME_VERSION }}-arm64 | |
| gate: swift-runner-ios | |
| xcuitest-platform: ios | |
| xcuitest-destination: generic/platform=iOS Simulator | |
| - name: Boot iOS test simulator | |
| id: ios-simulator | |
| uses: ./.github/actions/boot-ios-test-simulator | |
| with: | |
| runtime-version: ${{ env.IOS_RUNTIME_VERSION }} | |
| preferred-device-name: iPhone 17 Pro | |
| # Same command as ios.yml's targeted step minus every `-only-testing:` flag, so the | |
| # xctestrun's own test plan decides what runs — with one exception. | |
| # | |
| # `RunnerTests/testCommand` is not a test. It is the runner's server entry point: it | |
| # opens an NWListener and blocks in `XCTWaiter.wait(timeout: 24 * 60 * 60)` until a | |
| # client closes the session, which is exactly how production drives it | |
| # (src/platforms/apple/core/runner/runner-session.ts always passes it as the sole | |
| # `-only-testing:`). It compiles unconditionally — the `#if AGENT_DEVICE_RUNNER_UNIT_TESTS` | |
| # block ends well above it — so an unfiltered run reaches it in alphabetical order and | |
| # hangs the job until `timeout-minutes`. Its one escape hatch, | |
| # AGENT_DEVICE_RUNNER_NOOP_STARTUP, is an environment variable, and the comment above | |
| # the flag records that env plumbing into a simulator test process is not dependable; | |
| # `-skip-testing:` is the lever that works from here. | |
| # | |
| # A typo in that identifier silently re-arms the hang, so | |
| # `pnpm check:xctest-selection` validates `-skip-testing:` exactly like `-only-testing:`. | |
| - name: Run the full iOS runner XCTest suite | |
| run: | | |
| set -euo pipefail | |
| XCTESTRUN_PATH="$(find "$AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH/Build/Products" -maxdepth 1 -name '*.xctestrun' -print -quit)" | |
| test -n "$XCTESTRUN_PATH" | |
| mkdir -p "$(dirname "$RESULT_BUNDLE_PATH")" | |
| xcodebuild test-without-building \ | |
| -xctestrun "$XCTESTRUN_PATH" \ | |
| -destination "platform=iOS Simulator,id=${{ steps.ios-simulator.outputs.simulator-udid }}" \ | |
| -skip-testing:AgentDeviceRunnerUITests/RunnerTests/testCommand \ | |
| -resultBundlePath "$RESULT_BUNDLE_PATH" | |
| # Best-effort and never the job's verdict on its own; the step below is what asserts. | |
| # `--compact` first because a red night's summary is the large one, and the job summary | |
| # has a 1 MiB cap — the plain form is the fallback if an Xcode version rejects the flag. | |
| - name: Extract the test-results summary | |
| if: always() | |
| run: | | |
| set -uo pipefail | |
| [ -d "$RESULT_BUNDLE_PATH" ] || exit 0 | |
| xcrun xcresulttool get test-results summary --path "$RESULT_BUNDLE_PATH" --compact \ | |
| > "$RESULT_SUMMARY_PATH" 2>/dev/null || | |
| xcrun xcresulttool get test-results summary --path "$RESULT_BUNDLE_PATH" \ | |
| > "$RESULT_SUMMARY_PATH" 2>/dev/null || | |
| rm -f "$RESULT_SUMMARY_PATH" | |
| exit 0 | |
| # Reports the run AND is the lane's own liveness check. `xcodebuild` exits 0 when a | |
| # selection matches nothing, so "green" and "ran no tests" are the same log tail: a | |
| # build variant without the unit-test compile flag, an empty test plan, or a renamed | |
| # target would all read as a healthy night. The reporter asserts the executed count | |
| # equals what `pnpm check:xctest-selection` derives this lane reaches from the `#if` | |
| # guards (XCTEST_LANE above), so a guard that compiles a file of tests out of iOS reads | |
| # as red too. The failure list is capped so a badly red night cannot blow the 1 MiB | |
| # job-summary limit and lose the headline with it. | |
| - name: Report the run and assert it executed the source-derived set | |
| if: always() | |
| run: | | |
| set -euo pipefail | |
| if [ ! -s "$RESULT_SUMMARY_PATH" ]; then | |
| echo 'No usable test-results summary was produced, so nothing here can show a test ran.' | |
| echo 'The suite step above failed before or during the run; read its log first.' | |
| echo '### iOS runner full XCTest suite' >> "$GITHUB_STEP_SUMMARY" | |
| echo 'No result-bundle summary; see the run log and the uploaded bundle.' >> "$GITHUB_STEP_SUMMARY" | |
| exit 1 | |
| fi | |
| node --experimental-strip-types scripts/xctest-run-summary.ts | |
| - name: Upload result bundle | |
| if: always() | |
| uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 | |
| with: | |
| name: xctest-nightly-results-${{ github.run_id }}-${{ github.run_attempt }} | |
| path: .tmp/xctest-nightly | |
| if-no-files-found: warn |