XCTest Nightly #2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: XCTest Nightly | |
| # The full iOS runner XCTest suite (#1781 A7). The PR lane (ios.yml) names a hand-written | |
| # subset of the target's methods in an `-only-testing:` list; everything outside that list ran | |
| # nowhere at all. This lane drops the filter and runs the test plan whole, skipping only | |
| # `testCommand` — which is the runner's server entry point rather than a test (see the step | |
| # below). | |
| # | |
| # No count is quoted here on purpose. `pnpm check:xctest-selection` prints the live split | |
| # (declared / PR-selected / skipped / nightly-only) and is the only place those numbers are | |
| # derived rather than remembered. A hand-copied figure in a comment is the same class of | |
| # defect this lane exists to close: between this file being written and first review, the | |
| # suite gained three methods and the PR list gained five entries, so every number originally | |
| # written here was already wrong. | |
| # | |
| # Under the lane rule in #1781: | |
| # | |
| # - Catches: regressions in the runner-semantics tests the PR list leaves out — selector | |
| # matching, snapshot traversal, command journal, sequence execution, tv remote, keyboard, | |
| # scroll/gesture policy. Nothing else executes them. Note the reachable set is smaller | |
| # than the declared set: `testCommand` is skipped, and the `…OnTvOS` tests sit behind | |
| # `#if os(tvOS)` and cannot execute on this lane's iOS Simulator destination. | |
| # - Evidence: the list has never been re-derived since it was written by hand; the tests it | |
| # omits have been dark for their whole life, so their current state is unmeasured. The | |
| # first red night is the evidence this lane exists to produce. | |
| # - Cost: this is a public repository, so nothing here is billed. The real cost is one macOS | |
| # concurrency slot per night. The Apple runner build shares ios.yml's cache (identical | |
| # inputs and build variant), but scheduled runs only see DEFAULT-BRANCH caches, so the hit | |
| # depends on ios.yml's `push: main` run having populated the cache at the same source | |
| # hash — on a miss this lane also pays a full runner build. Adds nothing to any PR. | |
| # - Kill criterion: when A7's classification lands — pure Swift decision tests moved to a | |
| # fast unit target, stale tests deleted, the rest kept on the simulator — this lane goes | |
| # if the surviving set is small enough to run on every PR. | |
| # | |
| # Scheduled + manual only. It is deliberately NOT a PR gate: ios.yml is already over half of | |
| # this repo's CI spend (#1781 A9), and moving the unrun set onto it would make that worse for | |
| # tests whose flakiness on a shared runner is unknown. | |
| # | |
| # Note on `concurrency` below: it is keyed on `github.ref`, so a manual dispatch on the same | |
| # ref cancels an in-flight scheduled run. That is the intended trade — an operator asking for | |
| # a run wants the fresh one — but it means "dispatch while the nightly is running" loses the | |
| # nightly's result rather than queueing behind it. | |
| on: | |
| schedule: | |
| - cron: '30 4 * * *' | |
| workflow_dispatch: | |
| permissions: | |
| contents: read | |
| actions: read | |
| concurrency: | |
| group: ci-${{ github.workflow }}-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| full-suite: | |
| name: iOS Runner Full XCTest Suite | |
| runs-on: macos-26 | |
| # Measured, not guessed. A full local run of this exact command on matching toolchain | |
| # (Xcode 26.2, iOS 26.2 Simulator) executed 154 tests in 120s wall — the suite itself is | |
| # ~2 minutes, not the bulk of the job. What dominates the budget is the runner build on a | |
| # derived-data cache MISS, plus checkout and simulator boot. ios.yml carries 80 minutes | |
| # while doing strictly more than this lane (same build and boot, plus replays, prepare, | |
| # and the fixture E2E), so half of that is generous headroom for a cache-miss night and | |
| # still caps a wedged run at 45 minutes instead of two hours of held macOS slot. | |
| timeout-minutes: 45 | |
| env: | |
| # Must match ios.yml. The runtime version rides in the derived-data cache key, and | |
| # AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS is BOTH an input to the cache's build-variant | |
| # hash and the `-D AGENT_DEVICE_RUNNER_UNIT_TESTS` compile flag that makes the RunnerTests | |
| # methods exist at all. Dropping it would build a different variant (cache miss) whose | |
| # bundle contains no tests, and `xcodebuild` reports running zero tests as success — the | |
| # "silent 0-test green" this lane would then be. The test-count assertion below is the | |
| # backstop for exactly that. | |
| IOS_RUNTIME_VERSION: '26.2' | |
| AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS: '1' | |
| AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived | |
| RESULT_BUNDLE_PATH: ${{ github.workspace }}/.tmp/xctest-nightly/RunnerTests.xcresult | |
| RESULT_SUMMARY_PATH: ${{ github.workspace }}/.tmp/xctest-nightly/test-results-summary.json | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | |
| - name: Setup toolchain | |
| uses: ./.github/actions/setup-node-pnpm | |
| with: | |
| cache-dependency-path: | | |
| pnpm-lock.yaml | |
| examples/test-app/pnpm-lock.yaml | |
| # Byte-identical inputs to ios.yml's step, so both lanes restore the same cache entry | |
| # and only the first one to run on a commit pays for the build. | |
| - name: Restore and build iOS XCTest runner | |
| uses: ./.github/actions/setup-apple-runner-build | |
| with: | |
| derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }} | |
| cache-key-prefix: ios-runner-prebuilt | |
| cache-key-suffix: -ios-${{ env.IOS_RUNTIME_VERSION }} | |
| gate: swift-runner-ios | |
| xcuitest-platform: ios | |
| xcuitest-destination: generic/platform=iOS Simulator | |
| - name: Boot iOS test simulator | |
| id: ios-simulator | |
| uses: ./.github/actions/boot-ios-test-simulator | |
| with: | |
| runtime-version: ${{ env.IOS_RUNTIME_VERSION }} | |
| preferred-device-name: iPhone 17 Pro | |
| # Same command as ios.yml's targeted step minus every `-only-testing:` flag, so the | |
| # xctestrun's own test plan decides what runs — with one exception. | |
| # | |
| # `RunnerTests/testCommand` is not a test. It is the runner's server entry point: it | |
| # opens an NWListener and blocks in `XCTWaiter.wait(timeout: 24 * 60 * 60)` until a | |
| # client closes the session, which is exactly how production drives it | |
| # (src/platforms/apple/core/runner/runner-session.ts always passes it as the sole | |
| # `-only-testing:`). It compiles unconditionally — the `#if AGENT_DEVICE_RUNNER_UNIT_TESTS` | |
| # block ends well above it — so an unfiltered run reaches it in alphabetical order and | |
| # hangs the job until `timeout-minutes`. Its one escape hatch, | |
| # AGENT_DEVICE_RUNNER_NOOP_STARTUP, is an environment variable, and the comment above | |
| # the flag records that env plumbing into a simulator test process is not dependable; | |
| # `-skip-testing:` is the lever that works from here. | |
| # | |
| # A typo in that identifier silently re-arms the hang, so | |
| # `pnpm check:xctest-selection` validates `-skip-testing:` exactly like `-only-testing:`. | |
| - name: Run the full iOS runner XCTest suite | |
| run: | | |
| set -euo pipefail | |
| XCTESTRUN_PATH="$(find "$AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH/Build/Products" -maxdepth 1 -name '*.xctestrun' -print -quit)" | |
| test -n "$XCTESTRUN_PATH" | |
| mkdir -p "$(dirname "$RESULT_BUNDLE_PATH")" | |
| xcodebuild test-without-building \ | |
| -xctestrun "$XCTESTRUN_PATH" \ | |
| -destination "platform=iOS Simulator,id=${{ steps.ios-simulator.outputs.simulator-udid }}" \ | |
| -skip-testing:AgentDeviceRunnerUITests/RunnerTests/testCommand \ | |
| -resultBundlePath "$RESULT_BUNDLE_PATH" | |
| # Best-effort and never the job's verdict on its own; the step below is what asserts. | |
| # `--compact` first because a red night's summary is the large one, and the job summary | |
| # has a 1 MiB cap — the plain form is the fallback if an Xcode version rejects the flag. | |
| - name: Extract the test-results summary | |
| if: always() | |
| run: | | |
| set -uo pipefail | |
| [ -d "$RESULT_BUNDLE_PATH" ] || exit 0 | |
| xcrun xcresulttool get test-results summary --path "$RESULT_BUNDLE_PATH" --compact \ | |
| > "$RESULT_SUMMARY_PATH" 2>/dev/null || | |
| xcrun xcresulttool get test-results summary --path "$RESULT_BUNDLE_PATH" \ | |
| > "$RESULT_SUMMARY_PATH" 2>/dev/null || | |
| rm -f "$RESULT_SUMMARY_PATH" | |
| exit 0 | |
| # Reports the run AND is the lane's own liveness check. `xcodebuild` exits 0 when a | |
| # selection matches nothing, so "green" and "ran no tests" are the same log tail: a | |
| # build variant without the unit-test compile flag, an empty test plan, or a renamed | |
| # target would all read as a healthy night. Asserting the run executed tests is what | |
| # tells those apart. The failure list is capped so a badly red night cannot blow the | |
| # 1 MiB job-summary limit and lose the headline with it. | |
| - name: Report the run and assert it executed tests | |
| if: always() | |
| run: | | |
| set -euo pipefail | |
| if [ ! -s "$RESULT_SUMMARY_PATH" ]; then | |
| echo 'No usable test-results summary was produced, so nothing here can show a test ran.' | |
| echo 'The suite step above failed before or during the run; read its log first.' | |
| echo '### iOS runner full XCTest suite' >> "$GITHUB_STEP_SUMMARY" | |
| echo 'No result-bundle summary; see the run log and the uploaded bundle.' >> "$GITHUB_STEP_SUMMARY" | |
| exit 1 | |
| fi | |
| node --experimental-strip-types scripts/xctest-run-summary.ts | |
| - name: Upload result bundle | |
| if: always() | |
| uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 | |
| with: | |
| name: xctest-nightly-results-${{ github.run_id }}-${{ github.run_attempt }} | |
| path: .tmp/xctest-nightly | |
| if-no-files-found: warn |