investigate rust-ci matrix jobs that fail before any steps run

[dzianisv]

Summary

On PR #25, most rust-ci matrix jobs failed in 1-3 seconds with no steps executed at all. The jobs show failure but expose an empty steps array in the run metadata.

Repro

Seen on PR #25 on 2026-03-16:

• Workflow: rust-ci
• Run: 23166074105

Examples:

• Tests — ubuntu-24.04 - x86_64-unknown-linux-gnu
• Tests — windows-x64 - x86_64-pc-windows-msvc
• Lint/Build — ubuntu-24.04 - x86_64-unknown-linux-gnu
• Lint/Build — macos-15-xlarge - aarch64-apple-darwin

Observed metadata:

• jobs start and complete within ~1-3 seconds
• gh run view ... --json jobs reports conclusion: failure with steps: []
• this looks like a workflow/runner provisioning problem, not a normal test failure inside the job body

Why this matters

These failures make PRs appear red even before repository code is exercised. They block meaningful CI signal and hide actual code regressions.

Acceptance criteria

• identify why these matrix jobs fail before running any steps
• restore normal step execution for the affected matrix entries
• ensure future CI failures surface actionable logs instead of empty-step failures

[dzianisv]

Implemented the runner-availability fix on PR #32.

What changed:

• rust-ci now keeps openai/codex on its existing larger/custom runner layout
• forks fall back to standard hosted labels for the same target architecture
• Bazel macOS jobs now do the same instead of always requesting macos-15-xlarge

Fallback mapping in the patch:

• codex-linux-x64 -> ubuntu-24.04
• codex-linux-arm64 -> ubuntu-24.04-arm
• codex-windows-x64 -> windows-2025
• codex-windows-arm64 -> windows-11-arm
• macos-15-xlarge arm64 target -> macos-15
• macos-15-xlarge x86_64 target -> macos-15-intel

Validation done locally:

• YAML parse for both workflow files
• actionlint on the modified workflow files

PR:

• ci: unblock fork CI runner selection and npm staging #32 ci: unblock fork CI runner selection and npm staging #32

This PR is stacked on top of #31 so the workflow verification runs with both CI fixes in place.

[OpenCodeEngineer]

Pushed a follow-up fix on PR #32 at d3d11f1137fab1aabd5a9c70db8a5bf808fe4b5d to address the newly exposed hosted-CI fallout after the runner scheduling fix.

What changed on top of the runner fallback patch:

• codex-rs/cli/build.rs no longer uses expect() for CARGO_MANIFEST_DIR; it falls back to CODEX_CLI_VERSION=unknown
• codex-rs/core/src/codex.rs no longer uses expect() for reflection model info inside the reflection loop
• codex-rs/core/src/models_manager/manager.rs now satisfies the hosted clippy lints that became visible once fork jobs actually started running
• fork Bazel matrix selection is now expressed via fromJSON(...) include lists so forks only schedule the hosted lanes that are currently viable, while openai/codex keeps the full matrix

Validation on this SHA:

• just fmt
• cargo test -p codex-cli
• cargo clippy -p codex-cli --all-features --tests -- -D warnings

Additional signal:

• cargo test -p codex-core still fails in broad issue-fix codex-core late-suite failures in remote model and search-tool flows #26 areas (compact, remote-models, search-tool, rmcp, truncation, view-image). That does not change the runner diagnosis here; it confirms PR test(core): fix late-suite harness regressions #30 remains the next queue after PR ci: unblock fork CI runner selection and npm staging #32 is green.

I am actively babysitting PR #32 on the new SHA now. Current watch snapshot: 1 passed, 26 pending, 0 failed.

[OpenCodeEngineer]

Additional PR #32 follow-up pushed at 0f01a92c8.

The first hosted rerun exposed a second-stage lint issue after the runner fix:

• all completed red Lint/Build jobs were failing on the same clippy::single_component_path_imports error in codex-rs/core/tests/common/lib.rs
• the apparent arm64 GNU test failure was just a workflow cancellation after clippy had already turned the matrix red

Fix:

• remove the redundant use serde_json; import from codex-rs/core/tests/common/lib.rs

Local validation for the fix:

• just fmt
• cargo clippy -p core_test_support --all-features --tests -- -D warnings

I’ve pushed the fix and restarted PR babysitting on the new SHA.

[OpenCodeEngineer]

Another PR #32 follow-up is pushed at 365c5a7c6.

This batch addresses the broader hosted rust-ci clippy failures that became visible after the initial runner fix and the second-stage redundant-import fix:

• codex-rs/tui/tests/suite/model_switching_e2e.rs: helper-signature / type-complexity / redundant-clone lint cleanup
• codex-rs/core/tests/suite/cli_stream.rs: remove a needless_collect
• codex-rs/core/tests/suite/eval_swe_bench.rs: narrow expect_used expectations for the ignored Azure eval harness and replace deprecated Command::cargo_bin
• codex-rs/core/tests/suite/reflection.rs: same treatment for the ignored Azure reflection harness

Local validation:

• just fmt
• cargo clippy -p codex-tui --all-features --tests -- -D warnings
• cargo clippy -p codex-core --all-features --tests -- -D warnings with CARGO_BUILD_JOBS=1

The remaining known non-code failure class is the flaky/external fork Bazel fetch on Linux GNU, which is still the open thread in issue #33. I’m continuing to babysit PR #32 on the new SHA.

[OpenCodeEngineer]

Status update on issue #28:

A real follow-up code failure surfaced on PR #32 after the runner fallback changes exposed hosted Bazel and test lanes. The failing root cause was core/tests/common/lib.rs using include_str!("../../models.json") from the separate core/tests/common Bazel package, which does not have that relative path available at compile time.

I fixed this in commit e607fb8e4 by switching the shared test support to the same runtime resource lookup pattern we already use elsewhere:

• load bundled models.json via codex_utils_cargo_bin::find_resource!("../../models.json")
• share that loader between core/tests/common/lib.rs and core/tests/common/test_codex.rs
• remove the now-dead anyhow::Context import from test_codex.rs

Local validation on the final tree:

• cargo test -p core_test_support ✅
• bazel test //codex-rs/core/tests/common:common-unit-tests --test_output=errors ✅

PR #32 has been updated and CI restarted on SHA e607fb8e4e47353c0009486bab0eca00b0762cb0. I’m keeping the watcher on the new SHA until it settles.

[OpenCodeEngineer]

Pushed another follow-up on PR #32 at 1a8218d135a642244c182a77805fb7b32757bc83 (codex: fix PR #32 hosted test regressions).

What this batch fixes:

• codex-rs/core/tests/suite/models_cache_ttl.rs now writes provider_scope into seeded models_cache.json fixtures, so the cache-version tests exercise real cache hits instead of being rejected by the newer cache schema.
• codex-rs/app-server/tests/common/models_cache.rs now writes the same provider_scope field, which restores seeded-cache behavior for the full app-server model/list suite.
• codex-rs/tui/tests/suite/model_switching_e2e.rs and codex-rs/tui/tests/suite/model_availability_nux.rs now quote Windows TOML project paths correctly and prefer sibling/target binaries before falling back to cargo build, which removes the hosted-runner timeouts caused by in-test Cargo locks.
• .github/workflows/rust-ci.yml now prebuilds codex and test_stdio_server in the test profile before nextest.
• codex-rs/core/src/memories/prompts.rs now uses include_str!() prompt templates instead of Askama file-path resolution, which is Bazel-safe.
• codex-rs/tui/src/chatwidget/tests.rs now uses a Linux-specific experimental-popup snapshot so the Bubblewrap row no longer fights the macOS snapshot.

Local validation on this SHA:

• just fmt
• cargo test -p core_test_support
• cargo test -p codex-core --test all suite::models_cache_ttl::uses_cache_when_version_matches -- --nocapture --exact
• cargo test -p codex-core --test all suite::remote_models::remote_models_truncation_policy_with_tool_output_override -- --nocapture --exact
• cargo test -p codex-core --test all suite::rmcp_client::stdio_image_responses_are_sanitized_for_text_only_model -- --nocapture --exact
• cargo test -p codex-core memories::prompts::tests -- --nocapture
• cargo test -p codex-app-server --test all suite::v2::model_list -- --nocapture
• cargo test -p codex-tui --test all suite::model_switching_e2e::models_dev_provider_config_parses_custom_provider -- --nocapture --exact
• cargo test -p codex-tui --test all suite::model_switching_e2e::ollama_model_switch_then_prompt_uses_responses_api -- --nocapture --exact
• cargo test -p codex-tui --test all suite::model_availability_nux::resume_startup_does_not_consume_model_availability_nux_count -- --nocapture --exact
• cargo test -p codex-tui chatwidget::tests::experimental_popup_marks_reflection_enabled_when_feature_flag_is_enabled -- --nocapture

The rerun on PR #32 is already live on this SHA. Current state right after push: 4 checks passed, 23 pending, 0 failed. I’m continuing to babysit the rerun.

[OpenCodeEngineer]

Another follow-up is pushed on PR #32 at d6a06413f38b688f4529aebbd6c14e110a57a420 (codex: fix PR #32 windows arm node setup).

The fresh rerun on 1a8218d13 exposed a new workflow-only failure, not a Rust test regression:

• Tests — windows-11-arm - aarch64-pc-windows-msvc died in step 3 (actions/setup-node@v6) before rust-toolchain or nextest even started
• the job log showed the failure immediately after resolving codex-rs/node-version.txt to Node 22.22.0 and attempting the download on Windows ARM

Fix in d6a06413f:

• keep the existing setup-node path for all non-Windows-ARM test lanes
• install x64 Node explicitly on the Windows ARM test lane so js_repl setup uses the emulated x64 runtime instead of the failing default architecture path

Validation:

• local YAML parse of .github/workflows/rust-ci.yml after the edit

The new rerun is live on d6a06413f. Current snapshot right after the push: 2 checks passed, 25 pending, 0 failed. I’m still babysitting the PR.