Downstream test gate fails open: verification steps are always skipped

[pose]

The downstream job in test-provider-ci.yml is the only thing in our pipeline that validates a template change against real provider CI. It currently reports success while skipping every verification step.

Across the last 12 test-provider-ci runs, 24 of 24 downstream jobs skipped all verification and reported green.

Mechanism

.github/workflows/test-provider-ci.yml:78-87:

- name: Await PR opened for ${{ matrix.repo }}
  id: pr_opened
  timeout-minutes: 1
  continue-on-error: true # Fall through if a PR wasn't created.
  run: |
    until gh search prs --repo ${{ matrix.repo }} --match body "...from commit ${{ github.sha }}." --json url | grep url; do sleep 30; done;
    echo "pull_request_created=true" >> "${GITHUB_OUTPUT}"

1. gh search prs queries the search index, which lags minutes behind for a just-created PR. The PRs REST API does not.
2. timeout-minutes: 1 kills the step before the index catches up.
3. The line setting pull_request_created=true sits after the loop, so it never executes.
4. continue-on-error: true turns the timeout into a green step.
5. The eleven following steps are gated on if: steps.pr_opened.outputs.pull_request_created == 'true' and all skip.

Concretely, on #2408 the deploy job logged Created pull request #1713 and pull-request-operation = created, and the gate then logged The action 'Await PR opened for pulumi/pulumi-provider-boilerplate' has timed out after 1 minutes. That PR was never checked.

Why the workaround exists

#926: with no diff, no PR is opened and the until loop hangs forever. The timeout and continue-on-error were added to unblock that. The problem is they make two cases indistinguishable:

Situation	Should	Does
No PR created (no diff)	pass quietly	pass
PR created, not yet indexed	wait, then gate on checks	pass

A fix must keep #926 working, so continue-on-error cannot simply be deleted.

Suggested fix

Find the PR by head branch instead of by search. The branch name is deterministic and known to both jobs (update-workflows.yml:133): update-github-actions-workflows-${{ github.run_number }}.

branch="update-github-actions-workflows-${{ github.run_number }}"
number=$(gh pr list --repo "${{ matrix.repo }}" --head "$branch" --json number --jq '.[0].number // empty')

gh pr list uses the PRs REST API, which is read-your-writes consistent, so an empty result genuinely means no PR was created rather than "the index is slow". Then drop continue-on-error and branch explicitly: empty means skip, non-empty means gate on checks and fail on failure.

Note update-workflows.yml:32-35 already exposes a pull_request_created workflow_call output that the downstream job does not consume. It is not directly usable as-is because deploy is a matrix job and Actions does not provide per-leg outputs, but it is an alternative if deploy is split.

Separately, timeout-minutes: 10 on Await required checks succeed is too short once the gate actually fires. Provider CI on these repos runs 20-30 minutes.

The required-checks design itself is sound and should not be changed: sentinel is a real aggregator (needs: [test, prerequisites, lint], test needs build_sdks), so a build_sdks failure correctly prevents the Sentinel status from posting.

Verifying a fix

The hard part is proving a gate that is supposed to fail. Worth testing both directions: a deliberately broken native template must turn downstream red, a benign change must pass, and a no-diff change must skip quickly without hanging (#926). The clearest signal is whether Find PR number and Await required checks succeed actually run rather than showing skipped.

[pose]

Stronger evidence: the gate did not merely fail to verify, it discarded a failure it had already caught.

The downstream PR that #2408 generated was pulumi/pulumi-provider-boilerplate#1713. Its checks:

SUCCESS   prerequisites
SUCCESS   lint / lint
FAILURE   build_sdks (nodejs)
CANCELLED build_sdks (python)
CANCELLED build_sdks (dotnet)
CANCELLED build_sdks (go)
CANCELLED build_sdks (java)
SKIPPED   test
SKIPPED   sentinel

Timeline:

• 05:05:57 — PR Update codecov/codecov-action action to v5.5.1 #1713 created
• 05:10:50 — Update actions/setup-node action to v7 #2408 merged green, while build_sdks (nodejs) was still failing
• 07:12:24 — PR Update codecov/codecov-action action to v5.5.1 #1713 closed

So the existing native downstream provider reproduced the regression exactly, on the exact job, and left the evidence in a public PR for two hours. The downstream job reported success the whole time.

This also confirms the required-checks chain behaves as designed: build_sdks (nodejs) failed, so test and sentinel were skipped, so the Sentinel status never posted. Had discovery worked, gh pr checks --required --watch --fail-fast would have had a pending-forever required check and the job would have gone red. Only discovery is broken.

Worth drawing one conclusion for planning: coverage was not the gap here. Adding further downstream test providers would have produced additional unreported failures rather than catching this sooner. Fixing discovery is the prerequisite for any of that work to be meaningful.

[pose]

Blocked by pulumi/pulumi-xyz#2723