Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/native-cli-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@ jobs:
env:
EVENT_NAME: ${{ github.event_name }}
EVENT_REF_NAME: ${{ github.ref_name }}
BEFORE_SHA: ${{ github.event.before }}
INPUT_DRY_RUN: ${{ inputs.dry-run }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: scripts/resolve-native-cli-release-target.sh >> "$GITHUB_OUTPUT"
Expand Down
23 changes: 21 additions & 2 deletions docs/release-recovery-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,23 @@ approve the `cli-release` environment if GitHub requests approval:
gh run rerun <run-id> --repo <owner>/<repo> --failed
```

## Retrying after a cancelled or failed publish run

`native-cli-publish.yml` and `dispatcher-publish.yml` only evaluate publish/
release on push when HEAD's diff actually stamps the resolved version (the
release-please manifest/contract entry or the component's `CHANGELOG.md`
heading). Once a publish run is cancelled or fails, ordinary follow-up pushes
to main/v3-beta no longer retry it, because their HEAD commits do not stamp
that version. Recovery is limited to two paths:

- (a) Rerun the original run, but only when the conditions in "When rerun
recovery is valid" above hold (its `headSha` is the intended release commit
and the workflow definition there is healthy).
- (b) Trigger the workflow manually via `workflow_dispatch`, which bypasses
the push gate entirely and falls back to the same state-based evaluation
(missing release / missing assets / fully published) the gate used before
this change.

## Operational notes

### Preserved negative attestation specimen
Expand All @@ -104,8 +121,10 @@ verification. Do not pin this version; use beta.49 or later.
workflow concurrency group does not cancel it automatically, so it can block
the newer run indefinitely.
- Delete a broken release only after the version bump is merged. Deleting it
first leaves the historical version unresolved and can make later push builds
attempt to recreate it and fail.
first leaves the historical version unresolved; later push builds will not
attempt to recreate it on their own (see "Retrying after a cancelled or
failed publish run" above), so recovery still requires an explicit rerun or
`workflow_dispatch`.
- Do not grant GitHub Actions tag-ruleset bypass permissions. That does not
resolve the attestation invariant and expands bot authority unnecessarily.
- A broken release may remain published while the roll-forward release is cut;
Expand Down
26 changes: 19 additions & 7 deletions docs/version-series-realignment.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,14 +51,26 @@ Two automations react differently to the realignment commit:
fail with "no release-please commit found" until the release exists.
- `.github/workflows/dispatcher-publish.yml`
(`scripts/resolve-dispatcher-release-target.sh`) ignores commit subjects
entirely. On every push it derives the release tag from the
`dispatcherVersion` in the contract at HEAD and, when that release is
missing or lacks assets, builds the binaries and creates/publishes the
release with attestations at the pushed commit. This is what materializes
the realigned version as a real release.
entirely, but on push it only evaluates when HEAD's diff actually stamps
the resolved `dispatcherVersion` — a `+` line matching
`"dispatcherVersion": "<version>"` in the contract or a
`## [<version>]` heading in `cli/dispatcher/CHANGELOG.md`
(`release_commit_updates_dispatcher_version`). A `fix:`-titled realignment
commit still stamps both the contract and the CHANGELOG, so it passes this
gate even though its subject doesn't look like a release commit. When the
gate passes and the resolved release is missing or lacks assets, the
workflow builds the binaries and creates/publishes the release with
attestations at the pushed commit. This is what materializes the realigned
version as a real release.

If the release-please workflow failed before dispatcher-publish finished,
re-run it via `workflow_dispatch` once the release exists with all assets.
If the realignment push doesn't stamp the version (so dispatcher-publish
skips it on push), or the resulting dispatcher-publish run is cancelled or
fails, retry dispatcher-publish via `workflow_dispatch`, which bypasses the
push gate and falls back to the state-based evaluation described above.

If instead the release-please workflow failed before dispatcher-publish
finished, re-run it via `workflow_dispatch` once the release exists with all
assets.

## What not to touch

Expand Down
23 changes: 23 additions & 0 deletions scripts/resolve-dispatcher-release-target.sh
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,24 @@ dispatcher_release_is_prerelease() {
[ "$ref_name" = "v3-beta" ]
}

release_commit_updates_dispatcher_version() {
commit_sha=$1
version=$2
expected_contract_entry="\"dispatcherVersion\": \"$version\""
expected_changelog_heading="## [$version]"

commit_diff=$(git show --format= "$commit_sha" -- "$DISPATCHER_CONTRACT_PATH" cli/dispatcher/CHANGELOG.md 2>/dev/null || true)
printf '%s\n' "$commit_diff" \
| awk -v contract_entry="$expected_contract_entry" -v changelog_heading="$expected_changelog_heading" '
substr($0, 1, 1) == "+" && (index($0, contract_entry) > 0 || index($0, changelog_heading) > 0) {
found = 1
}
END {
exit found ? 0 : 1
}
'
}

DISPATCHER_CONTRACT_PATH=cli/dispatcher/dispatchercontract/dispatcher-contract.json

VERSION=$(jq -r '.dispatcherVersion' "$DISPATCHER_CONTRACT_PATH")
Expand Down Expand Up @@ -163,6 +181,7 @@ if [ "$EVENT_NAME" = "push" ]; then
fi

TARGET_SHA=$(dispatcher_release_target_sha "$RELEASE_TAG")
BUILD_SHA=$(git rev-parse HEAD)
IS_PRERELEASE=false
if dispatcher_release_is_prerelease "$VERSION" "$EVENT_REF_NAME"; then
IS_PRERELEASE=true
Expand All @@ -171,6 +190,10 @@ fi
if [ "$CAN_EVALUATE_DISPATCHER_RELEASE" != "true" ]; then
SHOULD_PUBLISH=false
SHOULD_RELEASE=false
elif [ "$EVENT_NAME" = "push" ] && ! release_commit_updates_dispatcher_version "$BUILD_SHA" "$VERSION"; then
echo "HEAD commit $BUILD_SHA does not stamp dispatcher version $VERSION; skipping dispatcher publish. Retry an incomplete release with workflow_dispatch." >&2
SHOULD_PUBLISH=false
SHOULD_RELEASE=false
elif release_is_published_with_dispatcher_assets "$RELEASE_TAG"; then
SHOULD_PUBLISH=false
SHOULD_RELEASE=false
Expand Down
68 changes: 8 additions & 60 deletions scripts/resolve-native-cli-release-target.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,21 +9,6 @@ EVENT_REF_NAME=${EVENT_REF_NAME:-}
INPUT_RELEASE_TAG=${INPUT_RELEASE_TAG:-}
INPUT_DRY_RUN=${INPUT_DRY_RUN:-false}
RELEASE_DATA=""
CLI_RELEASE_INPUT_PATHS="
cli/.go-version
cli/go.work
cli/go.work.sum
layout-contract.json
cli/common
cli/project-runner/cmd
cli/project-runner/go.mod
cli/project-runner/go.sum
cli/project-runner/internal
scripts/build-go-cli.sh
scripts/go-cli-toolchain.sh
scripts/package-go-cli.sh
scripts/verify-native-cli-release-assets.sh
"

is_semver_version() {
version=$1
Expand Down Expand Up @@ -118,33 +103,6 @@ release_is_published() {
[ "$is_draft" = "false" ]
}

latest_cli_asset_release_tag() {
excluded_tag=$1
release_list=$(gh release list --limit 100 --json tagName,isDraft)
release_tags=$(printf '%s\n' "$release_list" | jq -r '.[] | select(.isDraft == false) | .tagName')
for release_tag in $release_tags; do
if [ "$release_tag" = "$excluded_tag" ]; then
continue
fi

if release_has_all_cli_assets "$release_tag"; then
printf '%s\n' "$release_tag"
return
fi
done
}

cli_release_inputs_changed() {
base_ref=$1
head_ref=$2

if ! git diff --quiet "$base_ref" "$head_ref" -- $CLI_RELEASE_INPUT_PATHS; then
return 0
fi

return 1
}

release_commit_updates_cli_version() {
commit_sha=$1
version=$2
Expand Down Expand Up @@ -230,29 +188,19 @@ RELEASE_TARGET_SHA=$BUILD_SHA
if [ "$CAN_EVALUATE_CLI_RELEASE" != "true" ]; then
SHOULD_PUBLISH=false
SHOULD_RELEASE=false
elif release_is_published "$RELEASE_TAG"; then
SHOULD_RELEASE=false
else
SHOULD_RELEASE=true
fi

if [ "$CAN_EVALUATE_CLI_RELEASE" != "true" ]; then
elif [ "$EVENT_NAME" = "push" ] && ! release_commit_updates_cli_version "$BUILD_SHA" "$VERSION"; then
echo "HEAD commit $BUILD_SHA does not stamp project runner version $VERSION; skipping native CLI publish. Retry an incomplete release with workflow_dispatch." >&2
SHOULD_PUBLISH=false
SHOULD_RELEASE=false
elif release_is_published_with_cli_assets "$RELEASE_TAG"; then
SHOULD_PUBLISH=false
SHOULD_RELEASE=false
else
PREVIOUS_CLI_RELEASE_TAG=$(latest_cli_asset_release_tag "$RELEASE_TAG")
if [ -z "$PREVIOUS_CLI_RELEASE_TAG" ]; then
echo "No previous project runner asset release found; publishing native project runner assets." >&2
SHOULD_PUBLISH=true
elif release_commit_updates_cli_version "$BUILD_SHA" "$VERSION"; then
echo "Project runner release metadata changed in $BUILD_SHA; publishing native project runner assets." >&2
SHOULD_PUBLISH=true
elif cli_release_inputs_changed "$PREVIOUS_CLI_RELEASE_TAG" "$TARGET_SHA"; then
echo "Project runner release inputs changed since $PREVIOUS_CLI_RELEASE_TAG; publishing native project runner assets." >&2
SHOULD_PUBLISH=true
SHOULD_PUBLISH=true
if release_is_published "$RELEASE_TAG"; then
SHOULD_RELEASE=false
else
echo "Project runner release inputs are unchanged since $PREVIOUS_CLI_RELEASE_TAG; skipping native CLI publish." >&2
SHOULD_RELEASE=true
fi
fi

Expand Down
74 changes: 71 additions & 3 deletions scripts/test-resolve-dispatcher-release-target.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,23 @@ if [ "$1" = "rev-list" ] && [ "$2" = "-n" ] && [ "$3" = "1" ]; then
exit 1
fi

if [ "$1" = "show" ]; then
if [ "${DISPATCHER_BUILD_COMMIT_STAMPS:-false}" != "true" ]; then
exit 0
fi
case "${DISPATCHER_STAMP_KIND:-contract}" in
contract)
printf '%s\n' 'diff --git a/cli/dispatcher/dispatchercontract/dispatcher-contract.json b/cli/dispatcher/dispatchercontract/dispatcher-contract.json'
printf '+ "dispatcherVersion": "%s"\n' "$CURRENT_VERSION"
;;
changelog)
printf '%s\n' 'diff --git a/cli/dispatcher/CHANGELOG.md b/cli/dispatcher/CHANGELOG.md'
printf '+## [%s]\n' "$CURRENT_VERSION"
;;
esac
exit 0
fi

echo "unexpected git command: $*" >&2
exit 1
MOCK_GIT
Expand Down Expand Up @@ -81,27 +98,37 @@ write_contract() {
'}' > cli/dispatcher/dispatchercontract/dispatcher-contract.json
}

# Runs the resolver with mocked git/gh and returns stdout; stderr is captured to
# "$TMP_DIR/$name/stderr.txt" for tests that need to assert skip messages.
run_case() {
name=$1
release_state=$2
has_assets=$3
tag_sha=${4:-}
ref_name=${5:-v3-beta}
event_name=${6:-push}
build_commit_stamps=${7:-true}
stamp_kind=${8:-contract}
version=${9:-3.0.0}
work_dir="$TMP_DIR/$name"
mkdir -p "$work_dir"
(
cd "$work_dir"
write_contract "3.0.0"
write_contract "$version"
mock_bin="$work_dir/bin"
write_mock_commands "$mock_bin"
PATH="$mock_bin:$ORIGINAL_PATH" \
EVENT_NAME=push \
EVENT_NAME="$event_name" \
EVENT_REF_NAME="$ref_name" \
DISPATCHER_RELEASE_STATE="$release_state" \
DISPATCHER_RELEASE_HAS_ASSETS="$has_assets" \
DISPATCHER_TAG_SHA_VALUE="$tag_sha" \
"$SCRIPT"
DISPATCHER_BUILD_COMMIT_STAMPS="$build_commit_stamps" \
DISPATCHER_STAMP_KIND="$stamp_kind" \
CURRENT_VERSION="$version" \
"$SCRIPT" > output.txt 2> stderr.txt
)
cat "$work_dir/output.txt"
}

assert_contains() {
Expand All @@ -114,6 +141,18 @@ assert_contains() {
fi
}

assert_file_contains() {
file=$1
expected=$2
if ! grep -F -- "$expected" "$file" >/dev/null; then
echo "Expected $file to contain: $expected" >&2
echo "Actual content:" >&2
cat "$file" >&2
exit 1
fi
}

# Verifies a push with a stamping commit publishes and releases when no dispatcher release exists yet.
test_missing_release_publishes() {
output=$(run_case missing-release missing false)
assert_contains "$output" "publish=true"
Expand All @@ -122,34 +161,63 @@ test_missing_release_publishes() {
assert_contains "$output" "prerelease=true"
}

# Verifies a push with a stamping commit skips when the dispatcher release is already fully published.
test_complete_release_skips() {
output=$(run_case complete-release published true)
assert_contains "$output" "publish=false"
assert_contains "$output" "release=false"
}

# Verifies a push with a stamping commit uploads remaining assets onto a published release missing assets.
test_published_release_with_missing_assets_uploads_assets() {
output=$(run_case missing-assets published false)
assert_contains "$output" "publish=true"
assert_contains "$output" "release=false"
}

# Verifies the existing release tag SHA is reported when following up on a published release missing assets.
test_published_release_with_missing_assets_uses_existing_tag_sha() {
output=$(run_case missing-assets-followup published false release-sha)
assert_contains "$output" "publish=true"
assert_contains "$output" "release=false"
assert_contains "$output" "sha=release-sha"
}

# Verifies a push with a stamping commit on main is not treated as a prerelease.
test_main_branch_release_is_not_prerelease() {
output=$(run_case main-missing-release missing false "" main)
assert_contains "$output" "publish=true"
assert_contains "$output" "release=true"
assert_contains "$output" "prerelease=false"
}

# Verifies a push without a stamping commit skips publish/release and points to workflow_dispatch for retries.
test_push_unstamped_skips_and_reports_workflow_dispatch() {
output=$(run_case push-unstamped missing false "" v3-beta push false)
assert_contains "$output" "publish=false"
assert_contains "$output" "release=false"
assert_file_contains "$TMP_DIR/push-unstamped/stderr.txt" "does not stamp dispatcher version"
assert_file_contains "$TMP_DIR/push-unstamped/stderr.txt" "workflow_dispatch"
}

# Verifies a changelog-only stamp (version-series-realignment commits) still publishes on push.
test_push_changelog_only_stamp_publishes() {
output=$(run_case push-changelog-only missing false "" v3-beta push true changelog)
assert_contains "$output" "publish=true"
}

# Verifies workflow_dispatch bypasses the push gate and publishes via state-based evaluation when no release exists.
test_dispatch_unstamped_missing_release_publishes() {
output=$(run_case dispatch-missing missing false "" v3-beta workflow_dispatch false)
assert_contains "$output" "publish=true"
assert_contains "$output" "release=true"
}

test_missing_release_publishes
test_complete_release_skips
test_published_release_with_missing_assets_uploads_assets
test_published_release_with_missing_assets_uses_existing_tag_sha
test_main_branch_release_is_not_prerelease
test_push_unstamped_skips_and_reports_workflow_dispatch
test_push_changelog_only_stamp_publishes
test_dispatch_unstamped_missing_release_publishes
Loading