ci: emit canary heartbeat metric for external failure alarm#1812
Open
jesseturner21 wants to merge 1 commit into
Open
ci: emit canary heartbeat metric for external failure alarm#1812jesseturner21 wants to merge 1 commit into
jesseturner21 wants to merge 1 commit into
Conversation
github-actions
Bot
added
size/m
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
github-actions
Bot
removed
the
agentcore-harness-reviewing
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
Contributor
Package TarballHow to installgh release download pr-1812-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-0.24.1.tgz |
Contributor
Coverage Report
|
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
Every canary run now emits a Success=1|0 heartbeat metric to CloudWatch via an if: always() step. An alarm in the canary's AWS account watches the metric and alerts the team when the canary fails or stops reporting. The alarm treats missing data as breaching, so a heartbeat (rather than a failure-triggered push) also covers failure modes that produce no signal at all: runner never starts, broken credentials, silently disabled cron.
jesseturner21
force-pushed
the
canary-failure-alarm
branch
from
July 22, 2026 22:32
d0617eb to
863c461
Compare
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
jesseturner21
changed the title
notgitika
approved these changes
Jul 23, 2026
notgitika
left a comment
notgitika
left a comment
Contributor
There was a problem hiding this comment.
Smart approach! LGTM :)
notgitika
reviewed
Jul 23, 2026
| # manually dispatched against another region — so an off-region run can't | ||
| # starve the alarm into a false missing-data breach. | ||
| readonly MONITOR_REGION=us-east-1 | ||
| readonly NAMESPACE=AgentCoreCLI/Canary |
Contributor
There was a problem hiding this comment.
qq can we make this an env variable instead? this would prevent anyone configuring it themselves to have this value that could mess with our alarms
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a heartbeat step to the canary workflow: every run emits a
Success = 1|0metric to CloudWatch (AgentCoreCLI/Canarynamespace) viaif: always(). A CloudWatch alarm in the canary's AWS account watches the metric and alerts the team when the canary fails — or stops running — for 30+ minutes.Why a heartbeat instead of a failure-triggered push
A metric pushed only on failure goes silent in exactly the scenarios we most need to know about: runner never starts, broken credentials, silently disabled cron. The alarm treats missing data as breaching, so with a heartbeat, silence is itself the signal.
Notes for reviewers
Canary FailureGitHub issue + Slack path, which carries the per-failure detail (variant, commit, run URL). The alarm only fires after 30+ minutes of failure or silence.*/15cron — there's a comment at the cron line; if the schedule changes, the alarm period must change with it.cloudwatch:PutMetricDataon the workflow's AWS role (already granted, scoped to theAgentCoreCLI/Canarynamespace).