Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/scripts/emit-canary-heartbeat.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
#!/usr/bin/env bash
# Emits a per-run heartbeat metric for the canary workflow. A CloudWatch
# alarm in the AWS account the canary deploys to watches this metric and
# alerts the team when the canary fails or stops reporting.
#
# Called from canary.yml with `if: always()`: every matrix cell reports
# Success=1 or Success=0 on every run. The alarm treats missing data as
# breaching, so silence (dead runner, broken credentials, disabled cron)
# trips it just like an explicit failure does — which is why this step
# must never be made conditional on failure only.
#
# Usage: emit-canary-heartbeat.sh <job-status>
# job-status: GitHub's ${{ job.status }} — "success", "failure", or "cancelled"
set -euo pipefail

# The alarm lives in us-east-1. Always emit there — even when the canary is
# manually dispatched against another region — so an off-region run can't
# starve the alarm into a false missing-data breach.
readonly MONITOR_REGION=us-east-1
readonly NAMESPACE=AgentCoreCLI/Canary

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qq can we make this an env variable instead? this would prevent anyone configuring it themselves to have this value that could mess with our alarms

readonly METRIC_NAME=Success

STATUS="${1:?usage: emit-canary-heartbeat.sh <job-status>}"

# Anything other than success (failure, cancelled/timeout) counts as a failed
# canary run.
if [[ "$STATUS" == "success" ]]; then
VALUE=1
else
VALUE=0
fi

aws cloudwatch put-metric-data \
--region "$MONITOR_REGION" \
--namespace "$NAMESPACE" \
--metric-name "$METRIC_NAME" \
--value "$VALUE"

echo "Emitted ${NAMESPACE}/${METRIC_NAME}=${VALUE} (job status: ${STATUS}) to ${MONITOR_REGION}"
12 changes: 12 additions & 0 deletions .github/workflows/canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ name: Canary
on:
# Run every 15 minutes to catch dependency/service breakages (e.g. CDK
# schema mismatches) quickly instead of up to an hour later.
# NOTE: a CloudWatch alarm in the canary's AWS account expects a heartbeat
# metric at this cadence (see the "Emit canary heartbeat" step) — if you
# change this schedule, update the alarm period to match.
schedule:
- cron: '*/15 * * * *'
workflow_dispatch:
Expand Down Expand Up @@ -96,6 +99,15 @@ jobs:
# Smoke test for now, only runs strands-bedrock.test.ts
run: npx vitest run --project e2e e2e-tests/strands-bedrock.test.ts --retry 3

- name: Emit canary heartbeat
# Every run emits AgentCoreCLI/Canary :: Success = 1|0, watched by a
# CloudWatch alarm in the canary's AWS account. The alarm treats
# MISSING data as breaching, so this step must run on success AND
# failure — and if it stops running entirely, that absence is itself
# what fires the alarm.
if: always()
run: .github/scripts/emit-canary-heartbeat.sh "${{ job.status }}"

- name: Generate GitHub App Token
if: failure()
id: app-token
Expand Down
Loading