Add execution report heartbeat alerts (#130)

Author: Pigbibi

.github/workflows/execution-report-heartbeat.yml

@@ -0,0 +1,65 @@
+name: Execution Report Heartbeat
+
+on:
+  workflow_dispatch:
+    inputs:
+      lookback_hours:
+        description: "Report lookback window in hours."
+        required: false
+        type: string
+        default: "36"
+      fail_workflow_on_alert:
+        description: "Fail this workflow when an alert is emitted."
+        required: false
+        type: choice
+        default: "true"
+        options:
+          - "true"
+          - "false"
+  schedule:
+    - cron: "15 23 * * 1-5"
+
+env:
+  GCP_PROJECT_ID: interactivebrokersquant
+  GCP_WORKLOAD_IDENTITY_PROVIDER: projects/303168642265/locations/global/workloadIdentityPools/github-actions/providers/github-main
+  GCP_WORKLOAD_IDENTITY_SERVICE_ACCOUNT: ibkr-platform-deploy@interactivebrokersquant.iam.gserviceaccount.com
+
+jobs:
+  heartbeat:
+    name: Check execution report heartbeat
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      id-token: write
+    env:
+      RUNTIME_HEARTBEAT_NAME: InteractiveBrokersPlatform
+      RUNTIME_HEARTBEAT_REPORT_PLATFORM: interactive_brokers
+      RUNTIME_HEARTBEAT_REQUIRED_SERVICES: ${{ vars.RUNTIME_HEARTBEAT_REQUIRED_SERVICES }}
+      RUNTIME_HEARTBEAT_GCS_URIS: ${{ vars.RUNTIME_HEARTBEAT_GCS_URIS || vars.EXECUTION_REPORT_GCS_URI }}
+      RUNTIME_HEARTBEAT_LOOKBACK_HOURS: ${{ inputs.lookback_hours || vars.RUNTIME_HEARTBEAT_LOOKBACK_HOURS || '36' }}
+      RUNTIME_HEARTBEAT_FAIL_WORKFLOW_ON_ALERT: ${{ inputs.fail_workflow_on_alert || vars.RUNTIME_HEARTBEAT_FAIL_WORKFLOW_ON_ALERT || 'true' }}
+      RUNTIME_HEARTBEAT_ACCEPT_STATUSES: ${{ vars.RUNTIME_HEARTBEAT_ACCEPT_STATUSES }}
+      RUNTIME_HEARTBEAT_REJECT_STATUSES: ${{ vars.RUNTIME_HEARTBEAT_REJECT_STATUSES }}
+      CLOUD_RUN_SERVICE: ${{ vars.CLOUD_RUN_SERVICE }}
+      CLOUD_RUN_SERVICES: ${{ vars.CLOUD_RUN_SERVICES }}
+      CLOUD_RUN_SERVICE_TARGETS_JSON: ${{ vars.CLOUD_RUN_SERVICE_TARGETS_JSON }}
+      GLOBAL_TELEGRAM_CHAT_ID: ${{ vars.GLOBAL_TELEGRAM_CHAT_ID }}
+      CRISIS_ALERT_TELEGRAM_CHAT_IDS: ${{ vars.CRISIS_ALERT_TELEGRAM_CHAT_IDS }}
+      CRISIS_ALERT_TELEGRAM_API_BASE_URL: ${{ vars.CRISIS_ALERT_TELEGRAM_API_BASE_URL }}
+      TELEGRAM_TOKEN: ${{ secrets.TELEGRAM_TOKEN }}
+      CRISIS_ALERT_TELEGRAM_BOT_TOKEN: ${{ secrets.CRISIS_ALERT_TELEGRAM_BOT_TOKEN }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Authenticate to Google Cloud
+        uses: google-github-actions/auth@v3
+        with:
+          workload_identity_provider: ${{ env.GCP_WORKLOAD_IDENTITY_PROVIDER }}
+          service_account: ${{ env.GCP_WORKLOAD_IDENTITY_SERVICE_ACCOUNT }}
+
+      - name: Set up gcloud
+        uses: google-github-actions/setup-gcloud@v3
+
+      - name: Check recent execution report
+        run: python scripts/execution_report_heartbeat.py

README.md

@@ -391,6 +391,17 @@ The scheduled guard runs every 30 minutes. For a missed-run heartbeat, set
 The default leaves the heartbeat check off to avoid false alerts outside the
 active trading window.
 
+`Execution Report Heartbeat` (`.github/workflows/execution-report-heartbeat.yml`)
+is the stricter completion check. It runs on weekdays after the expected market
+window and verifies that a recent runtime report exists under
+`EXECUTION_REPORT_GCS_URI`. It reads the latest report JSON and alerts if no
+recent report exists or the recent reports have rejected statuses such as
+`error`. The deploy service account needs object read/list access on the report
+bucket.
+For slot deployments, `CLOUD_RUN_SERVICE_TARGETS_JSON` is used to require a
+recent acceptable report for each configured service; set
+`RUNTIME_HEARTBEAT_REQUIRED_SERVICES` when only a subset should be monitored.
+
 ### Deployment unit and naming
 
 - `QuantPlatformKit` is only a shared dependency; Cloud Run now deploys `InteractiveBrokersPlatform`.
@@ -669,6 +680,14 @@ OIDC/IAM/audience 配错、Cloud Run 返回 4xx/5xx、或容器在 app-level Tel
 `RUNTIME_GUARD_REQUIRE_SUCCESS=true`，并把 `RUNTIME_GUARD_LOOKBACK_MINUTES` 设成覆盖预期
 Scheduler 运行时间的窗口。默认不强制心跳，避免非交易窗口误报。
 
+更严格的完成检查是 `Execution Report Heartbeat`
+（`.github/workflows/execution-report-heartbeat.yml`）。它会在工作日预期市场窗口后检查
+`EXECUTION_REPORT_GCS_URI` 下最近的 runtime report JSON，读取 `status/stage/errors`，
+如果没有近期 report 或 report 状态为 `error` 等失败状态就发 Telegram。GitHub deploy
+service account 需要对 report bucket 有对象读取/列举权限。
+slot 部署会从 `CLOUD_RUN_SERVICE_TARGETS_JSON` 解析每个 service，并要求每个 service 都有近期
+可接受 report；如果只想监控部分服务，设置 `RUNTIME_HEARTBEAT_REQUIRED_SERVICES`。
+
 ### 部署单元和命名建议
 
 - `QuantPlatformKit` 只是共享依赖，不单独部署；Cloud Run 现在部署的是 `InteractiveBrokersPlatform`。