Bug: duplicate investigations when poller and event consumer both trigger for the same alarm

[AhmadHammad21]

Summary

When a Lambda function errors and triggers the aggregate CloudWatch alarm, both the event consumer and the proactive poller independently launch a full agent investigation for the same root cause. This produces two sessions, two Slack notifications, and two alert rows in the DB for a single incident.

Observed in logs:

agent.event_consumer - Processing event: aws.cloudwatch / CloudWatch Alarm State Change
agent.poller         - Poller: Lambda opendevops-test-failure error rate exceeds threshold

Two different dedup_key values were stored:

• lambda_errors:opendevops-test-failure — from the poller
• aws.cloudwatch:ae3abf074120 — from the event consumer

---

How dedup keys currently work

Event consumer (event_consumer.py)

The event consumer computes an MD5 fingerprint of the event's stable fields:

fingerprint = {
    "source": source,                          # e.g. "aws.cloudwatch"
    "detail-type": event.get("detail-type"),   # e.g. "CloudWatch Alarm State Change"
    "detail": _strip_volatile(event.get("detail", {})),
}
digest = hashlib.md5(json.dumps(fingerprint, sort_keys=True).encode()).hexdigest()[:12]
return f"{source}:{digest}"   # → "aws.cloudwatch:ae3abf074120"

_strip_volatile removes time-varying keys (time, timestamp, eventTime, requestId, startTime, endTime, updatedAt, stateTransitionedAt, lastUpdatedTime).

Poller (poller.py)

The poller constructs semantic keys directly:

• alarm:{alarmName} for CloudWatch alarms in ALARM state
• lambda_errors:{functionName} for Lambda error rate checks

It has an explicit cross-check to avoid re-investigating when the event consumer already handled the aggregate alarm:

from agent.event_infra import ALARM_NAME
if await is_recent_alert(f"alarm:{ALARM_NAME}"):
    logger.debug("Poller: skipping Lambda {} — aggregate alarm already handled", name)

Why the cross-check never fires

is_recent_alert does an exact key match in the alerts table. The event consumer stored the key as aws.cloudwatch:ae3abf074120. The poller checks for alarm:opendevops-aggregate-lambda-errors. These strings never match — the cross-check is effectively dead code.

---

Proposed fix

Change _event_dedup_key in event_consumer.py to use alarm:{alarmName} for CloudWatch alarm state change events instead of the MD5:

if source == "aws.cloudwatch" and "alarmName" in detail:
    return f"alarm:{detail['alarmName']}"

This makes the event consumer and poller share the same key namespace for alarms, so the poller's existing is_recent_alert(f"alarm:{ALARM_NAME}") cross-check works as intended.

---

User's concern with the proposed fix

"This leads to: if two different errors happen in Lambda, only one will be investigated, right?"

The concern: if the aggregate alarm fires twice within the 3-minute dedup window (e.g., function A errors at 9:30 → investigated, function B errors at 9:31 → key alarm:opendevops-aggregate-lambda-errors is still recent → skipped).

Counter-argument: CloudWatch alarms only fire on state transitions (OK → ALARM). If the alarm is already in ALARM state when function B errors, no second EventBridge event fires. The investigation triggered at 9:30 runs against all current Lambda errors and should detect both function A and B.

The scenario where two separate investigations would genuinely be warranted is ALARM → OK → ALARM within 3 minutes — errors resolved and reappeared. This is unusual in practice.

The user remained suspicious of this reasoning and chose not to apply the fix. The concern is that the trade-off analysis relies on the aggregate alarm behaving predictably, and any edge case (e.g., partial OK states, per-function dimension alarms) could cause a legitimate incident to be silently skipped.

---

Why using MD5 in the poller isn't a solution

The two systems start from different data sources:

• Event consumer hashes the raw EventBridge event payload (structured JSON delivered via SQS)
• Poller fetches data from the CloudWatch DescribeAlarms API (different field names, different structure)

There is no way to make both sides produce the same MD5 without them sharing the exact same payload, which they never do.

---

Next step suggestions

1. Store a secondary normalized key alongside the MD5. Add a normalized_key TEXT column to alerts. The event consumer writes both its MD5 dedup_key and a human-readable normalized_key (e.g., alarm:{alarmName}). is_recent_alert checks both columns. This removes the trade-off entirely — MD5 dedup stays precise, cross-system dedup works via the normalized key.

2. Widen is_recent_alert to accept multiple keys. Pass [f"alarm:{ALARM_NAME}", f"lambda_errors:{name}"] and match on any. Requires a small schema/query change but no new columns.

3. Consolidate into event-driven only. Disable the poller's alarm check entirely when event infra is enabled (SQS + EventBridge already covers alarm state changes). The poller would only handle Lambda error rate polling, which the event consumer doesn't cover. This avoids the namespace problem by removing the overlap.

4. Use alarm:{alarmName} with a longer dedup window. Apply the proposed fix but extend the dedup window for alarm events (e.g., 10 minutes instead of 3), reducing the risk of the ALARM→OK→ALARM edge case being swallowed.

---

Relevant files

File	Role
src/agent/event_consumer.py	_event_dedup_key(), _process_event()
src/agent/poller.py	_check_alarms(), _check_lambda_errors(), cross-check logic
src/agent/monitor_store.py	is_recent_alert(), add_alert()
src/agent/db/postgres.py	is_recent_alert SQL query
migrations/007_alerts_dedup_key.sql	dedup_key column on alerts