Skip to content

Alert recovery notifications (resolved alerts) #551

Description

@adminsyspro

Context

From discussion #549 (@DRO34): when an alert fires (e.g. high memory usage), there is no follow-up notification when the condition clears. Users have no way to know the alert is resolved without manually checking.

Feature

Add recovery/resolved notifications to the alerting system:

  • Track alert state transitions: firing -> resolved
  • When a metric drops back below the threshold (with a configurable hysteresis/cooldown to avoid flapping), send a recovery notification through the same channels as the original alert (email, webhook, etc.)
  • Include in the recovery notification: which alert resolved, when it fired, when it resolved, current metric value

Acceptance criteria

  • Alerts that return to normal trigger a "resolved" notification
  • Cooldown/hysteresis prevents rapid fire/resolve flapping
  • Recovery notifications sent via the same channel(s) as the original alert
  • Alert history shows both firing and resolved events

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions