Add error notification (Slack, Discord)

[Mr-DooSun]

Problem

When errors occur in production, there is no automated notification mechanism to alert the team through common communication channels. Issues may go undetected until users report them.

Proposed Solution

Add an error notification system:

• Webhook integration for Slack and Discord
• Configurable severity thresholds (e.g., only notify on 5xx errors or critical exceptions)
• Rate limiting to prevent notification storms
• Integration with the existing exception middleware for automatic triggering
• Channel routing (e.g., critical → #alerts, warnings → #monitoring)

Area

Core (_core)

Alternatives Considered

• External APM tools only (Sentry, Datadog): adds cost and external dependency — basic webhook notifications are complementary and free
• Email notifications: slower feedback loop than instant messaging

[vraj2131]

Hi, Can I take this one?

[Mr-DooSun]

@vraj2131 Sure, thanks for your interest! Please go ahead.

Before implementing, please share a brief approach first. The expected scope is:

• Slack/Discord webhook notification support
• configurable severity threshold
• basic rate limiting
• integration with the existing exception middleware
• tests for notification triggering and suppression behavior

[vraj2131]

Thanks! Here is the plan before I start:

• I'll add it as an optional piece of core infra (Slack and Discord webhook adapters, plus a no op fallback when it's disabled), following the same Protocol and Selector pattern the project already uses for things like LLM and Storage. That way it's toggleable through env vars and costs nothing when it's off.

• Webhook calls will go through the existing HttpClient, so there's no new HTTP dependency.

• Severity threshold and rate limiting will just be a couple of new settings plus a small in memory cooldown. This is not the slowapi rate limiting item from the roadmap, since that's for HTTP endpoints. This one just stops repeat error spam.

• It will hook into the existing exception handlers (generic_exception_handler and custom_exception_handler). If a webhook call fails, it will just get logged and never turn into a second error for the user.

• Tests will cover threshold gating, suppression on repeated errors, and payload building for both Slack and Discord.

• This shouldn't need a new domain, a new DB table, or an ADR, so I'll keep it as small as possible.

Let me know if that lines up with what you had in mind, then I'll get started.

[Mr-DooSun]

This plan lines up well — go ahead and get started. 🙌

The overall shape is exactly right: optional core infra behind the Protocol + Selector pattern with a no-op fallback (same as the LLM/Storage infra, ADR 042), reusing HttpClient, and hooking into the existing generic_exception_handler / custom_exception_handler. Keeping it out of a new domain/table/ADR is the right call.

Two small notes:

1. Channel routing is out of scope for this PR — don't worry about the critical → #alerts, warnings → #monitoring routing from the issue body. I've split it into a follow-up (Add channel routing for error notifications (Slack, Discord) #286) so you can keep this one focused. Feel free to Closes #17 in your PR.

2. Make the webhook send non-blocking. Since it fires from inside the exception handler (in the request/response path), please don't await the webhook call directly — a slow Slack/Discord endpoint would add latency to every 5xx response. Fire-and-forget (asyncio.create_task), a BackgroundTask, or offloading to a Taskiq task all work. (Optional: a one-line note that the in-memory cooldown is per-process, since it won't dedupe across multiple workers.)

Everything else — the settings-based threshold, the in-memory cooldown distinct from slowapi, and the test coverage — sounds good. Looking forward to the PR!

[vraj2131]

Thank you for your input, will start working on it