Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@ CHECKPOINT_BACKEND=memory
# Slack — leave unset to disable notifications
# SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxx/yyy/zzz

# Telegram — leave unset to disable notifications
# TELEGRAM_BOT_TOKEN=123456:ABC-... # from @BotFather
# TELEGRAM_CHAT_ID=-100123456789 # group/channel ID (negative = group)

# Proactive polling — set POLL_INTERVAL_SECONDS>0 to enable
# POLL_INTERVAL_SECONDS=300 # how often to check (seconds); 0 = disabled; 20 for dev testing
# POLL_ERROR_THRESHOLD=5.0 # Lambda error rate % that triggers an investigation
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ and gives actionable mitigation plans — without the AWS DevOps Agent price tag
- **Streaming responses** — FastAPI SSE endpoint streams agent tokens in real time as the LLM reasons; tool calls appear as they complete
- **Event-driven incident detection** — EventBridge → SQS → long-poll consumer; 9 EventBridge rules cover CloudWatch alarms, ECS task failures, Lambda async errors, RDS events, EC2 state changes, CodePipeline failures, and AWS Health events; runs alongside the metric poller — see [docs/event_detection.md](docs/event_detection.md)
- **Context enrichment** — before the LLM runs, deterministic boto3 calls fetch facts about the affected resource (alarm details, recent logs, function config, etc.) to reduce tool call count and speed up investigations
- **SNS alert delivery** — after each event-driven investigation, findings are published to a configured SNS topic as structured JSON; fires alongside Slack so both channels receive results simultaneously
- **SNS alert delivery** — after each event-driven investigation, findings are published to a configured SNS topic as structured JSON; fires alongside Slack and Telegram so all channels receive results simultaneously
- **Monitoring dashboard** — live incident feed showing all event-driven investigations: confidence level (or FAILED badge), affected service, root cause summary, SNS status; each alert links back to its original investigation session via **View investigation** so you can follow up without losing context — see [docs/monitoring.md](docs/monitoring.md)
- **AWS Configuration settings tab** — admin-only editable tab in Settings for SNS Topic ARN, SQS Queue URL, and AWS Region; shared org-wide (server-side `init.json`); includes an inline IAM permission checker per service
- **Web UI** — React + Vite SPA served by FastAPI:
Expand Down Expand Up @@ -208,6 +208,8 @@ docs/ # Feature reference — auth, schema, skills, databases,
| `INVESTIGATION_TIMEOUT` | `120` | Timeout in seconds |
| `TOOL_RESPONSE_MAX_CHARS` | `40000` | Truncate tool responses larger than this before feeding to the LLM; `0` disables |
| `SLACK_WEBHOOK_URL` | none | Slack incoming webhook URL; leave unset to disable notifications |
| `TELEGRAM_BOT_TOKEN` | none | Telegram bot token from @BotFather; leave unset to disable |
| `TELEGRAM_CHAT_ID` | none | Target chat/group/channel ID (negative number for groups) |
| `POLL_INTERVAL_MINUTES` | `0` | Proactive polling interval in minutes; `0` disables the poller |
| `POLL_ERROR_THRESHOLD` | `5.0` | Lambda error rate % that triggers an automatic investigation |
| `POLL_REINVESTIGATE_HOURS` | `1` | Cooldown period — skip re-investigating the same alarm within N hours |
Expand Down Expand Up @@ -247,7 +249,7 @@ docs/ # Feature reference — auth, schema, skills, databases,
- [ ] **Guardrails** — input/output validation, PII scrubbing, query scope enforcement
- [ ] **Multi-model escalation** — route simple queries to cheaper/smaller models, escalate hard investigations to larger ones
- [x] **Fun streaming labels** — contextual loading copy ("Digging through CloudTrail…", "Lemonizing metrics…", "Cooking up a root cause…")
- [x] **Slack notifications** — reactive: posts a color-coded Block Kit message to a Slack webhook after every investigation; proactive: background poller checks CloudWatch alarms and Lambda error rates on a configurable interval, auto-investigates new anomalies, and posts findings without any human trigger; set `SLACK_WEBHOOK_URL` and `POLL_INTERVAL_MINUTES` in `.env` to enable
- [x] **Slack & Telegram notifications** — reactive: posts after every investigation to Slack (Block Kit) and/or Telegram (HTML bot message); proactive: background poller checks CloudWatch alarms and Lambda error rates, auto-investigates, and delivers to both channels; set `SLACK_WEBHOOK_URL` and/or `TELEGRAM_BOT_TOKEN`+`TELEGRAM_CHAT_ID` in `.env`; see [docs/telegram.md](docs/telegram.md)
- [x] **Event-driven incident detection** — EventBridge → SQS → long-poll consumer; 9 EventBridge rules covering CloudWatch alarms, ECS, Lambda, RDS, EC2, CodePipeline, and AWS Health; runs in parallel with the metric poller; see [docs/event_detection.md](docs/event_detection.md)
- [x] **SNS alert delivery** — findings published to SNS after each event-driven investigation; fires alongside Slack; structured JSON payload for easy downstream processing
- [x] **Context enrichment** — deterministic boto3 calls per event type before LLM runs; reduces tool call count by front-loading relevant resource facts
Expand Down
3 changes: 2 additions & 1 deletion docs/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The Monitoring page shows a live feed of every incident detected and investigate

## Detection Modes

OpenDevOps Agent has two complementary detection modes. Both write to the same `/monitoring` page and Slack channel.
OpenDevOps Agent has two complementary detection modes. Both write to the same `/monitoring` page and can deliver to Slack and Telegram.

| | Proactive Polling | Event-Driven (EventBridge → SQS) |
|---|---|---|
Expand All @@ -18,6 +18,7 @@ OpenDevOps Agent has two complementary detection modes. Both write to the same `
| **Event types covered** | CloudWatch alarms in ALARM state + Lambda error rates | CloudWatch alarms, ECS task failures, Lambda async errors, RDS events, EC2 state changes, CodePipeline failures, GuardDuty findings, AWS Health events |
| **Appears in /monitoring** | Yes | Yes |
| **Slack notifications** | Yes (if `SLACK_WEBHOOK_URL` set) | Yes (if `SLACK_WEBHOOK_URL` set) |
| **Telegram notifications** | Yes (if `TELEGRAM_BOT_TOKEN` + `TELEGRAM_CHAT_ID` set) | Yes (if `TELEGRAM_BOT_TOKEN` + `TELEGRAM_CHAT_ID` set) |

**Recommendation:** enable both. Polling works immediately with zero AWS infrastructure and is faster for Lambda error detection. Event-driven catches a wider surface area (ECS, RDS, EC2, GuardDuty, etc.) once infrastructure is created.

Expand Down
150 changes: 150 additions & 0 deletions docs/telegram.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# Telegram Notifications

OpenDevOps Agent can send investigation results to a Telegram chat via a bot. Both the event consumer (EventBridge → SQS) and the proactive poller deliver to Telegram alongside Slack — they are independent and can both be active at the same time.

---

## Quick Setup

### 1. Create a bot

Open Telegram and message **@BotFather**:

```
/newbot
```

Follow the prompts — you'll receive a token like:

```
123456789:AAFxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

### 2. Get your chat ID

Add the bot to the group or channel you want alerts in, then send any message to it and open this URL in your browser (replace the token):

```
https://api.telegram.org/bot<TOKEN>/getUpdates
```

Find `"chat": { "id": -100123456789 }` in the response. **Group/channel IDs are negative numbers.**

For a personal chat: message the bot directly, then check `getUpdates` — your personal chat ID is a positive number.

### 3. Add to `.env`

```bash
TELEGRAM_BOT_TOKEN=123456789:AAFxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TELEGRAM_CHAT_ID=-100123456789
```

Restart the server. OpenDevOps will start delivering to Telegram immediately.

---

## Verify it works

Go to **Settings → Integrations → Telegram** and click **Send test message**. You should see a formatted test investigation appear in your chat within a few seconds.

You can also call the API directly:

```bash
curl -X POST http://localhost:8000/api/integrations/telegram/test \
-H "Authorization: Bearer <your-token>" # omit if auth is disabled
```

---

## Message format

Messages use Telegram's **HTML parse mode**. A successful investigation looks like:

```
🔍 OpenDevOps — Investigation Complete

🔴 HIGH confidence · RESOURCE_LIMIT

Root Cause
Lambda exhausted memory under burst traffic.

Services affected: Lambda, CloudWatch

Evidence
• MemoryUtilization reached 99%
• OOM errors spiked in CloudWatch logs

Mitigation Steps
1. Increase Lambda memory to 1024 MB
2. Add CloudWatch memory alarm at 80%

Session 1a2b3c4d
```

A failed investigation (agent hit the tool call limit or errored) looks like:

```
⚠️ OpenDevOps — Investigation Failed

Service: payment-service
Error: Investigation failed: agent timeout after 120s

Session 1a2b3c4d
```

---

## Delivery tracking

Every Telegram notification is recorded in the `alert_notifications` table:

```sql
SELECT channel, status, sent_at
FROM alert_notifications
WHERE alert_id = '<alert-uuid>'
AND channel = 'telegram';
```

The **Monitoring → alert detail** page shows this under "Notified via" with a green check (delivered) or red X (failed).

---

## Combining Slack and Telegram

Both channels fire independently. Set both `SLACK_WEBHOOK_URL` and `TELEGRAM_BOT_TOKEN` / `TELEGRAM_CHAT_ID` to receive notifications in both places. Each delivery is tracked separately in `alert_notifications`.

---

## Configuration reference

| Variable | Required | Description |
|---|---|---|
| `TELEGRAM_BOT_TOKEN` | Yes | Bot token from @BotFather |
| `TELEGRAM_CHAT_ID` | Yes | Target chat/group/channel ID |

Both must be set. If either is missing, Telegram delivery is silently skipped.

---

## Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| "Not configured" badge in Settings | One or both env vars missing or `(not set)` | Add both vars to `.env` and restart |
| Test message returns 400 | Vars not loaded yet | Verify `.env` has both vars; restart the server |
| Message not received | Wrong chat ID, or bot not in the group | Re-run `getUpdates` to confirm chat ID; make sure the bot is a member of the group/channel |
| `chat not found` in server logs | Chat ID is wrong or bot was never added | Add bot to the group first, then send a message before checking `getUpdates` |
| Notifications show as `failed` in alert detail | Bot API returned non-200 | Check server logs for the exact Telegram error message |

---

## Source files

| File | Purpose |
|---|---|
| `src/integrations/telegram.py` | Bot API client — message builder + HTTP delivery |
| `src/agent/event_consumer.py` | Delivers to Telegram after each event-driven investigation |
| `src/agent/poller.py` | Delivers to Telegram after each polling investigation |
| `src/api/routers/integrations.py` | `POST /api/integrations/telegram/test` endpoint |
| `src/config/appsettings.py` | `telegram_bot_token` + `telegram_chat_id` settings |
| `tests/integration/test_telegram.py` | Unit tests for message rendering and delivery |
95 changes: 95 additions & 0 deletions frontend/src/components/icons/IntegrationIcon.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
interface IconProps {
size?: number;
className?: string;
}

export function SlackIcon({ size = 16, className }: IconProps) {
return (
<svg width={size} height={size} viewBox="0 0 24 24" fill="none" className={className} aria-label="Slack">
{/* top-left: pink */}
<path d="M5.042 15.165a2.528 2.528 0 0 1-2.52 2.523A2.528 2.528 0 0 1 0 15.165a2.527 2.527 0 0 1 2.522-2.52h2.52v2.52z" fill="#E01E5A"/>
<path d="M6.313 15.165a2.527 2.527 0 0 1 2.521-2.52 2.527 2.527 0 0 1 2.521 2.52v6.313A2.528 2.528 0 0 1 8.834 24a2.528 2.528 0 0 1-2.521-2.522v-6.313z" fill="#E01E5A"/>
{/* top-right: green */}
<path d="M8.834 5.042a2.528 2.528 0 0 1-2.521-2.52A2.528 2.528 0 0 1 8.834 0a2.528 2.528 0 0 1 2.521 2.522v2.52H8.834z" fill="#36C5F0"/>
<path d="M8.834 6.313a2.528 2.528 0 0 1 2.521 2.521 2.528 2.528 0 0 1-2.521 2.521H2.522A2.528 2.528 0 0 1 0 8.834a2.528 2.528 0 0 1 2.522-2.521h6.312z" fill="#36C5F0"/>
{/* bottom-right: yellow */}
<path d="M18.956 8.834a2.528 2.528 0 0 1 2.522-2.521A2.528 2.528 0 0 1 24 8.834a2.528 2.528 0 0 1-2.522 2.521h-2.522V8.834z" fill="#2EB67D"/>
<path d="M17.688 8.834a2.528 2.528 0 0 1-2.523 2.521 2.527 2.527 0 0 1-2.52-2.521V2.522A2.527 2.527 0 0 1 15.165 0a2.528 2.528 0 0 1 2.523 2.522v6.312z" fill="#2EB67D"/>
{/* bottom-left: blue */}
<path d="M15.165 18.956a2.528 2.528 0 0 1 2.523 2.522A2.528 2.528 0 0 1 15.165 24a2.527 2.527 0 0 1-2.52-2.522v-2.522h2.52z" fill="#ECB22E"/>
<path d="M15.165 17.688a2.527 2.527 0 0 1-2.52-2.523 2.526 2.526 0 0 1 2.52-2.52h6.313A2.527 2.527 0 0 1 24 15.165a2.528 2.528 0 0 1-2.522 2.523h-6.313z" fill="#ECB22E"/>
</svg>
);
}

export function TelegramIcon({ size = 16, className }: IconProps) {
return (
<svg width={size} height={size} viewBox="0 0 24 24" fill="none" className={className} aria-label="Telegram">
<circle cx="12" cy="12" r="12" fill="#26A5E4"/>
<path d="M5.172 11.688 17.14 7.098c.561-.203 1.051.137.869.988l-2.017 9.504c-.148.672-.543.836-1.1.52l-2.992-2.205-1.445 1.39c-.16.16-.295.295-.605.295l.215-3.047 5.543-5.008c.24-.215-.053-.334-.375-.119L6.844 13.65 3.89 12.762c-.664-.207-.676-.664.28-1.074z" fill="white"/>
</svg>
);
}

export function GitHubIcon({ size = 16, className }: IconProps) {
return (
<svg width={size} height={size} viewBox="0 0 24 24" fill="currentColor" className={className} aria-label="GitHub">
<path d="M12 0C5.374 0 0 5.373 0 12c0 5.302 3.438 9.8 8.207 11.387.599.111.793-.261.793-.577v-2.234c-3.338.726-4.033-1.416-4.033-1.416-.546-1.387-1.333-1.756-1.333-1.756-1.089-.745.083-.729.083-.729 1.205.084 1.839 1.237 1.839 1.237 1.07 1.834 2.807 1.304 3.492.997.107-.775.418-1.305.762-1.604-2.665-.305-5.467-1.334-5.467-5.931 0-1.311.469-2.381 1.236-3.221-.124-.303-.535-1.524.117-3.176 0 0 1.008-.322 3.301 1.23A11.509 11.509 0 0 1 12 5.803c1.02.005 2.047.138 3.006.404 2.291-1.552 3.297-1.23 3.297-1.23.653 1.653.242 2.874.118 3.176.77.84 1.235 1.911 1.235 3.221 0 4.609-2.807 5.624-5.479 5.921.43.372.823 1.102.823 2.222v3.293c0 .319.192.694.801.576C20.566 21.797 24 17.3 24 12c0-6.627-5.373-12-12-12z"/>
</svg>
);
}

export function PagerDutyIcon({ size = 16, className }: IconProps) {
return (
<svg width={size} height={size} viewBox="0 0 24 24" fill="none" className={className} aria-label="PagerDuty">
<rect width="24" height="24" rx="4" fill="#06AC38"/>
<path d="M10.5 16.5H8.25V19.5H10.5V16.5Z" fill="white"/>
<path d="M10.5 4.5H7.5C6.675 4.5 6 5.175 6 6V14.25C6 15.075 6.675 15.75 7.5 15.75H10.5C12.984 15.75 15 13.734 15 11.25V9C15 6.516 12.984 4.5 10.5 4.5ZM10.5 13.5H8.25V6.75H10.5C11.742 6.75 12.75 7.758 12.75 9V11.25C12.75 12.492 11.742 13.5 10.5 13.5Z" fill="white"/>
</svg>
);
}

export function DatadogIcon({ size = 16, className }: IconProps) {
return (
<svg width={size} height={size} viewBox="0 0 24 24" fill="none" className={className} aria-label="Datadog">
<rect width="24" height="24" rx="4" fill="#632CA6"/>
<path d="M13.17 6.38 11.9 5.7l-3.84 2.08v4.1l1.27.69 3.84-2.08V6.38zm-1.27 3.5-1.3-.7V7.5l1.3.7v1.68zm1.27 1.02-1.27.7V9.92l1.27-.7v1.68z" fill="white"/>
<path d="m15.71 9.17-1.27-.7v4.1l-3.84 2.08v1.38l5.11-2.77V9.17z" fill="white"/>
<path d="M8.06 13.27v-1.38L6.79 11.2v4.1l5.11 2.77v-1.38l-3.84-2.08v-1.34z" fill="white"/>
</svg>
);
}

export function SnsIcon({ size = 16, className }: IconProps) {
return (
<svg width={size} height={size} viewBox="0 0 24 24" fill="none" className={className} aria-label="SNS">
<rect width="24" height="24" rx="4" fill="#FF9900"/>
<path d="M12 5a1 1 0 0 1 1 1v.27A5 5 0 0 1 17 11v3l1.707 1.707A1 1 0 0 1 18 17.5H6a1 1 0 0 1-.707-1.707L7 14v-3a5 5 0 0 1 4-4.9V6a1 1 0 0 1 1-1zm0 14a2 2 0 0 1-2-2h4a2 2 0 0 1-2 2z" fill="white"/>
</svg>
);
}

export function EmailIcon({ size = 16, className }: IconProps) {
return (
<svg width={size} height={size} viewBox="0 0 24 24" fill="none" className={className} aria-label="Email">
<rect width="24" height="24" rx="4" fill="#6B7280"/>
<path d="M4 8a1 1 0 0 1 1-1h14a1 1 0 0 1 1 1v8a1 1 0 0 1-1 1H5a1 1 0 0 1-1-1V8zm2.2 0 5.8 4 5.8-4H6.2zm11.8 1.5-5.4 3.73a1 1 0 0 1-1.2 0L6 9.5V16h12V9.5z" fill="white"/>
</svg>
);
}

const ICONS: Record<string, (props: IconProps) => JSX.Element> = {
slack: SlackIcon,
telegram: TelegramIcon,
github: GitHubIcon,
pagerduty: PagerDutyIcon,
datadog: DatadogIcon,
sns: SnsIcon,
email: EmailIcon,
};

export function IntegrationIcon({ name, size = 16, className }: IconProps & { name: string }) {
const Icon = ICONS[name.toLowerCase()];
if (!Icon) return null;
return <Icon size={size} className={className} />;
}
12 changes: 2 additions & 10 deletions frontend/src/pages/AlertDetailPage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { ArrowLeft, Clock, MessageSquare, Loader2, CheckCircle2, XCircle, Radio
import { cn } from '../lib/utils';
import { fetchAlert } from '../lib/api';
import type { Alert, AlertNotification } from '../types';
import { IntegrationIcon } from '../components/icons/IntegrationIcon';

const CHANNEL_LABELS: Record<string, string> = {
slack: 'Slack',
Expand All @@ -13,20 +14,11 @@ const CHANNEL_LABELS: Record<string, string> = {
pagerduty: 'PagerDuty',
};

const CHANNEL_ICONS: Record<string, string> = {
slack: '💬',
sns: '📡',
telegram: '✈️',
email: '📧',
pagerduty: '📟',
};

function NotificationRow({ n }: { n: AlertNotification }) {
const label = CHANNEL_LABELS[n.channel] ?? n.channel;
const icon = CHANNEL_ICONS[n.channel] ?? '🔔';
return (
<div className="flex items-center gap-2.5">
<span className="text-sm">{icon}</span>
<IntegrationIcon name={n.channel} size={16} />
<span className="text-[13px] text-gray-700 dark:text-[#CBD5E1] font-medium w-20">{label}</span>
{n.status === 'delivered'
? <CheckCircle2 size={13} className="text-emerald-500" />
Expand Down
Loading
Loading