diff --git a/.env.example b/.env.example index a626f4b..95e5a1b 100644 --- a/.env.example +++ b/.env.example @@ -70,7 +70,6 @@ CHECKPOINT_BACKEND=memory # POLL_ERROR_THRESHOLD=5.0 # Lambda error rate % that triggers an investigation # POLL_REINVESTIGATE_HOURS=1 # don't re-investigate the same alarm within N hours -# Event-driven detection — configure SNS/SQS via Settings → AWS Configuration after login -# SNS_TOPIC_ARN=arn:aws:sns:us-east-1:123456789012:opendevops-alerts +# Event-driven detection — configure SQS via Settings → AWS Configuration after login # SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/opendevops # EVENT_CONSUMER_ENABLED=false diff --git a/README.md b/README.md index 40e1039..0f32e79 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,8 @@ and gives actionable mitigation plans — without the AWS DevOps Agent price tag - **Streaming responses** — FastAPI SSE endpoint streams agent tokens in real time as the LLM reasons; tool calls appear as they complete - **Event-driven incident detection** — EventBridge → SQS → long-poll consumer; 9 EventBridge rules cover CloudWatch alarms, ECS task failures, Lambda async errors, RDS events, EC2 state changes, CodePipeline failures, and AWS Health events; uses a DLQ plus database-backed incident claims to avoid duplicate investigations; runs alongside the metric poller — see [docs/event_detection.md](docs/event_detection.md) - **Context enrichment** — before the LLM runs, deterministic boto3 calls fetch facts about the affected resource (alarm details, recent logs, function config, etc.) to reduce tool call count and speed up investigations -- **SNS alert delivery** — after each event-driven investigation, findings are published to a configured SNS topic as structured JSON; fires alongside Slack and Telegram so all channels receive results simultaneously -- **Monitoring dashboard** — live incident feed showing all event-driven investigations: confidence level (or FAILED badge), affected service, root cause summary, SNS status; each alert links back to its original investigation session via **View investigation** so you can follow up without losing context — see [docs/monitoring.md](docs/monitoring.md) -- **AWS Configuration settings tab** — admin-only editable tab in Settings for SNS Topic ARN, SQS Queue URL, and AWS Region; shared org-wide via database-backed app config; includes an inline IAM permission checker per service +- **Monitoring dashboard** — live incident feed showing all event-driven investigations: confidence level (or FAILED badge), affected service, root cause summary; each alert links back to its original investigation session via **View investigation** so you can follow up without losing context; real-time SSE push keeps the page live without polling — see [docs/monitoring.md](docs/monitoring.md) +- **AWS Configuration settings tab** — admin-only editable tab in Settings for SQS Queue URL and AWS Region; shared org-wide via database-backed app config; includes an inline IAM permission checker per service - **Web UI** — React + Vite SPA served by FastAPI: - **Chat page** — streaming responses, collapsible tool call inspector, cost/latency card, stop button; supports `?prompt=` deeplink for pre-seeded investigations from the Monitoring dashboard - **Session history sidebar** — lists all past conversations; click any to resume with full tool call inspector and cost card restored; new chat and delete (soft) buttons @@ -251,16 +250,14 @@ docs/ # Feature reference — auth, schema, skills, databases, - [x] **Fun streaming labels** — contextual loading copy ("Digging through CloudTrail…", "Lemonizing metrics…", "Cooking up a root cause…") - [x] **Slack & Telegram notifications** — reactive: posts after every investigation to Slack (Block Kit) and/or Telegram (HTML bot message); proactive: background poller checks CloudWatch alarms and Lambda error rates, auto-investigates, and delivers to both channels; set `SLACK_WEBHOOK_URL` and/or `TELEGRAM_BOT_TOKEN`+`TELEGRAM_CHAT_ID` in `.env`; see [docs/telegram.md](docs/telegram.md) - [x] **Event-driven incident detection** — EventBridge → SQS → long-poll consumer; 9 EventBridge rules covering CloudWatch alarms, ECS, Lambda, RDS, EC2, CodePipeline, and AWS Health; runs in parallel with the metric poller; see [docs/event_detection.md](docs/event_detection.md) -- [x] **SNS alert delivery** — findings published to SNS after each event-driven investigation; fires alongside Slack; structured JSON payload for easy downstream processing - [x] **Context enrichment** — deterministic boto3 calls per event type before LLM runs; reduces tool call count by front-loading relevant resource facts -- [x] **Monitoring dashboard** — live incident feed, per-service health summary, alert detail page; "View investigation" opens the original agent session for follow-up; failed investigations flagged separately; see [docs/monitoring.md](docs/monitoring.md) -- [x] **AWS Configuration settings tab** — admin-only editable tab for SNS/SQS/region config; shared org-wide via database-backed app config; inline IAM permission checker +- [x] **Monitoring dashboard** — live incident feed with real-time SSE push, per-service health summary (DB-backed, survives restarts), alert detail page; "View investigation" opens the original agent session for follow-up; failed investigations flagged separately; see [docs/monitoring.md](docs/monitoring.md) +- [x] **AWS Configuration settings tab** — admin-only editable tab for SQS/region config; shared org-wide via database-backed app config; inline IAM permission checker ### Later - [ ] **Observability** — OpenTelemetry traces for agent steps, tool call latency, LLM token usage - [ ] **Follow-up question suggestions** — after each investigation completes, generate 3 suggested follow-up questions in the background and surface them in the UI as clickable chips - [ ] **Session / user feedback loop** — thumbs up/down on investigations, feed signals back to the agent and to an internal quality dashboard -- [ ] **Telegram integration** — bot that accepts `/investigate` commands and streams findings back to a chat or group - [ ] **Knowledge base** — attach internal runbooks, post-mortems, and architecture docs so the agent grounds answers in org-specific context - [ ] **Multi-account AWS** — support multiple AWS profiles per org via `aws_profiles` table (schema already in place) - [ ] **Multi-cloud support** — extend tooling to GCP (Cloud Monitoring, Cloud Logging, GKE) and Azure (Monitor, Log Analytics, AKS); unified incident investigation across providers diff --git a/docs/event_detection.md b/docs/event_detection.md index 26e20f2..6df6605 100644 --- a/docs/event_detection.md +++ b/docs/event_detection.md @@ -19,9 +19,9 @@ AWS Service (ECS, Lambda, RDS, EC2…) → SQS queue (OpenDevOps, with DLQ) → SQS long-poll consumer (event_consumer_loop) → context_collectors (deterministic boto3 enrichment) - → agent.ainvoke (LLM investigation) - → monitor_store.add_alert (persist result) - → SNS publish + Slack notify + → investigation_runner.run_investigation (LLM investigation) + → monitor_store.add_alert (persist result + SSE push) + → Slack / Telegram notify ``` --- @@ -30,14 +30,13 @@ AWS Service (ECS, Lambda, RDS, EC2…) ### Option A — Init wizard (recommended) -Go to **Settings → AWS Configuration** after first login. Enter your AWS Region and optional SNS Topic ARN, save, then click **Run checks** to verify IAM permissions. Click **Create Infrastructure** to create the SQS queue, EventBridge rules, and aggregate Lambda CloudWatch alarm. +Go to **Settings → AWS Configuration** after first login. Enter your AWS Region, save, then click **Run checks** to verify IAM permissions. Click **Create Infrastructure** to create the SQS queue, EventBridge rules, and aggregate Lambda CloudWatch alarm. ### Option B — Manual `.env` ```bash -SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/opendevops-events +SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/opendevops-agent-events EVENT_CONSUMER_ENABLED=true # or leave unset — consumer also starts if SQS_QUEUE_URL is set -SNS_TOPIC_ARN=arn:aws:sns:us-east-1:123456789012:opendevops-alerts ``` The consumer starts automatically on server startup if either `EVENT_CONSUMER_ENABLED=true`, `SQS_QUEUE_URL` is set, or the init wizard has enabled event infrastructure in persistent app config. @@ -86,9 +85,9 @@ Context is appended to the investigation prompt, capped at 3 000 chars to stay w 2. Filters noise: only processes events where `_is_real_failure()` returns true (e.g. skips EC2 `running` state changes, healthy RDS events) 3. Deduplicates: builds a canonical incident key and atomically claims it in the database before the agent runs 4. Calls `collect_context(event)` to enrich the prompt -5. Generates a `session_id` and runs a full agent investigation via `agent.ainvoke`, persisting the session and messages to the database (`source = 'event'`) -6. Persists the result to the `alerts` table (linked to the session via `session_id`) via `monitor_store.add_alert()` -7. Delivers findings via SNS and Slack (if configured); failed investigations are flagged with status `'failed'` but are still persisted and notified +5. Generates a `session_id` and runs a full agent investigation via `investigation_runner.run_investigation`, persisting the session and messages to the database (`source = 'event'`) +6. Persists the result to the `alerts` table (linked to the session via `session_id`) via `monitor_store.add_alert()`, which also pushes the alert to all active SSE subscribers +7. Delivers findings via Slack and Telegram (if configured); failed investigations are flagged with status `'failed'` but are still persisted and notified. If the LLM completes the loop without calling `submit_investigation` (e.g. a weak model), a synthetic `_status: failed` result is produced so the alert is always persisted. 8. Deletes the SQS message only after successful processing, an intentional ignore, or a duplicate claim. Processing failures are left on the queue for retry and eventual DLQ redrive. The consumer is started as an `asyncio.Task` in the FastAPI lifespan and shut down cleanly on server stop. @@ -99,37 +98,12 @@ Deduplication uses canonical incident keys rather than hashes of whole event pay --- -## SNS Alert Delivery - -After each event-driven investigation, the result is published to the configured SNS topic (`SNS_TOPIC_ARN`). The message is a JSON payload: - -```json -{ - "service": "ECS / my-api", - "confidence": "HIGH", - "error": "Container exited with code 137 (OOM killed)", - "resolution": "Increase task memory from 512 MB to 1024 MB..." -} -``` - -SNS subscribers (email, Lambda, SQS fan-out, etc.) receive this payload. Configure subscribers in your AWS console. - -Slack notifications fire alongside SNS when `SLACK_WEBHOOK_URL` is set — both channels always receive the same result. - ---- - ## Permission Requirements -The IAM role/user used by the agent needs these additional permissions for event-driven detection: +The IAM role/user needs the permissions described in [iam_setup.md](iam_setup.md). For event-driven detection specifically: -``` -sns:Publish, sns:GetTopicAttributes -sqs:CreateQueue, sqs:SetQueueAttributes, sqs:GetQueueAttributes -sqs:ReceiveMessage, sqs:DeleteMessage, sqs:DeleteQueue, sqs:ListQueues -events:PutRule, events:PutTargets, events:ListRules -events:RemoveTargets, events:DeleteRule -cloudwatch:PutMetricAlarm, cloudwatch:DeleteAlarms, cloudwatch:DescribeAlarms -``` +- **Policy 1 (Operational)** — `SQSConsume` statement provides `ReceiveMessage`, `DeleteMessage`, `ChangeMessageVisibility` on `opendevops-agent-events` +- **Policy 2 (Setup)** — `SQSSetup`, `EventBridgeSetup`, and `CloudWatchAlarmSetup` statements with write actions scoped to `opendevops-*` resources Use **Settings → AWS Configuration → Run checks** to validate all permissions before going live. @@ -158,12 +132,12 @@ For SQLite, schema migrations are applied automatically on first start. | File | Purpose | |---|---| | `src/agent/event_consumer.py` | SQS long-poll loop, filtering, prompt building, delivery | +| `src/agent/investigation_runner.py` | Shared investigation runner (streaming loop, failed-result fallback) | | `src/agent/event_infra.py` | Create/teardown SQS queue + EventBridge rules | | `src/agent/context_collectors.py` | Per-event-type boto3 context enrichment | -| `src/agent/monitor_store.py` | In-process service status + DB-backed alerts, notifications, and incident claims | +| `src/agent/monitor_store.py` | In-process service status + DB-backed alerts, notifications, and incident claims; SSE subscriber broadcast | | `src/agent/init_store.py` | Persist init config to DB-backed `app_config` | | `src/agent/permission_checker.py` | IAM permission validation | -| `src/tools/sns.py` | SNS publish wrapper | | `src/api/routers/init.py` | Setup wizard API endpoints | | `migrations/005_alerts.sql` | PostgreSQL alerts table + indexes | | `migrations/012_incident_claims.sql` | PostgreSQL incident claim table for atomic deduplication | diff --git a/docs/iam_setup.md b/docs/iam_setup.md index 4ee5a57..8f7da2d 100644 --- a/docs/iam_setup.md +++ b/docs/iam_setup.md @@ -21,34 +21,46 @@ For a typical self-hosted deployment, create an IAM user and paste the keys into 1. Open **IAM → Users → Create user** in the AWS console 2. Username: `opendevops-agent` (or any name you prefer) 3. Select **"Attach policies directly"** -4. Create and attach the inline policy below (see [Step 2](#step-2--iam-policy)) +4. Create and attach **both** customer-managed policies from [Step 2](#step-2--iam-policies) below 5. Open the user → **Security credentials → Create access key** 6. Choose **"Other"** as the use case 7. Copy the **Access key ID** and **Secret access key** — you won't see the secret again +If you only want the chat interface and manual investigations (no event-driven monitoring), attach only **Policy 1** and skip Policy 2. + --- -## Step 2 — IAM policy +## Step 2 — IAM policies + +Two policies keep the least-privilege boundary clean: + +- **Policy 1 (Operational)** — read access across all resources, plus runtime SQS queue polling on the specific event queue. Required for all OpenDevOps features. +- **Policy 2 (Setup)** — write actions scoped to `opendevops-*` resources only. Required only if you use the Settings → AWS Configuration setup wizard to create SQS/EventBridge/CloudWatch infrastructure. -Create a **customer-managed policy** named `OpenDevOpsAgentPolicy` with this JSON: +### Policy 1 — OpenDevOpsAgentOperational ```json { "Version": "2012-10-17", "Statement": [ { - "Sid": "ReadOnlyInvestigation", + "Sid": "ReadAll", "Effect": "Allow", "Action": [ "cloudwatch:DescribeAlarms", "cloudwatch:DescribeAlarmHistory", "cloudwatch:GetMetricStatistics", "cloudwatch:GetMetricData", + "cloudwatch:ListMetrics", "logs:DescribeLogGroups", + "logs:DescribeLogStreams", "logs:FilterLogEvents", + "logs:GetLogEvents", "logs:StartQuery", + "logs:StopQuery", "logs:GetQueryResults", "cloudtrail:LookupEvents", + "cloudtrail:GetTrailStatus", "ecs:ListClusters", "ecs:DescribeClusters", "ecs:ListServices", @@ -64,42 +76,66 @@ Create a **customer-managed policy** named `OpenDevOpsAgentPolicy` with this JSO "rds:DescribeEvents", "iam:ListAttachedRolePolicies", "iam:ListRolePolicies", - "sts:GetCallerIdentity" + "sts:GetCallerIdentity", + "sqs:ListQueues", + "sqs:GetQueueUrl", + "sqs:GetQueueAttributes", + "events:ListRules", + "events:DescribeRule", + "events:ListTargetsByRule" ], "Resource": "*" }, { - "Sid": "EventMonitoringSetup", + "Sid": "SQSConsume", + "Effect": "Allow", + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:ChangeMessageVisibility" + ], + "Resource": "arn:aws:sqs:*:*:opendevops-agent-events" + } + ] +} +``` + +### Policy 2 — OpenDevOpsAgentSetup + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "SQSSetup", "Effect": "Allow", "Action": [ "sqs:CreateQueue", "sqs:DeleteQueue", - "sqs:GetQueueUrl", - "sqs:GetQueueAttributes", "sqs:SetQueueAttributes", - "sqs:ListQueues", - "sqs:ReceiveMessage", - "sqs:DeleteMessage", - "sqs:ChangeMessageVisibility", - "sqs:SendMessage", + "sqs:SendMessage" + ], + "Resource": "arn:aws:sqs:*:*:opendevops-*" + }, + { + "Sid": "EventBridgeSetup", + "Effect": "Allow", + "Action": [ "events:PutRule", "events:PutTargets", - "events:ListRules", "events:RemoveTargets", - "events:DeleteRule", - "cloudwatch:PutMetricAlarm", - "cloudwatch:DeleteAlarms" + "events:DeleteRule" ], - "Resource": "*" + "Resource": "arn:aws:events:*:*:rule/opendevops-*" }, { - "Sid": "SNSNotifications", + "Sid": "CloudWatchAlarmSetup", "Effect": "Allow", "Action": [ - "sns:Publish", - "sns:GetTopicAttributes" + "cloudwatch:PutMetricAlarm", + "cloudwatch:DeleteAlarms" ], - "Resource": "*" + "Resource": "arn:aws:cloudwatch:*:*:alarm:opendevops-*" } ] } @@ -107,15 +143,19 @@ Create a **customer-managed policy** named `OpenDevOpsAgentPolicy` with this JSO ### What each block does -| Block | Purpose | -|---|---| -| `ReadOnlyInvestigation` | Read-only access to CloudWatch, CloudTrail, ECS, Lambda, EC2, RDS, IAM — used by the investigation agent tools | -| `EventMonitoringSetup` | Create/manage the SQS queue and EventBridge rules during the setup wizard; poll SQS at runtime for incoming events | -| `SNSNotifications` | Optional — publish alert summaries to an SNS topic after investigations; check topic attributes during permission check. Remove this block if you're not using SNS. | +| Block | Policy | Purpose | +|---|---|---| +| `ReadAll` | Operational | Read access across CloudWatch, logs, CloudTrail, ECS, Lambda, EC2, RDS, IAM, STS, SQS, EventBridge — used by all investigation tools and bash-based AWS CLI calls | +| `SQSConsume` | Operational | Polls `opendevops-agent-events` at runtime for incoming EventBridge events | +| `SQSSetup` | Setup | Creates and tears down the SQS queue and DLQ; sends test events via the pipeline test script | +| `EventBridgeSetup` | Setup | Creates and tears down the 9 EventBridge rules that forward AWS events to SQS | +| `CloudWatchAlarmSetup` | Setup | Creates and tears down the aggregate `opendevops-lambda-errors-aggregate` CloudWatch alarm | + +**Why reads use `*`:** The agent can issue arbitrary AWS CLI commands via its bash tool. Scoping reads to specific resource ARNs would silently break any investigation that touches a resource not on the allowlist. Write actions are scoped to `opendevops-*` because the app only creates infrastructure under that prefix and nothing else. ### Minimal policy (investigation only, no event monitoring) -If you only want the chat interface and manual investigations — no automatic event detection — you can attach just the `ReadOnlyInvestigation` statement and skip the rest. +If you only want the chat interface and manual investigations — no automatic event detection — attach only **Policy 1** (`ReadAll` + `SQSConsume`). The `SQSConsume` statement is harmless if you never create the queue. --- @@ -156,7 +196,6 @@ After starting the app, the setup wizard's **Step 3 (Permission Check)** calls e | RDS | No | | EC2 | No | | IAM / STS | No | -| SNS | No | | CloudTrail | No | Required services must pass for event monitoring to work. Optional services are used when available — missing permissions will simply return empty results for that service. @@ -165,4 +204,4 @@ Required services must pass for event monitoring to work. Optional services are ## Using an IAM role instead of a user -If running on EC2, ECS, or Lambda, attach the same policy to the instance/task role instead. Remove `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_PROFILE` from `.env` — the SDK picks up the role automatically via the instance metadata service. +If running on EC2, ECS, or Lambda, attach the same policies to the instance/task role instead. Remove `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_PROFILE` from `.env` — the SDK picks up the role automatically via the instance metadata service. diff --git a/docs/monitoring.md b/docs/monitoring.md index 89daf8a..b82bf75 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -90,15 +90,14 @@ AWS service emits an event (ECS task stopped, Lambda errors, RDS failover…) → collect_context() — deterministic boto3 pre-fetch (logs, metrics, config) → agent.ainvoke() — full ReAct investigation → add_alert() — persists to alerts table → /monitoring - → SNS publish — fan-out to email, Lambda, etc. - → notify_slack() — posts to Slack + → notify_slack() — posts to Slack (+ Telegram if configured) ``` ### When to use - You want broad coverage beyond Lambda (ECS, RDS, EC2, GuardDuty, AWS Health) - You already have CloudWatch alarms configured on your resources -- You want SNS fan-out for email/PagerDuty/custom integrations +- You want near-real-time detection instead of polling-interval latency ### Why 2–5 minutes for alarm-based events? @@ -158,22 +157,6 @@ Before the LLM runs, `collect_context()` makes deterministic boto3 calls to pre- --- -## SQS vs SNS — What Each Does - -These are often confused. They serve different roles: - -| | SQS | SNS | -|---|---|---| -| **Direction** | Input to agent | Output from agent | -| **Role** | Receives EventBridge events for the agent to process | Fan-out investigated results to subscribers | -| **Who writes to it** | AWS EventBridge rules | Agent, after investigation completes | -| **Who reads from it** | `event_consumer_loop()` | Email, Lambda, Telegram, custom HTTP endpoints | -| **Required** | Yes, for event-driven detection | No — optional output channel | - -SQS is the inbox. SNS is the outbox. You need SQS for event-driven detection; SNS is optional and only relevant if you want to route investigation results to external systems beyond Slack. - ---- - ## Choosing Between the Two | Scenario | Recommendation | @@ -183,7 +166,6 @@ SQS is the inbox. SNS is the outbox. You need SQS for event-driven detection; SN | ECS, RDS, EC2, GuardDuty coverage | Event-driven required | | Already have CloudWatch alarms | Event-driven — reuses your existing alarms | | Want fastest Lambda detection | Both — polling at 1 min catches errors before the alarm trips | -| Want email/PagerDuty alerts | Event-driven + SNS | --- @@ -215,7 +197,6 @@ Each alert card shows: - **Service** — the affected AWS service and resource name - **Error** — the root cause summary from the agent - **Time** — when the event was detected -- **SNS badge** — shown if the finding was published to SNS Alerts are sorted by most recent first. Click any card to open the Alert Detail page. @@ -225,7 +206,7 @@ Shows the full investigation result: - Root cause in a highlighted panel - Resolution steps produced by the agent -- Timestamp and SNS notification status +- Timestamp and detection source **Investigate / View investigation button** — behaviour depends on whether the investigation session was persisted: @@ -240,7 +221,7 @@ If you open a "View investigation" chat and send at least one follow-up message, ### Service Health Panel -The top of the page shows tracked services and their last-known status (`healthy`, `error`, `unknown`). Updated in-memory each time the event consumer processes an event for a service. +The top of the page shows tracked services and their last-known status (`healthy`, `error`, `unknown`). Updated in-memory each time the event consumer processes an event for a service. On app restart, the panel is rebuilt from the most recent alert per service in the database — so status survives restarts without needing to reprocess events. ### Send Test Event (admin only) @@ -254,8 +235,11 @@ The **Send test event** button in the Monitoring page header pushes a synthetic GET /api/monitoring/alerts?limit=50 → list recent alerts GET /api/monitoring/alerts/{id} → single alert detail GET /api/monitoring/services → service health summary +GET /api/monitoring/stream → SSE stream of new alerts (real-time push) ``` +The SSE stream (`/api/monitoring/stream`) sends a `connected` event on open, an `alert` event for each new alert as it is added, and a `heartbeat` every 20 seconds to keep the connection alive. The frontend subscribes on page load and falls back to 30-second polling if the connection drops. + --- ## Source Files @@ -264,9 +248,10 @@ GET /api/monitoring/services → service health summary |---|---| | `frontend/src/pages/MonitoringPage.tsx` | Live incident feed UI | | `frontend/src/pages/AlertDetailPage.tsx` | Alert detail + investigate deeplink | -| `src/api/routers/monitoring.py` | REST endpoints | -| `src/agent/monitor_store.py` | In-process service status + DB-backed alert, notification, and incident claim persistence | -| `src/agent/poller.py` | Proactive polling loop | +| `src/api/routers/monitoring.py` | REST endpoints + SSE stream | +| `src/agent/monitor_store.py` | In-process service status + DB-backed alert, notification, and incident claim persistence; SSE subscriber broadcast | +| `src/agent/investigation_runner.py` | Shared investigation runner used by both poller and event consumer | +| `src/agent/poller.py` | Proactive polling loop (bounded ThreadPoolExecutor, 4 workers) | | `src/agent/event_consumer.py` | SQS long-poll consumer | | `src/agent/event_infra.py` | Create/teardown SQS + EventBridge + alarm | | `src/agent/context_collectors.py` | Per-event-type boto3 context enrichment | diff --git a/docs/slack_and_polling.md b/docs/slack_and_polling.md index 26b2683..a7f7410 100644 --- a/docs/slack_and_polling.md +++ b/docs/slack_and_polling.md @@ -78,6 +78,9 @@ Calls `list_lambda_functions` (capped at 20 functions) and checks each one's err ### Dedup The poller uses canonical incident keys and an atomic database claim before the agent runs. If the cooldown (`POLL_REINVESTIGATE_HOURS`) has not elapsed, the alarm or function is skipped. Autonomous polling requires SQLite or PostgreSQL; memory mode does not start the poller. +### Thread isolation +Boto3 calls in the poller run inside a bounded `ThreadPoolExecutor(max_workers=4)` named `poller`. This prevents poller threads from consuming the default executor used by the rest of the app, and caps concurrent boto3 calls to four threads even if multiple alarms fire simultaneously. + --- ## When is `submit_investigation` called? @@ -94,6 +97,8 @@ So the agent calls it exactly like any other tool — it decides when it has eno The function itself just returns `"Investigation result recorded."` — the real value is in the structured args the LLM was forced to populate. +If the LLM completes the ReAct loop without ever calling `submit_investigation` (a failure mode seen with some weaker models), the investigation runner synthesizes a `_status: failed` result so the alert is always persisted to the monitoring page instead of being silently dropped. + --- ## Message Format diff --git a/frontend/src/pages/InitPage.tsx b/frontend/src/pages/InitPage.tsx index cefa2bc..e07c18c 100644 --- a/frontend/src/pages/InitPage.tsx +++ b/frontend/src/pages/InitPage.tsx @@ -57,7 +57,6 @@ export default function InitPage() { // Step 2 const [region, setRegion] = useState('us-east-1'); - const [snsArn, setSnsArn] = useState(''); // Step 3 const [perms, setPerms] = useState>({}); @@ -78,7 +77,6 @@ export default function InitPage() { .then(r => r.json()) .then(d => { setRegion(d.aws_region || 'us-east-1'); - setSnsArn(d.sns_topic_arn || ''); if (d.initialized && (d.has_user || !d.auth_enabled)) { navigate('/', { replace: true }); return; @@ -114,7 +112,7 @@ export default function InitPage() { const r = await apiFetch('/api/init/setup', { method: 'POST', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ aws_region: region, sns_topic_arn: snsArn, sqs_queue_url: '', org_name: orgName.trim() }), + body: JSON.stringify({ aws_region: region, sqs_queue_url: '', org_name: orgName.trim() }), }); if (!r.ok) { setError('Failed to save configuration'); return; } setStep(3); @@ -279,20 +277,6 @@ export default function InitPage() { ))} -
- - setSnsArn(e.target.value)} - placeholder="arn:aws:sns:us-east-1:123456789012:alerts" - className={inp} - /> -

- Leave empty to skip SNS notifications — alerts will still appear in the monitoring page. -

-
{error &&

{error}

}