feat: event monitoring setup wizard, session persistence, and investigation pipeline

.env.example

@@ -59,8 +59,8 @@ CHECKPOINT_BACKEND=memory
 # Slack — leave unset to disable notifications
 # SLACK_WEBHOOK_URL=https://hooks.slack.com/services/xxx/yyy/zzz
 
-# Proactive polling — set POLL_INTERVAL_MINUTES>0 to enable
-# POLL_INTERVAL_MINUTES=5         # how often to check (minutes); 0 = disabled
+# Proactive polling — set POLL_INTERVAL_SECONDS>0 to enable
+# POLL_INTERVAL_SECONDS=300       # how often to check (seconds); 0 = disabled; 20 for dev testing
 # POLL_ERROR_THRESHOLD=5.0        # Lambda error rate % that triggers an investigation
 # POLL_REINVESTIGATE_HOURS=1      # don't re-investigate the same alarm within N hours
 

CLAUDE.md

@@ -99,7 +99,7 @@ Everything below is built and working in the codebase:
 - Starts automatically on app startup if `event_consumer_enabled=True`, `sqs_queue_url` is set, or `init.json` marks the wizard as completed.
 
 ### Proactive Polling
-- `polling_loop()` runs every `POLL_INTERVAL_MINUTES` minutes (disabled by default at 0); checks CloudWatch alarms in ALARM state and Lambda error rates above `POLL_ERROR_THRESHOLD`; auto-investigates new anomalies and posts to Slack. In-memory dedup via `_last_investigated` map.
+- `polling_loop()` runs every `POLL_INTERVAL_SECONDS` seconds (disabled by default at 0); checks CloudWatch alarms in ALARM state and Lambda error rates above `POLL_ERROR_THRESHOLD`; auto-investigates new anomalies and posts to Slack. In-memory dedup via `_last_investigated` map.
 
 ### Frontend
 - React 18 + TypeScript + Vite + Tailwind CSS + `@tailwindcss/typography`
@@ -211,7 +211,7 @@ JWT_EXPIRE_MINUTES=1440                # 24 hours
 SLACK_WEBHOOK_URL=                     # leave unset to disable
 
 # Proactive polling
-POLL_INTERVAL_MINUTES=0               # 0 = disabled
+POLL_INTERVAL_SECONDS=0               # 0 = disabled; set to e.g. 300 (5 min) to enable
 POLL_ERROR_THRESHOLD=5.0              # Lambda error rate % to trigger investigation
 POLL_REINVESTIGATE_HOURS=1            # dedup window
 
@@ -254,7 +254,7 @@ EventBridge rules (9 event types)
 ### Startup sequence
 ```
 db.init()  →  init_agent(checkpointer)
-  optional: asyncio.create_task(polling_loop())       if POLL_INTERVAL_MINUTES > 0
+  optional: asyncio.create_task(polling_loop())       if POLL_INTERVAL_SECONDS > 0
   optional: asyncio.create_task(event_consumer_loop()) if SQS configured or init.json present
 ```
 

README.md

@@ -13,7 +13,7 @@ and gives actionable mitigation plans — without the AWS DevOps Agent price tag
 - **Event-driven incident detection** — EventBridge → SQS → long-poll consumer; 9 EventBridge rules cover CloudWatch alarms, ECS task failures, Lambda async errors, RDS events, EC2 state changes, CodePipeline failures, and AWS Health events; runs alongside the metric poller — see [docs/event_detection.md](docs/event_detection.md)
 - **Context enrichment** — before the LLM runs, deterministic boto3 calls fetch facts about the affected resource (alarm details, recent logs, function config, etc.) to reduce tool call count and speed up investigations
 - **SNS alert delivery** — after each event-driven investigation, findings are published to a configured SNS topic as structured JSON; fires alongside Slack so both channels receive results simultaneously
-- **Monitoring dashboard** — live incident feed showing all event-driven investigations: confidence level, affected service, root cause summary, SNS status; each alert has a detail page with an **Investigate** button that opens a pre-seeded chat — see [docs/monitoring.md](docs/monitoring.md)
+- **Monitoring dashboard** — live incident feed showing all event-driven investigations: confidence level (or FAILED badge), affected service, root cause summary, SNS status; each alert links back to its original investigation session via **View investigation** so you can follow up without losing context — see [docs/monitoring.md](docs/monitoring.md)
 - **AWS Configuration settings tab** — admin-only editable tab in Settings for SNS Topic ARN, SQS Queue URL, and AWS Region; shared org-wide (server-side `init.json`); includes an inline IAM permission checker per service
 - **Web UI** — React + Vite SPA served by FastAPI:
   - **Chat page** — streaming responses, collapsible tool call inspector, cost/latency card, stop button; supports `?prompt=` deeplink for pre-seeded investigations from the Monitoring dashboard
@@ -251,7 +251,7 @@ docs/                  # Feature reference — auth, schema, skills, databases,
 - [x] **Event-driven incident detection** — EventBridge → SQS → long-poll consumer; 9 EventBridge rules covering CloudWatch alarms, ECS, Lambda, RDS, EC2, CodePipeline, and AWS Health; runs in parallel with the metric poller; see [docs/event_detection.md](docs/event_detection.md)
 - [x] **SNS alert delivery** — findings published to SNS after each event-driven investigation; fires alongside Slack; structured JSON payload for easy downstream processing
 - [x] **Context enrichment** — deterministic boto3 calls per event type before LLM runs; reduces tool call count by front-loading relevant resource facts
-- [x] **Monitoring dashboard** — live incident feed, per-service health summary, alert detail page, investigate deeplink to pre-seeded chat; see [docs/monitoring.md](docs/monitoring.md)
+- [x] **Monitoring dashboard** — live incident feed, per-service health summary, alert detail page; "View investigation" opens the original agent session for follow-up; failed investigations flagged separately; see [docs/monitoring.md](docs/monitoring.md)
 - [x] **AWS Configuration settings tab** — admin-only editable tab for SNS/SQS/region config; shared org-wide via server-side `init.json`; inline IAM permission checker
 
 ### Later

docs/databases.md

@@ -44,6 +44,10 @@ Uses `aiosqlite` for the app tables and `langgraph-checkpoint-sqlite` for the
 LangGraph checkpointer. Both share the same `.db` file via separate connections
 with WAL mode enabled.
 
+SQLite also stores app-level configuration such as init wizard completion and event
+monitoring infrastructure state in `app_config`. `data/init.json` is only a local
+cache/fallback.
+
 The file and its parent directory are created automatically on first start.
 
 **When to use:** Single-server deployments, personal use, hobbyist setups where
@@ -81,6 +85,8 @@ DATABASE_URL=postgresql://user:password@localhost:5432/opendevops
 Uses `psycopg3` + `AsyncConnectionPool` for the app tables and
 `langgraph-checkpoint-postgres` for the LangGraph checkpointer.
 The checkpointer schema is created automatically via `AsyncPostgresSaver.setup()`.
+Init wizard and event-infrastructure state are stored in the `app_config` table so
+all app instances share the same setup status.
 
 **When to use:** Production deployments, team environments, when you need full
 dashboard analytics, multi-instance horizontal scaling.

docs/event_detection.md

@@ -30,7 +30,7 @@ AWS Service (ECS, Lambda, RDS, EC2…)
 
 ### Option A — Init wizard (recommended)
 
-Go to **Settings → AWS Configuration** after first login. Enter your SNS Topic ARN and AWS Region, save, then click **Run checks** to verify IAM permissions. The SQS queue and EventBridge rules are created automatically the first time the server starts with valid config.
+Go to **Settings → AWS Configuration** after first login. Enter your AWS Region and optional SNS Topic ARN, save, then click **Run checks** to verify IAM permissions. Click **Create Infrastructure** to create the SQS queue, EventBridge rules, and aggregate Lambda CloudWatch alarm.
 
 ### Option B — Manual `.env`
 
@@ -40,7 +40,7 @@ EVENT_CONSUMER_ENABLED=true        # or leave unset — consumer also starts if
 SNS_TOPIC_ARN=arn:aws:sns:us-east-1:123456789012:opendevops-alerts
 ```
 
-The consumer starts automatically on server startup if either `EVENT_CONSUMER_ENABLED=true` or `SQS_QUEUE_URL` is set.
+The consumer starts automatically on server startup if either `EVENT_CONSUMER_ENABLED=true`, `SQS_QUEUE_URL` is set, or the init wizard has enabled event infrastructure in persistent app config.
 
 ---
 
@@ -50,15 +50,15 @@ The consumer starts automatically on server startup if either `EVENT_CONSUMER_EN
 
 | Rule | Events captured |
 |---|---|
-| `opendevops-cloudwatch-alarms` | CloudWatch Alarm state changes |
-| `opendevops-ecs-failures` | ECS task stopped with non-zero exit / OOM |
-| `opendevops-lambda-errors` | Lambda async invocation failures |
-| `opendevops-rds-events` | RDS instance failures and failovers |
+| `opendevops-alarm-state` | CloudWatch Alarm state changes to `ALARM` |
+| `opendevops-lambda-failure` | Lambda async invocation failures |
+| `opendevops-lambda-throttle` | Lambda throttling errors |
+| `opendevops-ecs-task-stopped` | ECS task stopped with non-zero exit / OOM |
 | `opendevops-ec2-state` | EC2 instance state changes |
-| `opendevops-asg-events` | Auto Scaling launch / terminate failures |
-| `opendevops-codepipeline` | CodePipeline stage failures |
-| `opendevops-ssm-compliance` | SSM patch compliance failures |
-| `opendevops-health-events` | AWS Health personal health events |
+| `opendevops-rds-events` | RDS instance failures and failovers |
+| `opendevops-health` | AWS Health personal health events |
+| `opendevops-codedeploy-failure` | CodeDeploy deployment failures |
+| `opendevops-guardduty` | GuardDuty findings |
 
 ---
 
@@ -82,13 +82,14 @@ Context is appended to the investigation prompt, capped at 3 000 chars to stay w
 
 `event_consumer_loop()` in `src/agent/event_consumer.py`:
 
-1. Receives up to 10 messages per poll with a 20-second long-poll wait
+1. Receives up to 5 messages per poll with a 20-second long-poll wait
 2. Filters noise: only processes events where `_is_real_failure()` returns true (e.g. skips EC2 `running` state changes, healthy RDS events)
-3. Calls `collect_context(event)` to enrich the prompt
-4. Runs a full agent investigation via `agent.ainvoke`
-5. Persists the result to the `alerts` table via `monitor_store.add_alert()`
-6. Delivers findings via SNS and Slack (if configured)
-7. Deletes the SQS message
+3. Deduplicates: checks an in-memory set (`_in_progress`) and the database to skip events already being investigated or investigated within the last 3 minutes
+4. Calls `collect_context(event)` to enrich the prompt
+5. Generates a `session_id` and runs a full agent investigation via `agent.ainvoke`, persisting the session and messages to the database (`source = 'event'`)
+6. Persists the result to the `alerts` table (linked to the session via `session_id`) via `monitor_store.add_alert()`
+7. Delivers findings via SNS and Slack (if configured); failed investigations are flagged with status `'failed'` but are still persisted and notified
+8. Deletes the SQS message
 
 The consumer is started as an `asyncio.Task` in the FastAPI lifespan and shut down cleanly on server stop.
 
@@ -120,9 +121,10 @@ The IAM role/user used by the agent needs these additional permissions for event
 ```
 sns:Publish, sns:GetTopicAttributes
 sqs:CreateQueue, sqs:SetQueueAttributes, sqs:GetQueueAttributes
-sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ListQueues
+sqs:ReceiveMessage, sqs:DeleteMessage, sqs:DeleteQueue, sqs:ListQueues
 events:PutRule, events:PutTargets, events:ListRules
 events:RemoveTargets, events:DeleteRule
+cloudwatch:PutMetricAlarm, cloudwatch:DeleteAlarms, cloudwatch:DescribeAlarms
 ```
 
 Use **Settings → AWS Configuration → Run checks** to validate all permissions before going live.
@@ -131,15 +133,19 @@ Use **Settings → AWS Configuration → Run checks** to validate all permission
 
 ## Persistence
 
-Completed investigations are stored in the `alerts` table across all three storage backends (memory, SQLite, PostgreSQL). The Monitoring dashboard reads from this table.
+Each event-driven investigation creates a full session (messages, tool calls, usage events) in the database, linked to the alert via `session_id`. This powers the "View investigation" button on the Monitoring page, which opens the original chat instead of starting a fresh one.
+
+Sessions created by the event consumer carry `source = 'event'` and are hidden from the sidebar by default. They become visible after the user sends a follow-up message in the chat (`user_interacted` is then set to `true`).
+
+The `alerts` table records `status` (`'completed'` or `'failed'`), `dedup_key` (MD5 fingerprint of stable event fields for cross-run deduplication), and `session_id` (FK to the session that produced it).
 
-For PostgreSQL, run the migration before starting the server:
+All three storage backends (memory, SQLite, PostgreSQL) support alerts. For PostgreSQL, run migrations before starting the server:
 
 ```bash
 uv run python scripts/setup_db.py
 ```
 
-For SQLite, the `alerts` table is created automatically on first start.
+For SQLite, schema migrations are applied automatically on first start.
 
 ---
 
@@ -151,7 +157,7 @@ For SQLite, the `alerts` table is created automatically on first start.
 | `src/agent/event_infra.py` | Create/teardown SQS queue + EventBridge rules |
 | `src/agent/context_collectors.py` | Per-event-type boto3 context enrichment |
 | `src/agent/monitor_store.py` | In-memory service status + DB-backed alert persistence |
-| `src/agent/init_store.py` | Persist init config to `data/init.json` |
+| `src/agent/init_store.py` | Persist init config to DB-backed `app_config` with `data/init.json` fallback/cache |
 | `src/agent/permission_checker.py` | IAM permission validation |
 | `src/tools/sns.py` | SNS publish wrapper |
 | `src/api/routers/init.py` | Setup wizard API endpoints |

docs/iam_setup.md

@@ -0,0 +1,168 @@
+# IAM Setup Guide
+
+This guide covers how to create an AWS IAM user (or role) with the permissions OpenDevOps Agent needs, and how to plug the credentials into your environment.
+
+---
+
+## Credential options
+
+| Method | When to use |
+|---|---|
+| **IAM user + access key** | Self-hosted server, Docker, Railway, Render, local dev without AWS CLI profiles |
+| **Named profile** (`~/.aws/credentials`) | Local dev with AWS CLI already configured |
+| **IAM role** | EC2 / ECS / Lambda — attach the role to the instance/task, no keys needed |
+
+For a typical self-hosted deployment, create an IAM user and paste the keys into `.env`.
+
+---
+
+## Step 1 — Create the IAM user
+
+1. Open **IAM → Users → Create user** in the AWS console
+2. Username: `opendevops-agent` (or any name you prefer)
+3. Select **"Attach policies directly"**
+4. Create and attach the inline policy below (see [Step 2](#step-2--iam-policy))
+5. Open the user → **Security credentials → Create access key**
+6. Choose **"Other"** as the use case
+7. Copy the **Access key ID** and **Secret access key** — you won't see the secret again
+
+---
+
+## Step 2 — IAM policy
+
+Create a **customer-managed policy** named `OpenDevOpsAgentPolicy` with this JSON:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "ReadOnlyInvestigation",
+      "Effect": "Allow",
+      "Action": [
+        "cloudwatch:DescribeAlarms",
+        "cloudwatch:DescribeAlarmHistory",
+        "cloudwatch:GetMetricStatistics",
+        "cloudwatch:GetMetricData",
+        "logs:DescribeLogGroups",
+        "logs:FilterLogEvents",
+        "logs:StartQuery",
+        "logs:GetQueryResults",
+        "cloudtrail:LookupEvents",
+        "ecs:ListClusters",
+        "ecs:DescribeClusters",
+        "ecs:ListServices",
+        "ecs:DescribeServices",
+        "ecs:ListTasks",
+        "ecs:DescribeTasks",
+        "lambda:ListFunctions",
+        "lambda:GetFunction",
+        "lambda:GetFunctionConfiguration",
+        "ec2:DescribeInstances",
+        "ec2:DescribeInstanceStatus",
+        "rds:DescribeDBInstances",
+        "rds:DescribeEvents",
+        "iam:ListAttachedRolePolicies",
+        "iam:ListRolePolicies",
+        "sts:GetCallerIdentity"
+      ],
+      "Resource": "*"
+    },
+    {
+      "Sid": "EventMonitoringSetup",
+      "Effect": "Allow",
+      "Action": [
+        "sqs:CreateQueue",
+        "sqs:DeleteQueue",
+        "sqs:GetQueueUrl",
+        "sqs:GetQueueAttributes",
+        "sqs:SetQueueAttributes",
+        "sqs:ListQueues",
+        "sqs:ReceiveMessage",
+        "sqs:DeleteMessage",
+        "sqs:ChangeMessageVisibility",
+        "sqs:SendMessage",
+        "events:PutRule",
+        "events:PutTargets",
+        "events:ListRules",
+        "events:RemoveTargets",
+        "events:DeleteRule",
+        "cloudwatch:PutMetricAlarm",
+        "cloudwatch:DeleteAlarms"
+      ],
+      "Resource": "*"
+    },
+    {
+      "Sid": "SNSNotifications",
+      "Effect": "Allow",
+      "Action": [
+        "sns:Publish",
+        "sns:GetTopicAttributes"
+      ],
+      "Resource": "*"
+    }
+  ]
+}
+```
+
+### What each block does
+
+| Block | Purpose |
+|---|---|
+| `ReadOnlyInvestigation` | Read-only access to CloudWatch, CloudTrail, ECS, Lambda, EC2, RDS, IAM — used by the investigation agent tools |
+| `EventMonitoringSetup` | Create/manage the SQS queue and EventBridge rules during the setup wizard; poll SQS at runtime for incoming events |
+| `SNSNotifications` | Optional — publish alert summaries to an SNS topic after investigations; check topic attributes during permission check. Remove this block if you're not using SNS. |
+
+### Minimal policy (investigation only, no event monitoring)
+
+If you only want the chat interface and manual investigations — no automatic event detection — you can attach just the `ReadOnlyInvestigation` statement and skip the rest.
+
+---
+
+## Step 3 — Add credentials to `.env`
+
+```bash
+# Option A — explicit keys (server, Docker, CI)
+AWS_ACCESS_KEY_ID=AKIA...
+AWS_SECRET_ACCESS_KEY=...
+AWS_REGION=us-east-1
+
+# Option B — named profile (local dev)
+AWS_PROFILE=opendevops-agent
+AWS_REGION=us-east-1
+```
+
+For Option B, add a profile to `~/.aws/credentials`:
+
+```ini
+[opendevops-agent]
+aws_access_key_id = AKIA...
+aws_secret_access_key = ...
+```
+
+---
+
+## Step 4 — Verify in the setup wizard
+
+After starting the app, the setup wizard's **Step 3 (Permission Check)** calls each service with a lightweight read and shows a pass/fail result for:
+
+| Service | Required |
+|---|---|
+| CloudWatch | Yes |
+| Lambda | Yes |
+| SQS | Yes |
+| EventBridge | Yes |
+| ECS | No |
+| RDS | No |
+| EC2 | No |
+| IAM / STS | No |
+| SNS | No |
+| CloudTrail | No |
+
+Required services must pass for event monitoring to work. Optional services are used when available — missing permissions will simply return empty results for that service.
+
+---
+
+## Using an IAM role instead of a user
+
+If running on EC2, ECS, or Lambda, attach the same policy to the instance/task role instead. Remove `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_PROFILE` from `.env` — the SDK picks up the role automatically via the instance metadata service.