fix: monitoring UI improvements, poller prompt quality, and timestamp fixes

CLAUDE.md

@@ -101,10 +101,10 @@ Everything below is built and working in the codebase:
 ### Event-Driven Detection
 - `event_consumer_loop()` long-polls SQS (20s wait), processes EventBridge events (CloudWatch alarm, ECS task failure, Lambda async error, RDS event, EC2 state change, CodePipeline failure, AWS Health), runs a full agent investigation per event, delivers results to SNS + Slack, persists to `alerts` table.
 - `context_collectors.collect_context()` pre-fetches resource facts deterministically before the LLM runs to reduce tool call count.
-- Starts automatically on app startup if `event_consumer_enabled=True`, `sqs_queue_url` is set, or `init.json` marks the wizard as completed.
+- Starts automatically on app startup if `event_consumer_enabled=True`, `sqs_queue_url` is set, or database-backed app config marks event infrastructure as enabled. Autonomous monitoring requires SQLite or PostgreSQL; memory mode is disabled for poller/consumer runs.
 
 ### Proactive Polling
-- `polling_loop()` runs every `POLL_INTERVAL_SECONDS` seconds (disabled by default at 0); checks CloudWatch alarms in ALARM state and Lambda error rates above `POLL_ERROR_THRESHOLD`; auto-investigates new anomalies and posts to Slack. In-memory dedup via `_last_investigated` map.
+- `polling_loop()` runs every `POLL_INTERVAL_SECONDS` seconds (disabled by default at 0); checks CloudWatch alarms in ALARM state and Lambda error rates above `POLL_ERROR_THRESHOLD`; auto-investigates new anomalies and posts to Slack/Telegram. Dedup uses canonical incident keys and durable DB-backed claims.
 
 ### Frontend
 - React 18 + TypeScript + Vite + Tailwind CSS + `@tailwindcss/typography`
@@ -225,10 +225,10 @@ POLL_REINVESTIGATE_HOURS=1            # dedup window
 # Event-driven detection
 SNS_TOPIC_ARN=                         # SNS publish target after investigations
 SQS_QUEUE_URL=                         # SQS queue for EventBridge events
-EVENT_CONSUMER_ENABLED=false           # also auto-starts if SQS_QUEUE_URL is set or init.json exists
+EVENT_CONSUMER_ENABLED=false           # also auto-starts if SQS_QUEUE_URL is set or app config enables infra
 
 # Misc
-DATA_DIR=data                          # server-side state files (init.json)
+DATA_DIR=data                          # reserved for future file-based state
 ```
 
 ---
@@ -262,7 +262,7 @@ EventBridge rules (9 event types)
 ```
 db.init()  →  init_agent(checkpointer)
   optional: asyncio.create_task(polling_loop())       if POLL_INTERVAL_SECONDS > 0
-  optional: asyncio.create_task(event_consumer_loop()) if SQS configured or init.json present
+  optional: asyncio.create_task(event_consumer_loop()) if SQS configured or app config enables infra
 ```
 
 ### Session continuity

README.md

@@ -10,11 +10,11 @@ and gives actionable mitigation plans — without the AWS DevOps Agent price tag
 - **Sandboxed bash execution tool** — agent can run whitelisted read-only AWS CLI, kubectl, and docker commands as a last resort when the structured tools fall short; every command validated against an allowlist before execution; never uses `shell=True`; hard 30-second timeout
   - Includes **CloudWatch Logs Insights** (`query_logs_insights`) — full query language support: `fields`, `filter`, `stats`, `sort`, `limit`; results include scanned MB
 - **Streaming responses** — FastAPI SSE endpoint streams agent tokens in real time as the LLM reasons; tool calls appear as they complete
-- **Event-driven incident detection** — EventBridge → SQS → long-poll consumer; 9 EventBridge rules cover CloudWatch alarms, ECS task failures, Lambda async errors, RDS events, EC2 state changes, CodePipeline failures, and AWS Health events; runs alongside the metric poller — see [docs/event_detection.md](docs/event_detection.md)
+- **Event-driven incident detection** — EventBridge → SQS → long-poll consumer; 9 EventBridge rules cover CloudWatch alarms, ECS task failures, Lambda async errors, RDS events, EC2 state changes, CodePipeline failures, and AWS Health events; uses a DLQ plus database-backed incident claims to avoid duplicate investigations; runs alongside the metric poller — see [docs/event_detection.md](docs/event_detection.md)
 - **Context enrichment** — before the LLM runs, deterministic boto3 calls fetch facts about the affected resource (alarm details, recent logs, function config, etc.) to reduce tool call count and speed up investigations
 - **SNS alert delivery** — after each event-driven investigation, findings are published to a configured SNS topic as structured JSON; fires alongside Slack and Telegram so all channels receive results simultaneously
 - **Monitoring dashboard** — live incident feed showing all event-driven investigations: confidence level (or FAILED badge), affected service, root cause summary, SNS status; each alert links back to its original investigation session via **View investigation** so you can follow up without losing context — see [docs/monitoring.md](docs/monitoring.md)
-- **AWS Configuration settings tab** — admin-only editable tab in Settings for SNS Topic ARN, SQS Queue URL, and AWS Region; shared org-wide (server-side `init.json`); includes an inline IAM permission checker per service
+- **AWS Configuration settings tab** — admin-only editable tab in Settings for SNS Topic ARN, SQS Queue URL, and AWS Region; shared org-wide via database-backed app config; includes an inline IAM permission checker per service
 - **Web UI** — React + Vite SPA served by FastAPI:
   - **Chat page** — streaming responses, collapsible tool call inspector, cost/latency card, stop button; supports `?prompt=` deeplink for pre-seeded investigations from the Monitoring dashboard
   - **Session history sidebar** — lists all past conversations; click any to resume with full tool call inspector and cost card restored; new chat and delete (soft) buttons
@@ -25,7 +25,7 @@ and gives actionable mitigation plans — without the AWS DevOps Agent price tag
   - **Team page** — admin-only user management: add, remove, and change roles
 - **Auth & RBAC** — optional password-based auth with `admin` and `user` roles; JWT tokens; first registered user auto-becomes admin; disabled by default (set `JWT_SECRET` to enable) — see [docs/auth.md](docs/auth.md)
 - **Three storage backends** — pick one via `CHECKPOINT_BACKEND` in `.env`; see [`docs/databases.md`](docs/databases.md)
-  - `memory` — zero config, no persistence; great for CI and quick testing
+  - `memory` — zero config, no persistence; great for CI and quick testing; autonomous polling/event monitoring is disabled in this mode
   - `sqlite` — local file, no external services; recommended for single-server and personal use
   - `postgres` — full production persistence via psycopg3 + `AsyncPostgresSaver`
   - Schema: `users`, `sessions`, `messages`, `tool_calls`, `usage_events` — see [`docs/schema.md`](docs/schema.md)
@@ -210,7 +210,7 @@ docs/                  # Feature reference — auth, schema, skills, databases,
 | `SLACK_WEBHOOK_URL` | none | Slack incoming webhook URL; leave unset to disable notifications |
 | `TELEGRAM_BOT_TOKEN` | none | Telegram bot token from @BotFather; leave unset to disable |
 | `TELEGRAM_CHAT_ID` | none | Target chat/group/channel ID (negative number for groups) |
-| `POLL_INTERVAL_MINUTES` | `0` | Proactive polling interval in minutes; `0` disables the poller |
+| `POLL_INTERVAL_SECONDS` | `0` | Proactive polling interval in seconds; `0` disables the poller |
 | `POLL_ERROR_THRESHOLD` | `5.0` | Lambda error rate % that triggers an automatic investigation |
 | `POLL_REINVESTIGATE_HOURS` | `1` | Cooldown period — skip re-investigating the same alarm within N hours |
 | `SUMMARIZATION_ENABLED` | `true` | Auto-compact sessions when they exceed the threshold |
@@ -221,7 +221,7 @@ docs/                  # Feature reference — auth, schema, skills, databases,
 | `SNS_TOPIC_ARN` | none | SNS topic to publish investigation findings to after each event-driven run |
 | `SQS_QUEUE_URL` | none | SQS queue URL for the event consumer to poll; also set via Settings → AWS Configuration |
 | `EVENT_CONSUMER_ENABLED` | `false` | Explicitly enable the SQS event consumer (also auto-starts if `SQS_QUEUE_URL` is set) |
-| `DATA_DIR` | `data` | Directory for server-side state files (`init.json`) |
+| `DATA_DIR` | `data` | Reserved data directory setting; init state is stored in the selected database backend |
 
 ## TODO / Roadmap
 
@@ -254,7 +254,7 @@ docs/                  # Feature reference — auth, schema, skills, databases,
 - [x] **SNS alert delivery** — findings published to SNS after each event-driven investigation; fires alongside Slack; structured JSON payload for easy downstream processing
 - [x] **Context enrichment** — deterministic boto3 calls per event type before LLM runs; reduces tool call count by front-loading relevant resource facts
 - [x] **Monitoring dashboard** — live incident feed, per-service health summary, alert detail page; "View investigation" opens the original agent session for follow-up; failed investigations flagged separately; see [docs/monitoring.md](docs/monitoring.md)
-- [x] **AWS Configuration settings tab** — admin-only editable tab for SNS/SQS/region config; shared org-wide via server-side `init.json`; inline IAM permission checker
+- [x] **AWS Configuration settings tab** — admin-only editable tab for SNS/SQS/region config; shared org-wide via database-backed app config; inline IAM permission checker
 
 ### Later
 - [ ] **Observability** — OpenTelemetry traces for agent steps, tool call latency, LLM token usage

docs/auth.md

@@ -17,6 +17,14 @@ Restart the server. The login page will appear on next browser load.
 
 **Without `JWT_SECRET`:** the app runs in dev/open mode — no login required, all users treated as admin. Suitable for local development or single-user installs on a trusted network.
 
+Auth works with all storage backends, but persistence follows the backend:
+
+| Backend | Auth behavior |
+|---|---|
+| `memory` | Users exist only until the server restarts; useful for quick local testing |
+| `sqlite` | Users persist in the local SQLite database |
+| `postgres` | Users persist in PostgreSQL and are suitable for production/team use |
+
 ---
 
 ## First-time setup

docs/databases.md

@@ -9,7 +9,7 @@ OpenDevOps Agent supports three storage backends. Pick one per deployment — se
 
 | Backend    | Persistence | External service | Dashboard | Best for                          |
 |------------|-------------|-----------------|-----------|-----------------------------------|
-| `memory`   | None        | None            | Counts only | CI, quick demos, local testing  |
+| `memory`   | None        | None            | Counts only; no autonomous monitoring | CI, quick demos, local testing  |
 | `sqlite`   | Local file  | None            | Full      | Single-server, personal use       |
 | `postgres` | Database    | PostgreSQL 14+  | Full      | Production, teams, multi-instance |
 
@@ -24,6 +24,13 @@ CHECKPOINT_BACKEND=memory
 Everything lives in Python dicts for the life of the process. On restart, all
 sessions and history are gone. The LangGraph checkpointer uses `MemorySaver`.
 
+Autonomous monitoring is disabled in memory mode. The poller and SQS event
+consumer require durable incident claims, so use SQLite or PostgreSQL for those
+features.
+
+Auth/register/login work in memory mode when `JWT_SECRET` is set, but users are
+lost on restart with the rest of the in-memory state.
+
 **When to use:** CI pipelines, smoke-testing, one-off demos.
 
 > **Dashboard limitation:** summary counts (sessions, queries, tool calls, cost) update
@@ -45,8 +52,7 @@ LangGraph checkpointer. Both share the same `.db` file via separate connections
 with WAL mode enabled.
 
 SQLite also stores app-level configuration such as init wizard completion and event
-monitoring infrastructure state in `app_config`. `data/init.json` is only a local
-cache/fallback.
+monitoring infrastructure state in `app_config`.
 
 The file and its parent directory are created automatically on first start.
 

docs/event_detection.md

@@ -4,7 +4,7 @@ OpenDevOps Agent supports two complementary detection modes that run simultaneou
 
 | Mode | Source | Trigger |
 |---|---|---|
-| **Proactive polling** | CloudWatch Alarms + Lambda metrics | Checked every N minutes by a background loop |
+| **Proactive polling** | CloudWatch Alarms + Lambda metrics | Checked every N seconds by a background loop |
 | **Event-driven consumer** | EventBridge → SQS | Real AWS infrastructure events delivered in near-real time |
 
 This page covers the event-driven consumer. For the polling loop see [slack_and_polling.md](slack_and_polling.md).
@@ -16,7 +16,7 @@ This page covers the event-driven consumer. For the polling loop see [slack_and_
 ```
 AWS Service (ECS, Lambda, RDS, EC2…)
   → CloudWatch → EventBridge rule
-      → SQS queue (OpenDevOps)
+      → SQS queue (OpenDevOps, with DLQ)
           → SQS long-poll consumer (event_consumer_loop)
               → context_collectors (deterministic boto3 enrichment)
               → agent.ainvoke (LLM investigation)
@@ -84,15 +84,19 @@ Context is appended to the investigation prompt, capped at 3 000 chars to stay w
 
 1. Receives up to 5 messages per poll with a 20-second long-poll wait
 2. Filters noise: only processes events where `_is_real_failure()` returns true (e.g. skips EC2 `running` state changes, healthy RDS events)
-3. Deduplicates: checks an in-memory set (`_in_progress`) and the database to skip events already being investigated or investigated within the last 3 minutes
+3. Deduplicates: builds a canonical incident key and atomically claims it in the database before the agent runs
 4. Calls `collect_context(event)` to enrich the prompt
 5. Generates a `session_id` and runs a full agent investigation via `agent.ainvoke`, persisting the session and messages to the database (`source = 'event'`)
 6. Persists the result to the `alerts` table (linked to the session via `session_id`) via `monitor_store.add_alert()`
 7. Delivers findings via SNS and Slack (if configured); failed investigations are flagged with status `'failed'` but are still persisted and notified
-8. Deletes the SQS message
+8. Deletes the SQS message only after successful processing, an intentional ignore, or a duplicate claim. Processing failures are left on the queue for retry and eventual DLQ redrive.
 
 The consumer is started as an `asyncio.Task` in the FastAPI lifespan and shut down cleanly on server stop.
 
+The setup wizard creates both `opendevops-agent-events` and `opendevops-agent-events-dlq`. The main queue has a redrive policy with `maxReceiveCount=5`; poison messages remain available in the DLQ for inspection instead of being dropped by the consumer.
+
+Deduplication uses canonical incident keys rather than hashes of whole event payloads. Examples: `cloudwatch_alarm:us-east-1:high-error-rate`, `lambda_errors:us-east-1:payment-processor`, and `lambda_error:us-east-1:payment-processor:<signature>`. Lambda EventBridge failures with different error signatures are separate incidents. Metric polling only sees function-level error rates, so those incidents are deduped at the function level during the cooldown window.
+
 ---
 
 ## SNS Alert Delivery
@@ -137,9 +141,9 @@ Each event-driven investigation creates a full session (messages, tool calls, us
 
 Sessions created by the event consumer carry `source = 'event'` and are hidden from the sidebar by default. They become visible after the user sends a follow-up message in the chat (`user_interacted` is then set to `true`).
 
-The `alerts` table records `status` (`'completed'` or `'failed'`), `dedup_key` (MD5 fingerprint of stable event fields for cross-run deduplication), and `session_id` (FK to the session that produced it).
+The `alerts` table records `status` (`'completed'` or `'failed'`), `dedup_key` (the canonical incident key), `session_id` (FK to the session that produced it), and `trigger_source` (`'poller'` or `'event_consumer'`). The `incident_claims` table stores the atomic pre-investigation claim used to prevent duplicate agent runs across workers and restarts.
 
-All three storage backends (memory, SQLite, PostgreSQL) support alerts. For PostgreSQL, run migrations before starting the server:
+Autonomous polling and event-driven monitoring require SQLite or PostgreSQL because incident claims must be durable. The memory backend remains available for chat, CI, and quick demos, but the poller, event consumer, and setup wizard are disabled in memory mode. For PostgreSQL, run migrations before starting the server:
 
 ```bash
 uv run python scripts/setup_db.py
@@ -156,9 +160,10 @@ For SQLite, schema migrations are applied automatically on first start.
 | `src/agent/event_consumer.py` | SQS long-poll loop, filtering, prompt building, delivery |
 | `src/agent/event_infra.py` | Create/teardown SQS queue + EventBridge rules |
 | `src/agent/context_collectors.py` | Per-event-type boto3 context enrichment |
-| `src/agent/monitor_store.py` | In-memory service status + DB-backed alert persistence |
-| `src/agent/init_store.py` | Persist init config to DB-backed `app_config` with `data/init.json` fallback/cache |
+| `src/agent/monitor_store.py` | In-process service status + DB-backed alerts, notifications, and incident claims |
+| `src/agent/init_store.py` | Persist init config to DB-backed `app_config` |
 | `src/agent/permission_checker.py` | IAM permission validation |
 | `src/tools/sns.py` | SNS publish wrapper |
 | `src/api/routers/init.py` | Setup wizard API endpoints |
 | `migrations/005_alerts.sql` | PostgreSQL alerts table + indexes |
+| `migrations/012_incident_claims.sql` | PostgreSQL incident claim table for atomic deduplication |