Repository navigation
feat: event monitoring setup wizard, session persistence, and investigation pipeline - #33
Merged
Merged
Conversation
… if already logged in
- Fix redirect loop: RedirectToSession now waits for authLoading before routing, eliminating the /init ↔ / race condition - Add require_admin auth guard to all protected init endpoints - Add org_name field to wizard Step 1; creates organizations record and links admin user via org_id on first registration - Subsequent users registered via /auth/register inherit the existing org - init_store: add in-memory cache, async DB persistence via app_config table, legacy field migration, and helper accessors - migrations/006_app_config.sql: new app_config table for init state - stop_event_consumer() added; called on teardown and re-provision - teardown_event_infra returns warnings/errors, handles managed vs external queues, and rolls back on partial failure - Better HTTP status codes in init router (409 conflicts, 500 errors) - login() called after Step 1 account creation so Steps 2-4 have auth
…in memory - InitPage: org name input moved from Step 1 to Step 2 (AWS config step) so it's shown regardless of whether the user is already logged in - /api/init/create-user no longer accepts org_name; /api/init/setup now creates the org when org_name is provided and none exists yet - init_store: replace file-based persistence with DB (app_config table) + in-memory _CACHE; sync load_init() reads from cache with zero I/O, safe for boto3 thread executors; cache is seeded from DB on startup via refresh_init_cache_from_db() in the FastAPI lifespan - Added reset_init_async() helper; /api/init/reset uses it directly
…users - docs/iam_setup.md: IAM policy JSON covering read-only investigation, event monitoring setup, and optional SNS notifications; includes instructions for IAM user vs role vs named profile - InitPage Step 3: add link to iam_setup.md so users know where to look before running the permission check - init.py /setup: after creating org, backfill org_id on any existing users that were created before the org (typically the Step 1 admin) - users.py /api/users POST: look up first org and pass org_id when creating team members, matching what auth.register already does - db/base.py + db/postgres.py: add assign_org_to_users_without_org()
Drop the *60 multiplication so the value is used directly as seconds. Makes dev testing easier (POLL_INTERVAL_SECONDS=20 instead of waiting a full minute). Update all references: config, poller loop, app startup check, Settings UI label, .env.example, and CLAUDE.md.
Three-layer approach to prevent the same incident triggering multiple investigations and Slack notifications: 1. DB dedup: add dedup_key column to alerts + is_recent_alert() query. Both poller and event consumer write a stable key per trigger (e.g. alarm:opendevops-lambda-errors-aggregate) and check it before running any investigation. If an alert with that key exists in the last 10 minutes, skip. Covers all cross-path duplicates. 2. Structural fix: skip _check_alarms() in the poller when event infra is active. EventBridge already handles alarm state changes via the SQS path — polling DescribeAlarms on top of that is redundant. 3. _check_lambda_errors() also checks is_recent_alert() before starting, so the third duplicate (lambda error rate path) is suppressed when the alarm or event consumer already handled the same function. Migration: 007_alerts_dedup_key.sql adds the column + index.
…ready handled
The lambda_errors:{fn} key and alarm:{ALARM_NAME} key are different, so
the previous dedup check missed the overlap. Now _check_lambda_errors
also checks is_recent_alert on the aggregate alarm key — if the event
consumer or poller already investigated that alarm in the last 10 min,
skip the per-function error rate investigation entirely.
10 min was too wide — different errors on the same Lambda within that window would be silently skipped. The actual race between event consumer and poller paths is ~2 min, so 3 min is enough to catch duplicates while allowing a distinct new error to re-trigger after the window closes.
Adds JWT BearerAuth security scheme to the OpenAPI schema so the Swagger UI shows the Authorize button. persistAuthorization keeps the token across page refreshes. All endpoints get the security requirement by default.
OAuth2PasswordBearer showed a username/password form in Swagger that called /auth/login with form data, but the endpoint expects JSON — so the Authorize flow silently failed. HTTPBearer shows a plain token input box that just sets Authorization: Bearer <token>, which works correctly with our JWT setup.
…ents - Persist full sessions for event-driven investigations (source='event'), linked to alerts via session_id — enables 'View investigation' from monitoring page to open the original agent chat - Add user_interacted flag: event sessions appear in sidebar only after the user sends a follow-up message (migrations 009, 010) - Add status column to alerts (migration 008): failed investigations are flagged and displayed with an orange FAILED badge instead of LOW confidence - MD5-based general dedup key across all AWS event types; in-memory _in_progress set closes the race window between check and save - Show original AWS error in Slack failure notifications; mark test events with TEST badge in Slack - Smart Investigate/View investigation button on monitoring and alert detail pages: navigates to existing session when available - Filter event sessions from sidebar (source=chat OR user_interacted=true) - Format investigation result as readable assistant message in persisted chat - Add status field to alert API responses and Alert type - Update docs: monitoring.md, event_detection.md, README
…all capture Both event consumer and poller now use astream(stream_mode="messages") — the same streaming path as the chat endpoint — so tool calls are captured and saved to the DB identically regardless of how an investigation is triggered. Dashboard tool-call counts and session history now reflect event-driven investigations.
…notify signatures
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch ships the full event monitoring setup wizard, wires alert rows back to the investigation sessions that produced them, and unifies all three agent invocation paths (web chat, event consumer, proactive poller) onto the same streaming pipeline.
What changed
1. Event monitoring setup wizard (Settings → AWS Configuration)
setup_event_infra()creates the SQS queue, nine EventBridge rules, and an aggregate Lambda CloudWatch alarm in one click/api/init/resetendpoint to re-run the wizard from scratch/iam-setup) with copy-paste policy JSON scoped to the minimum required permissionsorg_idnull bug fixedapp_configtable replacesinit.jsonfile; in-memory cache avoids repeated DB reads/initwhen bothhas_userandinitializedare true2. Session persistence for event-driven investigations
Every background investigation (event consumer + poller) now creates a full session in the DB — messages, tool calls, and usage events — linked back to the alert row via
session_id.New migrations:
008_alerts_status.sql—status TEXTonalerts(completed/failed)009_session_source_alert_session.sql—source TEXTonsessions;session_id UUID FKonalerts010_session_user_interacted.sql—user_interacted BOOLEANonsessionsSidebar promotion logic: event sessions (
source = 'event') are hidden from the sidebar by default. They appear only after the user sends a follow-up message in chat (user_interactedflips toTRUEviaON CONFLICTCASE logic). This keeps the sidebar clean while still allowing users to navigate back via the Monitoring page.Schema changes across all three DB backends (
postgres.py,sqlite.py,memory.py):upsert_session: acceptssourceparam;ON CONFLICTupgradesuser_interactedbut never downgradessourcelist_sessions:WHERE source = 'chat' OR user_interacted = TRUEadd_alert: acceptsstatusandsession_id3. Unified streaming path for all investigations
event_consumer._run_investigationandpoller._run_investigationnow useastream(stream_mode="messages")— the exact same streaming pattern as the chat endpoint. Previously both usedainvoke, which returned only the final state and missed all intermediate tool calls.Result: dashboard tool call counts, session history, and cost tracking now correctly reflect event-driven and polling investigations, not just web chat.
The
tc_accum/pending_calls/tool_calls_logextraction logic is identical across all three paths.4. Monitoring page improvements
session_idis set (opens the original chat); falls back to "Investigate" (new pre-seeded chat) otherwiseinfraEnabledstate check: empty-state copy changes based on whether event monitoring is set up5. Dedup and poller fixes
is_recent_alertcross-check)alertstable (previously Slack-only)POLL_INTERVAL_MINUTESrenamed toPOLL_INTERVAL_SECONDSfor precision6. Auth / Swagger fix
OAuth2PasswordBearertoHTTPBearerso Swagger UI's Authorize dialog accepts Bearer tokens correctly7. Docs
Updated:
docs/event_detection.md,docs/monitoring.md,docs/slack_and_polling.md,docs/databases.md,docs/schema.md,docs/ui.mdNew:
docs/iam_setup.mdKnown open issue
Duplicate investigations when poller and event consumer both trigger for the same alarm — tracked in #32.
The event consumer stores dedup keys as
aws.cloudwatch:{md5}while the poller checksalarm:{alarmName}. These namespaces never match, making the poller's cross-check dead code. A fix was proposed (usealarm:{alarmName}in the event consumer for CloudWatch alarm events) but not applied — there are trade-offs around the ALARM→OK→ALARM edge case that need more consideration. See the issue for full analysis and next step options.Test plan
Setup wizard
/, confirm redirect to/init/api/init/reset→ wizard resets; re-running creates infrastructure again without errorSession persistence
POST /api/init/test-event) or trigger a real Lambda errorProcessing event: aws.cloudwatch / ...SELECT id, source, user_interacted FROM sessions ORDER BY created_at DESC LIMIT 3— new row withsource = 'event',user_interacted = falseSELECT session_id, service, status FROM alerts ORDER BY created_at DESC LIMIT 3—session_idnon-nullSELECT tool_name, COUNT(*) FROM tool_calls GROUP BY tool_name— all intermediate tool calls present (not justsubmit_investigation)/chat/{session_id}with full conversation visibleDashboard counts
FAILED badge
MAX_TOOL_CALLS=1): alert shows orange FAILED badge, not red HIGH badgeMonitoring page — test event button
Poller
POLL_INTERVAL_SECONDS=30,POLL_ERROR_THRESHOLD=0.0, trigger a Lambda errorsession_idand correctstatussubmit_investigation)Regression
/iam-setuppage loads and policy JSON is copyable