Repository navigation
refactor: monitoring reliability, SNS removal, and IAM least-privilege docs - #44
Merged
Merged
Conversation
…reliability - Extract _run_investigation into src/agent/investigation_runner.py, eliminating ~130 lines of copy-paste between poller.py and event_consumer.py - Synthesise a failed result when agent completes without calling submit_investigation (handles models with weak tool-use support like Gemma) so the event is always persisted and notified rather than silently dropped - Add SSE push at GET /api/monitoring/stream; MonitoringPage switches from 30s polling to real-time alert delivery via EventSource with 30s fallback on error - Fix services grid to survive restarts by falling back to DB-derived status when in-memory dict is empty; fix status mapping so completed investigations show as healthy (green) instead of amber - Isolate poller boto3 calls to a dedicated ThreadPoolExecutor(max_workers=4) so poller ticks cannot saturate the shared default pool used by API requests
SNS was fully optional (poller never used it, event consumer gated on sns_topic_arn being set) and added IAM surface area with no benefit over the existing Slack and Telegram channels. - Delete src/tools/sns.py - Remove SNS block from event_consumer._deliver(); sns_sent hardcoded False - Remove sns_topic_arn from appsettings, init_store (_default + accessor), SetupBody, and /api/init/setup handler - Remove SNS permission check from permission_checker.check_permissions() - Remove SNS field from SettingsPage AWS Configuration tab and InitPage wizard - Remove SNS_TOPIC_ARN from .env.example - Clean up permission_checker line lengths while touching the file
…and investigation_runner notes Two-policy IAM split (Operational reads *, Setup writes scoped to opendevops-*). Removes all SNS mentions from docs, monitoring flow, and README roadmap. Adds SSE stream endpoint to monitoring API reference. Documents service health DB fallback, investigation_runner, and poller thread pool.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/agent/investigation_runner.py) — extracted ~130 lines of duplicated streaming logic from poller and event consumer; both now delegate to a singlerun_investigation()function. Also synthesizes a_status: failedresult when the LLM completes the loop without callingsubmit_investigation(fixes silent drop with weak models like Gemma).ThreadPoolExecutor(max_workers=4, thread_name_prefix="poller")isolates poller boto3 calls from the app's default executor, preventing thread pool exhaustion under concurrent alarms.get_services()rebuilds the in-memory service grid from thealertstable on restart instead of returning empty state. Also fixed a pre-existing bug wherecompletedalways showed asunknown(amber) instead ofhealthy.GET /api/monitoring/streampushes new alerts to the frontend via SSE as they arrive (heartbeat every 20s, falls back to 30s polling on disconnect). Replaces the previous 30s interval poll.scripts/test_dlq.py(end-to-end AWS script) andtests/integration/test_dlq_logic.py(7 unit tests, all passing) confirm the existingmaxReceiveCount=5redrive policy works correctly.src/tools/sns.py; removed all SNS config, UI fields, permission checks, and delivery code from the event consumer, poller, init store, settings page, init page, and env example.docs/iam_setup.md) — separate Operational policy (reads use*) and Setup policy (create/delete/put scoped toopendevops-*resources only). No hardcoded account IDs.monitoring.md,event_detection.md,slack_and_polling.md, andREADME.mdreflect all of the above.Test plan
scripts/test_pipeline.py, confirm alert appears on/monitoringwith correct confidence level and session linkLLM_MODELto a weak model (e.g.openrouter/google/gemma-4-26b-a4b-it), trigger an event; confirm aFAILEDalert is persisted rather than the investigation silently disappearing/monitoringservices panel shows last-known status without needing a new event/monitoringin the browser, trigger a test event via Send test event; confirm the new alert card appears without a page refreshuv run python scripts/test_dlq.py --profile <your-profile>with the server stopped; confirm✓ PASSoutputuv run pytest tests/integration/test_dlq_logic.py -v→ 7 passedSNS_TOPIC_ARNis absent from.env.example, Settings page, and Init page; confirm event consumer sends Slack notification onlydocs/iam_setup.md, attach to a test user; run Settings → AWS Configuration → Run checks; confirm all required services pass