Mission-Critical Incident Management System

An async Incident Management System for high-volume infrastructure signals. It ingests bursts, debounces noisy components, stores raw payloads for audit, tracks transactional incident workflow, and provides a React dashboard for responders.

Demo Screenshots

Live Incident Feed

Incident Detail And RCA

Raw Signals

Architecture

flowchart LR
  Producer[Signals / APIs / Queues / Caches / DBs] -->|HTTP JSON batches| FastAPI[FastAPI Ingestion API]
  FastAPI -->|bounded asyncio.Queue| Workers[Async Signal Workers]
  FastAPI --> Limiter[Token Bucket Rate Limiter]
  Workers -->|raw audit payloads| Mongo[(MongoDB Data Lake)]
  Workers -->|work items + RCA + MTTR| Postgres[(PostgreSQL Source of Truth)]
  Workers -->|debounce keys + hot dashboard| Redis[(Redis Cache)]
  Workers -->|minute buckets| Aggregations[(Aggregation Buckets)]
  React[React Dashboard] -->|REST APIs| FastAPI

Loading

Repository Layout

• backend/: FastAPI service, async workers, state/strategy patterns, tests.
• frontend/: React + Vite dashboard.
• sample-data/: sample JSON and a burst simulator.
• docs/: design and submission notes.
• prompts/: prompt/spec artifact required by the assignment.

Run With Docker Compose

docker compose up --build

Open:

• Frontend: http://localhost:5173
• Backend health: http://localhost:8000/api/health
• API docs: http://localhost:8000/docs

Simulate A Failure

In another terminal:

python -m venv .venv
.venv\Scripts\activate
pip install httpx
python sample-data/simulate_failure.py --api-url http://localhost:8000

The dashboard also has a Demo burst button that sends 120 cache signals for CACHE_CLUSTER_01. Those signals are linked to one debounced work item when they arrive inside the 10 second debounce window.

Backend APIs

• POST /api/signals: ingest one batch of signals.
• GET /api/incidents: active and recent incidents sorted by severity.
• GET /api/incidents/{id}: incident details with MongoDB raw signals.
• PATCH /api/incidents/{id}/status: transition incident state.
• POST /api/incidents/{id}/rca: create/update RCA and calculate MTTR.
• GET /api/health: health and queue depth.

Workflow Rules

Incidents move through:

OPEN -> INVESTIGATING -> RESOLVED -> CLOSED

The state pattern rejects invalid jumps. CLOSED is rejected until RCA is complete. RCA requires incident start/end, category, fix applied, and prevention steps. MTTR is calculated from the first signal time to RCA incident end time.

Backpressure And Resilience

The ingestion API accepts signals into a bounded asyncio.Queue. If the queue is full, it returns 503 with a clear backpressure message instead of crashing or growing memory indefinitely. Background workers process signals asynchronously and retry database writes with exponential backoff. A token-bucket limiter protects ingestion from cascading failures during very large bursts.

Throughput metrics are printed every 5 seconds:

[metrics] throughput=1234.20 signals/sec queue_depth=940

Data Handling

• MongoDB stores every raw signal as the audit/data-lake sink.
• PostgreSQL stores structured work items, RCA records, and aggregation buckets.
• Redis stores debounce keys and hot dashboard incident snapshots.

Tests

cd backend
pip install -r requirements.txt
pytest

Current tests cover mandatory RCA validation, invalid state transitions, valid close behavior, and RCA date validation.

Future Improvements

• Replace the in-memory queue with Kafka or RabbitMQ for durable event ingestion.
• Add authentication and role-based access control for incident responders.
• Add WebSocket/SSE updates for real-time dashboard refresh.
• Add alert escalation policies and responder assignment.
• Add distributed tracing for signal processing latency.
• Add Kubernetes manifests for production deployment.