An AI-powered SRE tool that remembers every past incident, auto-classifies errors from P0 to P3 across 10 categories, extracts structured signals, and tells you exactly what to fix — in seconds.
SentinelAI v2 upgrades from a single-error RDS-focused tool into a full-spectrum incident intelligence engine. It now handles every class of production error, from complete cluster outages to a missing Python package.
| Capability | v1 | v2 |
|---|---|---|
| Error classification | None | Yes, 10 categories auto-detected |
| Severity triage | Manual | Yes, P0–P3 auto-assigned |
| Triage checklist | No | Yes, category-specific playbooks |
| Signal extraction | No | Yes, stack traces, HTTP codes, thresholds |
| TTR estimation | No | Yes, memory-weighted estimate |
| Escalation logic | No | Yes, auto-flag P0/P1 |
| Bulk analysis | No | Yes, up to 10 errors at once |
| Seed incidents | 5 (RDS only) | 21 (all categories) |
| Memory namespace | incident-agent | sentinelai-v2 |
When production goes down at 2am, SREs waste 20–40 minutes re-reading Slack threads, Confluence runbooks, and old PagerDuty tickets to figure out if they've seen this before. Institutional knowledge lives in people's heads, not in the tools you need when things are on fire.
SentinelAI is an incident response agent powered by Hindsight persistent memory. It:
- Classifies every error into one of 10 categories with severity (P0–P3)
- Extracts signals — stack traces, HTTP codes, mentioned services, numeric thresholds
- Recalls similar past incidents from long-term memory using semantic search
- Scores recurrence risk (HIGH / MEDIUM / LOW) based on similarity to past patterns
- Hypothesizes root causes based on what caused this pattern before
- Recommends specific numbered fix steps from proven past resolutions
- Estimates TTR based on category, severity, and how many times it's been seen
- Learns every time you log a resolution — getting smarter with each incident
| Category | Default Severity | Example Signals |
|---|---|---|
| Database / Storage | P1 | connectionpool, rds, deadlock, max_connections |
| Memory / OOM | P1 | oomkilled, heap, out of memory, memory limit |
| Infrastructure / K8s | P1 | crashloopbackoff, pod, kubernetes, node |
| Network / HTTP | P2 | 502, ssl, dns, econnrefused, timeout |
| Auth / Security | P2 | 401, jwt, oauth, forbidden, token |
| Application Error | P2 | nullpointerexception, typeerror, traceback |
| Performance / Latency | P2 | p99, latency, cpu, throttle, spike |
| Data / Pipeline | P3 | kafka, schema, corrupt, lag, stream |
| Config / Environment | P3 | env, missing key, misconfigured, yaml |
| Dependency / Package | P3 | modulenotfounderror, import, version, pip |
"This looks like a database connection error. Check your connection string and make sure your database is running."
Database/Storage · P1 · HIGH RECURRENCE RISK
This exact ConnectionPoolTimeoutError pattern has occurred 3 times in 21 days on payment-api. Root cause is consistently RDSmax_connections=100being reset by IaC updates that revert the pgbouncer config.Fix:
- Immediately increase
max_connectionsto 500 in RDS parameter group- Re-deploy pgbouncer
- Verify IaC terraform module has the permanent fix committed
- Add CloudWatch alert at 80% connection saturation
TTR estimate: ~8 minutes (you've fixed this before)
User pastes error / stack trace / log output
↓
Error Classification Engine (10 categories × pattern matching)
↓
Signal Extraction (stack traces, HTTP codes, thresholds, services)
↓
Hindsight memory recall (semantic search, top-12 results)
↓
Similarity scoring (Jaccard + bigram) → recurrence count + risk score
↓
Groq LLM (qwen3-32b) with category context + memory injected into prompt
↓
Structured intelligence report:
severity · risk · root cause · fix steps · triage checklist · TTR estimate
↓
User logs resolution → stored back to Hindsight with category metadata
↓
Agent gets smarter for the next incident
| Layer | Tool |
|---|---|
| Memory | Hindsight by Vectorize |
| LLM | Groq — qwen/qwen3-32b |
| Backend | Python + FastAPI |
| Frontend | Vanilla HTML/CSS/JS (single file) |
git clone https://github.com/yourusername/sentinel-ai-incident-agent
cd sentinel-ai-incident-agentcp .env.example .env
# Edit .env with your API keysGet your keys:
- Hindsight: https://ui.hindsight.vectorize.io/
- Groq: https://groq.com
pip install -r requirements.txtpython seed_incidents.pyuvicorn main:app --reload --port 8000The app works out of the box with 21 pre-seeded synthetic incidents built into the code. Just run it and paste any error.
{
"error_text": "ConnectionPoolTimeoutError: could not connect to RDS...",
"service": "payment-api",
"environment": "production",
"severity_override": null
}{
"errors": [
{"error_text": "OOMKilled: pod exceeded 512Mi", "service": "checkout"},
{"error_text": "ModuleNotFoundError: cryptography", "service": "api"},
{"error_text": "CrashLoopBackOff: api-gateway", "service": "gateway"}
]
}{
"incident_id": "INC-20250113143022",
"error_text": "ConnectionPoolTimeoutError...",
"resolution": "Increased max_connections to 500, re-deployed pgbouncer",
"root_cause": "RDS parameter group reverted by infra update",
"time_to_resolve_minutes": 8,
"service": "payment-api",
"tags": ["database", "rds", "config-regression"]
}- Analyze tab: Paste any error → get severity classification + memory-based analysis + triage checklist
- Bulk Analyze tab: Paste multiple errors → ranked by severity
- Log Resolution tab: Store fixes for future recall
- Error Guide tab: Browse categories and detection patterns
- PagerDuty / OpsGenie webhook integration
- Slack bot interface for SRE teams
- Trend dashboard
- Multi-service memory namespacing
- Auto-generated runbook export
- Anomaly detection on TTR trends
This project was built for the AI Agents That Learn Using Hindsight hackathon, sponsored by Vectorize.
MIT