AIOps Observability Platform

Ironhack Cloud Engineering Bootcamp — Capstone Portfolio Project Author: Eric Borba

✅ Status: deployed and verified in us-east-1 (prod + dev). HTTPS, RDS, 3-AZ ECS, CrewAI→Bedrock agents, the human-in-the-loop action plane, and an Auth0-auth operator console are all live and tested. Badges/claims reflect only what is actually deployed.

🖥️ Live operator console: https://d2dp4bm5fu76cq.cloudfront.net — sign in with Google / GitHub (Auth0), then run analysis, trigger the autonomous agent (prod-denied vs dev-executed), and approve/reject supervised actions. Served via CloudFront (S3 console + ALB API on one origin), so it works on filtered networks.

---

Overview

An AIOps Observability Platform — the evolution of an instrumented, monitored order-processing service into a platform whose core value is an intelligent operations layer: AI agents (CrewAI on Amazon Bedrock) that don't just analyze infrastructure, but take supervised actions on it under a human-in-the-loop trust model.

Production-grade observability (structured logging, custom CloudWatch metrics, Golden-Signal dashboards, tiered alerting) is the foundation. The differentiator is the 3-tier AIOps agent model and an operator console that surfaces exactly what the agents see, decide, and do.

---

Architecture

Internet
   │
   ▼
CloudFront + WAF  ──────────────►  S3 (static AIOps Operator Console + landing page)
   │  (CDN, OWASP Top 10, rate limiting)
   ▼
Auth0 (OAuth2 / OIDC — Google / GitHub / email)
   │
   ▼
Application Load Balancer (HTTPS:443, ACM cert)
   │  → :8000
   ▼
ECS Fargate (private subnets, 3 tasks across 3 AZs)
├── FastAPI order service
├── Structured JSON logging → CloudWatch Logs
├── Custom metrics → CloudWatch Metrics
└── AIOps Agents (CrewAI → Amazon Bedrock / Nova Lite)
    ├── [ANALYSIS]   incident_analyst · fraud_detector · log_summarizer  (read-only)
    ├── [AUTONOMOUS] ecs_task_restarter  (non-prod scoped · circuit breaker >3/hr)
    └── [SUPERVISED] action requires human approval in the console
        └── approval token → decoupled executor Lambda → S3 Object Lock audit trail
   │
   ├── RDS PostgreSQL 15 (data subnets, Multi-AZ, encrypted)
   └── SNS alerts → email

Security & Compliance
├── WAF on ALB (OWASP + rate limit)   ├── AWS Config (7 compliance rules)
├── CloudTrail (multi-region audit)   ├── S3 account Block Public Access
├── IAM Access Analyzer · VPC Flow Logs · ECR scan-on-push
└── GuardDuty + Security Hub — IaC-ready, gated off (account not entitled)

Availability Zones: us-east-1a / 1b / 1c — 3 ECS tasks, Multi-AZ RDS.

---

AIOps Agent Model (human-in-the-loop)

Tier	Agents	Behavior
ANALYSIS	incident_analyst, fraud_detector, log_summarizer	Read-only, all on Bedrock. fraud_detector scans recent orders from RDS (computes per-customer velocity, flags the most suspicious — no synthetic input); incident_analyst applies RED/USE to live metrics + recent logs; log_summarizer summarizes the server-side log buffer. Never modify infrastructure.
AUTONOMOUS	ecs_task_restarter	Acts without approval — non-prod scope only; prod attempts are denied server-side. Circuit breaker disables after >3 actions/hour. Each action carries a data-driven justification (cites the live metrics that triggered it), surfaced in the console and the audit log.
SUPERVISED	action agents	Sends a single-use, time-limited approval request with a generated justification to the operator. Does not execute until a human approves in the console. Read-only agent identity is decoupled from a narrow-scoped executor Lambda. Every decision → immutable S3 (Object Lock) audit trail.

The AIOps Operator Console (static site in S3 behind CloudFront, Auth0-authenticated) shows the analysis feed, the autonomous action log, and pending approvals with Approve / Reject buttons — the human-in-the-loop, demoed live.

Frontend stack — intentionally zero-build: a single static index.html + vanilla JavaScript (app.js, no framework — no React/Vue), styled with Tailwind CSS via CDN, authenticated with the Auth0 SPA SDK (CDN, OIDC: Google / GitHub / email). No package.json, no Node/npm build, no bundler. It's hosted on S3 + CloudFront (OAC), and CloudFront also proxies the API paths to the ALB so the console and API share one origin (no CORS, works on filtered networks). The application complexity lives in the Python/FastAPI + CrewAI/Bedrock tier on ECS — the console is a deliberately dependency-free presentation layer (nothing to build, patch, or vuln-scan).

---

Repository Structure

.
├── README.md  ARCHITECTURE.md  SECURITY.md  RUNBOOK.md  COSTS.md  RETROSPECTIVE.md
├── terraform/
│   ├── modules/{networking,compute,monitoring,security}/
│   └── environments/{dev,prod}/
├── app/                      # FastAPI service + CrewAI/Bedrock agents
├── frontend/                 # static console: index.html + vanilla JS + Tailwind (CDN), no build → S3
├── .github/workflows/        # CI (validate/scan/test) · deploy (OIDC GitOps) · drift — all path-conditional
├── monitoring/               # dashboards, alarms, queries
├── tests/terratest/
├── scripts/                  # security-tests · load-test · inject-failure · deploy · cleanup
├── docs/{architecture,decisions,incident-reports}/
└── presentation/{slides.pdf,demo-script.md,screenshots/}

---

Requirements Coverage (capstone rubric)

Area	Approach
Infrastructure as Code (15%)	Modular Terraform, remote state, dev/prod environments
Application Platform (15%)	ECS Fargate, ALB, 3 tasks across 3 AZs, /health with instance/AZ
CI/CD & Automation (15%)	GitHub Actions, path-conditional: validate/tflint, Checkov/Trivy scan, Terratest; OIDC deploy-on-merge (no static keys) to dev/prod environments; nightly drift detection
Observability (10%)	CloudWatch logs/metrics/dashboards/alarms + SNS
Security & Compliance (10%)	WAF, GuardDuty, Security Hub, Config, CloudTrail, Auth0 OIDC, least-privilege IAM; STRIDE threat model + Prowler/Trivy testing
Cost Optimization (5%)	Allocation tags, FinOps Lambda, monthly projection, rightsizing
Advanced (20%)	Auto-scaling · RDS · ECS orchestration · DNS+SSL (HTTPS)
Excellence (10%)	FinOps dashboard · AIOps human-in-the-loop · security automation

---

Documentation

Doc	Contents
ARCHITECTURE.md	Components, network design, data flows, HA, scalability, decisions & trade-offs
SECURITY.md	STRIDE threat model, risk assessment, controls, testing results, IR plan, compliance
RUNBOOK.md	Deploy, update, monitor, troubleshoot, incident response, backup/recovery, scaling
COSTS.md	Monthly breakdown, tags, optimizations, savings, scaling projections, budgets
RETROSPECTIVE.md	What went well, challenges & fixes, lessons, future work

---

Prerequisites & setup

• Terraform ≥ 1.10 (S3-native state locking via use_lockfile), AWS CLI v2, Docker, an AWS identity with deploy permissions
• A Route53 public hosted zone (this project uses ironhack-bootcamp-ericborba.work.gd)
• No DB password to set — Terraform generates it (random_password) into Secrets Manager and injects it into the ECS task

Deployment

cd terraform/environments/prod      # or dev
terraform init && terraform apply
# build & push the image, then roll the service (see RUNBOOK.md §3)

Merges to main auto-deploy via GitOps (.github/workflows/deploy.yml), authenticated with GitHub OIDC (role aiops-prod-gha-deploy, repo secret AWS_DEPLOY_ROLE_ARN — no long-lived AWS keys). Jobs are path-conditional (dorny/paths-filter): a terraform/** change applies infra, app/** builds + rolls ECS, frontend/** syncs the console, monitoring/dashboards/** republishes the dashboard. Both CI and deploy also expose workflow_dispatch for on-demand full runs. Full steps and the model-switch procedure are in RUNBOOK.md.

Testing

• Infra: terraform validate + tflint + Checkov (CI); Terratest module validation; nightly drift detection
• Container/secrets: Trivy filesystem + secret + misconfig scan (CI) and image scan (evidence/security/)
• Cloud posture: Prowler CIS audit (evidence/security/)
• Security controls (CLI): ./scripts/security-tests.sh — live probes proving WAF blocks SQLi / Log4Shell / oversized requests (403), HTTP→HTTPS redirect + ACM, and RDS is private with credentials only in Secrets Manager. Each prints an AWS-console pointer. Optional waf-flood demonstrates the rate-based rule blocking a volumetric DDoS.
• Load & failure injection: ./scripts/load-test.sh (populates the dashboard + seeds orders) and ./scripts/inject-failure.sh cpu_spike|high_latency|error_flood|db_slow (drives alarms; gives the incident-analyst agent real data). See scripts/README.md.
• Manual: HTTPS redirect, WAF, and the AIOps guardrail/approval flow (see SECURITY.md §6)
• App: python -m compileall app + ruff (CI)

Cost summary

~$145/mo prod run-rate (dev ephemeral); deliberate FinOps choices (single NAT, right-sized Fargate, ephemeral dev) and budget alerts at 80%/100%. Full analysis in COSTS.md.

---

Predecessor

Evolves ce-project-2-instrumented-monitored-service — adds Bedrock-backed agents, the 3-tier action model, edge security (CloudFront + WAF), continuous threat detection, OAuth, and an operator console.

---

Contact

Eric Borba — eric.borba@gmail.com