Skip to content

Security: EricBorba/ce-capstone-aiops-observability-platform

Security

SECURITY.md

Security & Compliance

Covers the five required components — threat model, risk assessment, security controls, testing results, incident response plan — plus the implemented controls, IAM, network, secrets, compliance mapping, and known risks.


1. Security controls implemented

Domain Control
Edge AWS WAF on the ALB — OWASP Common Rule Set, Known Bad Inputs, SQLi managed groups, IP rate limit (1000 req / 5 min)
Transport HTTPS only — ACM cert, TLS 1.3 policy, HTTP 80 → 443 redirect; RDS rds.force_ssl=1
Network 3-tier VPC, least-privilege chained security groups, isolated data subnets (no internet route), VPC Flow Logs
Identity Least-privilege IAM per role; ECS task role is read-only + Bedrock invoke (no mutate); separate narrow executor-Lambda role is the only infra mutator
Data KMS encryption at rest (RDS, EBS, S3); S3 account-level Block Public Access; S3 Object Lock (Compliance) on the audit bucket
Secrets No hardcoded secrets; DB password Terraform-generated into Secrets Manager, injected via the ECS secrets block; Bedrock auth via task role (no API keys); .gitignore blocks .env, *.pem, key/credential files
Supply chain ECR scan-on-push; Trivy + Checkov in CI; GitHub OIDC deploy (no long-lived AWS keys); Dependabot-ready
Governance AWS Config (7 rules), multi-region CloudTrail w/ log-file validation, IAM Access Analyzer
AIOps safety Human-in-the-loop approvals (single-use 5-min tokens), non-prod scoping, circuit breaker, immutable audit trail, decoupled executor identity

2. Threat model (STRIDE)

Assets: order data/PII, DB credentials, approval tokens, container images, Terraform state, audit trail, the ECS task identity.

STRIDE Threat Mitigation Priority
Spoofing Forged/replayed requests to the API WAF rate limiting, HTTPS, Auth0 OIDC on the console High
Spoofing An agent impersonates a human to approve a SUPERVISED action Single-use, 5-min-TTL tokens issued server-side; the agent identity cannot approve or execute; approval requires the operator endpoint Critical
Tampering SQL injection via order endpoints SQLAlchemy parameterized queries + Pydantic validation + WAF SQLi rules Critical
Tampering Image tampered between build and deploy ECR scan-on-push, Trivy gate in CI, immutable :sha tags High
Tampering Terraform state altered S3 state encrypted + versioned, native state locking, restricted access High
Repudiation Operator denies approving an action Every decision written to S3 Object Lock (Compliance) + CloudTrail with operator identity & timestamp High
Info disclosure Public S3 bucket leaks data/state Account-level Block Public Access + per-bucket PAB + KMS Critical
Info disclosure PII in logs/metrics Structured logging, no PII in metric dimensions, restricted log access Medium
DoS Volumetric attack on the API WAF rate-based rule, ALB/Shield Standard, ECS autoscaling Medium
DoS Runaway autonomous agent floods ECS API Circuit breaker (>3 actions/hr disables the agent), non-prod scope High
Elevation Agent escalates to mutate prod infra Task role has no mutate perms; only the executor Lambda (narrow role) can act, and it refuses prod for autonomous and requires approval for supervised Critical

Full STRIDE detail (all categories, per-threat): see docs/decisions/ and the predecessor planning repo.


3. Risk assessment (top 5)

Risk = Likelihood × Impact (Low/Med/High).

# Risk Likelihood Impact Score Mitigation (control)
R1 AIOps agent takes an unauthorized/unsafe action on prod Med High High Decoupled executor identity; supervised approval gate; autonomous prod-deny; circuit breaker; immutable audit (§1, §2 Elevation)
R2 Injection / app exploit exfiltrates order data Med High High WAF (SQLi/CRS), parameterized queries, Pydantic, private DB subnets
R3 Vulnerable dependency in the container (e.g. CRITICAL CVE) High Med High ECR + Trivy scanning; documented accepted risks; image-slim roadmap (§5)
R4 Public exposure of S3 (state/audit/logs) Low High Med Account + bucket Block Public Access, KMS, Object Lock
R5 Credential leakage (DB password, keys) Low High Med No keys in code/image (task-role auth), .gitignore, secrets via CI/TF_VAR; rotation roadmap

4. Security controls for the top 5 risks

  • R1: aiops module — DynamoDB single-use tokens; action_executor Lambda re-validates approval state and refuses prod autonomous actions; S3 Object Lock audit; ECS task role carries zero ecs:Update*/write permissions.
  • R2: WAF managed rule groups on the ALB; SQLAlchemy ORM; Pydantic request models; RDS in isolated subnets reachable only from the ECS SG.
  • R3: ECR scan-on-push + Trivy in CI and at build; CRITICAL findings triaged (§6 testing results); accepted risks documented with compensating controls.
  • R4: aws_s3_account_public_access_block + per-bucket PAB + KMS SSE; Config rule s3-bucket-public-read/write-prohibited continuously checks.
  • R5: task-role-based auth for Bedrock/AWS (no static keys); DB password injected at deploy; comprehensive secret patterns in .gitignore; IAM Access Analyzer flags external access.

5. IAM, network, and secrets detail

IAM (least privilege):

  • ecs-task-role — CloudWatch put/get, Bedrock InvokeModel, read-only ecs/rds/elb describe, SNS publish, DynamoDB on the approvals table, lambda:InvokeFunction on the executor. No infra-mutate actions.
  • action-executor-role — the only role with ecs:UpdateService/StopTask, scoped DynamoDB + S3 audit writes. Decoupled from the agent identity.
  • Service roles (Config, RDS monitoring, VPC flow logs, Lambdas) each scoped to their function.

Network: see ARCHITECTURE.md §3 — chained SGs, isolated data tier, flow logs.

Secrets management: no secrets in git or the image. The DB password is generated by Terraform (random_password) into AWS Secrets Manager (aiops-<env>-db-credentials) and injected into the ECS task via the task-def secrets block (never in the task-def environment, never an input variable). All AWS/Bedrock access is via IAM roles. CI/CD authenticates to AWS with GitHub OIDC — no long-lived access keys exist. Roadmap: enable automatic rotation on the Secrets Manager secret.


6. Testing results

Per the security requirements: Prowler CIS audit + Trivy container scan + ≥1 manual check.

Trivy (container image scan) — evidence/security/trivy-image-scan.txt

Real scan of aiops-order-service:latest (Debian 13.5 base, 109 OS packages):

  • OS packages: 17 findings (2 CRITICAL, 15 HIGH) — e.g. perl-base CVE-2026-42496 (fix deferred upstream), curl CVE-2026-5773.
  • Python deps: chromadb 1.1.1 CRITICAL CVE-2026-45829 (pre-auth code execution) — transitive dependency of CrewAI, not exposed (we run no ChromaDB server); starlette HIGH CVE-2025-62727 (DoS, fixed in 0.49.1).

Triage / accepted risks:

  • chromadbaccepted with compensating controls: pulled in transitively by CrewAI, never instantiated or network-exposed in our app; no ChromaDB listener runs. Mitigation roadmap: slim the image / pin CrewAI extras to drop the vector-store dependency.
  • starlette DoS — mitigated by WAF rate limiting; upgrade tracked.
  • OS CVEs (perl-base, curl) — upstream fixes deferred/unavailable on Debian 13 at scan time; rebuild picks them up automatically.

Prowler (AWS CIS audit) — evidence/security/prowler-aiops.csv

Scoped assessment across IAM, S3, RDS, EC2, CloudTrail, VPC (198 checks). Command:

prowler aws --services iam s3 rds ec2 cloudtrail vpc --output-formats csv

Result: 607 PASS / 388 FAIL (FAIL by severity: 3 critical, 55 high, 196 medium, 134 low).

Context: this is a shared bootcamp account — Prowler is account-wide, so the majority of findings come from other lab resources, not this project. Our IaC directly implements the controls the relevant CIS checks look for: RDS encryption + SSL, S3 Block Public Access (account + bucket), CloudTrail multi-region with log-file validation, VPC Flow Logs, least-privilege IAM, and restricted-ssh. The 3 critical/55 high are triaged against project-owned resources first; project resources pass the encryption/public-access/logging controls. The CSV is the full evidence.

Security-control tests (CLI) — scripts/security-tests.sh

A repeatable suite that probes each control live and prints an AWS-console pointer; all pass against prod:

# Test Probe Result
1 WAF — SQL injection GET /orders?q=' OR 1=1 UNION SELECT… 403 blocked (rule aiops-prod-sqli) ✅
2 WAF — known-bad input Log4Shell ${jndi:ldap://…} in User-Agent 403 blocked (bad-inputs/common) ✅
3 WAF — abusive request 9 KB body → Common SizeRestrictions_BODY 403 blocked ✅
4 WAF — volumetric DDoS (waf-flood, opt-in) flood the ALB > 1000 req/5min from one IP → rate-based rule aiops-prod-rate-limit IP 403-blocked within seconds; auto-unblocks ~5 min after ✅
5 HTTPS enforced curl -I http://…/health 301https:443; 443 listener uses ACM cert ✅
6 RDS private + Secrets-only describe RDS/SG/task-def PubliclyAccessible=false; 5432 only from the ECS SG; DB creds in the task-def secrets block (none in environment); secret aiops-prod-db-credentials present ✅

DDoS proof (test 4) — while the flood is active, the rate-based rule lists the offending IP:

aws wafv2 get-rate-based-statement-managed-keys --scope REGIONAL --region us-east-1 \
  --web-acl-name aiops-prod-waf --web-acl-id <web-acl-id> \
  --rule-name RateLimit --query 'ManagedKeysIPV4.Addresses' --output json
# → ["<your-ip>/32"]   (rule name is "RateLimit"; metric name is aiops-prod-rate-limit)

Manual checks (verified live against prod)

  1. HTTP→HTTPS redirecthttp://…/health returns 301 to HTTPS. ✅
  2. AIOps autonomous guardrail — autonomous action against environment=prod returns denied: "autonomous agents may not act on production". ✅
  3. AIOps approval lifecycle — propose → pending → approve (executes via the executor Lambda, audited) and reject both behave correctly; an approved token is single-use. ✅

7. Compliance mapping (CIS-aligned)

CIS theme Control in this project
Logging CloudTrail multi-region + log-file validation; VPC Flow Logs; CloudWatch
Monitoring Config rules, alarms on unauthorized/abnormal activity, GuardDuty (IaC-ready)
IAM Least-privilege roles, no AdministratorAccess, Access Analyzer, iam-user-no-policies Config rule
Storage Block Public Access (account + bucket), KMS encryption, s3-bucket-public-* Config rules
Network restricted-ssh Config rule, isolated subnets, least-privilege SGs
Data rds-storage-encrypted, encrypted-volumes Config rules; SSL enforced

8. Known risks & mitigations

Risk Status Mitigation / roadmap
GuardDuty + Security Hub not enabled Account not entitled (SubscriptionRequiredException) IaC ready behind enable_* flags; enable once account permits
RDS single-AZ Free-tier account blocks Multi-AZ db_multi_az flag ready; 3-AZ HA holds for compute
chromadb CRITICAL CVE in image Accepted Transitive, unexposed; slim-image roadmap
DB password without automatic rotation Partially resolved Now Terraform-generated into Secrets Manager and injected via the ECS secrets block; automatic rotation not yet enabled
Single NAT gateway Accepted (cost) Egress-only impact; per-AZ NAT one flag away

9. Incident response plan (6-step)

Pre-written, not reactive. Applies to: DDoS, GuardDuty/WAF high finding, AIOps agent runaway, compromised credential, data exposure.

  1. Isolate — identify scope; if an agent is implicated, trip the circuit breaker / disable the executor Lambda; tighten the relevant security group; for DDoS, raise WAF rate limits / block offending IPs.
  2. Investigate — pull CloudWatch logs (correlation IDs), CloudTrail events, VPC Flow Logs, and the S3 audit trail; confirm blast radius and root cause.
  3. Contain — revoke/rotate affected credentials; scale or quarantine affected tasks; freeze deployments (concurrency lock in CI).
  4. Notify — SNS alert to the on-call email; record the incident in docs/incident-reports/.
  5. Remediate — apply the fix via the GitOps pipeline (PR → review → merge → apply); re-run Trivy/Prowler to confirm; restore service.
  6. Post-mortem — blameless RCA in docs/incident-reports/: timeline, root cause, what worked, action items; update controls/runbook.

See RUNBOOK.md for the operational commands behind each step.

There aren't any published security advisories