Operations Guide

Back to README · See also: Architecture · Compliance · API Reference

Docker Stack

Full local stack via single Compose file:

docker compose -f fraud_detection/infra/docker/docker-compose.yml up -d

Services

Service	Image	Port	Purpose
`api`	`fraud_detection:latest`	8000	FastAPI scoring service
`redis`	`redis:7.2-alpine`	6379	Feature store + blocklists
`postgres`	`postgres:16-alpine`	5432	Audit trail + profiles
`mlflow`	`ghcr.io/mlflow/mlflow:v2.15.0`	5000	Experiment tracking + model registry
`prometheus`	`prom/prometheus:v2.53.0`	9090	Metrics scrape + alerting
`grafana`	`grafana/grafana:11.0.0`	3000	Dashboards

Redis Configuration

maxmemory: 2gb
maxmemory-policy: allkeys-lru
save: ""          # no persistence — Redis is a cache, not source of truth
appendonly: no

Velocity data is disposable. On Redis restart, cold miss → feature defaults → confidence=LOW. Acceptable: next request re-populates velocity.

PostgreSQL Tuning

shared_buffers: 512MB
work_mem: 32MB
log_min_duration_statement: 100ms    # slow query log

Production: scale shared_buffers to 25% of instance RAM.

Environment Variables

Variable	Required	Default	Description
`REDIS_URL`	Yes	—	`redis://host:6379/0`
`DATABASE_URL`	Yes	—	asyncpg DSN (`postgresql://user:pass@host/db`)
`MLFLOW_TRACKING_URI`	Yes	—	MLflow server URL
`API_KEYS`	Yes	`dev_key_1`	Comma-separated valid tokens
`MODEL_VERSION`	Yes	`0.0.0-dev`	Semantic version, logged on every decision
`ARTIFACT_DIR`	Yes	`/app/artifacts`	Path to loaded model files
`RULES_PATH`	Yes	`/app/fraud_detection/configs/fraud_rules.yaml`	YAML rules config
`RATE_LIMIT_PER_MINUTE`	No	`1000`	Max requests per API key per minute (Redis Lua)
`MAXMIND_ACCOUNT_ID`	No	—	MaxMind GeoIP2 credentials
`MAXMIND_LICENSE_KEY`	No	—	MaxMind GeoIP2 credentials
`POSTGRES_PASSWORD`	No	`local_dev_only`	Override in production
`GRAFANA_PASSWORD`	No	`admin`	Override in production

Deployment

Container Build

docker build \
  -f fraud_detection/infra/docker/Dockerfile \
  -t fraud_detection:$(git rev-parse --short HEAD) \
  .

Multi-stage build: builder (gcc/g++ for LightGBM compile) → production (non-root appuser UID 1001).

Health Checks

# Liveness (is process alive?)
curl http://localhost:8000/health/live
# → {"status": "ok"}

# Readiness (models loaded + Redis connected?)
curl http://localhost:8000/health/ready
# → {"status": "ready", "checks": {"models_loaded": true, "rules_loaded": true, "redis": true}}

Kubernetes readiness probe uses /health/ready. Liveness uses /health/live. Startup probe allows 30s for model loading.

Model Loading at Startup

Models loaded in ThreadPoolExecutor during FastAPI lifespan (avoids blocking async event loop). Order:

Redis connection pool (50 connections, 10ms socket timeout)
PostgreSQL connection pool (asyncpg)
Load all model artifacts from ARTIFACT_DIR in parallel threads:
- lgbm_{rail}.lgb per rail
- lstm.onnx
- autoencoder.pt
- meta_learner.json
Initialise RulesEngine from RULES_PATH
Pre-warm MaxMind top-1000 IPs in Redis

/health/ready returns 503 until all 5 are loaded.

SIGHUP — Zero-Downtime Rules Hotpatch

To update fraud rules without redeployment:

Update configs/fraud_rules.yaml

Send SIGHUP to the API process:

kill -HUP $(pgrep -f "uvicorn fraud_detection")

RulesEngine.reload() called — reloads YAML atomically
New rules active for all subsequent requests
Zero downtime, zero redeploy

Log line on reload: rules_reloaded, n_rules=12

Monitoring

Prometheus Metrics

Key metrics for alerting:

fraud_request_latency_seconds{quantile="0.95"}   < 0.100   # SLA
fraud_request_latency_seconds{quantile="0.99"}   < 0.200   # hard timeout
fraud_decision_total{decision="BLOCK"}                      # block volume
fraud_decision_total{decision="REVIEW"}                     # review queue load
fraud_redis_miss_total                                      # fail-open events
fraud_feature_psi{tier="score"}                  < 0.10    # drift alert
fraud_rules_triggered_total{rule_id="aeps_ghost_txn"}       # rule fires

Prometheus config: infra/prometheus/prometheus.yml

Grafana Dashboards

Three dashboards (provision via infra/grafana/dashboards/):

1. Executive Dashboard

Fraud catch rate (%) — 24h rolling
False positive rate (%) — 24h rolling
Fraud amount caught (₹) — 24h rolling
Block / Review / Allow volume — real-time

2. Model Health Dashboard

P50/P95/P99 latency — time-series
Score PSI drift — per-feature heatmap
Redis miss rate (fail-open events)
Model version in production
Per-rail decision distribution

3. Operational Dashboard

Service uptime
Redis memory usage + eviction rate
PostgreSQL slow queries
Rule fire rates by rule_id
Error rate by endpoint

PSI Drift Schedule

Tier	Features	Check Freq	Alert Threshold
Score	`fraud_score` output	Every 1h	PSI > 0.10
Tier 1	Velocity, amount, MCC	Every 6h	PSI > 0.15
Tier 2	Entity profiles, IP risk	Daily	PSI > 0.20
Tier 3	Account tenure	Weekly	PSI > 0.25

PSI > 0.10: monitor. PSI > 0.20: moderate shift, review. PSI > 0.25: major shift, trigger retrain.

Code: src/monitoring/drift_detector.py

Runbooks

P0 — Catch Rate Drop > 10pp in 24h

Symptoms: fraud_catch_rate_24h drops from baseline by 10+ percentage points.

Immediate actions (< 30 min):

Check recent model deployment: mlflow models list-versions --name fraud_lgbm_upi
If new model deployed in last 24h → rollback immediately (see Rollback section)
Check score distribution: SELECT AVG(risk_score_int), STDDEV(risk_score_int) FROM model_decisions WHERE scored_at > NOW() - INTERVAL '1 hour'
Check rule fires: SELECT rule_id, COUNT(*) FROM model_decisions, jsonb_array_elements(rules_triggered) WHERE scored_at > NOW() - INTERVAL '1 hour' GROUP BY rule_id
Page ML Lead + Risk Officer

Root causes to investigate:

New fraud pattern not in training data → emergency retrain
Feature drift (PSI > 0.25 on Tier 1) → check velocity feature quality
Redis data corruption → flush + rebuild
Rules too permissive → tighten thresholds via YAML hotpatch

Timeline: Resolve within 4 hours or escalate to P-1 incident.

P1 — P95 Latency > 100ms

Symptoms: fraud_request_latency_seconds{quantile="0.95"} exceeds 100ms.

Diagnosis steps:

# 1. Check Redis latency
redis-cli --latency -i 1

# 2. Check slow queries
SELECT query, mean_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;

# 3. Check model inference time (SHAP can spike under load)
grep "scored" /var/log/fraud_detection/api.log | jq '.latency_ms' | sort -n | tail -20

# 4. Check CPU / thread pool saturation
# ThreadPoolExecutor max_workers=4; saturated = latency spike

Common fixes:

Redis slow → check eviction rate, increase maxmemory, or scale cluster
SHAP spike → SHAP is in-path (~2ms normally); check model size after retrain
Thread pool saturated → increase max_workers in _CPU_POOL (rebuild required)
Postgres slow → check index usage on model_decisions (see schema.sql)

P1 — Redis Fail-Open Rate Spike

Symptoms: fraud_redis_miss_total rate spikes. Responses show confidence=LOW.

Impact: Velocity features fall back to zeros. Model scores less accurate. NOT a complete outage — fail-open means transactions still scored (conservatively).

Actions:

Check Redis connection: redis-cli ping
Check Redis memory: redis-cli info memory | grep used_memory_human
If OOM: redis-cli info stats | grep evicted_keys — high = need more RAM
If connection issue: check REDIS_URL, network, security group
If planned Redis maintenance → alert is expected; suppress for maintenance window

P2 — Model Decision Audit Dead-Letter

Symptoms: audit_dead_letter log entries appearing. Indicates 3 retry attempts all failed.

Impact: Scoring decisions not persisted to PostgreSQL. Regulatory risk (RBI 8-year retention).

Actions:

Check PostgreSQL health: pg_isready -U fraud -d fraud_detection
Query dead-letter logs: grep "audit_dead_letter" /var/log/fraud_detection/api.log | jq '{txn_id, decision, error}'
Replay: dead-letter entries contain full payload (payload_keys). Write replay script using logged txn_id + decision + risk_score.
Fix root cause (disk full / connection pool exhausted / schema change)
Once fixed, replay all dead-lettered records in chronological order

Rollback Procedure

Trigger: Catch rate drop > 10pp post-deploy, P95 latency > 150ms, error rate > 0.1%.

Time to rollback: < 5 minutes.

# 1. Identify previous version
mlflow models search-model-versions --filter "name='fraud_lgbm_upi' AND tags.stage='production'"

# 2. Point production alias to previous version
mlflow models set-alias --model-name fraud_lgbm_upi \
  --alias production --version <PREV_VERSION>

# 3. Reload models without restart
curl -X POST http://localhost:8000/admin/reload-models \
  -H "Authorization: Bearer ${ADMIN_KEY}"

# 4. Verify
curl http://localhost:8000/health/ready
# Confirm model_version in response matches previous version

# 5. Monitor for 15 minutes
# Check fraud_catch_rate_24h recovers

Minimum 3 archived versions kept at all times to ensure at least 2 rollback targets.

Load Testing

k6 script: tests/load/fraud_scoring.js

# Run SLA validation at 100 req/s
k6 run --vus 50 --duration 60s fraud_detection/tests/load/fraud_scoring.js

# SLA gates
# P95 latency < 100ms
# Error rate < 0.1%
# No 503s after 30s warmup

Run before every production promotion. Results logged to CI artifacts.

Kafka Phase 2 (Planned)

When volume exceeds synchronous REST capacity:

# Additional compose profile: kafka
services:
  kafka:
    image: confluentinc/cp-kafka:7.7.0
    environment:
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_NODE_ID: 1
      # KRaft mode — no Zookeeper

Topics:

transactions.incoming — 24 partitions, key=user_id hash (ordering for LSTM)
transactions.scored — decisions + scores
transactions.fraud — BLOCK decisions only
transactions.dlq — dead-letter queue for replay

Consumer: aiokafka, at-least-once delivery. Scoring is idempotent — duplicate processing harmless.

ScoringService is a pure class with no HTTP/Kafka dependency. Both REST handler and Kafka consumer call await scoring_service.score(request). Zero modification to scoring logic for Phase 2.

Next: Compliance · Architecture

FilesExpand file tree

operations.md

Latest commit

History