Real-time ML System

Local-first foundation for a real-time fraud detection platform with:

• synthetic transaction generation (Spark)
• reproducible data workflow orchestration (DVC)
• model tracking and registry (MLflow)
• observability stack (Prometheus + Grafana)
• online serving API and UI (FastAPI + Redis)

Architecture diagram

flowchart LR
	A[Raw Transactions\nSpark Generator] --> B[Offline Features\nParquet + DVC]
	B --> C[Training Pipeline]
	C --> D[MLflow Registry\nModel Stage]
	B --> E[Redis Online Features]
	D --> F[FastAPI Inference API]
	E --> F
	F --> G[Prediction Clients]
	F --> H[Prometheus Metrics]
	H --> I[Grafana Dashboards]

Loading

The current repository includes runnable data generation, training, serving, monitoring, and a full local Docker runtime stack.

Repository layout

• configs/ - feature, model, and monitoring configs
• data/ - raw, processed, and feature datasets
• docker/ - container provisioning assets
• pipelines/ - orchestration code (planned)
• src/ - data, training, serving, and monitoring modules
• tests/ - smoke and integration checks
• dvc.yaml - stage graph for reproducible pipeline execution
• docker-compose.yml - local platform stack

Current implementation status

Implemented in code:

• src/data_pipeline/generate_data.py synthetic data generation
• src/data_pipeline/build_features.py, validate_data.py, push_to_redis.py
• src/training/train.py training and MLflow logging fallback
• src/serving/app.py FastAPI inference API with transaction UI at /ui
• src/monitoring/generate_monitoring_report.py monitoring summary output
• MLflow/Postgres/Redis/Prometheus/Grafana services in docker-compose.yml
• docker/mlflow.Dockerfile custom MLflow image with psycopg2-binary for Postgres backend support
• Prometheus scrape config and Grafana datasource provisioning

Verified in this workspace:

• feature generation, validation, training, Redis dry-run publishing, serving smoke tests, and monitoring report generation
• Docker-backed startup of MLflow, Redis, Prometheus, Grafana, and the API
• API/UI endpoint checks (/docs, /ui) and MLflow/Grafana availability checks
• static file checks for the updated code and docs

Planned but not yet implemented in this repo:

• broader CI and release automation
• richer end-to-end tests across data generation to prediction API

The DVC and Docker Compose files reference implemented components with a local development focus.

Prerequisites

• Python 3.13
• Java 21 (for Spark local runtime compatibility in current script)
• Docker Desktop with Compose v2
• Git

Local setup

1. Create environment

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip

2. Install dependencies (minimum to run current data pipeline)

pip install -r requirements/data.txt

Optional role-specific installs:

pip install -r requirements/training.txt
pip install -r requirements/serving.txt
pip install -r requirements/monitoring.txt

3. Generate synthetic raw data

python src/data_pipeline/generate_data.py

Output:

• data/raw/transactions.parquet

Serving API quick start

Run the serving API from the local venv:

.\.venv\Scripts\python.exe -m uvicorn serving.app:app --app-dir src --host 0.0.0.0 --port 8000

Key endpoints:

• GET /ui (interactive transaction form for fraud detection)
• GET /health/live
• GET /health/ready
• POST /predict
• GET /metrics

Primary local URLs:

• UI: http://localhost:8000/ui
• API docs: http://localhost:8000/docs

Run smoke tests:

.\.venv\Scripts\python.exe -m unittest tests/test_serving_smoke.py -v

Run the full stack via Docker Compose:

docker compose up -d

Monitoring quick start

Generate the monitoring report from current offline features and training metrics:

.\.venv\Scripts\python.exe src/monitoring/generate_monitoring_report.py

Output:

• data/processed/monitoring_report.json

Grafana dashboard provisioning is defined in:

• docker/grafana/provisioning/dashboards/dashboards.yml
• docker/grafana/dashboards/fraud-api-overview.json

Local platform services

Start core platform services:

docker compose up -d postgres mlflow redis prometheus grafana

Start full stack including API:

docker compose up -d

Endpoints:

• MLflow: http://localhost:5000
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin/admin)
• API UI: http://localhost:8000/ui
• API docs: http://localhost:8000/docs

Stop services:

docker compose down

Remove services and volumes:

docker compose down -v

DVC pipeline notes

dvc.yaml defines a complete pipeline, and the implemented scripts now cover the declared stages in the repository.

Safe command now:

dvc repro generate_raw_data

The remaining limitation is runtime infrastructure: MLflow, Redis, Prometheus, and Grafana still depend on Docker Desktop being available in this workspace.

Implemented executable stages include raw generation, feature building, validation, training, Redis publishing, and monitoring report generation.

Textbook documentation

• LaTeX book source
• Book build notes

Build PDF with Dockerized LaTeX (recommended in this workspace):

docker run --rm -v "${PWD}:/work" -w /work/docs/book blang/latex:ctanfull pdflatex -interaction=nonstopmode -halt-on-error main.tex
docker run --rm -v "${PWD}:/work" -w /work/docs/book blang/latex:ctanfull pdflatex -interaction=nonstopmode -halt-on-error main.tex