Skip to content
This repository was archived by the owner on Apr 18, 2026. It is now read-only.

README.md

Latest commit

 

History

History
101 lines (67 loc) · 4.55 KB

README.md

File metadata and controls

101 lines (67 loc) · 4.55 KB
Raw

Important

This repo is archived — now part of databricks-code-practice

All hands-on Databricks labs now live in one unified repo: jrlasak/databricks-code-practice.

This lab's new home: https://github.com/jrlasak/databricks-code-practice/tree/main/pipeline-labs/fintech-monitoring

Clone the new repo: git clone https://github.com/jrlasak/databricks-code-practice

The content below still works and is preserved for anyone with existing clones or links, but future updates land only in the new unified repo.

Lab: Real-Time Fintech Transaction Monitoring Pipeline

Independent educational resource; not endorsed by Databricks, Inc. "Databricks" and "Delta Lake" are trademarks of their respective owners.

Author

Jakub Lasak

Jakub Lasak — Helping you interview like seniors, execute like seniors, and think like seniors.

Business Scenario

Company: ModernPaymentsABC - a payment processor handling 500K+ transactions/day.

Problem: Fraud is rising, and overnight batch detection is too slow. The ops team needs real-time alerts for suspicious activity (velocity spikes, geo-anomalies), and compliance needs daily Suspicious Activity Reports (SARs).

Your Role: Senior Data Engineer building the end-to-end monitoring pipeline on Databricks.

Learning Objectives

By completing this lab, you will be able to:

  • Ingest Streaming JSON with Auto Loader and capture malformed data using the Rescued Data Column.
  • Implement Watermarked Deduplication to handle technical payment gateway retries.
  • Perform Stream-Static Joins to enrich real-time events with customer and merchant reference data.
  • Design a Rules Engine using Tumbling and Sliding windows for velocity detection.
  • Build a Medallion Architecture that serves dual SLAs: real-time streaming alerts and batch Gold reporting.
  • Optimize for Performance using Liquid Clustering.

Architecture Overview

  1. Bronze: Raw ingestion via Auto Loader + Watermarked Dedup.
  2. Silver: Enriched transactions + Real-time risk scoring & alerts.
  3. Gold: Aggregated merchant summaries and SAR pre-fill datasets.

Prerequisites

  • Basic knowledge of PySpark and Delta Lake.
  • Access to a Databricks workspace (Free Edition compatible).

⚠️ Before you start: Disable AI code suggestions in your Databricks workspace. Go to User Settings → Developer → AI-powered code completion and turn OFF:

  • Autocomplete as you type
  • Automatic Assistant Autocomplete This lab is designed to build muscle memory - auto-completions defeat the purpose.

How to Start

  1. Create a Databricks Account

  2. Import this repository to Databricks

    • In Databricks, go to the Workspace sidebar and click the "Repos" section, click "Add Repo".
      • Alternatively, go to your personal folder, click "create" and select "git folder".
    • Paste the GitHub URL for this repository.
    • Authenticate with GitHub if prompted, and select the main branch.
    • The repo will appear as a folder in your workspace, allowing you to edit, run notebooks, and manage files directly from Databricks.
    • For more details, see the official Databricks documentation: Repos in Databricks.

  3. Open the notebooks/ folder and run 00_Setup_Environment.py to create the Unity Catalog infrastructure and generate data.

  4. Follow the numbered notebooks (01 to 04) to build the pipeline.

    Each exercise includes a STUDENT EXERCISE area for your code and a commented-out SOLUTION for verification.

Certification Alignment

This lab prepares you for:

  • Databricks Data Engineer Associate: Auto Loader, Medallion, Delta basics.
  • Databricks Data Engineer Professional: Streaming, Watermarks, Windowing, Table optimization.