This project is a comprehensive data engineering solution built on Microsoft Azure, designed to process, transform, and analyze streaming and batch data related to Netflix. It follows a modern data pipeline architecture, leveraging Azure Databricks, Delta Live Tables, Azure Data Factory, and Azure Synapse Analytics to enable scalable, automated, and efficient data processing for business intelligence and analytics.
This project simulates a real-world data engineering scenario, enabling efficient data ingestion, transformation, storage, and visualization within a cloud-based ecosystem. The solution is designed to support incremental data ingestion, optimize data transformation through Delta Lake, and provide structured data models for downstream analytics.
Develop and implement a scalable, automated data pipeline that integrates Azure Databricks, Azure Data Factory, and Delta Live Tables to transform and incrementally load structured data into a star schema within Azure Data Lake, facilitating advanced analytics and reporting in Azure Synapse and Power BI.
Azure Databricks, Delta Live Tables, Azure Data Factory, Azure Data Lake Gen2, Azure Synapse Analytics, Power BI, GitHub, Azure Security (IAM, RBAC).
This project was developed as a learning initiative, inspired by Ansh Lamba's educational content, to gain hands-on experience with Azure’s data engineering capabilities.