pysaprk

End-to-end data pipeline transforming Olist e-commerce data through Azure cloud services. Implements medallion architecture (Bronze-Silver-Gold) with multi-source ingestion, Spark-based processing, and OLTP-to-OLAP optimization for analytics-ready datasets.

Project based on application of azure databricks

Databricks Medallion (Bronze/Silver/Gold) retail analytics + Looker Studio dashboard

Sample analysis done using pyspark on parking violations issued for fiscal year 2017 using the databricks platform

This project performed data wrangling, analysis, visualization as well as machine learning prediction on a hypothetical music app's user churn with pyspark.

Runnable data-engineering patterns, gotchas and utilities — SQL, Python/PySpark, shell and cloud. Each folder clones and runs in seconds, with captured output.

NBA shot predictions with PySpark and SparkML

ontains the code and examples for my article on Medium, which introduces the English SDK for Apache Spark, showcasing how to combine the power of Apache Spark with large language models (LLMs)

1.This is an end-to-end ETL pipeline to ingest data from AWS S3, validate it, process transformations using PySpark, and load final outputs into MySQL fact and dimension tables. 2.Designed a star schema and created optimized data marts using PySpark SQL and parquet processing. 3.Automated S3 operations (download, upload, archival) and implemented a