Distributed Data & Graph Analytics (PySpark, Spark SQL & GraphFrames)

A practical Big Data project leveraging Apache Spark, PySpark, Spark SQL, and GraphFrames to perform large-scale distributed graph processing, network analysis, and pattern discovery across interconnected datasets.

---

🚀 Key Highlights & Graph Algorithms

• Distributed Graph Representation: Modeling entities and relationships as distributed DataFrames of Vertices (nodes) and Edges (relationships).
• Graph Processing Algorithms:
  • PageRank: Identifying influential nodes within large network structures.
  • Connected Components: Detecting isolated subgraphs and clusters.
  • Shortest Path & Motif Finding: Querying complex structural patterns and multi-hop paths using declarative graph queries.
• Spark SQL Integration: Seamless combination of graph algorithms with relational Spark SQL transformations and window functions.

---

🛠️ Tech Stack

• Distributed Compute: Apache Spark 3.x
• Language / API: Python (PySpark) & Spark SQL
• Graph Framework: GraphFrames
• Environment: Jupyter Notebooks & Standalone PySpark Scripts

---

📁 Repository Structure

graphFrames/
├── notebooks/
│   └── spark_graph_analytics.ipynb # Interactive graph exploration & analysis
├── src/
│   └── graph_pipeline.py           # Automated PySpark pipeline
├── data/                           # Sample vertex and edge CSV datasets
└── README.md