Resilient Global Telemetry Platform PySpark Implementation

Companion code for the "Architecting and Implementing a Resilient Global Telemetry Platform" assignment (Big Data Platforms & Analytics, BITS Pilani Digital). This repo contains working, executed PySpark scripts for Part 3 of the assignment, run locally against a generated mock dataset.

What's in here

File	Assignment section	What it proves
src/01_avg_temp_per_model.py	Part 3.1 Transformations & Actions	Ingests telemetry, computes average engine temp per vehicle model, and prints the physical plan showing the narrow-vs-wide dependency / shuffle boundary.
src/02_salting_skew_fix.py	Part 3.2 Optimization / Salting	Confirms the 1000x skew actually exists in the mock data, then shows the salted two-stage aggregation produces the same correct result while spreading the hot keys across partitions.
src/03_checkpointing.py	Part 3.3 & 3.4 Fault Tolerance / Checkpointing	Uses a shuffle-based iterative loop (reduceByKey) so the RDD lineage genuinely grows each iteration, then proves checkpoint() truncates that lineage (42 lines → 2 lines in the run captured here).

Run logs from an actual local execution are saved in outputs_run1.txt, outputs_run2.txt, and outputs_run3.txt for reference.

Data

data/telemetry.csv is synthetic mock data (200 "normal" trucks + 3 deliberately "hot" trucks generating ~1000x more log rows, matching the skew scenario described in the assignment) generated with a fixed random seed for reproducibility.

Schema:

Column	Type	Example
vehicle_id	string	TRK-00042
vehicle_model	string	Volvo-FH16
engine_temp_c	double	88.4
speed_kmph	double	62.1
lat	double	12.87232
lon	double	77.62365
distance_segment_miles	double	3.71
event_ts	timestamp	2026-07-01T00:00:00

Running it yourself

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

python3 src/01_avg_temp_per_model.py
python3 src/02_salting_skew_fix.py
python3 src/03_checkpointing.py

Requires Java 11+ (Spark runs on the JVM under the hood) in addition to Python 3.10+.

Windows-specific setup

PySpark has a known extra requirement on Windows, even for purely local, single-machine runs with no real Hadoop cluster involved: it needs winutils.exe and a HADOOP_HOME environment variable pointing to it. Without this, you will see an error like:

java.io.IOException: Could not locate executable null\bin\winutils.exe

To fix it:

1. Download a winutils.exe matching your Hadoop version (the version PySpark 4.x bundles is Hadoop 3.x) from a trusted mirror such as https://github.com/cdarlint/winutils
2. Place it at e.g. C:\hadoop\bin\winutils.exe
3. Set the environment variable HADOOP_HOME=C:\hadoop (System Properties → Environment Variables, or setx HADOOP_HOME C:\hadoop in an admin PowerShell)
4. Restart your terminal/IDE so the environment variable is picked up, then re-run the scripts.

This is a Windows-only quirk of Spark's local file-handling code — macOS and Linux do not need this step.

Note: if you'd rather not set any of this up, the three outputs_run*.txt files in this repo already contain the real, executed output from each script (run on Linux with Java 21) — running the code yourself is a way to verify it, not a requirement to see the results.

Notes / honest caveats

• This runs on local[*] (single machine, all cores) — it demonstrates correct Spark semantics (lazy evaluation, shuffles, lineage, checkpointing) but not actual multi-node cluster behavior or true network-locality effects, which would require a real cluster (e.g. EMR, Databricks, or a multi-node Spark standalone setup).
• The mock dataset (160,000 rows) is small enough to run instantly on a laptop; it's sized for demonstrating correctness, not for benchmarking real performance at fleet scale.