Industry 4.0: Hybrid Storage Layer (IoT)

Data Engineering Project focused on ingestion, routing, and polyglot persistence of industrial sensor data, combining relational consistency (SQL) with scalable flexibility (NoSQL).

---

🏗️ Solution Architecture

This project implements a containerized microservices architecture to solve the Volume and Variety challenge of Industry 4.0 data. The system ingests raw data streams and uses a logical router to distribute information between relational and non-relational databases.

The Problem Solved

Modern manufacturing systems generate two distinct types of data that rarely coexist efficiently in a single database model:

1. Telemetry Data (Time-series): Highly structured, critical for condition monitoring.
2. Operational Logs & Events: Semi-structured, rapidly changing formats and volumes.

---

🗂️ Data Modeling

To support this heterogeneity, a Polyglot Persistence strategy was adopted:

1. Structured Data (MySQL)

Used to store critical sensor readings requiring referential integrity and ACID consistency.

2. Semi-Structured Data (MongoDB)

Used to store raw logs, event metadata, and machine settings, leveraging schema-less flexibility.

---

🛠️ Project Structure

industry40-storage-layer/
│
├── data/                    # Raw NASA dataset (Git ignored for performance)
│   └── README.md            # Dataset download instructions
│
├── docs/                    # Architecture and modeling diagrams
│   ├── architecture_docker.jpg
│   ├── mysql_erd.jpg
│   └── mongodb_schema.jpg
│
├── src/                     # Application Source Code
│   ├── __init__.py
│   ├── ingestion.py         # ETL: Data reading, cleaning, and transformation
│   ├── storage_layer.py     # Core: Hybrid routing logic (SQL/NoSQL)
│   ├── database_setup.sql   # DDL scripts for MySQL
│   └── analysis.py          # Analytics and Plotting script
│
├── docker-compose.yml       # Infrastructure orchestration (MySQL + MongoDB)
└── requirements.txt     # Python project dependencies

---

📊 Data Dictionary (NASA Dataset)

The project uses real-world data from the NASA Turbofan Engine Degradation Simulation. Below are the main sensors monitored by the ingestion system:

Sensor ID	Technical Description	Unit	Application in Project
Sensor 2	Total Temperature at LPC Outlet	°R (Rankine)	Thermal Monitoring
Sensor 7	Total Pressure at HPC Outlet	psia	Pressure Loss Detection
Sensor 12	Fan Rotation Speed	rpm	Vibration/Rotation Analysis
Sensor 21	LPT Coolant Bleed	lbm/s	Cooling Efficiency

---

🚀 How to Run

1. Prerequisites

• Docker and Docker Compose installed.
• Python 3.8+.
• NASA Dataset (instructions in the /data folder).

2. Spinning up Infrastructure

The command below starts the MySQL and MongoDB containers in an isolated network:

docker-compose up -d

3. Running the ETL Pipeline

This script reads the raw file, normalizes the data, and routes it to the appropriate databases.

pip install -r requirements.txt
python src/ingestion.py

Expected Console Output:

[ETL] Processing Cycle 1 of Unit 1...
[SQL] Inserted: Sensor T24 (Temperature) = 642.15 °R
[NoSQL] Operational Log saved: ID 65a91...
...
[INFO] 20,631 records processed in 4.2s.

4. Generating Analytics Reports

After loading, run the analysis script to visualize turbine degradation by retrieving historical data from SQL.

python src/analysis.py

🧠 Architecture Decisions

Why Containerization? Using Docker ensures the development environment is identical to production, eliminating compatibility issues between database versions.

Why Hybrid (SQL + NoSQL)? The diverse nature of IoT data requires hybrid solutions. Relational databases (MySQL) become inefficient for storing nested JSON logs with dynamic keys, while NoSQL databases (MongoDB) may not offer the necessary efficiency for complex aggregation queries on rigid time-series. Combining both offers the best of both worlds.

📚 References

• Design Science Research: Methodology used for artifact construction.

• Dataset: A. Saxena and K. Goebel (2008). "Turbofan Engine Degradation Simulation Data Set", NASA Ames Prognostics Data Repository.

• Theoretical Foundation: JIANG, Lihong et al. An IoT-Oriented Data Storage Framework in Cloud Computing Platform.