Global IoT Sensor Data Management Platform (Multi-Region)

A cloud-native IoT data management platform for ingesting, processing, and tracking terabytes of sensor data from distributed field locations worldwide.

🎯 Project Overview

This platform addresses the challenge of managing large-scale data collection from IoT sensors deployed globally, providing:

• Automated multi-region data ingestion with intelligent routing
• Comprehensive metadata tracking and delivery status monitoring
• Real-time dashboard for operational visibility
• Cost-optimized storage with 30-40% reduction through intelligent tiering
• Scalable architecture supporting TBs to PBs of data

🏗️ Architecture

┌─────────────┐
│   Sensors   │ (Global Distribution)
└──────┬──────┘
       │ Upload via S3 Transfer Acceleration
       ▼
┌─────────────────────────────────────┐
│     S3 Landing Zone (Regional)      │
│  - EU-CENTRAL-1 (Primary for MVP)   │
│  - Event-driven triggers            │
└──────────┬──────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│   Lambda: File Validator             │
│   - File type validation             │
│   - Size and integrity checks        │
│   - Initial metadata extraction      │
└──────────┬───────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│   DynamoDB: Tracking Table           │
│   - Real-time status updates         │
│   - Comprehensive metadata           │
│   - Query by multiple dimensions     │
└──────────┬───────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│   Lambda: Metadata Extractor         │
│   - WAV: duration, sample rate       │
│   - Images: resolution, EXIF         │
│   - Video: codec, bitrate            │
└──────────┬───────────────────────────┘
           │
           ▼
┌──────────────────────────────────────┐
│   Streamlit Dashboard                │
│   - Real-time status monitoring      │
│   - Data volume analytics            │
│   - Processing performance metrics   │
└──────────────────────────────────────┘

📦 Features

✅ Infrastructure

• Single-region S3 bucket with hierarchical organization
• DynamoDB for high-performance tracking
• Lambda functions for serverless processing
• CloudWatch for monitoring and logging

✅ Data Processing

• Automated file validation (WAV, JPEG/PNG, MP4/AVI)
• Metadata extraction for all file types
• Status tracking (PENDING → VALIDATED → READY_FOR_PROCESSING)
• Error handling and retry logic

✅ Monitoring & Visibility

• Real-time Streamlit dashboard
• Data volume and processing metrics
• Quality score tracking
• Alert system for failures

✅ Cost Optimization

• Intelligent-Tiering storage class
• Lifecycle policies for archival
• Lambda optimization for minimal execution time
• Cost monitoring and alerts

🚀 Quick Start

Prerequisites

• AWS Account with appropriate permissions
• Terraform >= 1.6
• Python 3.11+
• AWS CLI configured

Installation

1. Clone the repository

git clone <repository-url>
cd global-sensor-data-management

2. Configure AWS credentials

aws configure
export AWS_REGION=eu-central-1

3. Create Terraform variables

cd infrastructure/terraform
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your configuration

4. Deploy infrastructure

terraform init
terraform plan
terraform apply

5. Package and deploy Lambda functions

cd ../../scripts
./deploy.sh

6. Run the dashboard

cd ../src/dashboard
pip install -r requirements.txt

# Set environment variables
export AWS_REGION=eu-central-1
export DYNAMODB_TABLE_NAME=<your-table-name>

streamlit run app.py

7. Test with sample data

cd ../../scripts
python test-upload.py --bucket <your-bucket-name> --count 10 --type mixed

📁 Project Structure

global-sensor-data-management/
├── README.md                          # This file
├── .gitignore                        # Git ignore rules
├── docs/                     # Documentation
│   ├── README.md                     # Documentation overview
│   ├── PROJECT_SUMMARY.md            # Project summary
│   ├── architecture.md               # Detailed architecture
│   ├── deployment-guide.md           # Step-by-step deployment
│   ├── cost-analysis.md              # Cost projections
│   ├── operations-runbook.md         # Operations guide
│   └── Global_Sensor_Data_Management_Platform_SOW.pdf
├── infrastructure/
│   └── terraform/                    # Infrastructure as Code
│       ├── main.tf                   # Main configuration
│       ├── variables.tf              # Variable definitions
│       ├── s3.tf                     # S3 buckets
│       ├── dynamodb.tf               # DynamoDB tables
│       ├── lambda.tf                 # Lambda functions
│       ├── iam.tf                    # IAM roles and policies
│       ├── cloudwatch.tf             # Monitoring
│       ├── outputs.tf                # Output values
│       └── terraform.tfvars.example  # Example variables
├── src/
│   ├── lambda/                       # Lambda function code
│   │   ├── common/                   # Shared utilities
│   │   │   └── utils.py
│   │   ├── file_validator/           # File validation
│   │   │   ├── handler.py
│   │   │   └── requirements.txt
│   │   └── metadata_extractor/       # Metadata extraction
│   │       ├── handler.py
│   │       └── requirements.txt
│   └── dashboard/                    # Streamlit dashboard
│       ├── app.py                    # Main dashboard
│       └── requirements.txt
└── scripts/                          # Deployment & testing
    ├── deploy.sh                     # Lambda deployment script
    └── test-upload.py                # Upload test files

🔧 Configuration

Terraform Variables

Key configuration parameters in terraform.tfvars:

aws_region  = "eu-central-1"
environment = "production"
project_name = "global-sensor-platform"

# S3 Configuration
s3_bucket_prefix = "sensor-data"
enable_transfer_acceleration = true
lifecycle_glacier_days = 90

# Lambda Configuration
lambda_timeout = 300
lambda_memory = 512

# Alerting
alert_email = "your-email@example.com"

S3 Bucket Organization

Files are organized hierarchically:

s3://bucket-name/
├── project-name/
│   └── country/
│       └── client/
│           └── sensor-id/
│               └── YYYY-MM-DD/
│                   └── filename.ext

Example:

wildlife-monitoring/germany/client-a/sensor-001/2024-01-15/recording_001.wav

📊 Cost Estimates

Based on 5TB/month data ingestion:

Service	Monthly Cost	Notes
S3 Storage (Intelligent-Tiering)	$115	~5TB active data
S3 Transfer Acceleration	$40	Data ingestion
Lambda Executions	$15	File processing
DynamoDB	$25	On-demand pricing
Data Transfer	$30	Outbound transfers
CloudWatch	$10	Logs and metrics
Total	~$235/month	40% savings vs standard storage

See docs/architecture/cost-analysis.md for detailed breakdown.

🧪 Testing

Upload Test Files

# Upload 10 mixed files
python scripts/test-upload.py --bucket your-bucket-name --count 10 --type mixed

# Upload only WAV files
python scripts/test-upload.py --bucket your-bucket-name --count 5 --type wav

# Upload only images
python scripts/test-upload.py --bucket your-bucket-name --count 5 --type jpeg

Monitor Processing

1. Open the Streamlit dashboard: streamlit run src/dashboard/app.py
2. Watch files appear in real-time as they're processed
3. Check CloudWatch logs for Lambda execution details

📈 Success Metrics

• ✅ 99.9% successful ingestion rate
• ✅ <5 minutes end-to-end processing time (95th percentile)
• ✅ <1 minute dashboard latency
• ✅ 30-40% cost reduction vs baseline
• ✅ Zero data loss during normal operations

🗺️ Roadmap

Phase 2: Enhancement & Scale (Future)

• Multi-region deployment (US, APAC)
• AWS Step Functions for complex workflows
• Aurora Serverless for analytics
• QuickSight dashboards
• API Gateway for programmatic access

Phase 3: Advanced Features (Future)

• ML-based anomaly detection
• Automated quality scoring
• Predictive analytics
• Advanced visualizations

📝 Documentation

Comprehensive documentation is available in the docs/ directory:

• Architecture Documentation - Detailed system design
• Deployment Guide - Step-by-step deployment instructions
• Cost Analysis - Cost projections and optimization strategies
• Operations Runbook - Day-to-day operations guide

🤝 Contributing

This is a portfolio project demonstrating enterprise-grade AWS architecture and data engineering practices.

👤 Author

Omar Miranda

• Senior Systems Administrator / Data Engineer
• 20+ years experience in AWS Cloud Architecture, DevOps, and Data Engineering
• Location: Saarbrücken, Germany

Areas of Expertise:

• AWS Cloud Architecture & Infrastructure Management
• Big Data Engineering & Analytics Pipelines
• DevOps & CI/CD Implementation
• Cost Optimization & Performance Tuning
• Multi-Region Deployments & Data Platform Scaling

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Links

• Architecture Documentation
• Deployment Guide
• Cost Analysis
• Operations Runbook

---

Note: This is a Phase 1 MVP implementation demonstrating core capabilities. Production deployment requires additional security hardening, multi-region setup, and enterprise-grade monitoring.