Signalpipe is a fully managed observability pipeline for an Amazon EC2 instance, built entirely from AWS-managed services. CloudWatch continuously tracks CPU utilization, a configured alarm detects sustained overload, SNS fires an email alert the moment it happens, and CloudWatch Logs are streamed through Kinesis Data Firehose into an encrypted S3 bucket for secure, long-term archival.
Servers rarely fail instantly — resource exhaustion builds gradually, and without monitoring in place, teams find out about it from a user complaint rather than an alert. Signalpipe closes that gap with a clear separation of concerns: monitoring detects abnormal behavior, alerting tells the right person immediately, and logging preserves the detail needed for root-cause analysis afterward — three distinct disciplines working together as one pipeline.
The architecture is intentionally agentless and serverless, relying entirely on managed AWS services rather than custom infrastructure, and follows least-privilege IAM access throughout. It demonstrates the baseline observability pattern that real production environments build their alerting and compliance logging on top of.
This project implements a fully managed AWS monitoring, alerting, and log archival pipeline for an Amazon EC2 instance.
It uses:
- Amazon EC2
- Amazon CloudWatch (Metrics & Alarms)
- Amazon SNS (Email Notifications)
- Amazon CloudWatch Logs
- Amazon Kinesis Data Firehose
- Amazon S3 (Secure Log Archive)
The system detects sustained CPU overload, triggers automated email alerts, and archives logs securely for long-term retention — following production DevOps best practices.
- Project Overview
- Problem Statement
- Monitoring vs Logging vs Alerting
- High-Level Architecture
- Detailed Architecture Explanation
- AWS Services Used
- System Data Flow (Step-by-Step)
- EC2 Configuration Details
- CloudWatch Alarm Configuration
- SNS Notification System
- Logging & Archival Design
- Security Best Practices Applied
- Scalability & Extension Strategy
- Testing & Validation Process
- Production Relevance
- Interview Explanation (2-Minute Version)
- Resume-Ready Description
This project builds a production-ready monitoring, alerting, and logging pipeline for an Amazon EC2 instance.
The system continuously:
- Monitors CPU utilization
- Detects sustained abnormal usage
- Sends automated email alerts
- Archives logs securely in Amazon S3
The architecture reflects how real-world DevOps teams implement baseline observability in cloud environments.
In production systems:
- Servers rarely fail instantly
- Performance degrades gradually
- Resource exhaustion goes unnoticed
- Small issues escalate into outages
Without monitoring:
- Teams discover problems after users complain
- Downtime increases
- Root cause analysis becomes difficult
This project answers a critical operational question:
How do we detect infrastructure issues before customers experience downtime?
Understanding the distinction is essential.
Monitoring tracks numeric health indicators.
Example:
CPUUtilization = 82%
Monitoring answers:
- Is the system healthy right now?
Logging stores detailed event records.
Examples:
- Application errors
- System events
- Security logs
Logging answers:
- What exactly happened?
- Why did it happen?
Alerting sends notifications when abnormal conditions occur.
Example:
- Send email if CPU > 70% for 10 minutes
Alerting answers:
- Who needs to act immediately?
Monitoring detects abnormal behavior. Alerting triggers response. Logging supports root cause analysis.
This project prioritizes monitoring and alerting first, then extends to logging and archival.
Amazon EC2 (t2.micro)
│
▼
Amazon CloudWatch (Metrics)
│
▼
CloudWatch Alarm
│
▼
Amazon SNS Topic
│
▼
Email Notification
CloudWatch Logs
│
▼
Subscription Filter
│
▼
Amazon Kinesis Data Firehose
│
▼
Amazon S3 (Private | Encrypted | Long-Term Storage)
- Instance Type: t2.micro (Free-tier eligible)
- Represents production server
- Generates default infrastructure metrics
- No monitoring agent installed
Automatically collects:
- CPUUtilization
- NetworkIn
- NetworkOut
- DiskReadOps
- DiskWriteOps
No configuration required for default metrics.
Configuration:
- Threshold: CPU > 70%
- Period: 5 minutes
- Evaluation Periods: 2
Meaning:
CPU must remain above 70% for 10 continuous minutes before triggering.
This avoids false positives caused by temporary spikes.
Alarm states:
- OK
- ALARM
- INSUFFICIENT_DATA
When alarm enters ALARM state:
- CloudWatch publishes event to SNS
- SNS sends notification to subscribers
Configuration:
- Protocol: Email
- Subscription must be confirmed
SNS decouples monitoring logic from notification delivery.
Used to centralize logs from AWS-managed services.
Note:
EC2 does not send OS logs automatically without CloudWatch Agent.
Acts as a managed log delivery pipeline.
Capabilities:
- Buffers log data
- Compresses logs
- Retries failed deliveries
- Streams to S3
Fully managed and auto-scaling.
Used for:
- Long-term storage
- Compliance
- Auditing
Best practices applied:
- Private bucket
- Encryption enabled
- Lifecycle rules configured
- EC2 runs workload.
- AWS hypervisor collects metrics.
- CloudWatch stores metric data.
- Alarm evaluates CPU usage.
- CPU exceeds threshold for 10 minutes.
- Alarm changes to ALARM state.
- SNS sends email notification.
- Logs captured in CloudWatch Logs.
- Subscription filter streams logs to Firehose.
- Firehose delivers compressed logs to S3.
Entire workflow is automated and serverless.
Inbound:
- SSH (Port 22) restricted to specific IP
Outbound:
- Allow all (required for AWS API communication)
- No access keys stored
- Least privilege permissions
- Secure service-to-service authentication
All services are deployed within the same AWS region for compatibility and performance.
| Parameter | Value |
|---|---|
| Metric | CPUUtilization |
| Threshold | > 70% |
| Period | 5 Minutes |
| Evaluation Periods | 2 |
| Total Duration | 10 Minutes |
Why 70%?
- Below 70% is generally safe for burstable instances
- Sustained 70%+ indicates potential overload
Flow:
CloudWatch Alarm → SNS Topic → Email
Important:
Email subscriptions must be confirmed before receiving alerts.
CloudWatch Logs cannot directly stream to S3.
Firehose is required as the managed delivery service.
Benefits:
- Near real-time streaming
- Built-in compression
- Automatic scaling
- No infrastructure management
- IAM roles instead of access keys
- Least privilege policies
- S3 encryption enabled
- Private bucket configuration
- Log lifecycle management
- No hardcoded credentials
- Single-region consistency
This architecture can be extended to:
- Monitor multiple EC2 instances using Auto Scaling Groups
- Add memory/disk monitoring using CloudWatch Agent
- Integrate Slack or PagerDuty via SNS
- Implement centralized logging across multiple AWS accounts
- Deploy Multi-AZ infrastructure
- Add dashboards for real-time visualization
- Generate artificial CPU load
- Verify metric increase
- Confirm alarm transitions
- Confirm SNS subscription
- Trigger alarm
- Validate email receipt
- Check S3 bucket
- Verify compressed log files
- Validate timestamps and folder structure
This architecture reflects baseline observability patterns used in real organizations.
It demonstrates:
- Managed service utilization
- Cost-efficient design
- Secure architecture
- Scalable monitoring pipeline
It forms the foundation for enterprise-level observability systems.
This project implements a production-grade EC2 monitoring and logging system using AWS-managed services. CloudWatch collects infrastructure metrics and evaluates CPU thresholds using alarms. When sustained abnormal usage is detected, SNS triggers automated email alerts. Logs from AWS-managed services are centralized in CloudWatch Logs and streamed to S3 via Kinesis Data Firehose for secure long-term archival. The architecture avoids agents, uses IAM role-based access, and follows least-privilege security best practices.
Designed and implemented a production-grade AWS EC2 monitoring, alerting, and logging architecture using CloudWatch, SNS, S3, and Kinesis Data Firehose. Built an automated observability pipeline with CPU-based alarms, real-time email notifications, centralized log ingestion, and secure long-term archival while applying IAM least-privilege principles and cloud security best practices.