Skip to content

Latest commit

 

History

History
302 lines (266 loc) · 10.7 KB

File metadata and controls

302 lines (266 loc) · 10.7 KB

Pulse — OpenTelemetry Observability Stack

Vision

A ready-to-use, Docker-based observability stack built on OpenTelemetry standards. One docker compose up gives you full metrics, logs, and traces for any application. Based on a Bachelor thesis implementation of an OTel-native monitoring architecture.

This is NOT a SaaS — it's an open-source toolkit that makes it trivial to add production-grade observability to any project.

Architecture

┌──────────────┐     ┌──────────────────────┐     ┌─────────────┐
│  Your App    │     │  OTel Collector      │     │  Backends   │
│              │     │                      │     │             │
│  OTel Agent  │────▶│  Receivers           │────▶│ Prometheus  │──┐
│  (auto-      │OTLP │  Processors          │     │ (Metrics)   │  │
│  instrument) │     │  Exporters           │     │             │  │
│              │     │                      │────▶│ Jaeger      │  ├──▶ Grafana
│              │     │                      │     │ (Traces)    │  │    (Unified
│              │     │                      │────▶│ OpenSearch  │  │     Dashboard)
│              │     │                      │     │ (Logs)      │──┘
└──────────────┘     └──────────────────────┘     └─────────────┘

Tech Stack

  • OpenTelemetry Collector: Central hub for receiving, processing, and exporting telemetry
  • Prometheus: Metrics storage and querying (PromQL)
  • Jaeger: Distributed tracing UI and storage
  • OpenSearch: Log aggregation and full-text search
  • Grafana: Unified dashboards connecting all backends
  • Docker Compose: Single-command deployment

Repository Structure

pulse-observability/
├── docker-compose.yml              # Main stack definition
├── docker-compose.demo.yml         # Demo app overlay
├── .env.example                    # Environment variables
├── README.md                       # Main documentation
│
├── collector/
│   ├── otel-collector-config.yml   # OTel Collector pipeline config
│   └── README.md                   # Collector configuration docs
│
├── prometheus/
│   ├── prometheus.yml              # Prometheus scrape config
│   ├── alert-rules.yml             # Example alerting rules
│   └── README.md
│
├── grafana/
│   ├── provisioning/
│   │   ├── datasources/
│   │   │   └── datasources.yml    # Auto-provision Prometheus, Jaeger, OpenSearch
│   │   └── dashboards/
│   │       ├── dashboards.yml     # Dashboard provisioning config
│   │       ├── system-overview.json     # System metrics dashboard
│   │       ├── jvm-metrics.json         # JVM performance dashboard
│   │       ├── http-requests.json       # HTTP request metrics & latency
│   │       ├── trace-analysis.json      # Trace overview dashboard
│   │       └── log-analysis.json        # Log patterns dashboard
│   └── README.md
│
├── opensearch/
│   ├── opensearch.yml              # OpenSearch node config
│   ├── index-template.json         # Log index template
│   └── README.md
│
├── jaeger/
│   └── README.md                   # Jaeger config notes
│
├── demo/
│   ├── spring-boot-app/            # Sample Spring Boot app with OTel agent
│   │   ├── Dockerfile
│   │   ├── pom.xml
│   │   └── src/
│   │       └── main/java/com/pulse/demo/
│   │           ├── DemoApplication.java
│   │           ├── controller/
│   │           │   ├── OrderController.java    # REST endpoints
│   │           │   └── HealthController.java
│   │           ├── service/
│   │           │   ├── OrderService.java       # Business logic with delays
│   │           │   └── NotificationService.java
│   │           └── model/
│   │               └── Order.java
│   ├── load-generator/
│   │   ├── Dockerfile
│   │   └── generate-load.sh        # Script that hits demo endpoints
│   └── README.md                   # How to run the demo
│
├── docs/
│   ├── architecture.md             # Detailed architecture explanation
│   ├── getting-started.md          # Quick start guide
│   ├── instrumenting-your-app.md   # How to add OTel to your own app
│   ├── configuration-guide.md      # Deep dive into each config file
│   ├── thesis-context.md           # Background from the thesis
│   ├── diagrams/                   # Architecture diagrams (.excalidraw + .svg)
│   └── images/                     # Dashboard screenshots
│
├── scripts/
│   ├── setup.sh                    # First-time setup script
│   ├── start.sh                    # Start the stack
│   ├── stop.sh                     # Stop the stack
│   ├── demo.sh                     # Start stack + demo app + load generator
│   └── health-check.sh             # Verify all services are running
│
└── .gitignore

Docker Compose Services

Core Stack (docker-compose.yml)

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    ports:
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver
      - "8889:8889"   # Prometheus exporter (collector's own metrics)
    
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    
  jaeger:
    image: jaegertracing/jaeger:latest  # v2 all-in-one
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317"         # OTLP gRPC (internal)
    
  opensearch:
    image: opensearchproject/opensearch:2
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node
      - plugins.security.disabled=true
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=Pulse@2026!
    
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2
    ports:
      - "5601:5601"
    
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=pulse
      - GF_AUTH_ANONYMOUS_ENABLED=true

Demo Overlay (docker-compose.demo.yml)

services:
  demo-app:
    build: ./demo/spring-boot-app
    environment:
      - OTEL_SERVICE_NAME=pulse-demo
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
      - JAVA_TOOL_OPTIONS=-javaagent:/app/opentelemetry-javaagent.jar
    ports:
      - "8080:8080"
    
  load-generator:
    build: ./demo/load-generator
    depends_on:
      - demo-app

OTel Collector Config

The collector config should demonstrate the 3-pillar pipeline:

Receivers

  • OTLP (gRPC on 4317, HTTP on 4318)

Processors

  • batch (group telemetry for efficient export)
  • memory_limiter (prevent OOM)
  • resource (add common attributes)
  • filter (example log level filtering)

Exporters

  • prometheusremotewrite → Prometheus (metrics)
  • otlp → Jaeger (traces)
  • opensearch → OpenSearch (logs)

Pipelines

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [opensearch]

Demo App

A simple Spring Boot 3 app that generates realistic telemetry:

  • GET /api/orders — list orders (fast)
  • POST /api/orders — create order (medium latency, ~200ms)
  • GET /api/orders/{id} — get order (sometimes slow, simulates DB query)
  • POST /api/orders/{id}/notify — send notification (slow, ~500ms, sometimes fails)
  • GET /health — health check

The app uses OpenTelemetry Java Agent for auto-instrumentation (zero code changes). The agent generates:

  • HTTP request metrics (latency, status codes, throughput)
  • JVM metrics (heap, GC, threads)
  • Traces for every request (with spans for service calls)
  • Logs forwarded via OTel

Grafana Dashboards (pre-provisioned)

1. System Overview

  • Request rate (req/sec)
  • Error rate (%)
  • P50/P95/P99 latency
  • Active services
  • JVM heap usage

2. JVM Metrics

  • Heap used vs committed vs max
  • GC pause time
  • Thread count
  • Class loading

3. HTTP Requests

  • Request duration histogram
  • Status code distribution
  • Top endpoints by latency
  • Error breakdown

4. Trace Analysis

  • Service map (if available)
  • Trace duration distribution
  • Slowest traces
  • Links to Jaeger for detail view

5. Log Analysis

  • Log level distribution (pie chart)
  • Error logs timeline
  • Log volume over time
  • Links to OpenSearch Dashboards

README.md Content

Badges

  • Docker Compose, OpenTelemetry, Prometheus, Grafana, Jaeger, OpenSearch

Sections

  1. What is Pulse? (2 paragraphs)
  2. Architecture diagram (embedded image)
  3. Quick Start (3 commands: clone, copy .env, docker compose up)
  4. Access the UIs (table with URLs)
  5. Demo (how to run demo app + load generator)
  6. Add Observability to Your App (link to docs)
  7. Stack Components (brief description of each tool)
  8. Configuration (link to docs)
  9. Thesis Context (brief mention + link to docs/thesis-context.md)
  10. License (MIT)

Quick Start should be:

git clone https://github.com/TheLoop705/pulse-observability.git
cd pulse-observability
cp .env.example .env
docker compose up -d

# Optional: Run demo app with load generator
docker compose -f docker-compose.yml -f docker-compose.demo.yml up -d

Access UIs table:

Service URL Credentials
Grafana http://localhost:3000 admin / pulse
Prometheus http://localhost:9090
Jaeger http://localhost:16686
OpenSearch Dashboards http://localhost:5601
Demo App http://localhost:8080

Key Points

  • Everything must work with a single docker compose up -d
  • Pre-provisioned Grafana datasources + dashboards (no manual setup)
  • Demo app demonstrates real telemetry without requiring the user to instrument anything
  • Load generator creates realistic traffic patterns
  • All configs well-commented
  • docs/ folder explains the WHY behind each decision (thesis context)
  • .env.example for all configurable values
  • Health check script to verify stack is running
  • Professional README with architecture diagram description