Skip to content

README.md

Latest commit

 

History

History
642 lines (516 loc) · 25.7 KB

README.md

File metadata and controls

642 lines (516 loc) · 25.7 KB
Raw

LLM Observability Stack

A comprehensive observability stack for monitoring LLM inference and training workloads on Kubernetes with NVIDIA GPUs. This project provides Prometheus metrics collection, Grafana dashboards, GPU monitoring with DCGM, Kubernetes cluster monitoring, and custom LLM metrics.

Prometheus Grafana Kubernetes NVIDIA Oracle Cloud

Table of Contents

Overview

Monitoring LLM workloads requires visibility into multiple layers:

  • GPU Metrics: Utilization, memory, temperature, power, health status, XID errors
  • Inference Metrics: Latency, throughput, tokens per second, KV cache, queue depth
  • Training Metrics: Loss, learning rate, throughput, gradient norm, checkpoints
  • Kubernetes Metrics: Pods, nodes, deployments, services, health conditions
  • System Metrics: CPU, memory, network, disk I/O

This stack provides end-to-end observability for production LLM deployments.

Architecture

+-------------------+     +-------------------+     +-------------------+
|   GPU Node 1      |     |   GPU Node 2      |     |   Load Balancer   |
|   (2x A10 GPU)    |     |   (2x A10 GPU)    |     |                   |
+-------------------+     +-------------------+     +-------------------+
        |                         |                         |
        v                         v                         v
+---------------------------------------------------------------+
|                     OKE Kubernetes Cluster                     |
|  +----------------------------------------------------------+ |
|  |  Namespaces:                                              | |
|  |  - llm-observability (Prometheus, Grafana, AlertManager)  | |
|  |  - nim-bench (vLLM, TGI inference servers)                | |
|  |  - training (LLM training workloads)                      | |
|  |  - load-gen (Load generators)                             | |
|  +----------------------------------------------------------+ |
+---------------------------------------------------------------+
        |
        v
+-------------------+
|  FSS Storage      |
|  (Model cache,    |
|  checkpoints)     |
+-------------------+

Components

Component Version Port Purpose
Prometheus 2.47.0 9090 Metrics collection and storage
Grafana 10.2.0 3000 Visualization and dashboards
DCGM Exporter 3.3.0 9400 NVIDIA GPU metrics
kube-state-metrics 2.10.1 8080 Kubernetes cluster metrics
AlertManager 0.26.0 9093 Alert routing and notifications
vLLM 0.6.4 8000 LLM inference server
TGI 2.0.4 8000 HuggingFace Text Generation
Training Workload Custom 8000 PyTorch training with metrics

Test Environment

Component Specification
Platform Oracle Cloud Infrastructure (OCI)
Kubernetes Oracle Kubernetes Engine (OKE)
GPU Nodes 2 worker nodes
GPU Model NVIDIA A10 (2 per node)
GPU Memory 24 GB GDDR6 per GPU
Total GPUs 4x NVIDIA A10 (96 GB VRAM)
Storage OCI File Storage Service (FSS)

Quick Start

Prerequisites

  • Kubernetes cluster with GPU nodes
  • NVIDIA GPU drivers installed
  • GPU node labels: nvidia.com/gpu: "true"
  • GPU node taints: nvidia.com/gpu:NoSchedule
  • kubectl configured

Step 1: Create Namespace and RBAC

kubectl apply -f deployments/common/00-namespace.yaml

Step 2: Deploy DCGM Exporter (GPU Metrics)

kubectl apply -f deployments/dcgm-exporter/

Step 3: Deploy kube-state-metrics (Kubernetes Metrics)

kubectl apply -f deployments/kube-state-metrics/

Step 4: Deploy Prometheus

kubectl apply -f deployments/prometheus/

Step 5: Deploy Grafana

kubectl apply -f deployments/grafana/

Step 6: Deploy AlertManager (Optional)

kubectl apply -f deployments/alertmanager/

Step 7: Access Dashboards

# Access URLs (with LoadBalancer)
Grafana: http://<EXTERNAL_IP>:3000 (admin/admin)
Prometheus: http://<EXTERNAL_IP>:9090

One-Command Deployment

./scripts/deploy-all.sh

Dashboards

1. Cluster Management Home (Default)

URL: /d/cluster-home/cluster-management-home

Executive overview dashboard providing a single-pane-of-glass view for LLM infrastructure management. Shows cluster health, active training jobs with loss metrics, inference server status, and GPU utilization across all nodes.

Cluster Management Home

Section Metrics
Cluster Overview Total Nodes, GPUs, Avg GPU Util, Running/Pending/Failed Pods
Training Jobs Active Jobs, Progress %, Loss, Throughput (tok/s)
LLM Inference Servers, Requests/sec, Latency, Tokens Generated
Health & Alerts GPU Errors, Thermal/Power Violations, Node Conditions
GPU Utilization Per-node GPU usage over time
Workload Status Training jobs table, Inference servers table

2. GPU Cluster Overview

URL: /d/gpu-cluster-overview/gpu-cluster-overview

Comprehensive GPU monitoring dashboard showing per-GPU health status, utilization, memory, temperature, and power metrics. Distinguishes between busy (>5% util) and idle GPUs with detailed breakdowns.

GPU Cluster Overview

Section Metrics
Cluster Summary Total GPUs, Busy/Idle GPUs, Cluster Avg Util, Memory Used/Free
GPU Health Alerts XID Errors, Thermal Throttling, Power Violations, Row Remap Failures
GPU Health Table Per-GPU: Node, Util %, Temp, XID Errors, Thermal/Power Violations
Busy GPUs Details and timeline for GPUs with >5% utilization
Idle GPUs Details and available memory for idle GPUs

3. OKE Cluster Overview

URL: /d/oke-cluster-overview/oke-cluster-overview

Kubernetes cluster health monitoring showing node status, pod distribution by namespace, and resource allocation. Includes node health alerts and detailed namespace resource breakdown.

OKE Cluster Overview

Section Metrics
Cluster Summary Nodes, Pods, Services, Deployments, Namespaces
Node Health Alerts DiskPressure, MemoryPressure, PIDPressure, NotReady
Pod Status Pods by Phase (Running/Pending), Pods by Namespace
Node Details Nodes Ready, Pods per Node, Node Info Table
Namespace Overview Resources by Namespace (Pods, Services, Deployments)

4. Training Cluster Dashboard

URL: /d/training-cluster/training-cluster-dashboard

Comprehensive LLM training monitoring with loss curves, throughput metrics, timing analysis, and scalability metrics. Tracks training progress, learning rate schedules, gradient norms, and GPU resource utilization.

Training Cluster - Loss Metrics

Training overview with loss metrics, learning rate schedule, throughput, and timing analysis

Training Cluster - Scalability

GPU resources, gradient optimization, model configuration, and scalability metrics

Section Metrics
Overview Status, Epoch, Step, Progress %, Loss, Tokens/sec, Learning Rate, GPUs
Loss Metrics Training/Validation loss over time, Learning rate schedule
Throughput Tokens/sec, Samples processed, Total samples/tokens
Timing Batch time breakdown (Forward/Backward/Optimizer/Data Loading)
GPU Resources Memory usage per GPU, Memory %, GPU Utilization (DCGM)
Gradients Gradient norm over time, Gradient clipping events, Checkpoint stats
Model Config Parameters, Trainable params, Batch size, Sequence length, World size
Scalability Throughput vs GPUs, Scaling efficiency %, Communication overhead

5. LLM Inference Dashboard

URL: /d/llm-inference-v2/llm-inference

Comprehensive inference server monitoring with throughput metrics, latency percentiles (P50/P90/P99), request queue status, and KV cache utilization. Tracks tokens generated and request rates in real-time.

LLM Inference Dashboard

Section Metrics
Overview Requests/sec, Running/Waiting requests, KV Cache %, Tokens Total
Throughput Prompt tokens/s, Generation tokens/s over time
Latency P50/P90/P99 latency distribution, Average request latency
Queue & Cache Request queue status (Running/Waiting/Swapped), KV Cache usage
Tokens Cumulative prompt/generated tokens, Token rate over time

6. GPU Overview Dashboard

URL: /d/gpu-overview/gpu-overview

Single-node GPU monitoring:

Section Metrics
Gauges GPU Utilization, Temperature, Power Usage, Memory Usage
Time Series GPU Utilization Over Time, Memory Over Time
Thermal Temperature trends per GPU
Power Power usage trends per GPU

7. vLLM Inference Metrics Dashboard

URL: /d/vllm-inference/vllm-inference-metrics

vLLM-specific monitoring dashboard showing detailed token throughput, request latency percentiles, queue status, KV cache utilization, and preemption events. Optimized for vLLM inference server monitoring.

vLLM Inference Metrics

Section Metrics
Throughput Generation tokens/s, Prompt tokens/s over time
Latency P50/P95/P99 request latency distribution
Queue Running/Waiting/Swapped requests over time
Cache GPU/CPU KV cache usage %
Token Rate Prompt/Generation tokens rate
Requests Requests per second over time
Preemptions Total preemption events over time

8. LLM Inference (Simple)

URL: /d/llm-inference/llm-inference

Lightweight inference dashboard:

Section Metrics
Stats Tokens/sec, P95 Latency, Queue Depth, Error Rate
Latency P50/P95/P99 distribution over time
Requests Requests per second over time
Tokens Token generation rate by model

Metrics Reference

GPU Metrics (DCGM Exporter)

Metric Description Unit Alert Threshold
DCGM_FI_DEV_GPU_UTIL GPU compute utilization % -
DCGM_FI_DEV_FB_USED Framebuffer memory used bytes -
DCGM_FI_DEV_FB_FREE Framebuffer memory free bytes < 10% warning
DCGM_FI_DEV_FB_TOTAL Total framebuffer memory bytes -
DCGM_FI_DEV_GPU_TEMP GPU temperature °C > 80°C warning, > 85°C critical
DCGM_FI_DEV_MEMORY_TEMP Memory temperature °C -
DCGM_FI_DEV_POWER_USAGE Power draw W -
DCGM_FI_DEV_SM_CLOCK SM clock speed MHz -
DCGM_FI_DEV_MEM_CLOCK Memory clock speed MHz -
DCGM_FI_DEV_MEM_COPY_UTIL Memory copy utilization % -
DCGM_FI_DEV_ENC_UTIL Encoder utilization % -
DCGM_FI_DEV_DEC_UTIL Decoder utilization % -
DCGM_FI_DEV_XID_ERRORS XID error count count > 0 critical
DCGM_FI_DEV_THERMAL_VIOLATION Thermal throttling events count > 0 warning
DCGM_FI_DEV_POWER_VIOLATION Power throttling events count > 0 warning
DCGM_FI_DEV_RELIABILITY_VIOLATION Reliability violations count > 0 critical
DCGM_FI_DEV_BOARD_LIMIT_VIOLATION Board limit violations count > 0 warning
DCGM_FI_DEV_LOW_UTIL_VIOLATION Low utilization events count -
DCGM_FI_DEV_SYNC_BOOST_VIOLATION Sync boost violations count -
DCGM_FI_DEV_ROW_REMAP_FAILURE Row remap failures count > 0 critical
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS Correctable remapped rows count -
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS Uncorrectable remapped rows count > 0 critical
DCGM_FI_DEV_ECC_SBE_AGG_TOTAL Single-bit ECC errors count -
DCGM_FI_DEV_ECC_DBE_AGG_TOTAL Double-bit ECC errors count > 0 critical
DCGM_FI_DEV_PCIE_REPLAY_COUNTER PCIe replay count count -
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL NVLink bandwidth bytes -
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION Total energy consumed mJ -
DCGM_FI_PROF_GR_ENGINE_ACTIVE Graphics engine active ratio -
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE Tensor core active ratio -
DCGM_FI_PROF_PIPE_FP32_ACTIVE FP32 pipe active ratio -
DCGM_FI_PROF_PIPE_FP16_ACTIVE FP16 pipe active ratio -
DCGM_FI_PROF_DRAM_ACTIVE DRAM active ratio -

vLLM Inference Metrics

Metric Type Description
vllm:request_success_total Counter Total successful requests
vllm:prompt_tokens_total Counter Total prompt tokens processed
vllm:generation_tokens_total Counter Total tokens generated
vllm:e2e_request_latency_seconds Histogram End-to-end request latency
vllm:time_to_first_token_seconds Histogram Time to first token (TTFT)
vllm:time_per_output_token_seconds Histogram Time per output token (TPOT)
vllm:request_queue_time_seconds Histogram Time spent in queue
vllm:request_prefill_time_seconds Histogram Prefill phase time
vllm:request_decode_time_seconds Histogram Decode phase time
vllm:request_inference_time_seconds Histogram Total inference time
vllm:request_prompt_tokens Histogram Prompt tokens per request
vllm:request_generation_tokens Histogram Generated tokens per request
vllm:num_requests_running Gauge Currently running requests
vllm:num_requests_waiting Gauge Requests waiting in queue
vllm:num_requests_swapped Gauge Requests swapped to CPU
vllm:gpu_cache_usage_perc Gauge GPU KV cache usage %
vllm:cpu_cache_usage_perc Gauge CPU KV cache usage %
vllm:gpu_prefix_cache_hit_rate Gauge GPU prefix cache hit rate
vllm:cpu_prefix_cache_hit_rate Gauge CPU prefix cache hit rate
vllm:avg_prompt_throughput_toks_per_s Gauge Avg prompt throughput
vllm:avg_generation_throughput_toks_per_s Gauge Avg generation throughput
vllm:num_preemptions_total Counter Total preemptions
vllm:iteration_tokens_total Histogram Tokens per iteration

Training Metrics

Metric Type Description
training_status Gauge Status (1=running, 0=stopped, 2=completed)
training_epoch_current Gauge Current epoch number
training_step_current Gauge Current training step
training_total_steps Gauge Total training steps
training_progress_percent Gauge Training progress %
training_loss_current Gauge Current batch loss
training_loss_average Gauge Running average loss
training_learning_rate Gauge Current learning rate
training_tokens_per_second Gauge Training throughput (tok/s)
training_samples_per_second Gauge Samples per second
training_samples_total Counter Total samples processed
training_tokens_total Counter Total tokens processed
training_batch_time_seconds Histogram Time per batch
training_forward_time_seconds Gauge Forward pass time
training_backward_time_seconds Gauge Backward pass time
training_optimizer_time_seconds Gauge Optimizer step time
training_data_load_time_seconds Gauge Data loading time
training_gradient_norm Gauge Gradient L2 norm
training_gradient_norm_clipped_total Counter Gradient clipping events
training_gpu_memory_used_bytes Gauge GPU memory used (per GPU)
training_gpu_memory_reserved_bytes Gauge GPU memory reserved
training_gpu_memory_percent Gauge GPU memory usage %
training_checkpoint_time_seconds Gauge Checkpoint save time
training_checkpoint_total Counter Total checkpoints saved
training_checkpoint_size_bytes Gauge Checkpoint size
training_model_parameters_total Gauge Total model parameters
training_trainable_parameters_total Gauge Trainable parameters
training_batch_size Gauge Batch size
training_sequence_length Gauge Sequence length
training_world_size Gauge Distributed world size
training_rank Gauge Current worker rank
training_communication_time_seconds Gauge All-reduce time

Kubernetes Metrics (kube-state-metrics)

Metric Description
kube_node_info Node information
kube_node_status_condition Node conditions (Ready, DiskPressure, MemoryPressure)
kube_pod_info Pod information
kube_pod_status_phase Pod phase (Running, Pending, Failed)
kube_pod_container_resource_requests Container resource requests
kube_pod_container_resource_limits Container resource limits
kube_deployment_status_replicas Deployment replicas
kube_deployment_status_replicas_available Available replicas
kube_daemonset_status_desired_number_scheduled DaemonSet desired
kube_daemonset_status_number_ready DaemonSet ready
kube_service_info Service information
kube_namespace_created Namespace creation time

Alerting

GPU Health Alerts

Alert Severity Condition
GPUXIDError critical DCGM_FI_DEV_XID_ERRORS > 0
GPUHighTemperature warning DCGM_FI_DEV_GPU_TEMP > 80 for 5m
GPUCriticalTemperature critical DCGM_FI_DEV_GPU_TEMP > 85 for 2m
GPUThermalThrottling warning DCGM_FI_DEV_THERMAL_VIOLATION > 0
GPUPowerThrottling warning DCGM_FI_DEV_POWER_VIOLATION > 0
GPUMemoryExhausted critical DCGM_FI_DEV_FB_USED/FB_TOTAL > 0.95
GPURowRemapFailure critical DCGM_FI_DEV_ROW_REMAP_FAILURE > 0
GPUECCError critical DCGM_FI_DEV_ECC_DBE_AGG_TOTAL > 0

Kubernetes Health Alerts

Alert Severity Condition
NodeNotReady critical kube_node_status_condition{condition="Ready",status="false"}
NodeDiskPressure warning kube_node_status_condition{condition="DiskPressure",status="true"}
NodeMemoryPressure warning kube_node_status_condition{condition="MemoryPressure",status="true"}
PodFailed warning kube_pod_status_phase{phase="Failed"} > 0
PodPending warning kube_pod_status_phase{phase="Pending"} > 0 for 10m

Inference Alerts

Alert Severity Condition
HighInferenceLatency warning P99 latency > 10s for 5m
InferenceQueueBacklog warning vllm:num_requests_waiting > 20 for 5m
KVCacheExhausted critical vllm:gpu_cache_usage_perc > 0.95

Training Alerts

Alert Severity Condition
TrainingLossSpike warning Loss increased by 50%
TrainingStalled critical No progress for 10m
GradientExploding critical training_gradient_norm > 100

Workload Deployment

Inference Workload (vLLM)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama
  namespace: nim-bench
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.6.4
        args:
          - "--model=/models/llama-3-8b-instruct"
          - "--served-model-name=llama-3-8b-instruct"
        resources:
          limits:
            nvidia.com/gpu: "1"
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule

Training Workload

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-training
  namespace: training
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
    spec:
      containers:
      - name: training
        image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
        resources:
          limits:
            nvidia.com/gpu: "1"
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule

Project Structure

llm-observability-stack/
├── README.md
├── LLM_Observability_Complete_Guide.md
├── deployments/
│   ├── common/
│   │   └── 00-namespace.yaml
│   ├── prometheus/
│   │   ├── 01-prometheus-config.yaml
│   │   ├── 02-prometheus-rules.yaml
│   │   └── 03-prometheus-deployment.yaml
│   ├── grafana/
│   │   ├── 01-grafana-config.yaml
│   │   ├── 02-grafana-dashboards.yaml
│   │   └── 03-grafana-deployment.yaml
│   ├── dcgm-exporter/
│   │   └── 01-dcgm-exporter.yaml
│   ├── kube-state-metrics/
│   │   └── 01-kube-state-metrics.yaml
│   ├── alertmanager/
│   │   ├── 01-alertmanager-config.yaml
│   │   └── 02-alertmanager-deployment.yaml
│   └── llm-exporter/
│       ├── llm_exporter.py
│       ├── Dockerfile
│       └── 01-llm-exporter-deployment.yaml
├── dashboards/
│   ├── cluster-home.json          # Executive overview dashboard
│   ├── gpu-cluster-overview.json  # Detailed GPU cluster metrics
│   ├── gpu-overview.json          # Single-node GPU monitoring
│   ├── llm-inference.json         # Simple inference dashboard
│   ├── llm-inference-v2.json      # Detailed inference dashboard
│   ├── oke-cluster-overview.json  # Kubernetes cluster health
│   ├── training-cluster.json      # Training job monitoring
│   ├── vllm-inference.json        # vLLM-specific metrics
│   └── images/                    # Dashboard screenshots
│       ├── 01-cluster-home.png
│       ├── 02-gpu-cluster-overview.png
│       ├── 03-llm-inference.png
│       ├── 04-training-cluster-loss.png
│       ├── 05-vllm-inference.png
│       ├── 06-oke-cluster-overview.png
│       └── 07-training-cluster-scalability.png
├── scripts/
│   ├── deploy-all.sh
│   ├── cleanup.sh
│   └── port-forward.sh
└── docs/
    ├── METRICS.md
    ├── ALERTING.md
    └── DEPLOYMENT_STEPS.md

Access URLs

Service URL Credentials
Grafana http://<GRAFANA_IP>:3000 admin/admin
Prometheus http://<GRAFANA_IP>:9090 -
AlertManager http://<GRAFANA_IP>:9093 -

Dashboard URLs

Dashboard URL Description
Cluster Management Home http://<GRAFANA_IP>:3000/d/cluster-home Executive overview (default)
GPU Cluster Overview http://<GRAFANA_IP>:3000/d/gpu-cluster-overview Detailed GPU cluster metrics
GPU Overview http://<GRAFANA_IP>:3000/d/gpu-overview Single-node GPU gauges
OKE Cluster Overview http://<GRAFANA_IP>:3000/d/oke-cluster-overview Kubernetes cluster health
Training Cluster http://<GRAFANA_IP>:3000/d/training-cluster Training job monitoring
LLM Inference (Detailed) http://<GRAFANA_IP>:3000/d/llm-inference-v2 Comprehensive inference metrics
LLM Inference (Simple) http://<GRAFANA_IP>:3000/d/llm-inference Basic inference monitoring
vLLM Inference http://<GRAFANA_IP>:3000/d/vllm-inference vLLM-specific metrics

Best Practices

  1. GPU Node Tolerations: Always add nvidia.com/gpu tolerations for workloads
  2. Prometheus Annotations: Add prometheus.io/scrape: "true" to all pods with metrics
  3. Shared Memory: Use /dev/shm with emptyDir for inference servers
  4. Resource Limits: Set GPU limits explicitly (nvidia.com/gpu: "1")
  5. Health Checks: Implement /health and /metrics endpoints
  6. Dashboard Refresh: Use 15-30s refresh intervals
  7. Metric Deduplication: Use max by (UUID) for DCGM metrics to avoid duplicates

Troubleshooting

Issue Cause Solution
Pods pending on GPU nodes Missing toleration Add nvidia.com/gpu toleration
No GPU metrics DCGM not running Check DCGM DaemonSet on GPU nodes
Duplicate GPU metrics Multiple scrape jobs Use max by (UUID) in queries
No Kubernetes metrics kube-state-metrics missing Deploy kube-state-metrics
404 on inference Wrong model name Check --served-model-name
Prometheus not scraping Missing annotations Add prometheus.io/scrape: "true"

References

Document Version: 2.0 Last Updated: 2026-01-19 Author: Deepak Soni - AI CoE Team