Add Observability Plugin for Comprehensive Monitoring

[mrgb7]

Description

Add a new observability plugin that provides comprehensive monitoring capabilities for Kubernetes clusters created with playground.

Features

The observability plugin should install and configure the following components:

Metrics

• Victoria metrics single for metrics collection and storage
• Node Exporter for node-level metrics
• Kube-state-metrics for Kubernetes object metrics
• Custom ServiceMonitors for application metrics

Traces

• Jaeger or OpenTelemetry Collector for distributed tracing
• Trace ingestion and storage capabilities
• Integration with application tracing libraries
• Otel collector to collect traces from the cluster

Logs

• Fluent Bit or Fluentd for log aggregation
• Victoria logs storage and indexing
• Log parsing and enrichment capabilities
• Service Monitor
• Alert Manager

Dashboards

• Grafana for visualization and dashboarding
• Pre-configured dashboards for:
  • Cluster overview and health
  • Node and pod resource utilization
  • Application performance metrics
  • Network and storage metrics
  • Custom application dashboards

Implementation Requirements

• Follow the existing plugin architecture pattern in internal/plugins/

• Support configurable installation options (e.g., lightweight vs full observability stack)

• Provide reasonable default configurations

• Include health checks and dependency validation

• Support both development and production configurations

• Integrate with existing cluster management commands

• update ingress plugin to expose all components UI e.g. victoria metrics and logs , jaeger , grafana with local dns like argocd and also the tls

• Enable pvc for the components data

• update chaos.sh to include test install for the observability

• Update chaos.sh to add test for all observability components and check if each K8s objects are up and running

• Acceptance Criteria

• Plugin can be installed via playground cluster plugin add observability

• All monitoring components are properly configured and running

• Grafana dashboards are accessible and populated with data

• Documentation includes setup and usage instructions

• Tests cover plugin installation and basic functionality

• Plugin follows existing code patterns and conventions

Additional Context

This plugin would significantly enhance the development and debugging experience by providing comprehensive visibility into cluster and application performance.