refactor: split deployment seeding and dashboard responsibilities

.github/workflows/ci.yml

@@ -12,4 +12,11 @@ jobs:
 
       - uses: azure/setup-helm@v4
 
-      - run: helm lint helm/streamline
+      - name: Install validation dependencies
+        run: |
+          helm plugin install https://github.com/helm-unittest/helm-unittest.git --version v1.1.2 --verify=false
+          sudo apt-get update
+          sudo apt-get install -y shellcheck
+
+      - name: Run complete validation
+        run: make test lint

AUDIT.md

@@ -0,0 +1,77 @@
+# Clean Code and SRP Audit
+
+## Summary
+
+- **Highest-leverage split:** separate `docker/seed-data.sh` into runtime
+  transport/readiness and demo-fixture units so SRE changes do not edit product
+  sample data.
+- `helm/streamline/templates/grafana-dashboards.yaml` packages three dashboards
+  for different operator personas in one review unit.
+- The chart's long `values.yaml` is a deliberate public configuration surface,
+  not an SRP violation; splitting it would fragment the Helm API.
+- Monitoring rules and dashboard JSON are long but cohesive observability
+  artifacts and should remain independent rather than share a generic library.
+- Cross-repo metric-name drift is higher severity than local structure, but it
+  requires a core/deploy contract decision and is therefore deferred.
+
+## Findings
+
+| ID | Location | Category | Severity | Actors in conflict | Cost | Size | Behavior risk |
+|---|---|---|---|---|---|---|---|
+| DEP-SRP-1 | `docker/seed-data.sh:1-181` | SRP | P2 | demo/product content; runtime operations | Readiness, HTTP transport, topic provisioning, and four fixture catalogs change in one script; an ops retry change conflicts with demo-content edits. | M | Medium |
+| DEP-SRP-2 | `helm/streamline/templates/grafana-dashboards.yaml:1-179` | SRP | P2 | platform overview; consumer operations; topic operations | Three independently reviewed dashboards share one template and cannot be tested or changed in isolation. | M | Low |
+| DEP-CC-1 | `docker/seed-data.sh:13-45` | Hidden error policy | P2 | demo idempotency; deployment diagnostics | Topic/message HTTP failures are intentionally swallowed, but the function names and final success summary do not expose partial failure. | S | Medium |
+| DEP-D-1 | `helm/streamline/templates/grafana-dashboards.yaml`; core `src/metrics/mod.rs` | Cross-repo contract | P1 | core metrics owners; deployment/SRE | Several dashboard and alert queries use metric names absent from core, producing silent empty panels. | M | High |
+
+## Actor and State Partition
+
+### `docker/seed-data.sh`
+
+| Partition | Functions/state | Actor/axis |
+|---|---|---|
+| Runtime | `HOST`, `HTTP_PORT`, `BASE_URL`, `TOTAL_MESSAGES`, `wait_for_server`, `create_topic`, `produce_message` | SRE; transport, readiness, retry/error policy |
+| Fixtures | `TOPICS`, `create_topics`, `seed_events`, `seed_logs`, `seed_metrics`, `seed_orders` | Product/demo; sample domain content |
+| Orchestration/presentation | banner, call ordering, final summary | Demo operator; workflow presentation |
+
+Resulting units: `seed-runtime.sh`, `seed-fixtures.sh`, and the existing
+`seed-data.sh` as the orchestration entry point. The new units own real
+decisions and are independently characterizable; no forwarding class or
+interface is introduced.
+
+### Grafana dashboard template
+
+The three ConfigMaps share chart metadata but not dashboard content. Resulting
+units: `grafana-overview.yaml`, `grafana-consumer-lag.yaml`, and
+`grafana-topics.yaml`, with one narrowly named Helm helper for the shared
+Grafana discovery metadata.
+
+## Ordered Refactor Sequence
+
+1. Add shell characterization coverage for topic/message call counts and final
+   summary output.
+2. Add Helm unit coverage for all three dashboard ConfigMaps.
+3. Move runtime and fixture functions unchanged into sourced shell units.
+4. Name retry constants and fixture orchestration after the move.
+5. Move each Grafana ConfigMap unchanged into its own template.
+6. Consolidate only the shared Grafana discovery metadata in a Helm helper.
+7. Run `make test lint` after every commit.
+
+## Deferred
+
+- Metric names and dashboard PromQL require a cross-repo contract decision; do
+  not rename core or deploy metrics in this repository alone.
+- Live smoke tests require a buildable/published Streamline image and remain
+  outside this local structural refactor.
+- The seeder's swallowed HTTP failures are preserved until product decides
+  whether idempotent reruns or fail-fast diagnostics are the public behavior.
+
+## Out of Scope
+
+- `helm/streamline/values.yaml`: one public chart-configuration actor despite
+  its length.
+- `helm/streamline/values.schema.json`: schema for that same public contract.
+- Grafana dashboard JSON and Prometheus alert files: each has one
+  observability/presentation axis.
+- Docker Compose variants and raw Kubernetes manifests: separate deployment
+  products that only resemble one another; deduplicating them would couple
+  different operational actors.

Makefile

@@ -1,4 +1,4 @@
-.PHONY: build test lint clean help docker helm-lint smoke-test helm-template helm-test helm-validate
+.PHONY: build test lint clean help docker helm-lint smoke-test helm-template helm-test helm-validate shell-syntax shell-tests shellcheck
 
 help: ## Show this help
 	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}'
@@ -8,7 +8,7 @@ build: docker ## Build Docker image
 docker: ## Build Streamline Docker image
 	docker build -t streamline:dev .
 
-test: helm-lint helm-validate ## Run validation tests
+test: helm-lint helm-validate helm-test shell-syntax shell-tests ## Run validation tests
 	docker compose config --quiet
 	docker compose -f docker-compose.demo.yml config --quiet
 	docker compose -f docker-compose.test.yml config --quiet
@@ -17,7 +17,7 @@ smoke-test: docker ## Run smoke tests against a live Streamline instance
 	docker compose -f docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from smoke-test
 	docker compose -f docker-compose.test.yml down -v
 
-lint: helm-lint ## Run all linting
+lint: helm-lint shellcheck ## Run all linting
 
 helm-lint: ## Lint Helm chart
 	helm lint helm/streamline
@@ -43,6 +43,15 @@ helm-validate: ## Validate all Helm templates render with various value combinat
 helm-test: ## Run helm-unittest tests (requires helm-unittest plugin)
 	helm unittest helm/streamline
 
+shell-syntax: ## Validate tracked shell script syntax
+	@for file in $$(git ls-files '*.sh'); do bash -n "$$file"; done
+
+shell-tests: ## Run shell characterization tests
+	bash tests/seed-data_test.sh
+
+shellcheck: ## Run ShellCheck on tracked shell scripts
+	shellcheck -x -P docker $$(git ls-files '*.sh')
+
 clean: ## Clean up containers
 	docker compose down -v 2>/dev/null || true
 
@@ -51,4 +60,3 @@ up: ## Start Streamline via Docker Compose
 
 down: ## Stop Streamline
 	docker compose down
-# bump base image to latest Alpine

demos/edge-pilot.sh

@@ -30,7 +30,10 @@ echo "Step 1: Starting edge server + MQTT bridge..."
 docker compose -f "$COMPOSE" up -d streamline-edge
 echo "  Waiting for health..."
 for i in $(seq 1 30); do
-    curl -sf http://localhost:9094/health >/dev/null 2>&1 && break || sleep 1
+    if curl -sf http://localhost:9094/health >/dev/null 2>&1; then
+        break
+    fi
+    sleep 1
 done
 echo "  ✅ Edge server ready (Kafka:9092, HTTP:9094, MQTT:1883)"
 

docker/Dockerfile.seed

@@ -12,7 +12,7 @@ RUN apt-get update && apt-get install -y \
     jq \
     && rm -rf /var/lib/apt/lists/*
 
-COPY seed-data.sh /seed-data.sh
+COPY seed-data.sh seed-runtime.sh seed-fixtures.sh /
 RUN chmod +x /seed-data.sh
 
 ENTRYPOINT ["/seed-data.sh"]

docker/seed-data.sh

@@ -1,152 +1,13 @@
 #!/usr/bin/env bash
 set -euo pipefail
 
-HOST="${STREAMLINE_HOST:-localhost}"
-KAFKA_PORT="${STREAMLINE_KAFKA_PORT:-9092}"
-HTTP_PORT="${STREAMLINE_HTTP_PORT:-9094}"
-BASE_URL="http://${HOST}:${HTTP_PORT}"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 
-TOPICS=("demo-events" "demo-logs" "demo-metrics" "demo-orders")
-TOTAL_MESSAGES=0
-
-# --- Helpers ---
-
-wait_for_server() {
-  echo "⏳ Waiting for Streamline to be ready at ${BASE_URL}/health ..."
-  local retries=0
-  until curl -sf "${BASE_URL}/health" > /dev/null 2>&1; do
-    retries=$((retries + 1))
-    if [ "$retries" -ge 60 ]; then
-      echo "❌ Streamline did not become ready after 60 seconds"
-      exit 1
-    fi
-    sleep 1
-  done
-  echo "✅ Streamline is healthy"
-  echo ""
-}
-
-create_topic() {
-  local topic="$1"
-  local partitions="${2:-1}"
-  echo "  Creating topic '${topic}' (partitions: ${partitions})..."
-  curl -sf -X POST "${BASE_URL}/v1/topics" \
-    -H "Content-Type: application/json" \
-    -d "{\"name\":\"${topic}\",\"partitions\":${partitions}}" \
-    > /dev/null 2>&1 || true
-}
-
-produce_message() {
-  local topic="$1"
-  local message="$2"
-  curl -sf -X POST "${BASE_URL}/v1/topics/${topic}/messages" \
-    -H "Content-Type: application/json" \
-    -d "{\"value\":$(echo "$message" | jq -Rs .)}" \
-    > /dev/null 2>&1 || true
-  TOTAL_MESSAGES=$((TOTAL_MESSAGES + 1))
-}
-
-# --- Create Topics ---
-
-create_topics() {
-  echo "📦 Creating demo topics..."
-  create_topic "demo-events" 2
-  create_topic "demo-logs" 1
-  create_topic "demo-metrics" 2
-  create_topic "demo-orders" 2
-  echo "✅ Topics created"
-  echo ""
-}
-
-# --- Seed: demo-events ---
-
-seed_events() {
-  echo "📨 Seeding demo-events (user activity events)..."
-  local events=(
-    '{"event":"user.signup","user_id":"u-1001","email":"alice@example.com","plan":"free","ts":"2025-01-15T09:00:00Z"}'
-    '{"event":"user.login","user_id":"u-1001","ip":"192.168.1.10","ts":"2025-01-15T09:05:00Z"}'
-    '{"event":"user.signup","user_id":"u-1002","email":"bob@example.com","plan":"pro","ts":"2025-01-15T09:10:00Z"}'
-    '{"event":"page.view","user_id":"u-1001","page":"/dashboard","duration_ms":1200,"ts":"2025-01-15T09:12:00Z"}'
-    '{"event":"user.login","user_id":"u-1002","ip":"10.0.0.42","ts":"2025-01-15T09:15:00Z"}'
-    '{"event":"feature.used","user_id":"u-1001","feature":"export_csv","ts":"2025-01-15T09:20:00Z"}'
-    '{"event":"page.view","user_id":"u-1002","page":"/settings","duration_ms":800,"ts":"2025-01-15T09:22:00Z"}'
-    '{"event":"user.upgrade","user_id":"u-1001","from":"free","to":"pro","ts":"2025-01-15T09:30:00Z"}'
-    '{"event":"user.signup","user_id":"u-1003","email":"charlie@example.com","plan":"enterprise","ts":"2025-01-15T09:35:00Z"}'
-    '{"event":"user.logout","user_id":"u-1002","ts":"2025-01-15T09:40:00Z"}'
-  )
-  for msg in "${events[@]}"; do
-    produce_message "demo-events" "$msg"
-  done
-  echo "  ✅ ${#events[@]} events produced"
-}
-
-# --- Seed: demo-logs ---
-
-seed_logs() {
-  echo "📝 Seeding demo-logs (application log lines)..."
-  local logs=(
-    '{"level":"INFO","service":"api-gateway","msg":"Server started on port 8080","ts":"2025-01-15T09:00:01Z"}'
-    '{"level":"INFO","service":"auth-service","msg":"Connected to database","ts":"2025-01-15T09:00:02Z"}'
-    '{"level":"WARN","service":"api-gateway","msg":"Rate limit approaching for client 10.0.0.42","ts":"2025-01-15T09:05:30Z"}'
-    '{"level":"INFO","service":"order-service","msg":"Processing order ORD-5001","ts":"2025-01-15T09:10:00Z"}'
-    '{"level":"ERROR","service":"payment-service","msg":"Payment gateway timeout after 30s","trace_id":"abc-123","ts":"2025-01-15T09:10:05Z"}'
-    '{"level":"INFO","service":"payment-service","msg":"Retry succeeded for payment PAY-7890","trace_id":"abc-123","ts":"2025-01-15T09:10:08Z"}'
-    '{"level":"DEBUG","service":"cache-service","msg":"Cache hit ratio: 94.2%","ts":"2025-01-15T09:15:00Z"}'
-    '{"level":"WARN","service":"auth-service","msg":"Failed login attempt for user unknown@test.com","ts":"2025-01-15T09:20:00Z"}'
-    '{"level":"INFO","service":"api-gateway","msg":"Health check passed","ts":"2025-01-15T09:25:00Z"}'
-    '{"level":"INFO","service":"order-service","msg":"Order ORD-5001 shipped","ts":"2025-01-15T09:30:00Z"}'
-  )
-  for msg in "${logs[@]}"; do
-    produce_message "demo-logs" "$msg"
-  done
-  echo "  ✅ ${#logs[@]} log entries produced"
-}
-
-# --- Seed: demo-metrics ---
-
-seed_metrics() {
-  echo "📊 Seeding demo-metrics (system metrics)..."
-  local metrics=(
-    '{"metric":"cpu_usage_percent","host":"web-01","value":42.5,"ts":"2025-01-15T09:00:00Z"}'
-    '{"metric":"memory_usage_mb","host":"web-01","value":1024,"ts":"2025-01-15T09:00:00Z"}'
-    '{"metric":"http_requests_total","host":"web-01","value":15230,"method":"GET","status":200,"ts":"2025-01-15T09:00:00Z"}'
-    '{"metric":"cpu_usage_percent","host":"web-02","value":67.8,"ts":"2025-01-15T09:00:00Z"}'
-    '{"metric":"disk_usage_percent","host":"db-01","value":71.2,"mount":"/data","ts":"2025-01-15T09:00:00Z"}'
-    '{"metric":"http_latency_p99_ms","host":"web-01","value":245,"endpoint":"/api/orders","ts":"2025-01-15T09:05:00Z"}'
-    '{"metric":"memory_usage_mb","host":"web-02","value":1820,"ts":"2025-01-15T09:05:00Z"}'
-    '{"metric":"connection_pool_active","host":"db-01","value":18,"max":50,"ts":"2025-01-15T09:05:00Z"}'
-    '{"metric":"cpu_usage_percent","host":"web-01","value":38.1,"ts":"2025-01-15T09:10:00Z"}'
-    '{"metric":"http_requests_total","host":"web-02","value":8920,"method":"POST","status":201,"ts":"2025-01-15T09:10:00Z"}'
-  )
-  for msg in "${metrics[@]}"; do
-    produce_message "demo-metrics" "$msg"
-  done
-  echo "  ✅ ${#metrics[@]} metrics produced"
-}
-
-# --- Seed: demo-orders ---
-
-seed_orders() {
-  echo "🛒 Seeding demo-orders (e-commerce order records)..."
-  local orders=(
-    '{"order_id":"ORD-5001","customer_id":"u-1001","status":"confirmed","items":[{"sku":"WIDGET-A","qty":2,"price":29.99}],"total":59.98,"ts":"2025-01-15T09:10:00Z"}'
-    '{"order_id":"ORD-5002","customer_id":"u-1002","status":"confirmed","items":[{"sku":"GADGET-B","qty":1,"price":149.00},{"sku":"CABLE-C","qty":3,"price":9.99}],"total":178.97,"ts":"2025-01-15T09:12:00Z"}'
-    '{"order_id":"ORD-5001","customer_id":"u-1001","status":"processing","ts":"2025-01-15T09:15:00Z"}'
-    '{"order_id":"ORD-5003","customer_id":"u-1003","status":"confirmed","items":[{"sku":"WIDGET-A","qty":10,"price":29.99}],"total":299.90,"ts":"2025-01-15T09:18:00Z"}'
-    '{"order_id":"ORD-5002","customer_id":"u-1002","status":"processing","ts":"2025-01-15T09:20:00Z"}'
-    '{"order_id":"ORD-5001","customer_id":"u-1001","status":"shipped","carrier":"FastShip","tracking":"FS-98765","ts":"2025-01-15T09:25:00Z"}'
-    '{"order_id":"ORD-5004","customer_id":"u-1001","status":"confirmed","items":[{"sku":"PREMIUM-D","qty":1,"price":499.00}],"total":499.00,"ts":"2025-01-15T09:28:00Z"}'
-    '{"order_id":"ORD-5002","customer_id":"u-1002","status":"shipped","carrier":"QuickPost","tracking":"QP-12345","ts":"2025-01-15T09:30:00Z"}'
-    '{"order_id":"ORD-5003","customer_id":"u-1003","status":"processing","ts":"2025-01-15T09:32:00Z"}'
-    '{"order_id":"ORD-5001","customer_id":"u-1001","status":"delivered","ts":"2025-01-15T09:45:00Z"}'
-  )
-  for msg in "${orders[@]}"; do
-    produce_message "demo-orders" "$msg"
-  done
-  echo "  ✅ ${#orders[@]} order records produced"
-}
-
-# --- Main ---
+# shellcheck source-path=SCRIPTDIR
+# shellcheck source=seed-runtime.sh
+source "$SCRIPT_DIR/seed-runtime.sh"
+# shellcheck source=seed-fixtures.sh
+source "$SCRIPT_DIR/seed-fixtures.sh"
 
 echo ""
 echo "╔══════════════════════════════════════════════╗"
@@ -155,14 +16,11 @@ echo "╚═══════════════════════
 echo ""
 
 wait_for_server
-create_topics
+seed_topics
 
 echo "🌱 Seeding sample data..."
 echo ""
-seed_events
-seed_logs
-seed_metrics
-seed_orders
+seed_all_fixtures
 
 echo ""
 echo "══════════════════════════════════════════════"