Skip to content

README.md

Latest commit

 

History

History
491 lines (377 loc) · 23.1 KB

README.md

File metadata and controls

491 lines (377 loc) · 23.1 KB
Raw

Is it Observable

Is It observable Logo

Watch the episode kagent OpenTelemetry Dynatrace

📺 Watch the episode: https://www.youtube.com/@Isitobservable — Kagent: Build and Observe Kubernetes-Native AI Agents

Episode : Kagent — Build and Observe Kubernetes-Native AI Agents

Everyone is shipping AI agents. Almost no one can see what they do. This episode builds a fully observable agentic platform on Kubernetes with kagent — the CNCF Sandbox framework where every agent, model and tool is a Kubernetes CRD — and then makes every reasoning step, tool call and delegation hop visible in Dynatrace through OpenTelemetry.

You will stand up a cluster, deploy the full observability + service-mesh stack, run an SRE agent that troubleshoots a broken gRPC route on live cluster state, and compose a BMAD crew of six specialist agents that ships a real Todo app on GitHub — issues, milestone, code, PR and review — all watched live in two Dynatrace dashboards. Because kagent emits span-derived GenAI telemetry (gen_ai.*, no custom OTLP metrics), you see token usage, tool calls and A2A delegation without adding a single line to the agents.

Components used in this episode:

  • kagent v0.9.11 — Kubernetes-native agentic framework (agents/models/tools as CRDs)
  • OpenTelemetry Operator — manages the two collectors that carry all telemetry
  • Two OpenTelemetry Collectors — a DaemonSet node-ingest tier + a StatefulSet gateway (the sole Dynatrace egress)
  • Dynatrace Operator v1.9.1 (kubernetes-monitoring mode) — k8s entity enrichment + log/metric ingest
  • Istio (ambient profile) + Gateway API / GAMMA — the mesh that carries the gRPC-route demo
  • OpenTelemetry Demo — the OTel-native microservices workload (plays the "app" + workload generator)
  • Ollama (qwen3.6:latest, self-hosted) — the tool-calling model behind every agent
  • cert-manager — webhook CA prerequisite for the OTel Operator
  • kagent GitHub MCP server + mcp-k8s-networking — the tools the agents act through

Everything is destination Dynatrace, and the whole repo ships no environment-specific values in git — every IP, tenant and token comes from deploy/.env, so you can run all of it in your cluster.

Prerequisite

Install the CLIs and have a Dynatrace tenant ready before you start:

  • kubectl — talk to the cluster
  • helm 3.x — install kagent, Istio, the operators
  • git — clone this repo
  • jq — parse JSON from the agent A2A endpoints
  • curl — hit the kagent controller / A2A APIs
  • clusterctl — only for the Cluster API provisioning path
  • gcloud — only for the GKE provisioning path
  • the kagent CLI — optional, for driving agents from your terminal

You also need:

  • A Kubernetes cluster (this episode provisions one with Cluster API on Proxmox; GKE works too).
  • A Dynatrace SaaS tenant + two API tokens (see Getting started below).
  • An Ollama endpoint hosting a tool-calling model (qwen3.6:latest in this episode). kagent runs on tool calls — a model that can't call tools will fail the SRE and BMAD steps.
  • A GitHub account and a fine-grained PAT scoped to a single demo repo (for the BMAD crew).

Getting started

All environment-specific values live in one git-ignored file. Copy the template, fill it in, and hydrate it into your shell before applying anything:

git clone https://github.com/isItObservable/Kagent.git
cd Kagent
cp deploy/.env.example deploy/.env
$EDITOR deploy/.env            # OLLAMA_HOST, K8S_CLUSTER_NAME, DT_*, GITHUB_*, ...
set -a && . deploy/.env && set +a

deploy/.env is git-ignored — never commit real endpoints, tenants or tokens. A CI grep-gate fails the build on a stray IP, dt0c01. token or github_pat_.

1. Dynatrace Tenant — start a trial

If you don't already have one, create a free trial tenant at https://www.dynatrace.com/trial. Save your tenant API base URL (including the /api suffix) into deploy/.env:

# e.g. https://abc12345.live.dynatrace.com/api
DT_ENDPOINT=https://YOUR_TENANT.live.dynatrace.com/api
DT_TENANT=YOUR_TENANT

2. Create the Dynatrace API Tokens

You need two tokens. Create them under Access Tokens in your Dynatrace tenant.

Operator token (DT_DATA_INGEST_TOKEN) — used by the Dynatrace Operator / DynaKube for Kubernetes monitoring. Scopes:

  • Create ActiveGate tokens
  • Read entities
  • Read settings
  • Write settings
  • Access problem and event feed, metrics, and topology
  • Read configuration
  • Write configuration
  • Paas integration - Installer download

Ingest data token (DT_API_TOKEN) — used by the gateway collector to export OTLP to Dynatrace. Scopes:

  • Ingest metrics (metrics.ingest)
  • Ingest logs (logs.ingest)
  • Ingest OpenTelemetry traces (openTelemetryTrace.ingest)
  • Ingest events (events.ingest)

Put both into deploy/.env:

DT_API_TOKEN=dt0c01.XXXX          # ingest (OTLP/metrics/logs) — used by the collector
DT_DATA_INGEST_TOKEN=dt0c01.YYYY  # DynaKube data-ingest — used by the operator

3. Spin up a k8s cluster

Two documented on-ramps — pick one. Everything downstream is identical once you have a kubeconfig. The model always stays external on your Ollama endpoint; the cluster never hosts the LLM. Full details live in deploy/provisioning/ — the exact commands are inlined below so you can run the episode without opening another file.

Option A — Cluster API + Proxmox (CAPMOX, primary)

The homelab reference cluster (observable-kagent). You need a management cluster with the CAPI + CAPMOX + in-cluster IPAM providers, a Proxmox API token with VM create/clone rights, and a CAPMOX-ready golden template (Ubuntu 24.04 + kubeadm — see the proxmox-template skill). Full walkthrough: deploy/provisioning/capi-proxmox.md.

# 3A.1 — provider + cluster inputs (RFC5737 example ranges — use your own LAN).
#         VIP / node pool / MetalLB pool / DHCP reserve MUST NOT overlap.
export PROXMOX_URL="https://your-proxmox:8006/api2/json"
export PROXMOX_TOKEN="user@pam!tokenid=xxxxxxxx-...-xxxxxxxxxxxx"
export CLUSTER_NAME="${K8S_CLUSTER_NAME}"          # e.g. observable-kagent
export KUBERNETES_VERSION="v1.31.4"
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=3
export CONTROL_PLANE_ENDPOINT_IP="192.0.2.72"      # your VIP
export NODE_IP_RANGES="192.0.2.73-192.0.2.90"      # node pool
export GATEWAY="192.0.2.1"; export IP_PREFIX=24; export DNS_SERVERS="192.0.2.1"
export CP_DISK_STORAGE="local-lvm" # etcd fsync (board policy)
export WORKER_DISK_STORAGE="zetlab-vmdisks"        # shared NFS for live-migration
export TEMPLATE_ID=9000; export PROXMOX_SOURCENODE="pve1"

# 3A.2 — initialize CAPI providers on the MANAGEMENT cluster (once).
clusterctl init --infrastructure proxmox --ipam in-cluster

# 3A.3 — generate + apply the workload-cluster manifests.
clusterctl generate cluster "${CLUSTER_NAME}" \
  --infrastructure proxmox \
  --kubernetes-version "${KUBERNETES_VERSION}" \
  --control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
  --worker-machine-count "${WORKER_MACHINE_COUNT}" \
  > "cluster-${CLUSTER_NAME}.yaml"
kubectl apply -f "cluster-${CLUSTER_NAME}.yaml"

# 3A.4 — wait, then pull the new cluster's kubeconfig.
clusterctl describe cluster "${CLUSTER_NAME}"
clusterctl get kubeconfig "${CLUSTER_NAME}" > "${CLUSTER_NAME}.kubeconfig"
export KUBECONFIG="$PWD/${CLUSTER_NAME}.kubeconfig"

# 3A.5 — install a CNI (Cilium) + a MetalLB pool from your reserved range.
#         (GKE users skip this — see Option B.) Then you are ready to deploy.

Serialized worker builds: zetlab-vmdisks (shared NFS) re-introduces cfs-lock serialization on parallel CAPI worker provisioning — expect workers to build one at a time. Dev clusters that prefer fast parallel rebuilds may set WORKER_DISK_STORAGE=local-lvm (opt-out).

Option B — GKE (managed, alternative)

No Proxmox? Bring a managed cluster; GKE ships a CNI + LoadBalancer already, so you skip Cilium/MetalLB. Full walkthrough: deploy/provisioning/gke.md.

# 3B.1 — cluster inputs.
export GCP_PROJECT="your-gcp-project"
export GKE_REGION="europe-west1"
export CLUSTER_NAME="${K8S_CLUSTER_NAME}"
export GKE_NODE_COUNT=3
export GKE_MACHINE_TYPE="e2-standard-4"            # 4 vCPU / 16 GB — headroom for otel-demo + kagent

# 3B.2 — create the cluster (Workload Identity on for clean metadata).
gcloud config set project "${GCP_PROJECT}"
gcloud container clusters create "${CLUSTER_NAME}" \
  --region "${GKE_REGION}" \
  --num-nodes "${GKE_NODE_COUNT}" \
  --machine-type "${GKE_MACHINE_TYPE}" \
  --cluster-version "1.31" \
  --workload-pool "${GCP_PROJECT}.svc.id.goog" \
  --release-channel stable

# 3B.3 — pull credentials into your kubeconfig.
gcloud container clusters get-credentials "${CLUSTER_NAME}" --region "${GKE_REGION}"

Either way, confirm kubectl is pointed at the new cluster before deploying:

kubectl cluster-info && kubectl config current-context

Deploy most of the components

One ordered, idempotent installer runs the entire platform stack (steps 00→09). Every step is kubectl apply + an explicit kubectl wait, so it is safe to re-run:

export KUBECONFIG=/path/to/${K8S_CLUSTER_NAME}.kubeconfig
./deploy/deploy-all.sh          # runs steps 00 → 09 in order
./deploy/validate.sh            # pass/fail acceptance checks
# Script What it installs
00 00-preflight.sh CLI + cluster reachability + required-var checks
01 01-cert-manager.sh cert-manager (OTel Operator webhook prerequisite)
02 02-otel-operator.sh OpenTelemetry Operator
03 03-dynatrace-operator.sh Dynatrace Operator v1.9.1, kubernetes-monitoring mode
04 04-istio-ambient.sh Istio ambient mesh (ztunnel + istio-cni + istiod)
05 05-gateway-api.sh Gateway API CRDs + GAMMA (mesh routing)
06 06-otel-demo.sh OpenTelemetry Demo — standard chart, no Dynatrace auto-injection
07 07-kagent.sh kagent (OTLP → gateway) + Ollama ModelConfig
08 08-collectors.sh DaemonSet agent + StatefulSet gateway + Istio telemetry wiring
09 09-kagent-github-mcp.sh kagent GitHub MCP ToolServer + github-mcp-token Secret

Resume or isolate a step: START_AT=06 ./deploy/deploy-all.sh or ONLY=08 ./deploy/deploy-all.sh.

Deploy the Workload Generator

The OpenTelemetry Demo ships its own load generator — it drives continuous, OTel-native traffic through the microservices so the traces, the gRPC-route demo and the dashboards all have live data. Step 06 deploys it as part of the demo; confirm it is running:

kubectl -n otel-demo get deploy load-generator frontend-proxy

Architecture: Two Collectors, One Egress

The architectural crux: two collectors live in the default namespace, and every workload points at them. The DaemonSet handles per-node ingest; the StatefulSet gateway aggregates and is the single door to Dynatrace — one credential surface.

                                  kagent OTLP + Istio ambient + otel-demo SDKs
        node logs/traces │                         │ OTLP
┌─────────────────────────┐   forward   ┌──────────▼───────────────────┐
│ otel-agent (DaemonSet)  │────OTLP────▶ │ otel-gateway (StatefulSet)   │──OTLP──▶ Dynatrace
│ per-node: hostmetrics,  │             │ cluster metrics, tail-sampling│  (Api-Token)
│ filelog, k8sattributes  │             │ SINGLE Dynatrace egress       │
└─────────────────────────┘             └──────────────────────────────┘
DynaKube ActiveGate (kubernetes-monitoring) ─────────────────────────────▶ Dynatrace (k8s metrics)

The OTel Operator names each collector Service <CR-name>-collector, so the CRs are named otel-agent and otel-gateway to yield exactly:

  • DaemonSet (node logs/traces, host/kubelet metrics): otel-agent-collector.default.svc.cluster.local:4317
  • StatefulSet gateway (metrics aggregation, tail sampling, sole Dynatrace egress): otel-gateway-collector.default.svc.cluster.local:4317

Wiring — every producer exports to the gateway:

  • kagent OTLP → otel-gateway-collector.default:4317 (deploy/values/kagent-values.yaml).
  • Istio ambient telemetry → the same gateway (deploy/values/istio-telemetry.yaml + meshConfig extension provider, patched in step 08).
  • otel-demo app SDKs → the gateway (deploy/values/otel-demo-values.yaml).
  • DaemonSet agent → forwards node logs/traces to the gateway (it never talks to Dynatrace).
  • Only the gateway exports to Dynatrace, and it keeps all GenAI spans via tail-based sampling.

Why otel-demo stays OTel-native. The story is "OTel-native app → our collector → Dynatrace backend." We do not layer Dynatrace OneAgent on the demo workloads; DynaKube stays in kubernetes-monitoring mode so it enriches k8s entities without double-instrumenting the apps. This is the honest, portable demo.

Step-by-Step: The Kagent Walkthrough

Full hands-on companion in TUTORIAL.md. The three steps build on each other: a brain, then a brain with tools, then a crew of them shipping software.

Step 1 — The brain: a ModelConfig for Ollama

Every agent needs a model, declared as a ModelConfig. Prove the brain works before anything else. → manifests/01-modelconfig-mac-studio.yaml

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: mac-studio-ollama
  namespace: kagent
spec:
  model: qwen3.6:latest          # MUST support function/tool calling
  provider: Ollama
  ollama:
    host: http://${OLLAMA_HOST}  # hydrated from deploy/.env — no IP in git
envsubst < manifests/01-modelconfig-mac-studio.yaml | kubectl apply -f -
kubectl get modelconfig mac-studio-ollama -n kagent

An agent references the model by name — the field is modelConfig (a plain string), not modelConfigRef.

Step 2 — An SRE agent with MCP + a gRPC-route demo

An agent with a model but no tools can only talk. Tools come from MCP servers. We deploy an SRE networking agent whose tools include henrikrexed/mcp-k8s-networking (Gateway API / Istio route inspection) plus kagent's built-in read-only k8s tools.

# 2.1 — deploy + register the MCP server (RemoteMCPServer)
envsubst < manifests/sre/10-mcp-k8s-networking-server.yaml | kubectl apply -f -
kubectl -n kagent get remotemcpserver mcp-k8s-networking     # ACCEPTED True, status.tools non-empty

# 2.2 — deploy the SRE agent (toolNames explicitly enumerated — small-model discipline)
kubectl apply -f manifests/sre/11-sre-networking-agent.yaml
kubectl get agent sre-networking-agent -n kagent             # READY True, ACCEPTED True

The deterministic demo. Break a gRPC route in otel-demo, then let the agent diagnose it. The checkout GRPCRoute backendRef.port is set to 8080 (frontend HTTP) instead of 5050 (checkout gRPC) — gRPC/h2c to an HTTP/1 listener fails silently:

kubectl apply -f manifests/demos/grpc-misroute/broken.yaml

Ask the SRE agent (kagent UI or A2A): "Traffic to checkout is failing — what is broken in the network path?" It reads the GRPCRoute through mcp-k8s-networking, resolves backendRef to checkout:8080, compares it to the checkout Service (gRPC 5050), and reports backendRef.port 8080 should be 5050. Apply the fix and watch the otel-demo traces recover in Dynatrace:

kubectl apply -f manifests/demos/grpc-misroute/fix.yaml

Step 3 — A BMAD crew that ships a Todo app on GitHub

The payoff: orchestration. In kagent, an agent can be a tool of another agent — when one lists another as type: Agent, the orchestrator's LLM delegates over the A2A protocol. There is no Team/Ensemble CRD; the crew is the graph of references. Six BMAD specialists, each acting on the world through kagent's built-in GitHub MCP server, run the full software-delivery loop and build a Todo app.

Agent BMAD role GitHub tools (least-privilege)
bmad-analyst Mary — requirements → issues issues RW, repo RO
bmad-pm John — milestone + plan issues RW, repo RO
bmad-architect Winston — design → dev-task issues issues RW, repo RO
bmad-developer Amelia — code → branch → PR repo RW, PRs RW, issues RO
bmad-qa Quinn — review the PR PRs RW, issues RW
bmad-orchestrator BMad Master — conduct the chain type: Agent → the five above

No agent gets a merge_* tool — merge stays a human/CI step, so the crew can never merge its own PR. One token, one repo = the entire blast radius.

# 3.1 — install the GitHub MCP server (fine-grained PAT: Contents/Issues/Pull requests RW)
kubectl -n kagent create secret generic github-mcp-token --from-literal=token=$GITHUB_TOKEN
helm install github-mcp oci://ghcr.io/kagent-dev/kagent/helm/github-mcp-server \
  -n kagent -f manifests/bmad-crew/values-github-mcp.yaml
kubectl -n kagent get toolserver                              # each Accepted, status.tools non-empty

# 3.2 — deploy the crew
kubectl apply -k manifests/bmad-crew
kubectl get agents -n kagent -l app.kubernetes.io/part-of=bmad-crew   # all six READY True

# 3.3 — one kickoff to the orchestrator runs the whole chain (see TUTORIAL.md §6.3)

Expected artifacts on ${GITHUB_DEMO_REPO}: requirement issues → a "Todo app v1" milestone → dev-task issues → a feat/todo-app branch + PR → a QA review.

kagent vs. the Sympozium Ensemble. Same crew, same Todo-app workflow, opposite construction. Sympozium uses one Ensemble CRD that generates N agents; kagent is a reference-based graph of independently-authored agents that delegate over A2A. Ensemble = declarative roster; kagent = compositional graph.

Testing / Validating the Deployment

1. Acceptance checks

deploy/validate.sh runs a non-destructive pass/fail sweep mirroring the acceptance criteria — component health, both collector Services, the gateway/agent rollouts, that kagent's OTLP endpoint points at otel-gateway-collector:4317, and that otel-demo is serving traffic:

./deploy/validate.sh

2. Confirm the collector wiring by hand

kubectl -n default get svc otel-agent-collector otel-gateway-collector
kubectl -n default rollout status statefulset/otel-gateway-collector
kubectl -n default rollout status daemonset/otel-agent-collector
kubectl -n kagent get deploy kagent-controller -o yaml | grep otel-gateway-collector

3. Agents are healthy and reachable

kubectl get agents -n kagent                                  # READY True, ACCEPTED True
kubectl -n kagent get remotemcpserver,toolserver             # Accepted, status.tools non-empty

kagent exposes each agent as an A2A JSON-RPC endpoint on the controller (:8083), or use the kagent UI at kagent-ui:8080. A quick single-agent smoke test ("Inspect the cluster with your tools and name three namespaces you observed.") should return real cluster state — proof the model + MCP loop works.

Observability Reference: gen_ai.* Signals & Dashboards

kagent emits GenAI spans (gen_ai.*, ADK gcp.vertex.agent) and logs — there are no kagent OTLP metrics, so every efficiency signal is span-derived. The span tree is invoke_agent → call_llm → generate_content (+ execute_tool); crew work adds A2A delegation hops between agents.

Two Dynatrace dashboards (dashboards/) tell the story on live telemetry:

Dashboard ID What it shows
kagent Health 4b46b649 engine up, runs succeeding vs. erroring, pod/workload resource health (DynaKube k8s metrics)
kagent Agentic Efficiency dfe6dfc4 token usage (gen_ai.usage.*), tool calls, A2A delegation hops, latency per turn — all span-derived

Both are templated on ${DT_TENANT} / ${K8S_CLUSTER_NAME} — no tenant baked in. Deploy per dashboards/README.md.

Signal notes (dashboards/EFFICIENCY-SIGNALS.md):

  • Tokens: gen_ai.usage.* lands on both call_llm and generate_content spans — sum on one to avoid a 2× double-count. Deduped, a typical crew run is ~10,659 tokens/run.
  • Model attribution: filter on gen_ai.request.model (the real qwen3.6), not gen_ai.system (ADK labels the provider gemini).
  • Latency: run p50 ≈ 62s / p90 ≈ 176s; LLM call p50 ≈ 35s / p90 ≈ 102s (local model, multi-hop).
  • Delegation and memory panels stay empty on single-agent captures — they populate once the BMAD crew runs.

Resources

This repo ships no environment-specific values — every IP, tenant and token lives in deploy/.env (git-ignored). Copy deploy/.env.example, fill it in, and run it in your own cluster.