Skip to content

TUTORIAL.md

Latest commit

 

History

History
491 lines (382 loc) · 25 KB

TUTORIAL.md

File metadata and controls

491 lines (382 loc) · 25 KB
Raw

kagent Tutorial — a fully observable agentic platform, from cluster to BMAD crew

The hands-on companion to the episode. By the end you will have provisioned a Kubernetes cluster with Cluster API (or GKE), deployed a full OpenTelemetry + Dynatrace + Istio ambient + otel-demo stack, wired kagent and the mesh to collectors in the default namespace, run an SRE agent that troubleshoots a broken gRPC route, and composed a BMAD crew that builds a real Todo app on GitHub — issues, milestone, code, PR and review — all watched live in two Dynatrace dashboards.

Portability first. This repo ships no environment-specific values in git — no IPs, tokens, tenants or secrets. Everything variable lives in deploy/.env (copied from deploy/.env.example) and is hydrated at apply time. You can run the whole thing in your cluster.

Validated on the episode cluster (${K8S_CLUSTER_NAME}, kagent v0.9.11, models self-hosted on the Mac Studio ${OLLAMA_HOST} over Ollama, qwen3.6:latest). Real outputs are quoted in §8 Validation.

Table of contents

  1. Prerequisites
  2. Provision the cluster (Cluster API or GKE)
  3. Deploy the platform stack
  4. Collectors in the default namespace + wiring
  5. Step 1 — the brain: a ModelConfig for Ollama
  6. Step 2 — an SRE agent with MCP + a gRPC-route demo
  7. Step 3 — a BMAD crew that ships a Todo app on GitHub
  8. Observe it all — Dynatrace dashboards
  9. Validation — what we actually saw
  10. Cleanup
  11. Troubleshooting

0. Prerequisites

  • Tools: kubectl, helm 3.x, git, clusterctl (for the Cluster API path), and optionally the kagent CLI and gcloud (for the GKE path).

  • Accounts / endpoints:

    • A Proxmox cluster you control or a GKE project.
    • A Dynatrace tenant (SaaS) + two API tokens (ingest + data-ingest).
    • A GitHub account and a fine-grained PAT scoped to one demo repo (§6).
    • An Ollama endpoint hosting a tool-calling model (this episode: the Mac Studio at ${OLLAMA_HOST}, qwen3.6:latest). kagent runs on tool calls — a model that can't call tools will fail §5 and §6.

  • Config: copy the env template and fill in your own values — this is the only place your environment-specific data lives:

    cp deploy/.env.example deploy/.env
$EDITOR deploy/.env            # OLLAMA_HOST, K8S_CLUSTER_NAME, DT_*, GITHUB_*, ...
set -a && . deploy/.env && set +a

    deploy/.env is git-ignored. Never commit real endpoints, tenants or tokens; a CI grep-gate fails the build on a stray IP / dt0c01. / github_pat_.

1. Provision the cluster (Cluster API or GKE)

Two documented on-ramps to a conformant cluster; everything downstream is identical once you have a kubeconfig. Model hosting stays external on the Mac Studio Ollama endpoint in both paths — the cluster never hosts the LLM.

1a. Primary — Cluster API + Proxmox (CAPMOX)

You supply your own Proxmox details in deploy/.env (PROXMOX_URL, credentials as a Secret, node/template IDs, IP ranges). No real IPs live in the repo — the render templates carry placeholder ranges only.

# bootstrap the management cluster with the Proxmox + in-cluster IPAM providers
clusterctl init --infrastructure proxmox --ipam in-cluster

# render your workload cluster from values.env, then apply
deploy/01-provision-capi.sh          # renders clusters/${K8S_CLUSTER_NAME}/*.yaml and applies

The provisioning scripts under deploy/ are owned by ProxOps and follow the org's proven observable-llm/Sympozium CAPI pattern. This tutorial documents the contract; the scripts hold the exact commands.

1b. Alternative — GKE

No Proxmox? Bring a managed cluster and skip straight to §2:

gcloud container clusters create ${K8S_CLUSTER_NAME} \
  --project ${GKE_PROJECT} --region ${GKE_REGION} \
  --num-nodes 3 --machine-type e2-standard-4
gcloud container clusters get-credentials ${K8S_CLUSTER_NAME} --region ${GKE_REGION}

2. Deploy the platform stack

One ordered, idempotent deploy — every step is kubectl apply + an explicit kubectl wait, so it is safe to re-run. Ordering is load-bearing because of CRD + webhook readiness.

deploy/02-platform-stack.sh          # runs the ordered install below
# Component Notes
1 cert-manager webhooks/CA for the OTel Operator; wait Ready first
2 OpenTelemetry Operator manages our collector CRs (§3); depends on cert-manager
3 Dynatrace Operator v1.9.1 PINNED, mode kubernetes-monitoring — k8s entity/enrichment + log/metric ingest, not full-stack OneAgent on the apps
4 Gateway API CRDs + GAMMA standard channel Gateway API; GAMMA = mesh binding of Service→routes for Istio ambient
5 Istio (ambient profile) ztunnel + istio-cni + istiod; ambient dataplane
6 Collectors in default ns DaemonSet + StatefulSet (§3)
7 kagent (Helm) values overridden to export to the default-ns collector (§3)
8 otel-demo native OTel SDK instrumentation, no Dynatrace auto-injection

Why otel-demo stays OTel-native (D4). The story is "OTel-native app → our collector → Dynatrace backend." We do not layer Dynatrace OneAgent on the demo workloads — that would double-instrument and skew the picture. DynaKube stays in kubernetes-monitoring mode so it enriches k8s entities without instrumenting the demo. This is the honest, portable demo.

3. Collectors in the default namespace + wiring

The architectural crux: two collectors live in the default namespace, and every workload points at them — there are no per-namespace collectors.

        node logs/traces │                    kagent OTLP + Istio ambient
                          ▼                                 │ OTLP
┌─────────────────────────┐   forward    ┌─────────────────▼────────────┐
│ otel-agent (DaemonSet)  │─────OTLP────▶ │ otel-gateway (StatefulSet)   │──OTLP/HTTP──▶ Dynatrace
│ per-node: hostmetrics,  │              │ cluster metrics, tail-sampling│   (+Api-Token)
│ filelog, k8sattributes  │              │ SINGLE Dynatrace egress       │
└─────────────────────────┘              │ (one credential surface)      │
                                         └──────────────────────────────┘
DynaKube ActiveGate (kubernetes-monitoring) ─────────────────────────────▶ Dynatrace (k8s metrics)

Topology note. kagent and Istio ambient export directly to the StatefulSet gateway (the sole Dynatrace egress). The DaemonSet handles node-local logs/traces + host/kubelet metrics and forwards them to the same gateway.

Wiring each producer to the default-ns collector:

  • kagent — observability/kagent-otel-values.yaml points the kagent OTLP exporter at otel-agent-collector.default.svc.cluster.local:4317:

    helm upgrade kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  -n kagent --reuse-values -f observability/kagent-otel-values.yaml

  • Istio ambient — set mesh telemetry (ztunnel/waypoint access logs + metrics) to export to the same default-ns collector via the Telemetry API / meshConfig defaultProviders → an OTLP provider named for the default-ns gateway.

  • otel-demo — repoint its built-in OTLP endpoint env at the default-ns DaemonSet.

Rationale: one collector plane, one Dynatrace egress, one credential surface.

kagent emits GenAI spans (gen_ai.*, ADK gcp.vertex.agent) and logs — there are no kagent OTLP metrics, so the agentic-efficiency dashboard is span-derived (see dashboards/EFFICIENCY-SIGNALS.md).

4. Step 1 — the brain: a ModelConfig for Ollama

Every agent needs a model, declared as a ModelConfig. First we limit kagent to just this — prove the brain works before anything else. → manifests/01-modelconfig-mac-studio.yaml

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: mac-studio-ollama
  namespace: kagent
spec:
  model: qwen3.6:latest          # MUST support function/tool calling
  provider: Ollama
  ollama:
    host: http://${OLLAMA_HOST}  # hydrated from deploy/.env — no IP in git
  # apiKeySecret / apiKeySecretKey omitted — keyless local Ollama
envsubst < manifests/01-modelconfig-mac-studio.yaml | kubectl apply -f -
kubectl get modelconfig mac-studio-ollama -n kagent

An agent references the model by name — the field is modelConfig (a plain string), not modelConfigRef.

5. Step 2 — an SRE agent with MCP + a gRPC-route demo

An agent with a model but no tools can only talk. Tools come from MCP servers. We deploy a single SRE networking agent whose tools include the henrikrexed/mcp-k8s-networking MCP server (Gateway API / Istio route inspection) plus kagent's built-in k8s tools.

5.1 — deploy the MCP server + register it. The server runs in-cluster and is registered as a RemoteMCPServer. → manifests/sre/10-mcp-k8s-networking-server.yaml

envsubst < manifests/sre/10-mcp-k8s-networking-server.yaml | kubectl apply -f -
kubectl -n kagent get remotemcpserver mcp-k8s-networking    # ACCEPTED True, status.tools non-empty

5.2 — deploy the SRE agent. toolNames are explicitly enumerated (small-model discipline — qwen3.6 degrades with a large tool surface). → manifests/sre/11-sre-networking-agent.yaml

kubectl apply -f manifests/sre/11-sre-networking-agent.yaml
kubectl get agent sre-networking-agent -n kagent            # READY True, ACCEPTED True

5.3 — the troubleshooting demo (deterministic). Break a gRPC route in otel-demo, then let the agent diagnose it. → manifests/demos/grpc-misroute/

# Break it: the checkout GRPCRoute backendRef.port is 8080 (frontend HTTP)
# instead of 5050 (checkout gRPC). gRPC/h2c to an HTTP/1 listener fails.
kubectl apply -f manifests/demos/grpc-misroute/broken.yaml

Ask the SRE agent (kagent UI or A2A) — the prompts that drive the demo:

  1. "Traffic to the checkout service is failing — what is broken in the network path?"
  2. "Show me the gRPC routes for checkout and which backend + port they resolve to."
  3. "Compare the route's target port to the port the checkout service exposes."
  4. "What is the root cause and what is the fix?"

The agent reads the GRPCRoute via mcp-k8s-networking, resolves the backendRef to checkout:8080, compares it to the checkout Service (gRPC 5050), and reports backendRef.port 8080 should be 5050. Apply the fix and watch traffic recover in the otel-demo traces in Dynatrace:

kubectl apply -f manifests/demos/grpc-misroute/fix.yaml

Why gRPC specifically: it rides HTTP/2 (h2c in-mesh), so pointing the route at an HTTP/1-only port is a silent, protocol-level failure. The agent has to reason about protocol + port, not just reachability.

6. Step 3 — a BMAD crew that ships a Todo app on GitHub

The payoff — orchestration. In kagent, an agent can be a tool of another agent: when one lists another as type: Agent, the orchestrator's LLM delegates over the A2A protocol. There is no Team/Ensemble CRD — the crew is the graph of references.

v2 makes the crew operate GitHub. Six BMAD specialists, each acting on the world through kagent's built-in GitHub MCP server, run the full software-delivery loop and build a Todo app — mirroring the Sympozium episode one-for-one.

Agent BMAD role GitHub tools (least-privilege)
bmad-analyst Mary — requirements → issues issues RW, repo RO
bmad-pm John — milestone + plan issues RW, repo RO
bmad-architect Winston — design → dev-task issues issues RW, repo RO
bmad-developer Amelia — code → branch → PR repo RW, PRs RW, issues RO
bmad-qa Quinn — review the PR PRs RW, issues RW
bmad-orchestrator BMad Master — conduct the chain type: Agent → the five above, + issues RO

No agent gets a merge_* tool — merge stays a human/CI step, so the crew can never merge its own PR.

6.1 — bring your own GitHub PAT → a Kubernetes Secret. The crew authenticates to GitHub with your own Personal Access Token, which you store as the Secret github-mcp-token (key token) in the kagent namespace. The token is never committed — the manifests reference the Secret; only you ever hold the value.

Create a fine-grained PAT (github.com → Settings → Developer settings → Fine-grained tokens), then scope it by which path you're running — least privilege:

Path Repository access Permissions
Read-only issue demo (analyst lists/reads issues) only ${GITHUB_DEMO_REPO} Issues: Read (+ Metadata: Read, auto)
Full write demo (crew opens issues/branches/PRs) only ${GITHUB_DEMO_REPO} Issues: RW · Contents: RW · Pull requests: RW 
# paste YOUR token — it stays only in this Secret, never in a manifest or git
export GITHUB_TOKEN='<your-fine-grained-PAT>'   # from GitHub → Fine-grained tokens
kubectl -n kagent create secret generic github-mcp-token \
  --from-literal=token="$GITHUB_TOKEN" \
  --dry-run=client -o yaml | kubectl apply -f -
unset GITHUB_TOKEN

The demo repo is public, so the read-only path authenticates even with a token that has no extra permissions — keep it minimal. Rotate the PAT after the demo, and if a token is ever pasted into a chat/ticket, treat it as compromised and rotate. To rotate: recreate the Secret (same command, new token) and re-apply §6.2.

6.2 — the GitHub MCP server. All six agents reference ONE self-hosted server — github-mcp (GitHub's official ghcr.io/github/github-mcp-server) registered as the github-mcp-remote RemoteMCPServer in 07-github-mcp.yaml, which the crew's kustomization applies in §6.3. It reads the PAT from the github-mcp-token Secret (§6.1), runs GITHUB_TOOLSETS=issues,pull_requests,repos + GITHUB_READ_ONLY=0, and least privilege is enforced per agent by the toolNames each Agent lists — no agent is granted merge_pull_request (merge stays human/CI).

# 07-github-mcp.yaml keeps GITHUB_PERSONAL_ACCESS_TOKEN empty in git; inject at apply:
GITHUB_PAT="$(kubectl -n kagent get secret github-mcp-token -o jsonpath='{.data.token}' | base64 -d)" \
  envsubst < manifests/bmad-crew/07-github-mcp.yaml | kubectl apply -f -

# VALIDATION GATE — confirm the server registered the write tool surface
kubectl -n kagent get remotemcpserver github-mcp-remote \
  -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}{"  "}{.status.discoveredTools[*].name}'
# → Accepted=True  ... list_issues issue_read issue_write create_pull_request create_branch pull_request_read ...

This github-mcp-server build uses consolidated tool names (issue_read/issue_write, pull_request_read/pull_request_review_write) — verify 01–06 toolNames against the live status.discoveredTools. values-github-mcp.yaml documents a hosted alternative (token only in the Secret via headersFrom) that needs a kagent GitHub-MCP Helm chart not yet published, so the self-hosted server above is what deploys today. One token, one repo = the blast radius.

6.3 — deploy the crew.

kubectl apply -k manifests/bmad-crew
kubectl get agents -n kagent -l app.kubernetes.io/part-of=bmad-crew
# all six: READY True, ACCEPTED True

6.4 — run the Todo-app demo. One kickoff to the orchestrator runs the whole chain:

kubectl run a2a --rm -i --restart=Never --image=curlimages/curl:8.10.1 -n kagent \
  --command -- curl -s -m 300 -X POST -H 'Content-Type: application/json' \
  --data '{"jsonrpc":"2.0","id":"1","method":"message/send","params":{"message":
    {"role":"user","messageId":"m1","kind":"message",
     "parts":[{"kind":"text","text":"Build a simple Todo app in the '"$GITHUB_DEMO_REPO"' repository. Run the full BMAD workflow: analyst scopes requirements as GitHub issues, PM creates a milestone and plan, architect creates dev-task issues, developer opens a PR with working code, QA reviews the PR. Coordinate the whole chain and give me a summary with links to every issue and PR you created."}]}}}' \
  http://kagent-controller.kagent:8083/api/a2a/kagent/bmad-orchestrator

Expected artifacts on ${GITHUB_DEMO_REPO}: requirement issues → a "Todo app v1" milestone → dev-task issues → a feat/todo-app branch + PR → a QA review.

6.5 How this compares to the Sympozium Ensemble

Dimension Sympozium BMAD Ensemble kagent BMAD crew
Roster definition One Ensemble CRD lists agentConfigs[]; controller generates one Agent per entry No parent resource — each Agent authored independently
Composition style One-to-many generative (1 manifest → N members) Peer-to-peer / reference-based graph
Transport between agents Sympozium controller / IPC bridge (delegate_to_persona) A2A protocol (JSON-RPC/HTTP) + agent-as-tool
Acting on the world SkillsPack tools GitHub MCP server (issues, PRs, repo)

Ensemble = declarative roster; kagent = compositional graph. Same crew, same Todo-app workflow, opposite construction. Keep the crew in one namespace — cross-namespace agent refs still have rough edges (kagent-dev/kagent#841).

7. Observe it all — Dynatrace dashboards

Every hop in the crew — invoke_agent, execute_tool (including the GitHub MCP calls), generate_content — is a span. Two Dynatrace dashboards (dashboards/) tell the story on live telemetry:

Both are templated on ${DT_TENANT} / ${K8S_CLUSTER_NAME} — no tenant baked in. Deploy them per dashboards/README.md. A watch-out from building these: gen_ai.usage.* lands on both call_llm and generate_content spans — sum on one to avoid a 2× token double-count.

8. Validation — what we actually saw

kagent exposes each agent as an A2A JSON-RPC endpoint on the controller (:8083). Invoke with message/send, or use the kagent UI at kagent-ui:8080.

Single agent + MCP. Prompt: "Inspect the cluster with your tools and name three namespaces you observed." → completed, 1323 tokens:

"I observed the default, kube-system, and istio-system namespaces, along with several others including kagent, kgateway-system, metallb-system …"

The agent read real cluster state through its read-only k8s tools. The model + MCP loop works.

Crew delegation over A2A. The orchestrator log shows it calling qwen3.6, resolving the analyst's A2A card, and delegating over A2A:

POST http://${OLLAMA_HOST}/api/chat            200   (orchestrator decides)
GET  http://bmad-analyst.kagent:8080/.well-known/agent-card.json  200
a2a.client.card_resolver - Successfully fetched agent card ... 'name': 'bmad_analyst'
POST http://kagent-controller.kagent:8083/api/tasks  201   (task handed to analyst)

Real type: Agent delegation firing over A2A with a self-hosted model.

📝 Episode note. In the Sympozium episode the local model's orchestrator frequently never emitted a delegate call. Here kagent's agent-as-tool

  • A2A delegation does fire with qwen3.6 — a nice on-screen contrast. Multi-hop crews are still slow on a local model, so keep demo tasks small and timeouts generous (200–300s). Live GitHub artifact counts / token totals are captured at record time on the dashboards.

9. Cleanup

kubectl delete -k manifests/bmad-crew
helm uninstall github-mcp -n kagent
kubectl -n kagent delete secret github-mcp-token
kubectl delete -f manifests/sre/11-sre-networking-agent.yaml
kubectl delete -f manifests/sre/10-mcp-k8s-networking-server.yaml
kubectl delete -f manifests/demos/grpc-misroute/fix.yaml
kubectl delete modelconfig mac-studio-ollama -n kagent
# platform stack + cluster: tear down with deploy/99-teardown.sh (ProxOps)

10. Troubleshooting

Symptom Likely cause / fix
ACCEPTED=False on an Agent Schema error — check field names: modelConfig (string, not modelConfigRef); RemoteMCPServer vs MCPServer fields don't mix. kubectl describe agent <name>.
ToolServer has empty status.tools GitHub PAT missing/expired or wrong scope; the MCP endpoint didn't resolve. Recreate github-mcp-token with a repo-scoped fine-grained PAT.
Agent references a tool that never fires The referenced ToolServer isn't Accepted, or toolNames names a tool not in status.tools. Trim to what actually resolves.
Developer create_branch/push_files 403s PAT scope too narrow — needs Contents: RW, not just Issues.
Orchestrator answers itself instead of delegating Local models sometimes skip delegation. Strengthen the systemMessage, keep the task small, give it time.
Crew / SRE call is very slow Multi-hop delegation × a local model. Expected. Keep prompts short; client timeout 200–300s.
gRPC route "looks up" but checkout fails Protocol mismatch — a health check on the wrong port can pass while gRPC/h2c fails. Compare route port to the Service's grpc/h2c port.

Manifest inventory

Path Kind Purpose
deploy/.env.example — the one place environment-specific values live
deploy/*.sh — ordered, idempotent provisioning + stack deploy (ProxOps)
manifests/01-modelconfig-mac-studio.yaml ModelConfig Mac Studio Ollama model (the brain)
manifests/sre/10-mcp-k8s-networking-server.yaml Deployment + RemoteMCPServer mcp-k8s-networking tool backend
manifests/sre/11-sre-networking-agent.yaml Agent SRE networking troubleshooting agent
manifests/demos/grpc-misroute/{broken,fix}.yaml GRPCRoute deterministic break/fix for the SRE demo
manifests/bmad-crew/values-github-mcp.yaml Helm values GitHub MCP server → github-mcp-server-* ToolServers
manifests/bmad-crew/01..06-*.yaml Agent the six BMAD v2 agents (GitHub-operating)
manifests/bmad-crew/kustomization.yaml — kubectl apply -k the whole crew
observability/21-otel-agent-daemonset.yaml OpenTelemetryCollector default-ns DaemonSet (node ingest)
observability/22-otel-gateway-statefulset.yaml OpenTelemetryCollector default-ns StatefulSet (Dynatrace egress)
observability/kagent-otel-values.yaml Helm values point kagent OTLP at the default-ns collector
observability/10-dynakube.yaml DynaKube Dynatrace k8s-monitoring
dashboards/*.json Dynatrace health + agentic-efficiency dashboards

Sources

Research consolidated in doc kagent-research. Primary docs: https://kagent.dev/docs/kagent/concepts. GitHub MCP server: https://github.com/kagent-dev/kagent (contrib/tools/github-mcp-server). Networking MCP: https://github.com/henrikrexed/mcp-k8s-networking.