Skip to content
BranchesTags

Repository files navigation

Is it Observable

sympozium.ai logo

Episode :Sympozium

This repository contains the files utilized during the tutorial presented in the dedicated IsItObservable episode related to Sympozium

What you'll build

  1. A Kubernetes workload cluster — either on Proxmox (Cluster API + the CAPMOX provider) or on GKE — with a CNI, a LoadBalancer (MetalLB on Proxmox; built-in on GKE) and a default StorageClass.
  2. Sympozium installed on it (controller, apiserver, webhook, NATS). Telemetry is sent to the episode's own collectors (below), not Sympozium's built-in collector.
  3. Two local models on a Mac Studio (or any Ollama host): qwen3.5:122b (the heavyweight — architect, dev/review, testing, devops) and qwen3.6:latest (the fast model — manager, analysis, planning, stories, challenger, observability).
  4. The bmad-ensemble — the full BMAD Crew (a Crew Manager + nine phase leads) as one Ensemble, with a custom bmad SkillPack, persistent memory, and a delegation/handoff flow ported from the Paperclip‑Bmad‑Crew.
  5. Guardrails — a SympoziumPolicy (tool gating, sub‑agent caps, sandbox bounds, feature gates) referenced by the Ensemble, plus an optional Agent Sandbox (gVisor) so every tool call runs with kernel‑level isolation.
  6. A Slack channel so you can task the team and watch the code → review → merge loop live.
  7. End‑to‑end OpenTelemetry into Dynatrace — every agent run, the controller, apiserver and mcp‑bridge emit OTLP to two collectors in the default namespace (an oteld daemonset for logs + node metrics, and an otel gateway statefulset for OTLP + cluster scraping), deployed via the OpenTelemetry Operator. Both ship to Dynatrace, alongside the Dynatrace Operator/DynaKube for infrastructure monitoring and a Dynatrace MCP the O11y Engineer queries directly.
                         ┌───────────────────────┐
        kickoff ───────▶ │   CTO / Crew Manager  │ ──── escalate ───▶ CEO
                         └───────────┬───────────┘
              routes (delegate_to_persona) to the right phase lead
   ┌──────────────┬──────────────┬───────────────┬───────────────┬──────────────┐
   ▼              ▼              ▼               ▼               ▼              ▼
 Mary  ──▶  John (PM)  ──▶  Winston   ──▶  Story Writer ──▶  Amelia    ──▶  Testing
(analysis) (planning)   (architect)     (stories)        (dev+review)    Architect
   Phase 1     Phase 2     Phase 3         Phase 3/4        Phase 4        Phase 4
                                                              │ PR + HANDOFF ──▶ CTO merges
   Challenger  ── adversarial quality gate (supervises all phases) ──────────────┐
   O11y Engineer ── observability (supervises architecture + devops) ────────────┘
   DevOps Engineer ── CI/CD, K8s, IaC (Phase 3/4)

   Every handoff is a GitHub PR/issue comment, mirrored to Slack, and recorded in MEMORY.md.

Table of contents

Concepts in 60 seconds

Sympozium is a Kubernetes‑native multi‑agent platform from the creator of k8sgpt and llmfit.

Idea What it means
Ephemeral agent Pod One pod per agent run, created and torn down per task.
CRD‑driven policy Agents, skills, schedules, channels and packs are all Custom Resources.
Job‑backed execution Each run is a Kubernetes Job — lifecycle, retries and TTL managed by the platform.
Isolated skill sidecars Every skill runs in its own sidecar with ephemeral, least‑privilege RBAC, GC'd at run end. "Give the agent tools, not trust."
Channels Bridges to Slack/Telegram/Discord/WhatsApp, routed through NATS JetStream.

Ensemble

An Ensemble is Sympozium's top‑level unit of teamwork: a single Ensemble CRD (sympozium.ai/v1alpha1) that bundles several pre‑configured agent personas (spec.agentConfigs) — each with its own system prompt, model, skills, tool policy, schedule and memory seeds — and activates them together. Activating it stamps out one Agent per persona (bmad-ensemble-cto, -architect, -code-reviewer, … — one per agentConfigs[] entry) — plus the matching Schedules, secrets and memory ConfigMaps — all carrying ownerReferences back to the Ensemble, so deleting the Ensemble garbage‑collects the whole crew.

Beyond personas, an Ensemble can also declare typed relationships between them — delegation (request a peer and await its result), sequential (one finishes before the next starts), supervision (one observes another) and stimulus (kick off the team once every persona is serving) — together with a workflow type: autonomous (the default — each persona runs on its own schedule), pipeline, or delegation (runtime hand‑offs via the built‑in delegate_to_persona tool). See the Ensembles concept doc.

This episode uses workflowType: delegation. The Crew Manager (CTO) routes each incoming task to the right phase lead through the built‑in delegate_to_persona tool, the BMAD phases chain through sequential edges, and the Challenger and O11y Engineer attach as supervision edges across phases — the crew's reporting hierarchy expressed as first‑class relationships. On top of that runtime routing, every persona still emits a structured BMAD HANDOFF block at the end of a run and mirrors it to GitHub (issues, PRs, review comments), a shared Slack thread, and its own MEMORY.md — so the coordination is observable, auditable, and survives pod restarts, exactly what we want to watch in an IsItObservable episode.

BMAD

BMAD ("Breakthrough Method for Agile AI‑Driven Development") gives the team a repeatable lifecycle, delivered here as a custom bmad SkillPack (sympozium/skills/bmad.yaml) mounted on every agent:

Phase Owner(s) What happens
1 Analysis Mary (brainstormer) Research, market/domain analysis, product briefs.
2 Planning John (product-manager) PRD lifecycle + epics/stories with acceptance criteria.
3 Solutioning Winston (architect) Architecture decisions + implementation‑readiness.
3/4 Stories Story Writer Decompose epics into implementation‑ready stories (GWT).
4 Implementation + Review Amelia (code-reviewer) Implement one story/PR, then adversarial review (Blind Hunter, Edge Case Hunter, Acceptance Auditor).
4 Test Testing Architect Generate + run unit/API/E2E tests; 100% pass before done.
Cross‑cutting Challenger / O11y Engineer / DevOps Adversarial quality gate, observability, CI/CD + infra.
Routing + merge Crew Manager (cto) Route work to phase leads; merge approved PRs.

The reviewer's three layers are Blind Hunter, Edge Case Hunter and Acceptance Auditor.

📖 Deep dive: Ensembles, Skills & Sidecars, Channels.

Prerequisites

  • A Kubernetes cluster — see Step 1 for Proxmox or GKE.
  • An Ollama host (a Mac Studio in this episode, but any machine with enough memory) reachable from the cluster network — it serves the two Qwen models.
  • For Proxmox: a management/bootstrap cluster with Cluster API + the CAPMOX provider, and a Proxmox VM template (cloud‑init image with kubeadm/kubelet/containerd installed).
  • Optional: an NFS server (for csi-driver-nfs persistent volumes) and a registry mirror (Harbor or similar) to cache container images.
  • A GitHub repo for the team to work on, plus a token with repo scope.
  • A Slack workspace where you can install a bot app.
  • Local tools: kubectl, helm, envsubst, and the sympozium CLI: 
    curl -fsSL https://deploy.sympozium.ai/install.sh | sh
# or: brew tap AlexsJones/sympozium && brew install sympozium

Step 1 — Create your cluster

Pick the path that fits your environment. Both end with a KUBECONFIG pointing at a Ready cluster.

Option A — Proxmox (Cluster API + CAPMOX)

Provision a workload cluster on your own Proxmox using Cluster API and the CAPMOX provider. You supply every environment‑specific value; nothing is hard‑coded.

1. Prepare the management cluster (one‑time): install Cluster API with the Proxmox provider.

clusterctl init --infrastructure proxmox

2. Store your Proxmox API credentials as the secret the manifest references:

kubectl create secret generic proxmox-credentials \
  --from-literal=url="https://YOUR-PROXMOX:8006/api2/json" \
  --from-literal=token="USER@pam!TOKENID" \
  --from-literal=secret="YOUR-TOKEN-SECRET"
# (field names follow your CAPMOX version's ProxmoxCluster credentialsRef schema)

3. Fill in your details in infrastructure/cluster-api/workload-cluster/values.env — Proxmox node(s), VM template ID, the control‑plane VIP, the node IP range, gateway, DNS, your registry mirror host and your SSH public key:

# excerpt — see values.env for the full list
CLUSTER_NAME=observable-llm
K8S_VERSION=v1.31.0
PROXMOX_SOURCE_NODE=pve
PROXMOX_TEMPLATE_ID=9000
CONTROL_PLANE_VIP=192.0.2.10
NODE_IP_RANGE=192.0.2.11-192.0.2.20
NODE_GATEWAY=192.0.2.1
DNS_SERVER=192.0.2.2
HARBOR_HOST=harbor.example.internal     # or your own registry mirror
SSH_PUBLIC_KEY="ssh-ed25519 AAAA... you@example.com"

4. Render and apply the cluster (the manifest uses ${VAR} placeholders that envsubst fills from values.env):

set -a; . infrastructure/cluster-api/workload-cluster/values.env; set +a
envsubst < infrastructure/cluster-api/workload-cluster/capi_llm_cluster.yaml | kubectl apply -f -
kubectl get cluster "$CLUSTER_NAME" -w        # wait for Provisioned

5. Get the kubeconfig and install a CNI so the nodes go Ready:

clusterctl get kubeconfig "$CLUSTER_NAME" > observable-llm.kubeconfig
export KUBECONFIG=$PWD/observable-llm.kubeconfig

# install any CNI — Cilium is a good fit (Istio-ambient + kgateway compatible):
helm repo add cilium https://helm.cilium.io && helm repo update
helm install cilium cilium/cilium -n kube-system
kubectl get nodes -w                          # all nodes Ready

MetalLB, csi-driver-nfs and cert‑manager are installed in the next step by deployment.sh.

The manifest pulls images through an optional registry mirror (${HARBOR_HOST}). To pull straight from the public registries, point HARBOR_HOST at a host you don't use and remove the certs.d / imageRepository blocks from the manifest.

Option B — GKE

No Proxmox? Spin up a managed cluster on GKE. On GKE you get a CNI, a LoadBalancer and a default StorageClass out of the box, so you can skip MetalLB and csi-driver-nfs.

gcloud container clusters create sympozium \
  --region=europe-west1 \
  --num-nodes=3 \
  --machine-type=e2-standard-4
gcloud container clusters get-credentials sympozium --region=europe-west1

Then run deployment.sh --skip-platform (Step 3) — it still installs cert‑manager and the Sympozium control plane. The NATS PVC binds against GKE's default standard StorageClass automatically.

The agents run on local Ollama models, so even on GKE the Mac Studio (Step 2) must be reachable from the cluster's egress — over a VPN/tunnel, or via a publicly reachable Ollama endpoint.

Step 2 — Serve the local models

Both models run locally on a Mac Studio (or any Ollama host) — no cloud API keys. Ollama exposes an OpenAI‑compatible API, so Sympozium talks to it with the provider set to ollama.

Model Crew personas Why
qwen3.5:122b architect, code-reviewer (Amelia, dev+review), testing-architect, devops-engineer The heavyweight — multi‑file implementation, architecture and deep reasoning.
qwen3.6:latest cto, brainstormer, product-manager, story-writer, challenger, o11y-engineer Faster turn‑around for routing, research, planning, review and observability.

On the Ollama host, bind it to all interfaces (so cluster pods can reach it over the network) and pull both models:

OLLAMA_HOST=0.0.0.0 ollama serve     # bind to all interfaces
ollama pull qwen3.6:latest
ollama pull qwen3.5:122b
ollama list                          # confirm both are present

The cluster reaches Ollama at http://$MACSTUDIO_IP:11434/v1.

⚠️ Pick an address outside your LoadBalancer pool. Set MACSTUDIO_IP to the Ollama host's address on your LAN, and make sure it is not inside the MetalLB range you choose in Step 3 — or a LoadBalancer Service could be handed that IP and hijack the agents' egress to Ollama.

How the local endpoint reaches the agents. The Ensemble carries the Ollama endpoint directly in spec.baseURL, so the controller propagates it (as spec.agents.default.baseURL) to every generated Agent (every bmad-ensemble-<persona>) — no post‑apply patch needed. The manifest ships a <MACSTUDIO_IP> placeholder; deployment.sh substitutes $MACSTUDIO_IP at apply time. The Ollama host is external (not a cluster node); egress to port 11434 is already allowed by the default network policies.

Step 3 — Deploy everything with deployment.sh

deployment.sh is the one‑shot orchestrator. It runs preflight checks, then: platform pieces (MetalLB, csi‑driver‑nfs + StorageClass, cert‑manager, optional kgateway) → observability (OpenTelemetry Operator + the two default‑namespace collectors + Dynatrace Operator/DynaKube + Dynatrace MCP) → Sympozium control plane via Helm (the published deploy.sympozium.ai/charts repo, pointed at the otel gateway collector) → the bmad SkillPack and the bmad-guardrails SympoziumPolicy → the Ollama + Slack secrets → the bmad-ensemble Ensemble (with the macstudio baseURL substituted in, and — with --with-sandbox — the gVisor Agent Sandbox enabled) → verification.

It needs kubectl, helm and envsubst. Pinned versions: cert‑manager v1.17.1, MetalLB v0.14.8, csi‑driver‑nfs v4.9.0, Gateway API v1.2.1, kgateway v2.0.0, Dynatrace Operator 1.9.0, OpenTelemetry Operator latest.

Dynatrace credentials (optional — the collectors deploy regardless but only export with real creds):

export DT_TENANT_URL=https://abc12345.live.dynatrace.com   # OTLP/API base, NO trailing /api
export DT_API_TOKEN=dt0c01.XXXXXXXX...                     # data-ingest + API token (collectors + DynaKube)
# optional:
export DT_DATA_INGEST_TOKEN=dt0c01.YYYY...                 # DynaKube metric ingest (defaults to DT_API_TOKEN)
export DT_MCP_TOKEN=dt0s16.ZZZZ...                         # platform token for the Dynatrace MCP
export DT_MCP_ENVIRONMENT=https://abc12345.apps.dynatrace.com  # MCP platform URL (.apps. domain)
export DT_CLUSTER_NAME=observable-llm                      # replaces CLUSTER_NAME_TO_REPLACE in the manifests

Skip the whole observability stack with --skip-observability (or SKIP_OBSERVABILITY=true).

Already have MetalLB / storage (e.g. GKE, or a cluster with add‑ons)

Skip the platform step — but install cert‑manager first (the Sympozium admission webhook needs it, and --skip-platform skips it too):

export KUBECONFIG=/path/to/your.kubeconfig
export MACSTUDIO_IP=<ollama-host-ip>       # REQUIRED
export SLACK_BOT_TOKEN=xoxb-...            # optional, needed to demo Slack
export SLACK_APP_TOKEN=xapp-...            # optional, enables Socket Mode
export DT_TENANT_URL=https://abc12345.live.dynatrace.com   # optional, Dynatrace backend
export DT_API_TOKEN=dt0c01....            # optional, Dynatrace data-ingest + API token

# cert-manager is required by BOTH the Sympozium webhook AND the OpenTelemetry Operator,
# and --skip-platform skips it, so install it first:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.1/cert-manager.yaml
kubectl -n cert-manager rollout status deploy/cert-manager-webhook --timeout=180s

./deployment.sh --skip-platform

Let the script build the platform (bare Proxmox cluster)

export KUBECONFIG=/path/to/observable-llm.kubeconfig
export MACSTUDIO_IP=<ollama-host-ip>
export METALLB_IP_RANGE=192.0.2.240-192.0.2.250   # a free block on your LAN, outside MACSTUDIO_IP
export NFS_SERVER=192.0.2.5                        # your NFS server
export NFS_SHARE=/export/sympozium
export SLACK_BOT_TOKEN=xoxb-...
export SLACK_APP_TOKEN=xapp-...
export DT_TENANT_URL=https://abc12345.live.dynatrace.com   # Dynatrace backend (see above)
export DT_API_TOKEN=dt0c01....
export DT_CLUSTER_NAME=observable-llm

./deployment.sh                 # full stack
# ./deployment.sh --with-kgateway   # also install kgateway + Gateway API

Flags: --skip-platform (cluster pieces already provisioned), --skip-observability (no OTel operator/collectors or Dynatrace), --skip-ensemble (control plane only), --with-kgateway (install kgateway + Gateway API), --with-sandbox (enable the gVisor Agent Sandbox — see Step 4½). Run ./deployment.sh --help for the full header.

Step 4 — Inside the bmad-ensemble

The Ensemble (sympozium/ensemble/ensemble-bmad-ensemble.yaml) is the full BMAD Crew — a Crew Manager plus nine phase leads, ported from the Paperclip‑Bmad‑Crew. Activating it stamps out one Agent per agentConfigs[] entry.

Persona (slug) BMAD role / phase Model Key Sympozium skills
Crew Manager (cto) Routing + merge — spans all phases qwen3.6:latest bmad, github-gitops, memory
Mary (brainstormer) Phase 1 Analysis — research, briefs qwen3.6:latest bmad, memory
John (product-manager) Phase 2 Planning — PRD, epics/stories qwen3.6:latest bmad, github-gitops, memory
Winston (architect) Phase 3 Solutioning — architecture qwen3.5:122b bmad, software-dev, github-gitops, memory
Story Writer (story-writer) Phase 3/4 — story decomposition qwen3.6:latest bmad, github-gitops, memory
Amelia (code-reviewer) Phase 4 — implement + adversarial review qwen3.5:122b bmad, software-dev, code-review, github-gitops, memory
Testing Architect (testing-architect) Phase 4 — test generation qwen3.5:122b bmad, software-dev, code-review, github-gitops, memory
Challenger (challenger) Cross-cutting quality gate qwen3.6:latest bmad, code-review, memory
DevOps Engineer (devops-engineer) Phase 3/4 — CI/CD, K8s, IaC qwen3.5:122b bmad, software-dev, github-gitops, memory
O11y Engineer (o11y-engineer) Cross-cutting observability (OTel/Dynatrace) qwen3.6:latest bmad, software-dev, memory

Shape (abridged — see the manifest for full system prompts and seeds):

apiVersion: sympozium.ai/v1alpha1
kind: Ensemble
metadata: { name: bmad-ensemble, namespace: default }
spec:
  enabled: true
  workflowType: delegation            # the CTO routes work to phase leads at runtime
  policyRef: bmad-guardrails          # cluster-wide guardrails (Step 4½)
  baseURL: "http://<MACSTUDIO_IP>:11434/v1"   # local Ollama endpoint for every Agent
  authRefs: [{ provider: ollama, secret: bmad-ollama-key }]
  channelConfigs: { slack: bmad-slack-tokens }
  skillParams: { github-gitops: { repo: "isItObservable/Sympozium" } }
  stimulus:                           # injected into the CTO once every pod is Serving
    name: kickoff
    prompt: "Review the repo, route work to the right phase lead, keep the workflow moving."
  relationships:                      # routing + phase flow + supervision (the crew hierarchy)
    - { source: kickoff, target: cto, type: stimulus }
    - { source: cto, target: architect, type: delegation, condition: "architecture / solutioning" }
    - { source: product-manager, target: architect, type: sequential }
    - { source: challenger, target: code-reviewer, type: supervision }
    # …one delegation edge per phase lead, sequential edges for the BMAD flow, and
    #   challenger/o11y supervision edges — see the manifest.
  agentConfigs:
    - name: code-reviewer             # "Amelia" — implement + adversarial review
      displayName: "Code Reviewer / Developer (Amelia)"
      model: qwen3.5:122b             # the heavyweight local model
      systemPrompt: |
        You are Amelia. In dev mode you implement the next ready story (one PR, tests
        for every AC); in review mode you review with Blind Hunter / Edge Case Hunter /
        Acceptance Auditor. You never merge — the CTO does.
      skills: [bmad, software-dev, code-review, github-gitops, memory]
      toolPolicy: { deny: [delete_branch, merge_pull_request] }
      schedule: { type: sweep, interval: "1h", task: "Implement the next ready story, then review open PRs…" }
      memory: { enabled: true, seeds: ["Implement one story per PR; tests for every AC.", "Review layers: Blind Hunter, Edge Case Hunter, Acceptance Auditor."] }
      channels: [slack]
    # … cto, brainstormer, product-manager, architect, story-writer, testing-architect,
    #   challenger, devops-engineer, o11y-engineer (see the manifest for each).

The flow

The Crew Manager (CTO) routes each incoming task to the right phase lead via delegate_to_persona, and work chains through the BMAD phases; the Challenger and O11y Engineer supervise across phases:

kickoff ▶ CTO ─routes▶ Mary(analysis) ▶ John(PRD) ▶ Winston(arch) ▶ Story Writer
                         ▶ Amelia(dev+review) ▶ Testing Architect ▶ CTO merges
   Challenger ── adversarial gate (supervises planning / architecture / code)
   O11y Engineer ── observability (supervises architecture + devops)
   DevOps Engineer ── CI/CD, K8s, IaC

Note the tool policies that enforce roles: only the CTO can merge_pull_request; every other persona denies merge_pull_request and delete_branch. Every handoff is posted as a PR/issue comment and mirrored to Slack so a human can watch the crew live, and the platform's workflowType: delegation + typed relationships make the routing and supervision first‑class (and observable).

deployment.sh applies the SympoziumPolicy, then this Ensemble (with the Ollama baseURL substituted in), then creates bmad-ollama-key (placeholder) and bmad-slack-tokens. To watch the crew come up:

sympozium                                  # TUI → Personas tab shows bmad-ensemble
kubectl -n default get ensemble bmad-ensemble        # Personas / Installed / Workflow / Phase
kubectl -n default get agent | grep bmad-ensemble    # the 10 generated Agents

Agent or Ensemble — do you hand‑write the agents?

No. You don't author individual Agent resources for a team — the Ensemble pre‑packages the whole team and the controller stamps out the Agents for you. Applying one Ensemble produces one Agent per agentConfigs[] entry (bmad-ensemble-cto, -architect, -code-reviewer, … — ten in all), each with its own SympoziumSchedule and memory ConfigMap, all owned by the Ensemble (so kubectl delete ensemble cleans the lot up). A standalone Agent CR is for a single one‑off agent; the moment you want a coordinated group with shared providers, policy, channels and memory seeds, you use an Ensemble. Edit spec.agentConfigs[] and re‑apply to add or change a teammate — never touch the generated Agents directly (the controller owns them and will reconcile your changes away).

Step 4½ — Guardrails & sandboxed execution

A bmad-ensemble writes code and runs commands, so it ships with two layers of containment — both referenced by the Ensemble, both observable.

Guardrails: SympoziumPolicy

sympozium/ensemble/sympoziumpolicy-bmad.yaml defines a SympoziumPolicy named bmad-guardrails that the Ensemble references via spec.policyRef; the controller applies it to every generated Agent. It is the cluster‑side backstop on top of each persona's per‑agent toolPolicy:

apiVersion: sympozium.ai/v1alpha1
kind: SympoziumPolicy
metadata: { name: bmad-guardrails, namespace: default }
spec:
  toolGating:                 # allow the dev-team tools; deny/ask the rest
    defaultAction: allow
    rules:
      - { tool: execute_command, action: allow }   # build + test
      - { tool: write_file,      action: allow }
      - { tool: fetch_url,       action: allow }
  subagentPolicy: { maxDepth: 2, maxConcurrent: 3 }     # cap sub-agent fan-out
  sandboxPolicy:                                         # bounds when the sandbox runs
    required: false
    maxCPU: "4"
    maxMemory: 8Gi
    agentSandboxPolicy: { required: false, defaultRuntimeClass: gvisor, allowedRuntimeClasses: [gvisor, kata] }
  featureGates: { code-execution: true, file-access: true, sub-agents: true, browser-automation: false }
  imagePolicy: { allowedRegistries: ["ghcr.io/sympozium-ai/", "docker.io/library/"] }

It's observable. Every tool call the policy blocks is emitted as the OTel metric sympozium.access.denied (tagged with agent/instance/tool), so you can graph "denied tool calls per agent" right next to the run metrics — the guardrail proves itself in your backend.

kubectl -n default get sympoziumpolicy bmad-guardrails
kubectl -n default get agent bmad-ensemble-code-reviewer -o jsonpath='{.spec.policyRef}{"\n"}'   # -> bmad-guardrails

Sandboxed execution: Agent Sandbox (optional)

For kernel‑level isolation, run each agent's tool calls inside a Kubernetes Agent Sandbox (gVisor or Kata). The Ensemble carries this in spec.agentSandbox, propagated to every generated Agent:

./deployment.sh --with-sandbox            # patches spec.agentSandbox onto the Ensemble
# equivalently:
kubectl -n default patch ensemble bmad-ensemble --type=merge \
  -p '{"spec":{"agentSandbox":{"enabled":true,"runtimeClass":"gvisor"}}}'

Prerequisites: the agent‑sandbox CRDs installed, and a gvisor (or kata) RuntimeClass on the nodes — on GKE enable GKE Sandbox; on Proxmox install gVisor (runsc) on each node. Without a matching RuntimeClass the sandboxed pods stay Pending, so the tutorial leaves the sandbox off by default and bmad-guardrails keeps sandboxPolicy.required: false.

Observability of the sandbox — and a real gap. Sandboxed runs still emit the full agent telemetry (sympozium.agent.*, sympozium.tool.invocations, gen_ai.usage.*) and a sympozium.pod.create event, and the pod's runtimeClassName=gvisor is visible in the Kubernetes API. What Sympozium does not emit is per‑sandbox resource usage (the gVisor sandbox's own CPU/memory/syscall counts) — for that you fall back to kube-state-metrics + cAdvisor (or the runtime's own metrics). That's exactly the kind of "missing observability area" this episode is meant to surface.

Step 5 — Showcase agent memory

Memory is what turns the crew's stateless model calls into a team that gets better over time. Each agent has persistent memory enabled and seeded — no external database; it lives in etcd as a <agent>-memory ConfigMap holding a MEMORY.md file that is mounted read‑only into the pod, prepended to the agent's context, and patched after each run.

The memory.seeds in the Ensemble give each agent starting context from run #1 (e.g. the Crew Manager knows its routing map; Amelia knows the three review layers; the O11y Engineer knows "change management first"). Over time the agents accumulate learned memory on top of the seeds.

Two layers of memory. This episode uses per‑persona memory — each agent's own MEMORY.md. An Ensemble can also enable optional pack‑level shared memory: a store the whole team reads and writes through the workflow_memory_search / workflow_memory_store / workflow_memory_list tools, with per‑persona read/write access control, plus an advanced Synthetic Membrane layer (selective permeability, provenance tracking, token budgets, circuit breakers, time decay). Here the shared layer is intentionally GitHub + Slack, so every cross‑agent fact stays visible to humans too — but see the Ensembles doc if you want the agents to share a private memory pool instead.

See it working

# What Amelia (code-reviewer / developer) remembers
kubectl -n default get configmap bmad-ensemble-code-reviewer-memory \
  -o jsonpath='{.data.MEMORY\.md}'

# …or the Challenger's accumulated defect patterns
kubectl -n default get configmap bmad-ensemble-challenger-memory \
  -o jsonpath='{.data.MEMORY\.md}'

Talking points for the episode: continuity across runs with no external DB; seeds vs. learned memory; and how Amelia and the Challenger accumulate recurring‑defect patterns across PRs to speed up future reviews.

Step 6 — Connect Slack

Channels let you task the team and watch it work from Slack. Sympozium's Slack channel uses Socket Mode, so you don't need a public webhook URL. (Adapted from the Sympozium Channels docs.)

6a. Create the Slack app

  1. Create a Slack app → enable Socket Mode.
  2. App Home → Messages Tab: enable it and allow users to message the app.
  3. Event Subscriptions (bot events): message.im, message.channels, app_mention.
  4. Generate tokens: a bot token xoxb-… (OAuth & Permissions) and an app token xapp-… (Basic Information → App‑Level Tokens, scope connections:write).
  5. Reinstall the app after changing scopes/events.

6b. Provide the tokens

Export both before running deployment.sh — it provisions the bmad-slack-tokens secret and binds the slack channel to every agent:

export SLACK_BOT_TOKEN=xoxb-...
export SLACK_APP_TOKEN=xapp-...
./deployment.sh --skip-platform

Fallback: if you omit SLACK_APP_TOKEN, Sympozium falls back to the Slack Events API, which needs a publicly reachable webhook URL. Use Socket Mode for the tutorial.

6c. Verify

kubectl -n default get agent | grep bmad-ensemble
for a in cto brainstormer product-manager architect story-writer \
         code-reviewer testing-architect challenger devops-engineer o11y-engineer; do
  kubectl -n default get agent "bmad-ensemble-$a" \
    -o custom-columns=NAME:.metadata.name,CHANNELS:.status.channels --no-headers
done
kubectl -n default logs -l sympozium.ai/channel=slack -f

Then DM the bot (or @mention it in a channel) — it spawns an AgentRun and replies in‑thread.

Step 7 — Run the team

  1. Open or @mention a feature request in Slack (or file an issue in the target repo).
  2. The Crew Manager (cto) routes it to the right phase lead — research to Mary, requirements to John, architecture to Winston, story decomposition to the Story Writer.
  3. Amelia (code-reviewer, qwen3.5:122b) implements the next ready story, writes tests, and opens a single PR with a HANDOFF block; the Testing Architect adds coverage.
  4. Amelia (review mode) and the Challenger run the adversarial layers and post APPROVE or REQUEST_CHANGES; the O11y Engineer checks the observability story.
  5. On APPROVE the Crew Manager merges; on REQUEST_CHANGES it loops back to implementation.
  6. Watch the whole crew in Slack, and the spans in your observability backend.
kubectl -n default get agentruns -w

Observability

Because it's an IsItObservable episode, we don't use Sympozium's built‑in collector. deployment.sh deploys the OpenTelemetry Operator (cert‑manager is its prerequisite — already installed in the platform step) and two collectors in the default namespace (infrastructure/observability/):

Collector Mode Receives Forwards
oteld daemonset node logs (filelog from /var/log/pods) + its own Prometheus metrics, k8s‑enriched Dynatrace (otlphttp)
otel statefulset (gateway) OTLP traces/metrics/logs from the app + cluster‑wide Prometheus scrape (kubernetes-pods) Dynatrace (otlphttp)

Both run the contrib image otel/opentelemetry-collector-contrib:0.154.0 under the otelcontribcol ServiceAccount (rbac.yaml). Their pipelines all follow the rule memory_limiter first, batch last, with k8sattributes + resource + transform (Dynatrace dt.kubernetes.* enrichment) and cumulativetodelta on metrics. They read the Dynatrace endpoint + token from a dynatrace Secret (dynatrace_oltp_url / dt_api_token) that deployment.sh creates from DT_TENANT_URL / DT_API_TOKEN.

The episode values.yaml points the whole control plane and every agent instance at the gateway (observability.endpoint and defaultInstance.otlpEndpoint → otel-collector.default.svc.cluster.local:4317), tagged deployment.environment=isitobservable-episode. So agent runs, the controller, apiserver and mcp‑bridge flow OTLP → otel gateway → Dynatrace; you'll see gen_ai.usage.* token metrics and end‑to‑end agent traces across the whole BMAD loop.

Dynatrace Operator, DynaKube & the Dynatrace MCP

On top of the OTLP path, deployment.sh installs the Dynatrace Operator (Helm, oci://public.ecr.aws/dynatrace/dynatrace-operator, csidriver.enabled=false) and a DynaKube (infrastructure/observability/dynatrace/dynakube.yaml) for infrastructure + Kubernetes API monitoring and metadata enrichment (apiUrl: <DT_TENANT_URL>/api, ActiveGate with kubernetes-monitoring, routing, dynatrace-api, metrics-ingest; Istio support on). It needs a dynakube Secret in the dynatrace namespace (apiToken, optional dataIngestToken).

It also deploys the Dynatrace MCP as a dynatrace-mcp MCPServer CR (sympozium/mcp/dynatrace-mcp.yaml, image mcp/dynatrace-mcp-server) wired into the O11y Engineer agent (spec.agentConfigs[].mcpServers). That gives the agent a read‑only slice of Dynatrace — list_problems, execute_dql, generate_dql_from_natural_language, find_entity_by_name, chat_with_davis_copilot … — so it can validate live telemetry and surface problems during its observability sweeps. Credentials come from a dynatrace-mcp-secret (DT_ENVIRONMENT = the .apps. platform URL, DT_PLATFORM_TOKEN).

agents/control plane ──OTLP──▶  otel (gateway, default ns) ──▶ Dynatrace
nodes (logs+metrics) ─filelog▶  oteld (daemonset, default ns) ─▶ Dynatrace
infra/K8s API        ──────────  DynaKube / ActiveGate ────────▶ Dynatrace
O11y Engineer agent  ◀──query── Dynatrace MCP (dt.* tools) ◀──── Dynatrace

What you can actually graph

Sympozium emits these OTel signals — enough to build a dashboard per agent and watch the BMAD loop:

Signal What it tells you
sympozium.agent.runs, sympozium.agent.run.completed run throughput per agent / per persona
sympozium.agent.duration_ms how long each agent run takes (per persona — e.g. architect vs review vs o11y)
sympozium.tool.invocations which tools each agent uses, how often
sympozium.access.denied policy in action — tool calls blocked by bmad-guardrails
sympozium.pod.create, sympozium.errors scheduling activity and failures
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.chat local‑model token cost per run

Attributes like sympozium.instance.name, sympozium.agentrun.id and deployment.environment let you slice all of the above by agent and by episode.

Coverage & gaps (the point of this episode)

This episode is meant to exercise most of Sympozium's surface — Ensemble, Agents, SkillPacks, Schedules, Channels, memory, SympoziumPolicy, and the Agent Sandbox — and call out where observability runs thin. The biggest gap today: when the Agent Sandbox is on, Sympozium does not emit per‑sandbox resource metrics (the gVisor sandbox's own CPU/memory/syscalls). You can see that a run was sandboxed (runtimeClassName=gvisor, sympozium.pod.create) and its agent‑level telemetry, but for sandbox resource cost you currently fall back to kube-state-metrics + cAdvisor. Channel sidecars also emit limited OTLP today. These are tracked as follow‑ups for the episode.

Because telemetry routes through the two default‑namespace collectors (not Sympozium's built‑in one), the collectors' own health is observable too: the otel gateway exposes enableMetrics: true and both collectors self‑scrape their :8888 internal metrics — watch otelcol_exporter_send_failed_* and otelcol_processor_dropped_* to confirm data is actually reaching Dynatrace.

Ready‑made dashboards & notebook (dtctl)

You don't have to build the views by hand. infrastructure/observability/dynatrace/ ships three dtctl‑ready artifacts, every tile grounded in the signals above (all DQL validated live on the episode tenant, coalesce(new, old) semconv so they survive the gen_ai.system → gen_ai.provider.name rename):

Artifact What it shows
Notebook — notebooks/notebook-sympozium-observability.yaml Metrics → logs → traces walkthrough (the agentrun → agent.run → gen_ai.chat / execute_tool / skill.exec tree).
Dashboard — Control Plane Health — dashboards/dashboard-control-plane-health.yaml Component CPU/memory, restarts/OOM, run throughput, errors by reason, allowed‑vs‑denied, handoff latency, delegation, collector pipeline.
Dashboard — Agent / Ensemble Run — dashboards/dashboard-agent-ensemble-run.yaml Token consumption (total/input/output), top tools · models · skills, slowest tools & skills, config‑tuning KPIs, and a Memory section (reads vs writes, retrieve vs store latency).

Deploy them once your dtctl context points at your tenant (.apps. platform host):

cd infrastructure/observability/dynatrace
for f in dashboards/*.yaml notebooks/*.yaml; do dtctl apply -f "$f"; done

Each file pins a stable id, so re‑applying updates the document instead of creating a duplicate. Full prerequisites, token scopes and a round‑trip edit workflow are in infrastructure/observability/dynatrace/README.md.

Cleanup

kubectl -n default delete ensemble bmad-ensemble         # removes the agents + their instances
kubectl -n default delete sympoziumpolicy bmad-guardrails
helm -n sympozium-system uninstall sympozium sympozium-crds
# Observability:
kubectl -n default delete opentelemetrycollector otel oteld
kubectl delete -f infrastructure/observability/opentelemetry-collector/rbac.yaml
kubectl -n sympozium-system delete mcpserver dynatrace-mcp
kubectl delete -f infrastructure/observability/dynatrace/dynakube.yaml 2>/dev/null || true
helm -n dynatrace uninstall dynatrace-operator
kubectl delete -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
# Proxmox: tear down the workload cluster from the management cluster:
kubectl delete cluster "$CLUSTER_NAME" -n default
# GKE: gcloud container clusters delete sympozium --region=europe-west1

References

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages