Kiali — LightSpeed Troubleshooting Evaluation

OSSM evaluation results (OLS): RESULTS_OSSM.md
Latest evaluation results: RESULTS.md
OSSM evaluation guide: OSSM.md
Full setup & run guide: DEVELOPMENT.md

End-to-end evaluation of the OpenShift LightSpeed troubleshooting agent against a live Kiali / Istio cluster.

Conversations are defined in scenarios/conversations.yaml. Each one deploys a broken workload via a setup script, asks the agent to diagnose and fix it, then cleans up. Responses are scored with LLM-based metrics (custom:answer_correctness, custom:keywords_eval, custom:tool_eval).

---

Architecture

flowchart LR
    BOOKINFO["Bookinfo app"]
    ISTIO["Istio mesh"]
    KIALI["Kiali"]
    MCP["MCP Server :8089"]
    OLS["OLS Service :8080"]
    EVAL["lightspeed-eval"]
    JUDGE["Judge LLM"]

    BOOKINFO --- ISTIO
    ISTIO --- KIALI
    KIALI -->|REST API| MCP
    MCP -->|tools| OLS
    OLS -->|tool calls| MCP
    EVAL -->|query| OLS
    EVAL -->|score| JUDGE

Loading

Component	Role
Kubernetes MCP Server	Exposes Kiali observability tools via MCP protocol so OLS can query the mesh
OpenShift Lightspeed (OLS)	The AI troubleshooting agent under evaluation
lightspeed-evaluation	Sends scenario queries to OLS and scores responses with the judge LLM
Judge LLM	Independent model (Claude Opus / Gemini) that scores agent correctness

---

LLM Configuration

Two LLM providers are supported. Switch with PROVIDER=openai (default) or PROVIDER=google.

Provider	OLS backend	Judge LLM	Credentials
openai (default)	Gemini via OpenAI-compat endpoint	Claude Opus via Vertex AI	~/.openai/openai_api_key.txt + ~/.gcp/gcp_credentials.txt
google	Gemini via Google Vertex AI	Gemini via Vertex AI	~/.gcp/gcp_credentials.txt

See DEVELOPMENT.md for the full credential setup guide.

---

Quick start

# 1. Set up credentials (see DEVELOPMENT.md §2)

# 2. Install the evaluation framework
make setup

# 3. Terminal 1 — MCP server
make run-mcp

# 4. Terminal 2 — OLS service
make run-ols               # openai (default)
make run-ols PROVIDER=google

# 5. Terminal 3 — Run evaluations
make all
make all PROVIDER=google

# 6. Generate report
make generate-results

---

Makefile reference

Setup

Target	Description
make setup	Create venv/ and install the evaluation framework
make setup-dashboard	Clone and install the web dashboard
make check-provider	Show active provider, OLS config, and system config

Services

Target	Description
make run-ols [PROVIDER=…]	Run the LightSpeed service container (port 8080)
make run-mcp	Start the Kubernetes MCP server with Kiali toolset (port 8089)
make run-dashboard	Start the evaluation dashboard (port 5173)
make run-dashboard OLS_ENV=true	Start the OSSM evaluation dashboard (port 5173)

Evaluation

Target	Description
make all [PROVIDER=…]	Run all conversations
make fix_bookinfo_fault_injection	Run one conversation
make fix_bookinfo_routing	Run one conversation
make check_mesh_status	Run one conversation
make troubleshoot_latency_trace	Run one conversation
make generate-results [PROVIDER=…]	Generate RESULTS.md
make clean-results [PROVIDER=…]	Wipe results/<provider>/

OSSM (OpenShift LightSpeed)

See OSSM.md for the full guide.

Target	Description
make test	Run OSSM conversations (OpenAI, custom:answer_correctness)
make check_mesh_status-test	Run the OSSM mesh status conversation
make run-dashboard OLS_ENV=true	Start the OSSM web dashboard
make generate-ossm-results	Generate RESULTS_OSSM.md from ossm/results/

Overridable variables

Variable	Default	Description
PROVIDER	openai	LLM provider (openai or google)
KIALI_ENDPOINT	https://kiali-istio-system.apps-crc.testing/	Kiali UI/API URL
OLS_IMAGE	quay.io/openshift-lightspeed/lightspeed-service-api:latest	OLS image
WAIT_SECONDS	200	Seconds to wait after setup/cleanup for metrics
OLS_ENV	false	Set to true for OSSM dashboard paths (ossm/conversations.yaml, ossm/results/)

---

Project layout

.
├── Makefile                          # Setup and service targets
├── DEVELOPMENT.md                    # Step-by-step setup and run guide
├── scenarios/
│   ├── scenarios.mk                  # Evaluation targets (included by Makefile)
│   ├── conversations.yaml            # All evaluation conversations
│   ├── fix_bookinfo_fault_injection/ # Fault injection scenario
│   ├── fix_bookinfo_routing/         # Traffic routing scenario
│   ├── troubleshoot_latency_trace/   # Latency / trace scenario
│   └── check_mesh_status/            # (no setup/cleanup needed)
├── ossm/
│   ├── ossm_scenarios.mk             # OSSM evaluation targets
│   ├── conversations.yaml            # OSSM conversations (OLS reporting)
│   └── results/                      # OSSM evaluation output
├── system/
│   ├── system_openai.yaml            # Judge + API config for openai provider
│   └── system_google.yaml            # Judge + API config for google provider
├── olsconfig/
│   ├── olsconfig-openai.yaml         # OLS config for openai provider
│   └── olsconfig-google.yaml         # OLS config for google provider
├── mcp_config.toml                   # Kubernetes MCP server config
├── scripts/
│   ├── generate_results.py           # RESULTS.md generator
│   └── generate_ossm_results.py      # RESULTS_OSSM.md generator
├── dashboard/                        # git-ignored — from make setup-dashboard
└── results/                          # evaluation output
    ├── <conv>_<model>.md             # committed per-run detail pages
    └── evaluation_*.csv / *.json     # git-ignored raw data

---

Conversations

All conversations are defined in scenarios/conversations.yaml with optional setup_script / cleanup_script (resolved relative to the YAML file).

Conversation	Category	Description
check_mesh_status	Health	Agent reports mesh and service health
fix_bookinfo_fault_injection	Fault injection	100% HTTP 503 on ratings — diagnose and fix
fix_bookinfo_routing	Traffic routing	reviews-v3 weight 0 — find and fix
troubleshoot_latency_trace	Latency / tracing	3-second delay on ratings — trace and fix