Skip to content

fix(monitoring): pin Grafana to 11.6.11 + fix prometheus datasource URL - #146

Open
0xbuidler wants to merge 2 commits into
chutesai:mainfrom
0xbuidler:fix/grafana-plugin-compatibility
Open

fix(monitoring): pin Grafana to 11.6.11 + fix prometheus datasource URL#146
0xbuidler wants to merge 2 commits into
chutesai:mainfrom
0xbuidler:fix/grafana-plugin-compatibility

Conversation

@0xbuidler

@0xbuidler 0xbuidler commented Aug 17, 2026

Copy link
Copy Markdown

Summary

Fix two monitoring stack issues affecting Grafana dashboards:

  • Pin Grafana to 11.6.11 to ensure compatibility with newer frontend plugins such as the Infinity datasource.
  • Point the Grafana Prometheus datasource at the actual Kubernetes service name, prometheus-server.

Changes

  • Set grafana.image.tag to 11.6.11 in monitoring-values.yaml.j2.
  • Change the Prometheus datasource URL from: http://prometheus to http://prometheus-server

Why

The monitoring deployment was using Grafana 11.6.0 together with newer Grafana plugins. This could leave datasources successfully provisioned and healthy on the backend while dashboard variables failed to resolve them correctly in the frontend.

Additionally, the Grafana Prometheus datasource was configured to query:

http://prometheus

while the monitoring chart actually creates the Prometheus server service as:

prometheus-server

This caused Grafana queries to fail with DNS errors such as:

lookup prometheus: no such host

Using prometheus-server makes the datasource match the service already created by the monitoring chart, without renaming Kubernetes resources or affecting other consumers.

Testing

Redeployed the monitoring stack with:

ansible-playbook \
  -i ~/chutes/inventory.yml \
  -i ~/chutes/tee-inventory.yml \
  playbooks/deploy-charts.yml \
  --tags monitoring-charts

Verified:

  • Grafana runs version 11.6.11.
  • Infinity datasource loads successfully.
  • Chutes Monitor cluster variables resolve and display the available TEE clusters.
  • Prometheus datasource resolves prometheus-server successfully.
  • Grafana Prometheus queries no longer fail with DNS lookup errors.
  • Existing TEE federation monitoring continues to work.

Scope

This keeps the changes limited to the generated monitoring Helm values and does not rename existing Kubernetes services or alter the Prometheus deployment topology.

@0xbuidler 0xbuidler changed the title fix(monitoring): pin Grafana to 11.6.11 fix(monitoring): pin Grafana to 11.6.11 + fix prometheus datasource URL Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant