MON-4059: Update TelemeterClientFailures alert by slashpai · Pull Request #550 · openshift/telemeter

slashpai · 2024-11-19T08:19:14Z

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Ref: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md#diagnosis

Use the new `metricsclient_http_requests_total` metric which would tell the difference between 4xx errors (e.g. bad pull secret) and 5xx (issue on Red Hat side). Signed-off-by: Jayapriya Pai <janantha@redhat.com>

openshift-ci-robot · 2024-11-19T08:19:19Z

@slashpai: This pull request references MON-4059 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

Details

In response to this:

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-11-19T08:21:20Z

@slashpai: This pull request references MON-4059 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

Details

In response to this:

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Ref: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md#diagnosis

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

simonpasquier

I'd recommend to create 2 alerting rules: one for the federate_from client and another one for the federate_to client. They can have the same name but different descriptions.

simonpasquier · 2024-11-19T10:17:32Z

-                  sum by (namespace) (
-                    rate(federate_requests_total{job="telemeter-client"}[15m])
-                  ) > 0.2
+                  sum by(client, status_code) (rate(metricsclient_http_requests_total{status_code!~"200"}[15m])) > 0


(suggestion) we need to preserve the namespace label. Let's also evaluate the rate of failed requests.

Suggested change

sum by(client, status_code) (rate(metricsclient_http_requests_total{status_code!~"200"}[15m])) > 0

sum by(client, status_code,namespace) (rate(metricsclient_http_requests_total{status_code!~"2..",job="telemeter-client"}[15m]))

/

on(client, namespace) group_left() sum by(client, namespace) (rate(metricsclient_http_requests_total{job="telemeter-client"}[15m])) > 0.2

openshift-bot · 2025-04-22T01:00:35Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2025-05-31T00:30:41Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2025-06-30T08:00:34Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci · 2025-06-30T08:01:17Z

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

simonpasquier · 2025-06-30T08:28:37Z

/reopen

openshift-ci · 2025-06-30T08:28:48Z

@simonpasquier: Reopened this PR.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot · 2025-06-30T08:28:51Z

@slashpai: This pull request references MON-4059 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Ref: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md#diagnosis

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

simonpasquier · 2025-06-30T08:28:52Z

remove-lifecycle rotten

simonpasquier · 2025-06-30T08:28:57Z

/remove-lifecycle rotten

openshift-ci · 2025-06-30T08:29:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: slashpai

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [slashpai]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2025-09-29T01:01:25Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2025-10-29T08:30:16Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2025-11-29T00:01:12Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci · 2025-11-29T00:01:39Z

@openshift-bot: Closed this PR.

Details

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

simonpasquier · 2025-12-01T08:52:28Z

/reopen
/remove-lifecycle rotten

simonpasquier · 2025-12-01T08:52:48Z

/lifecycle frozen

openshift-ci · 2025-12-01T08:52:58Z

@simonpasquier: The lifecycle/frozen label cannot be applied to Pull Requests.

Details

In response to this:

/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2025-12-01T08:54:08Z

@simonpasquier: Reopened this PR.

Details

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot · 2025-12-01T08:54:09Z

@slashpai: This pull request references MON-4059 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

Use the new metricsclient_http_requests_total metric which would tell the difference between 4xx errors (e.g. bad pull secret) and
5xx (issue on Red Hat side).

Ref: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md#diagnosis

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2026-03-02T01:01:14Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

juzhao · 2026-03-09T03:02:28Z

/remove-lifecycle stale

juzhao · 2026-03-09T08:50:08Z

/test e2e-aws-ovn

openshift-ci · 2026-03-09T12:12:27Z

@slashpai: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

MON-4059: Update TelemeterClientFailures alert

6ff5685

Use the new `metricsclient_http_requests_total` metric which would tell the difference between 4xx errors (e.g. bad pull secret) and 5xx (issue on Red Hat side). Signed-off-by: Jayapriya Pai <janantha@redhat.com>

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 19, 2024

openshift-ci Bot requested review from simonpasquier and thibaultmg November 19, 2024 08:21

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 19, 2024

simonpasquier reviewed Nov 19, 2024

View reviewed changes

openshift-ci Bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 22, 2025

openshift-ci Bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 31, 2025

openshift-ci Bot closed this Jun 30, 2025

openshift-ci Bot reopened this Jun 30, 2025

openshift-ci Bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jun 30, 2025

openshift-ci Bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 29, 2025

openshift-ci Bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 29, 2025

openshift-ci Bot closed this Nov 29, 2025

openshift-ci Bot reopened this Dec 1, 2025

openshift-ci Bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Dec 1, 2025

openshift-ci Bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 2, 2026

openshift-ci Bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 9, 2026

-                  sum by(client, status_code) (rate(metricsclient_http_requests_total{status_code!~"200"}[15m])) > 0
+                  sum by(client, status_code,namespace) (rate(metricsclient_http_requests_total{status_code!~"2..",job="telemeter-client"}[15m]))
+                  /
+                  on(client, namespace) group_left() sum by(client, namespace) (rate(metricsclient_http_requests_total{job="telemeter-client"}[15m])) > 0.2

Conversation

slashpai commented Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 19, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 19, 2024 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonpasquier left a comment

Choose a reason for hiding this comment

Uh oh!

simonpasquier Nov 19, 2024

Choose a reason for hiding this comment

Uh oh!

openshift-bot commented Apr 22, 2025

Uh oh!

openshift-bot commented May 31, 2025

Uh oh!

openshift-bot commented Jun 30, 2025

Uh oh!

openshift-ci Bot commented Jun 30, 2025

Uh oh!

simonpasquier commented Jun 30, 2025

Uh oh!

openshift-ci Bot commented Jun 30, 2025

Uh oh!

openshift-ci-robot commented Jun 30, 2025 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonpasquier commented Jun 30, 2025

Uh oh!

simonpasquier commented Jun 30, 2025

Uh oh!

openshift-ci Bot commented Jun 30, 2025

Uh oh!

openshift-bot commented Sep 29, 2025

Uh oh!

openshift-bot commented Oct 29, 2025

Uh oh!

openshift-bot commented Nov 29, 2025

Uh oh!

openshift-ci Bot commented Nov 29, 2025

Uh oh!

simonpasquier commented Dec 1, 2025

Uh oh!

simonpasquier commented Dec 1, 2025

Uh oh!

openshift-ci Bot commented Dec 1, 2025

Uh oh!

openshift-ci Bot commented Dec 1, 2025

Uh oh!

openshift-ci-robot commented Dec 1, 2025 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented Mar 2, 2026

Uh oh!

juzhao commented Mar 9, 2026

Uh oh!

juzhao commented Mar 9, 2026

Uh oh!

openshift-ci Bot commented Mar 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

slashpai commented Nov 19, 2024 •

edited

Loading

openshift-ci-robot commented Nov 19, 2024 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Nov 19, 2024 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Jun 30, 2025 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Dec 1, 2025 •

edited by openshift-ci Bot

Loading