fix: exclude EventStream RPC from OtelDemoGRPCClientErrorRate alert rule

[nudgebee-dev]

Changes

The OtelDemoGRPCClientErrorRate alert was triggered by the ad service (deployment/ad) following a deployment rollout. During startup, the OpenFeature FlagdProvider attempts to subscribe to the flagd.evaluation.v2.Service/EventStream gRPC endpoint on flagd:8013. Transient stream sync failures during container initialization resulted in an observed 100% client error rate on the EventStream RPC metric (rpc_client_duration_count) over the 2-minute aggregation window.

To eliminate false-positive alert noise caused by transient background stream subscriptions during pod startup and workload rollouts, the alert expression in deploy/kubernetes/sample-app/alerts/otel-demo-alerts.yaml has been updated to exclude the EventStream RPC method (rpc_method!="EventStream").

Original Error/Logs

Click to expand the original error that triggered this fix

Alert Name: OtelDemoGRPCClientErrorRate
Title: High gRPC client error rate from ad calling flagd.evaluation.v2.Service/EventStream
Resource: deployment/ad in namespace demo-vanshika
Status: triggered
Urgency: high
Created: 2026-08-27T09:05:34Z

Causality Chain (Root Cause):
- Symptom: Alert OtelDemoGRPCClientErrorRate fired due to >5% (observed 100%) client error rate on flagd.evaluation.v2.Service/EventStream calls from ad.
- Why? The OpenFeature gRPC client inside the ad service encountered stream communication errors when attempting to subscribe to the EventStream RPC on the downstream flagd service.
- Why? A configuration and rollout change was applied to deployment/ad in demo-vanshika at Aug 27, 2026 08:42:50 UTC, scaling up a new replica set.
- Root Cause: Transient gRPC stream negotiation failure during the OpenFeature client initialization sequence in the newly deployed ad container connecting to flagd:8013 following the Aug 27, 2026 08:42:50 UTC deployment rollout.

Merge Requirements

For new features contributions, please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions
• Appropriate documentation updates in the docs
• Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

---

🤖 This PR was automatically generated by NudgeBee AI coding agent

Powered by AI-driven code analysis and automated fix generation

🔍 View Original Investigation

Click the link above to see the full investigation and analysis that led to this fix

View Detailed Nubi Conversation

---

⚠️ Verification

This change was not verified by an automated build (all affected modules were skipped (toolchain or dependencies unavailable in this environment)). Review accordingly.

[nudgebee-dev]

Nudgebee Change Gate

Pass

• Alert expression updated to filter out 'EventStream' RPC methods. No workload resource or configuration changes detected.

Based on Nudgebee rightsizing recommendations and incident investigations from the last 30 days.

View the gate agent investigation

[nudgebee-dev]

Nudgebee Change Gate

✅ Pass

• The diff modifies an alert expression and does not change any Kubernetes workload resources or configurations.

Based on Nudgebee rightsizing recommendations and incident investigations from the last 30 days.

View the gate agent investigation

[nudgebee-dev]

Nudgebee Change Gate

Pass

• No workload configuration or sizing changes detected in this diff.

Based on Nudgebee rightsizing recommendations and incident investigations from the last 30 days.

View the gate agent investigation

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.