fix: high gRPC client error rate from ad calling flagd.evaluation.v2.Service/EventStream

[VanshikaR7]

Changes

The flagd-ui service updates the shared demo.flagd.json configuration file by writing directly to it using File.write!. This in-place modification triggers a REMOVE event in flagd's fsnotify watcher, which breaks active EventStream connections for clients like the ad service, resulting in a 100% gRPC error rate.

To fix this, the file writing mechanism in src/flagd-ui/lib/flagd_ui/storage.ex has been updated to use an atomic rename (writing to a temporary file first, then renaming it to the target file). This ensures fsnotify detects a CREATE or WRITE event instead of a REMOVE, maintaining stable EventStream connections.

Original Error/Logs

Click to expand the original error that triggered this fix

Title: High gRPC client error rate from ad calling flagd.evaluation.v2.Service/EventStream
Description: gRPC client error rate exceeded 5% when calling downstream service.
Caller: ad
Target: flagd.evaluation.v2.Service/EventStream
Deployment: ad
Pod: ad-66df4f848c-68pkp
Error Rate: 100%

Causality Chain (Root Cause):
- Symptom: 100% gRPC client error rate from ad calling flagd.evaluation.v2.Service/EventStream.
- Why? The ad service is unable to maintain a successful gRPC EventStream connection to the downstream flagd service.
- Why? The flagd service's internal flag synchronization was disrupted because its configuration file watcher detected the file being removed.
- Root Cause: The demo.flagd.json file was modified or replaced in the shared volume in a way that triggered a REMOVE event in flagd's fsnotify watcher at 08:48:03 UTC, breaking the active EventStream for connected clients.

Merge Requirements

For new features contributions, please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions
• Appropriate documentation updates in the docs
• Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

---

🤖 This PR was automatically generated by NudgeBee AI coding agent

Powered by AI-driven code analysis and automated fix generation

🔍 View Original Investigation

Click the link above to see the full investigation and analysis that led to this fix

View Detailed Nubi Conversation

[nudgebee-dev]

Nudgebee Change Gate

✅ Pass

• No Kubernetes workloads or manifests were modified in this diff.

Based on Nudgebee rightsizing recommendations and incident investigations from the last 30 days.

View the gate agent investigation

[gemini-code-assist]

Code Review

This pull request updates the write_state/1 function in FlagdUi.Storage to write the state to a temporary file first and then rename it to the target path, ensuring atomic file writes. There are no review comments, and I have no additional feedback to provide.

[VanshikaR7]

Nudgebee Automated Followup

Updated CHANGELOG.md to document the fix for the high gRPC client error rate from ad calling flagd.evaluation.v2.Service/EventStream, fulfilling the PR merge requirements. The code changes in src/flagd-ui/lib/flagd_ui/storage.ex were verified to correctly implement atomic file writes using File.write! and File.rename!. All CI checks are passing and there were no actionable review comments.

Changes:

CHANGELOG.md                         | 3 +++
 src/flagd-ui/lib/flagd_ui/storage.ex | 4 +++-
 2 files changed, 6 insertions(+), 1 deletion(-)

Review comments: 2 acknowledged

Commit: 79543545b96e

[nudgebee-dev]

Nudgebee Change Gate

✅ Pass

• No Kubernetes manifest changes detected in this PR.

Based on Nudgebee rightsizing recommendations and incident investigations from the last 30 days.

View the gate agent investigation

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.