pepr watcher may enter into a healthy but non-functioning state

[corang]

Environment

Device and OS: Azure Kubernetes Ubuntu
App version: 1.1.5
Kubernetes distro being used: AKS 1.35.1

Steps to reproduce

1. k8s api responds with TooManyRequests

Expected result

pepr either keeps retrying with a backoff or goes unhealthy so it can be restarted

Actual Result

pepr reports healthy but stops functioning

Visual Proof (screenshots, videos, text, etc)

https://defense-unicorns.slack.com/archives/C08KECJFRDF/p1783615036367229

Severity/Priority

Med/High? Not great when pepr starts silently failing in a cluster

[corang]

Worth noting this cluster has a 2nd instance of pepr that's also an operator style like core's and it didn't have these same issues