Investegate incident in NHN 25.02.2025

[havardelnan]

On the 25.02 a rollout of the agent v2 resulted in a outage in the ror-api.

Root cause:
mongodb stoped replying to queries in a timely manner due to missing indexes on the resourcev2 collection.

Other issues observed:
RabbitMQ had a que that had no consumers making it fill the memory allocated.
The api starts slowly due to ldap servers queried from dns not responding.

Remediation:

• implement indexes on resourcev2 collection #36
• Implement rate limiting/memory monitoring #32
• taint unavailable ldap-servers #33
• investegate possible memory leak in resource v2 endpoint handling #34
• Make all ques registered for micro services autodelete #38
• Create 500 page that links to alternative management documentation #39