HA setup, inactive icingadb-redis keeps growing

[erik911]

Setup:
Icinga2 master A ( HA ) -> IcingaDB A ( HA ) -> Redis A
Icinga2 master B ( HA ) -> IcingaDB B ( HA ) -> Redis B

Icinga DB version: v1.5.1
Redis server v=8.2.6

Issue:
Dump file ( dump.rdb ) and memory usage of Redis database on server with inactive IcingaDB node keeps growing significantly. Active IcingaDB instance keeps size of dump file and memory usage of Redis database within 'normal' values.

For example:
Redis A 2MB
Redis B 400MB

After force switch of active IcingaDB node A to node B dump file size is reduced to 2MB as well. But, A will start growing.

Is it normal behavior? If yes, is it possible to implement command for manual switch of active node?

[oxzi]

Thanks for creating this issue.

No, this is definitely not the expected behavior. Could you please share a few details about your configuration and paste the (redacted) logs of the two Icinga 2 and Icinga DB nodes?

[yhabteab]

The icinga:runtime and icinga:runtime:state Redis streams won't be consumed by the passive (inactive) Icinga DB instance, and both streams can cache up to a million entries each before Redis starts to evict old entries. So, depending on the size of your environment and e.g., the size of check outputs, it will definitely consume a lot more memory than the active instance.

So, the question is: Why don't we always consume and discard both streams on the passive instance too instead of caching them endlessly, but then purge them all together when the passive instance becomes active?

[julianbrost]

So, the question is: Why don't we always consume and discard both streams on the passive instance too instead of caching them endlessly, but then purge them all together when the passive instance becomes active?

I don't see a reason why this shouldn't be done. Just from my memory without looking at the code: it's probably similar code for processing the history and runtime updates that deletes entries after they were processed. For runtime updates, we don't need to catch up from the runtime update stream but can just do a full sync which is why the runtime updates can just be discarded every time that's done. And that combination probably resulted in the oversight that this issue describes.

[julianbrost]

For the record: we've received an e-mail¹ of an entity that ran into the same issue but doesn't want to comment on GitHub themselves. (So count this as a generic +1 comment. 🙈)

Footnotes

1. Internal reference: ref/IP/64630 (but that hardly contains any new information over this issue with this comment) ↩