Notifications configuration discovery and health monitoring

[nilmerg]

Even though Icinga Notifications will add high availability with v1.0, coupling more than one with Icinga DB will not be supported in the foreseeable future. But with the notification component being a very integral part of monitoring itself, we have to ensure it works seamlessly in our recommended HA architecture.

HA Responsibility

In case the connection to Icinga Notifications cannot be established, Icinga DB must pass on responsibility to another instance.

Health Monitoring

The database table icingadb_instance is already used by Icinga DB Web to identify the responsible instance and verifies its heartbeat. What I'd like to see is a flag in the same table that allows Web to verify a working notification component.

Configuration Discovery

One aspect of our recommendation is how Icinga DB connects to Icinga Notifications: by a unix domain socket.
Our packages will set this up by default and so it's always the same path. This brought me to the idea that Icinga DB may also indicate the availability of said socket path inside icingadb_instance so that Web can ensure the connection will work. Since the process needs access to the socket so see it, it's username should also be reported by this and written to the database so that the UI has a guarantee that both settings are valid values.

Goals

This should allow Web to do the following:

• Indicate in the Health section whether the notification component is active
• Provide a simple toggle to enable/disable the notification component (as otherwise it will require the user to enter an address and a username)