Domino Grafana
Grafana is a powerful widely used open source solution for visualizing statics and logs.
This repository is intended to provide the basic functionality to collect statistics and logs from HCL Domino.
The current development focus is Domino on Linux including containers.
But the servertask is also available on Windows and there is an exporter project for Windows as well.
There are multiple components involved on the Grafana stack side in an integration.
On first glance it looks like a lot of moving parts.
But this project contains all the scripts and glue to make them work together in an easy to setup stack.
- Grafana itself is the visual representation and dashboarding component
- Prometheus stores statistics data from servers providing a data source for Grafana
- Prometheus Node Exporter for Linux presents stats in a Prometheus compatible format to be collected by Prometheus on a defined endpoint
- Windows Exporter presents Windows stats and is a Prometheus community project
- Grafana Loki stores log data from servers providing data source for Grafana
- Promtail is the data collecting component sending logs to Loki
The project contains simple setup steps for the individual components. The first step should be always to clone the project.
mkdir -p /local/github
cd /local/github
git clone https://github.com/nashcom/domino-grafana.git
cd /local/github/domino-grafana
The servertask domprom is the core component of this project.
It needs to be deployed on each Domino server.
By default when started the servertask writes it's .prom files into <notesdata>/domino/stats.
When using a container image the installation is just another build option in the build menu.
On Linux and Windows native the binary needs to be copied into the binary directory manually.
The following example assumes the default location of the binary directory. It copies the file, sets the execution permissions and creates a start-up link.
cp domprom /opt/hcl/domino/notes/latest/linux
chmod 755 /opt/hcl/domino/notes/latest/linux/domprom
cd /opt/hcl/domino/bin
ln -s tools/startup domprom
The Prometheus Node Exporter collects OS level statistics and also the Domino .prom files written by domprom.
Depending on your environment the node_exporter runs:
- as part of the container image to only collect Domino statistics
- on your Linux host to collect OS statistics and Domino statistics
For natively installed Domino servers and Docker/Podman hosts running Domino as a container, the Node Exporter runs on host level to have full access OS level system statistics and to collect Domino statistics from .prom files.
There is currently no out of the box package with a systemd script available in Linux distributions. The binary is usually downloaded directly from the Prometheus Node Exporter GitHub Page.
This project contains a convenience script to
- Download and install the Node Exporter
- Configure a systemd service to run the Node Export
- Provide an out of the box configuration of the Node Exporter collecting Domino statistics from .prom files automatically.
The systemd setup script is part of the GitHub project and located here:
exporter/install_node_exporter.sh
On Kubernetes (K8s) there is usually already an instance of Grafana and Prometheus involved collecting the OS statistics. In that case the Node Exporter metrics end-point only provides Domino statistics.
The Domino container project provides a build option to install domprom and a current version of the Node Exporter automatically.
To start the Node Exporter just to to collect Domino statistics on Kubernetes (K8s) specify the following environment parameter for your Domino container.
export NODE_EXPORTER_OPTIONS=default
Prometheus queries ("scrapes") the Metrics endpoint at http://server:9100/metrics.
Ensure port 9100 is exposed (Docker port configuration and Linux firewall).
By default the port can be queried by any server or user who can connect to the server. Usually this is not a problem, because only numeric statistic data is provided. No PII should be included in the statistic data.
In case statistics should only be available to the collecting Prometheus servers, use a corporate or Linux firewall.
Prometheus is the base which collects ("scrapes") metrics endpoints which are served by the Node Exporter and stores them in it's internal time series database. Grafana is the graphical interface which can get statistics data from multiple back-ends -- In our case Prometheus.
You can use an existing instance of Prometheus to collect data and Grafana to display statistics. If you don't have an existing infrastructure already, this project provides a Docker Compose stack to bring up all required components.
- Prometheus instance running internally on port 3030
- Grafana instance exposing port 3000 by default to allow hosting Grafana on an existing machine
- NGINX to provide TLS termination for the Grafana graphical interface
To bring up the compose stack you need a TLS/SSL certificate stored in the certs directory
NGINX is expecting the certificate and key in
- /local/certs/cert.pem certificate chain including root
- /local/certs/key.pem private key
Note: CertMgr can create exportable MicroCA certificates. It can be decrypted using an OpenSSL command line. It prompts for the password and stores it unencrypted.
openssl pkey -in encrypted_key -out key.pem
When configuring a new instance copy prometheus.yml.example into prometheus.yml and specify the servers to scrape.
The example below also is helpful for your own Prometheus instance, where want to add those new targets to collect statistics from.
The following example defines two Domino servers to scrape in a single job. The Node Exporter by default listens on port 9100.
The prometheus.yml YAML based configuration is mounted into the Prometheus container to tell Prometheus which back-ends to scrape.
global:
scrape_interval: 60s
scrape_configs:
- job_name: domino
static_configs:
- targets:
- notes.example.com:9100
labels:
pod: notes.example.com
namespace: domino
- targets:
- domino.example.com:9100
labels:
pod: domino.example.com
namespace: domino
The job should be called domino which turns into a label.
Targets could be configured in one step, but this does not allow to specify additional labels.
The target turns into the instance variable and isn't a stable variable over time.
On Kubernetes (K8s) the instance might be just the IP address of the pod.
K8s uses the namespace and pod label in dashboards.
To align Kubernetes with Docker and native Linux deployments set namespace and pod labels in Prometheus scrape configuration.
There are two different Alertmanger options:
This is the classical option which uses Prometheus data directy.
It also requires alert rules defined in alerts.yml files.
The email configuration is defineed in prometheus.yml.
See prometheus.yml.example for details.
The Alertmanager is an optional container which is only started if specifying the alertmanger profile.
Newer option integrated into Grafana. The email configuration is part of Grafana. Create an .env file based on env.example to configure the SMTP settings.
Once those two configuration steps are provided, the compose stack can be started:
To start Grafana and Prometheus run the compose stack in detached mode.
docker compose up -d
The default user and password is admin/admin and should be changed immediately. Once the server is started, log into the portal. The GUI prompts to change the password.
Grafana provides a way to provision data sources and dashboards when first started. This project provides the data source definition and also default dashboards, which you will find in the Example folder in the Grafana GUI.
They are updated over time and are intended as reference dashboards to copy into your own dashboards. Over time we hopefully have reusable and published dashboards which can be imported via Grafana GUI.
The provisioning resources can be found in the grafana/provisioning folder.
The primary goal of this project is to provide the tooling to collect the metrics and logs from Domino.
But of course it would make sense to also provide components to reuse in dashboards
It would also make sense to provide a basic Domino dashboard in the official Grafana Dashboard catalog
The best starting point for a Domino on Linux dashboard would be begin with the Node Exporter Full Dashboard. A good way is to copied the dashboard and added Domino statistics to it from the same data source.
The dashboard is also an excellent example to learn building dashboards.
To get the dashboard installed, enter the dashboard URL or the number of the dashboard which is 1860
There is also a Windows Node Exporter Dashboard which is using the Windows Exporter in a similar way.
In contrast to the existing Domino Stats Publishing functionality based on a push model (initially introduced for New Relic integration in Domino 10) Prometheus expects data being collected from a server using a pull model.
In addition the statistics names have to be in a specific Prometheus format.
Basically only alphanumeric chars and the underscore can be used.
This leads to a small add-on component available for Linux and Windows.
domprom is a small C-API based servertask, collecting all LONG and NUMBER statistics from the Domino server to present them in a file on disk with Prometheus compatible metrics names.
Specially the dots in the names of Domino statistics need to be converted to underscores.
The exported stats are written to a file on disk, which can be included in an existing log collection.
The Node Exporter on Linux allows to include additional .prom files.
Those log files could be also presented via Domino HTTP directory on Windows or leveraging any other solution like NGINX.
Prometheus Node Exporter for Linux is a native for Linux OS level metrics. It is a single binary available on GitHub, which should run on the host level to collect all relevant Linux statistics.
The node_exporter can be configured to send additional log files, which is the best choice for Domino metrics collection on Linux to ensure metrics are in-line between the OS and Domino. The file generated by domprom can just be included and is send along with the Linux level stats.
See the setup and systemd scripts in /exporter for details.
Windows Exporter is a native for Linux OS level metrics. It is a single binary available on GitHub, which should run on the host level to collect all relevant Windows statistics.
The Windows Exporter can be configured to send additional log files, which is the best choice for Domino metrics collection on Windows to ensure metrics are in-line between the OS and Domino.
Data is stored in the following Docker volumes. In contrast all configuration is stored in native volumes.
- grafana-data
- prometheus-data
- loki-data (currently unused)
In earlier versions the data was located in native volumes, which was more complicated to setup. When you migrate from an earlier version you have to move the data.
When copying files the files might have the wrong security context even located in the right directory. Relabeling the binary should help:
Example:
/usr/sbin/restorecon -v /usr/local/bin/node_exporter
The install script runs the relabeling if restorecon is installed.