diff --git a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/monitoring.md b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/monitoring.md index 30351764c..fe6a58b64 100644 --- a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/monitoring.md +++ b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/monitoring.md @@ -4,202 +4,210 @@ title: Monitoring keywords: - Monitoring - Pool metrics -description: This guide explains about the IO engine pool metrics exporter. + - Node metrics + - Replica metrics + - Volume metrics + - Replicated PV Mayastor Stats +description: This guide explains about the IO engine metrics exporter and obs-callhome stats exporter. --- -# Monitoring +# Mayastor metrics reference -## Pool Metrics Exporter +Replicated PV Mayastor exposes metrics from two different sources: the metrics exporter sidecar in the I/O-engine pod, and the call-home stats container in the call-home pod. -The IO engine pool metrics exporter runs as a sidecar container within every I/O-engine pod and exposes pool usage metrics in Prometheus format. These metrics are exposed on port 9502 using an HTTP endpoint/metrics and are refreshed every five minutes. +## Source: I/O-engine pod — metrics exporter container -### Supported Pool Metrics +Exposed at port `9502`, path `/metrics`. -| Name | Type | Unit | Description | -| :--- | :--- | :--- | :--- | -| disk_pool_total_size_bytes | Gauge | Integer | Total size of the pool | -| disk_pool_used_size_bytes | Gauge | Integer | Used size of the pool | -| disk_pool_status | Gauge | Integer | Status of the pool (0, 1, 2, 3) = {"Unknown", "Online", "Degraded", "Faulted"} | -| disk_pool_committed_size | Gauge | Integer | Committed size of the pool in bytes | +The metrics below — grouped by resource: diskpools, nodes, replicas, and volumes — come from this exporter. Each is a live Prometheus gauge, polled inline with the Prometheus client's scrape cycle rather than cached. -**Example Metrics** -``` -# HELP disk_pool_status disk-pool status -# TYPE disk_pool_status gauge -disk_pool_status{node="worker-0",name="mayastor-disk-pool"} 1 -# HELP disk_pool_total_size_bytes total size of the disk-pool in bytes -# TYPE disk_pool_total_size_bytes gauge -disk_pool_total_size_bytes{node="worker-0",name="mayastor-disk-pool"} 5.360320512e+09 -# HELP disk_pool_used_size_bytes used disk-pool size in bytes -# TYPE disk_pool_used_size_bytes gauge -disk_pool_used_size_bytes{node="worker-0",name="mayastor-disk-pool"} 2.147483648e+09 -# HELP disk_pool_committed_size_bytes Committed size of the pool in bytes -# TYPE disk_pool_committed_size_bytes gauge -disk_pool_committed_size_bytes{node="worker-0", name="mayastor-disk-pool"} 9663676416 -``` - -## Stats Exporter Metrics - -When [eventing](./eventing.md) is activated, the stats exporter operates within the **obs-callhome-stats** container, located in the **callhome** pod. The statistics are made accessible through an HTTP endpoint at port `9090`, specifically using the `/stats` route. - -### Supported Stats Metrics - -| Name | Type | Unit | Description | -| :--- | :--- | :--- | :--- | -| pools_created | Guage | Integer | Total successful pool creation attempts | -| pools_deleted | Guage | Integer | Total successful pool deletion attempts | -| volumes_created | Guage | Integer | Total successful volume creation attemtps | -| volumes_deleted | Guage | Integer | Total successful volume deletion attempts | - -## Integrating Exporter with Prometheus Monitoring Stack - -1. To install, add the Prometheus-stack helm chart and update the repo. - -**Command** -``` -helm repo add prometheus-community https://prometheus-community.github.io/helm-charts -helm repo update -``` - -Then, install the Prometheus monitoring stack and set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues to false. This enables Prometheus to discover custom ServiceMonitor for Replicated PV Mayastor. - -**Command** -``` -helm install mayastor prometheus-community/kube-prometheus-stack -n openebs --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false -``` - -2. Install the ServiceMonitor resource to select services and specify their underlying endpoint objects. - -**ServiceMonitor YAML** -``` -apiVersion: monitoring.coreos.com/v1 -kind: ServiceMonitor -metadata: - name: mayastor-monitoring - labels: - app: mayastor -spec: - selector: - matchLabels: - app: mayastor - endpoints: - - port: metrics -``` - -:::info -Upon successful integration of the exporter with the Prometheus stack, the metrics will be available on the port 9090 and HTTP endpoint /metrics. -::: - -## CSI Metrics Exporter - -| Name | Type | Unit | Description | -| :--- | :--- | :--- | :--- | -| kubelet_volume_stats_available_bytes | Gauge | Integer | Size of the available/usable volume (in bytes) | -| kubelet_volume_stats_capacity_bytes | Gauge | Integer | The total size of the volume (in bytes) | -| kubelet_volume_stats_used_bytes | Gauge | Integer | Used size of the volume (in bytes) | -| kubelet_volume_stats_inodes | Gauge | Integer | The total number of inodes | -| kubelet_volume_stats_inodes_free | Gauge | Integer | The total number of usable inodes. | -| kubelet_volume_stats_inodes_used | Gauge | Integer | The total number of inodes that have been utilized to store metadata. | - - -## Performance Monitoring Stack - -Earlier, the pool capacity/state stats were exported, and the exporter used to cache the metrics and return when Prometheus client queried. This was not ensuring the latest data retuns during the Prometheus poll cycle. - -In addition to the capacity and state metrics, the metrics exporter also exports performance statistics for pools, volumes, and replicas as Prometheus counters. The exporter does not pre-fetch or cache the metrics, it polls the IO engine inline with the Prometheus client polling cycle. +## Diskpool metrics -:::important -Users are recommended to have Prometheus poll interval not less then 5 minutes. -::: +| Metric | Type | Labels | Description | +|---|---|---|---| +| `diskpool_alert_attention_reason` | gauge | `name`, `node`, plus reason labels (`io_error`, `io_error_exc`, `io_stall_intermittent`, `io_stall_intermittent_exc`, `io_stalled`, `unknown`) | Collection of reasons contributing to an attention-level alert | +| `diskpool_alert_critical_reason` | gauge | same reason labels as above | Collection of reasons contributing to a critical alert | +| `diskpool_alert_notice_reason` | gauge | same reason labels as above | Collection of reasons contributing to a notice alert | +| `diskpool_alert_warning_reason` | gauge | same reason labels as above | Collection of reasons contributing to a warning alert | +| `diskpool_bytes_read` | gauge | `name=`, `node=` | Total bytes read on the pool | +| `diskpool_bytes_written` | gauge | `name=`, `node=` | Total bytes written on the pool | +| `diskpool_committed_size_bytes` | gauge | `name`, `node` | Committed size of the pool, in bytes | +| `diskpool_disk_capacity_bytes` | gauge | `name`, `node` | Capacity of the pool's underlying device | +| `diskpool_io_alert_status` | gauge | `name`, `node` | DiskPool alert status | +| `diskpool_io_error_count` | gauge | `name`, `node` | Count of I/O errors for the pool | +| `diskpool_io_error_threshold` | gauge | `name`, `node` | Threshold for I/O errors in the pool | +| `diskpool_io_stall_transition_count` | gauge | `name`, `node` | Count of I/O stall transitions in the pool | +| `diskpool_io_stall_transition_threshold` | gauge | `name`, `node` | Threshold for I/O stall transitions in the pool | +| `diskpool_io_stalled` | gauge | `name`, `node` | Stalled I/O operations in the pool | +| `diskpool_max_expandable_size` | gauge | `name`, `node` | Maximum capacity to which this pool can be expanded, in bytes | +| `diskpool_num_read_ops` | gauge | `name=`, `node=` | Number of read operations on the pool | +| `diskpool_num_write_ops` | gauge | `name=`, `node=` | Number of write operations on the pool | +| `diskpool_read_latency_us` | gauge | `name=`, `node=` | Total read latency for all IOs on the pool, in microseconds | +| `diskpool_status` | gauge | `name`, `node` | Status of the pool: `0` = Unknown, `1` = Online, `2` = Degraded, `3` = Faulted, `4` = Suspected | +| `diskpool_total_size_bytes` | gauge | `name`, `node` | Total size of the pool, in bytes | +| `diskpool_used_size_bytes` | gauge | `name`, `node` | Used size of the pool, in bytes | +| `diskpool_write_latency_us` | gauge | `name=`, `node=` | Total write latency for all IOs on the pool, in microseconds | -The following sections describes the raw resource metrics counters. +> The alert-reason metrics aren't separate counters per reason — each is a single metric where the reason labels (`io_error`, `io_stalled`, etc.) indicate which condition is contributing to that alert level. -### DiskPool IoStat Counters +## Node metrics + +| Metric | Type | Labels | Description | +|---|---|---|---| +| `mayastor_node_cordoned` | gauge | `node_id` | Indicates if the Mayastor node is cordoned | +| `mayastor_node_draining` | gauge | `node_id` | Indicates if the Mayastor node is draining | +| `mayastor_node_online` | gauge | `node_id` | Indicates if the Mayastor node is online | + +## Replica metrics + +| Metric | Type | Labels | Description | +|---|---|---|---| + | `replica_bytes_read` | gauge | `name=`, `node=`, `pool_name=`, `pool_uuid=`, `pv_name=` | Total bytes read on the replica | + | `replica_bytes_written` | gauge | `name=`, `node=`, `pool_name=`, `pool_uuid=`, `pv_name=` | Total bytes written on the replica | + | `replica_num_read_ops` | gauge | `name=`, `node=`, `pool_name=`, `pool_uuid=`, `pv_name=` | Number of read operations on the replica | + | `replica_num_write_ops` | gauge | `name=`, `node=`, `pool_name=`, `pool_uuid=`, `pv_name=` | Number of write operations on the replica | + | `replica_read_latency_us` | gauge | `name=`, `node=`, `pool_name=`, `pool_uuid=`, `pv_name=` | Total read latency for all IOs on the replica, in microseconds | + | `replica_write_latency_us` | gauge | `name=`, `node=`, `pool_name=`, `pool_uuid=`, `pv_name=` | Total write latency for all IOs on the replica, in microseconds | -| Metric Name | Metric Type | Labels/Tags | Metric Unit | Description | -| :--- | :--- | :--- | :--- | :--- | -|diskpool_num_read_ops | Gauge | `name`=``, `node`=`` | Integer | Number of read operations | -|diskpool_bytes_read | Gauge | `name`=``, `node`=`` | Integer | Total bytes read on the pool | -|diskpool_num_write_ops | Gauge | `name`=``, `node`=`` | Integer | Number of write operations on the pool | -|diskpool_bytes_written | Gauge | `name`=``, `node`=`` | Integer | Total bytes written on the pool | -|diskpool_read_latency_us | Gauge | `name`=``, `node`=`` | Integer | Total read latency for all IOs on Pool in usec. | -|diskpool_write_latency_us | Gauge | `name`=``, `node`=`` | Integer | Total write latency for all IOs on Pool in usec. | +## Volume metrics + +| Metric | Type | Labels | Description | +|---|---|---|---| +| `volume_bytes_read` | gauge | `node=`, `pv_name=` | Total bytes read through the volume target | +| `volume_bytes_written` | gauge | `node=`, `pv_name=` | Total bytes written through the volume target | +| `volume_num_read_ops` | gauge | `node=`, `pv_name=` | Number of read operations through the volume target | +| `volume_num_write_ops` | gauge | `node=`, `pv_name=` | Number of write operations through the volume target | +| `volume_read_latency_us` | gauge | `node=`, `pv_name=` | Total read latency for all IOs through the volume target, in microseconds | +| `volume_write_latency_us` | gauge | `node=`, `pv_name=` | Total write latency for all IOs through the volume target, in microseconds | + +## Deriving IOPS, latency, and throughput + +The metrics above are raw, cumulative per-resource totals (typically monotonically increasing values). To get IOPS, average latency, or throughput for a dashboard, compute them across two successive polls of the *same* resource (pool, replica, or volume): -### Replica IoStat Counters +**R/W IOPS** -| Metric Name | Metric Type | Labels/Tags | Metric Unit | Description | -| :--- | :--- | :--- | :--- | :--- | -|replica_num_read_ops | Gauge | `name`=``, `pool_id`=`` `pv_name`=``, `node`=`` | Integer | Number of read operations on replica | -|replica_bytes_read | Gauge | `name`=``, `pv_name`=``, `node`=`` | Integer | Total bytes read on the replica | -|replica_num_write_ops | Gauge | `name`=``, `pv_name`=``, `node`=`` | Integer | Number of write operations on the replica | -|replica_bytes_written | Gauge | `name`=``, `pv_name`=``, `node`=`` | Integer | Total bytes written on the Replica | -|replica_read_latency_us | Gauge | `name`=``, `pv_name`=``, `node`=`` | Integer | Total read latency for all IOs on replica in usec. | -|replica_write_latency_us | Gauge | `name`=``, `pv_name`=``, `node`=`` | Integer | Total write latency for all IOs on replica in usec. | - -### Target/Volume IoStat Counters - -| Metric Name | Metric Type | Labels/Tags | Metric Unit | Description | -| :--- | :--- | :--- | :--- | :--- | -|volume_num_read_ops | Gauge | `pv_name`=`` | Integer | Number of read operations through vol target | -|volume_bytes_read | Gauge | `pv_name`=`` | Integer | Total bytes read through vol target | -|volume_num_write_ops | Gauge | `pv_name`=`` | Integer | Number of write operations through vol target | -|volume_bytes_written | Gauge | `pv_name`=`` | Integer | Total bytes written through vol target | -|volume_read_latency_us | Gauge | `pv_name`=`` | Integer | Total read latency for all IOs through vol target in usec. | -|volume_write_latency_us | Gauge | `pv_name`=`` | Integer | Total write latency for all IOs through vol target in usec. | - -:::note -If you require IOPS, Latency, and Throughput in the dashboard, use the following consideration while creating dashboard json config. -::: - -## R/W IOPS Calculation - -`num_read_ops` and `num_write_ops` for all resources in stats response are available. - -``` -write_iops = num_write_ops (current poll) - num_write_ops (previous_poll) / poll period (in sec) -``` - -``` -read_iops = num_read_ops (current poll) - num_read_ops (previous_poll) / poll period (in sec) -``` - -## R/W Latency Calculation - -`write_latency` (sum of all IO's read_latency) and `read_latency` (sum of all IO and read_latency) are available. +``` +write_iops = (num_write_ops_current - num_write_ops_previous) / poll_period_sec +read_iops = (num_read_ops_current - num_read_ops_previous) / poll_period_sec +``` + +**R/W latency** + +``` +read_latency_avg = (read_latency_current - read_latency_previous) / (num_read_ops_current - num_read_ops_previous) +write_latency_avg = (write_latency_current - write_latency_previous) / (num_write_ops_current - num_write_ops_previous) +``` +> If `num_*_ops_current == num_*_ops_previous` (no I/O between polls), the average latency can’t be computed; treat it as 0/NaN in dashboards. +**R/W throughput** + +``` +read_throughput = (bytes_read_current - bytes_read_previous) / poll_period_sec +write_throughput = (bytes_written_current - bytes_written_previous) / poll_period_sec +``` + +**Handling counter resets** + +These counters are not persistent across I/O engine restarts — they reset to zero when the engine restarts. If the current-poll value is lower than the previous-poll value, the formulas above will go negative. In that case, fall back to: ``` -read_latency_avg = read_latency (current poll) - read_latency (previous poll) / num_read_ops (current poll) - num_read_ops (previous_poll) +iops (r/w) = num_ops (r/w) / poll_period_sec +latency_avg (r/w) = latency_sum (r/w) / num_ops (r/w) +throughput (r/w) = bytes_read/written / poll_period_sec ``` -``` -write_latency_avg = write_latency (current poll) - write_latency (previous poll) / num_write_ops (current poll) - num_write_ops (previous_poll) +### Sample scrape + +``` +# HELP diskpool_alert_attention_reason Collection of reason for attention alert +# TYPE diskpool_alert_attention_reason gauge +diskpool_alert_attention_reason{io_error="0",io_error_exc="0",io_stall_intermittent="0",io_stall_intermittent_exc="0",io_stalled="0",name="pool-on-node-2-477115",node="node-2-477115",unknown="0"} 0 +# HELP diskpool_bytes_read Total bytes read on the pool +# TYPE diskpool_bytes_read gauge +diskpool_bytes_read{name="pool-on-node-2-477115",node="node-2-477115"} 3198976 +# HELP diskpool_bytes_written Total bytes written on the pool +# TYPE diskpool_bytes_written gauge +diskpool_bytes_written{name="pool-on-node-2-477115",node="node-2-477115"} 0 +# HELP diskpool_committed_size_bytes Committed size of the pool in bytes +# TYPE diskpool_committed_size_bytes gauge +diskpool_committed_size_bytes{name="pool-on-node-2-477115",node="node-2-477115"} 1073741824 +# HELP diskpool_disk_capacity_bytes Capacity of the Pool's underlying device +# TYPE diskpool_disk_capacity_bytes gauge +diskpool_disk_capacity_bytes{name="pool-on-node-2-477115",node="node-2-477115"} 10737418240 +# HELP diskpool_status Status of the pool +# TYPE diskpool_status gauge +diskpool_status{name="pool-on-node-2-477115",node="node-2-477115"} 1 +# HELP diskpool_total_size_bytes Total size of the pool in bytes +# TYPE diskpool_total_size_bytes gauge +diskpool_total_size_bytes{name="pool-on-node-2-477115",node="node-2-477115"} 10724835328 +# HELP diskpool_used_size_bytes Used size of the pool in bytes +# TYPE diskpool_used_size_bytes gauge +diskpool_used_size_bytes{name="pool-on-node-2-477115",node="node-2-477115"} 1073741824 +# HELP mayastor_node_online Indicates if Mayastor node is online +# TYPE mayastor_node_online gauge +mayastor_node_online{node_id="node-2-477115"} 1 +# HELP replica_bytes_read Total bytes read on the replica +# TYPE replica_bytes_read gauge +replica_bytes_read{name="77c600a8-6709-4779-9a8f-7e86b8cf005e",node="node-2-477115",pool_name="",pool_uuid="",pv_name="pvc-86e8ccf8-3fcd-4d5a-847c-60016f722403"} 3145728 +# HELP replica_num_read_ops Number of read operations on the replica +# TYPE replica_num_read_ops gauge +replica_num_read_ops{name="77c600a8-6709-4779-9a8f-7e86b8cf005e",node="node-2-477115",pool_name="",pool_uuid="",pv_name="pvc-86e8ccf8-3fcd-4d5a-847c-60016f722403"} 24 +# HELP volume_bytes_read Total bytes read from the volume +# TYPE volume_bytes_read gauge +volume_bytes_read{node="node-2-477115",pv_name="pvc-86e8ccf8-3fcd-4d5a-847c-60016f722403"} 9437184 +# HELP volume_num_read_ops Number of read operations on the volume +# TYPE volume_num_read_ops gauge +volume_num_read_ops{node="node-2-477115",pv_name="pvc-86e8ccf8-3fcd-4d5a-847c-60016f722403"} 72 +# HELP volume_read_latency_us Total read latency on the volume in usec +# TYPE volume_read_latency_us gauge +volume_read_latency_us{node="node-2-477115",pv_name="pvc-86e8ccf8-3fcd-4d5a-847c-60016f722403"} 149780 ``` -## R/W Throughput Calculation +> Truncated for brevity — every metric in the disk pool, node, replica, and volume tables above follows this same `# HELP` / `# TYPE` / sample-line pattern on a live scrape. -`bytes_read/written` (total bytes read/written for a bdev) are available. +## Source: call-home pod — obs-callhome-stats container -``` -read_throughput = bytes_read (current poll) - bytes_read (previous_poll) / poll period (in sec) -``` +Exposed at port `9090`, path `/stats`. These are cumulative event counters (lifecycle actions), distinct from the per-resource I/O metrics above. -``` -write_throughput = bytes_written (current poll) - bytes_written (previous_poll) / poll period (in sec) -``` +| Metric | Type | Labels | Description | +|---|---|---|---| +| `nexus` | counter | `action="created"` | Total nexus creation events | +| `nexus` | counter | `action="deleted"` | Total nexus deletion events | +| `nexus` | counter | `action="rebuild_started"` | Total nexus rebuild-started events | +| `nexus` | counter | `action="rebuild_ended"` | Total nexus rebuild-ended events | +| `pool` | counter | `action="created"` | Total pool creation events | +| `pool` | counter | `action="deleted"` | Total pool deletion events | +| `volume` | counter | `action="created"` | Total volume creation events | +| `volume` | counter | `action="deleted"` | Total volume deletion events | -### Handling Counter Reset +> `nexus`, `pool`, and `volume` are each a single counter metric — the different lifecycle events are distinguished by the `action` label rather than being separate metric names. -The performance stats are not persistent across IO engine restart, this means the counters will be reset upon IO engine restart. Users will receive lesser values for all the resources residing on that particular IO engine due to reset. So using the above logic would yield negative values. Hence, the counter current poll is less than the counter previous poll. In this case, do the following: +### Sample scrape ``` -iops (r/w) = num_ops (r/w) / poll cycle -``` - -``` -latency_avg(r/w) = latency (r/w) / num_ops -``` - -``` -throughput (r/w) = bytes_read/written / poll_cycle (in secs) -``` - -[Learn more](https://kubernetes.io/docs/concepts/storage/volume-health-monitoring/) \ No newline at end of file +# HELP nexus Nexus stats +# TYPE nexus counter +nexus{action="created"} 6 +nexus{action="deleted"} 4 +nexus{action="rebuild_ended"} 7 +nexus{action="rebuild_started"} 7 +# HELP pool Pool stats +# TYPE pool counter +pool{action="created"} 6 +pool{action="deleted"} 0 +# HELP volume Volume stats +# TYPE volume counter +volume{action="created"} 2 +volume{action="deleted"} 0 +``` + +## Preconfigured monitoring stack + +If you'd rather not wire up Prometheus, Grafana, and Alertmanager from scratch, OpenEBS maintains a monitoring Helm chart with sample dashboards and alert rules built on top of these metrics: + +[github.com/openebs/monitoring](https://github.com/openebs/monitoring) + +## Notes + +- All four resource groups (diskpool, node, replica, volume) are scoped by labels — always filter/group by the relevant label (`node`, `name`, `pv_name`, etc.) rather than reading a metric as a global value. +- The I/O-engine metrics (port 9502) and the call-home stats (port 9090) are separate exporters on separate ports/paths — don't assume both live in the same scrape target. + +[Learn more](https://kubernetes.io/docs/concepts/storage/volume-health-monitoring/)