|
| 1 | +import {Note} from "@/components/mdx"; |
| 2 | + |
| 3 | +export const description = 'Scrape NetBird client connection health with Prometheus through the local /metrics endpoint on the client daemon.' |
| 4 | + |
| 5 | +# Local Metrics Endpoint |
| 6 | + |
| 7 | +The NetBird client daemon can expose a Prometheus `/metrics` endpoint on the local machine. It reports whether the client is connected to Management and Signal, how many peers it knows and how many of those are connected, the latency to each directly connected peer, and how long connection establishment, sync processing, and logins take. |
| 8 | + |
| 9 | +The endpoint is opt-in and off by default. When enabled, it binds to `127.0.0.1:9191` unless you set another address. It is independent of [client metrics push](/manage/client-metrics): nothing here is sent anywhere, the data is only served to whatever scrapes the endpoint. |
| 10 | + |
| 11 | +<Note> |
| 12 | + Available since NetBird <strong>v0.78.0</strong>. |
| 13 | +</Note> |
| 14 | + |
| 15 | +## Enabling the endpoint |
| 16 | + |
| 17 | +```bash |
| 18 | +netbird up --enable-local-metrics |
| 19 | +``` |
| 20 | + |
| 21 | +To listen on a different port: |
| 22 | + |
| 23 | +```bash |
| 24 | +netbird up --enable-local-metrics --local-metrics-address 127.0.0.1:9300 |
| 25 | +``` |
| 26 | + |
| 27 | +To disable it again: |
| 28 | + |
| 29 | +```bash |
| 30 | +netbird up --enable-local-metrics=false |
| 31 | +``` |
| 32 | + |
| 33 | +The setting is stored in the [profile](/client/profiles) config, so it survives restarts and applies per profile. Switching profiles starts, stops, or rebinds the endpoint to match the profile you switch to. |
| 34 | + |
| 35 | +<Note> |
| 36 | + `netbird up` ignores configuration flags when the client is already connected. Run `netbird down` first, then `netbird up` with the flags. |
| 37 | +</Note> |
| 38 | + |
| 39 | +Once the client is up, check the endpoint: |
| 40 | + |
| 41 | +```bash |
| 42 | +curl http://127.0.0.1:9191/metrics |
| 43 | +``` |
| 44 | + |
| 45 | +## Exposing it beyond localhost |
| 46 | + |
| 47 | +The endpoint has no authentication. Anything that can reach it can read your peer names, the latency to each of them, and your connectivity state. Keep it on a loopback address and let the scraper run on the same host. |
| 48 | + |
| 49 | +If a scraper genuinely has to reach it from elsewhere, binding to a non-loopback address requires root on Linux and macOS, and administrator privileges on Windows. The daemon runs with those privileges itself, so an unprivileged local user must not be able to publish this to the network: |
| 50 | + |
| 51 | +```bash |
| 52 | +sudo netbird down |
| 53 | +sudo netbird up --enable-local-metrics --local-metrics-address 0.0.0.0:9191 |
| 54 | +``` |
| 55 | + |
| 56 | +The daemon logs a warning for every non-loopback bind. Put the endpoint behind a firewall rule, or a reverse proxy that adds authentication, if you do this. |
| 57 | + |
| 58 | +## Scraping with Prometheus |
| 59 | + |
| 60 | +```yaml |
| 61 | +scrape_configs: |
| 62 | + - job_name: netbird-client |
| 63 | + static_configs: |
| 64 | + - targets: ['127.0.0.1:9191'] |
| 65 | +``` |
| 66 | +
|
| 67 | +## Metrics reference |
| 68 | +
|
| 69 | +### Connection state |
| 70 | +
|
| 71 | +These are read from the daemon at scrape time and are always present while the daemon runs. |
| 72 | +
|
| 73 | +| Metric | Type | Labels | Description | |
| 74 | +| --- | --- | --- | --- | |
| 75 | +| `netbird_management_connected` | gauge | — | `1` when connected to the management service, `0` otherwise. | |
| 76 | +| `netbird_signal_connected` | gauge | — | `1` when connected to the signal service, `0` otherwise. | |
| 77 | +| `netbird_peers` | gauge | — | Number of peers known to this client, matching the total in `netbird status`. Includes peers that are currently offline. | |
| 78 | +| `netbird_peers_connected` | gauge | `connection_type` | Number of connected peers, split into `p2p` and `relay`. | |
| 79 | +| `netbird_peer_latency_seconds` | gauge | `peer` | Round-trip latency to a directly connected peer, labeled with its FQDN. Relayed connections carry no latency measurement and produce no series. | |
| 80 | + |
| 81 | +### Connection establishment and management interactions |
| 82 | + |
| 83 | +These are recorded as the client runs, so they appear only once the engine is up and the corresponding event has happened at least once. They reset when the daemon restarts. |
| 84 | + |
| 85 | +| Metric | Type | Labels | Description | |
| 86 | +| --- | --- | --- | --- | |
| 87 | +| `netbird_peer_connection_stage_duration_seconds` | histogram | `stage`, `connection_type`, `attempt_type` | Duration of peer connection establishment stages. | |
| 88 | +| `netbird_sync_duration_seconds` | histogram | — | Duration of processing a sync message from the management service. | |
| 89 | +| `netbird_sync_phase_duration_seconds` | histogram | `phase` | Duration of an individual sync processing phase, for example `routes_apply`, `filtering`, or `added_peers`. | |
| 90 | +| `netbird_login_duration_seconds` | histogram | `success` | Duration of logins to the management service, split by whether the login succeeded. | |
| 91 | + |
| 92 | +Label values for `netbird_peer_connection_stage_duration_seconds`: |
| 93 | + |
| 94 | +| Label | Values | |
| 95 | +| --- | --- | |
| 96 | +| `stage` | `signaling_to_connection`, `connection_to_wg_handshake`, `total` | |
| 97 | +| `connection_type` | `ice_p2p`, `ice_turn`, `relay` | |
| 98 | +| `attempt_type` | `initial` for the first connection to a peer, `reconnection` for a later one | |
| 99 | + |
| 100 | +<Note> |
| 101 | + Every label is a bounded enum except `peer` on `netbird_peer_latency_seconds`, which carries one series per directly connected peer. On a client with many direct connections this is the one metric whose cardinality grows with your network. |
| 102 | +</Note> |
| 103 | + |
| 104 | +## Grafana dashboard |
| 105 | + |
| 106 | +NetBird ships a [client dashboard](/selfhosted/observability/dashboards#client) that graphs everything above. Import [`client.json`](https://github.com/netbirdio/netbird/blob/main/infrastructure_files/observability/grafana/dashboards/client.json) into Grafana and point it at the Prometheus datasource that scrapes your clients. |
| 107 | + |
| 108 | +## MDM |
| 109 | + |
| 110 | +Both settings can be enforced through [MDM](/client/mdm-integration): |
| 111 | + |
| 112 | +| Key | Type | Description | |
| 113 | +| --- | --- | --- | |
| 114 | +| `enableLocalMetrics` | boolean | Turn the local `/metrics` endpoint on or off. | |
| 115 | +| `localMetricsAddress` | string | Listen address for the endpoint, for example `127.0.0.1:9191`. | |
| 116 | + |
| 117 | +An MDM-supplied address is applied as-is and is not subject to the privilege check, since the policy already comes from an administrator. |
0 commit comments