telemetry: native multi-process Prometheus aggregation (prometheus_mp)

docs/telemetry.md

@@ -14,7 +14,8 @@ Custom telemetry exporter plug-ins can be created according to [src/plugins/tele
 2. **Shared Memory Buffer**: Statically-linked built in implementation of telemetry exporter. Uses shared memory cyclic buffer for efficient event storage and export.
 3. **Telemetry Readers**: C++ and Python applications to read and display telemetry data from the cyclic buffer.
 4. **Prometheus exporter**: EXPERIMENTAL (beta) Prometheus compatible telemetry exporter, see [src/plugins/telemetry/prometheus/README.md](../src/plugins/telemetry/prometheus/README.md).
-5. **DOCA exporter**: EXPERIMENTAL DOCA/CollectX telemetry exporter. Drives one or more delivery backends (`NIXL_TELEMETRY_DOCA_BACKENDS`, default `scrape`): a local Prometheus scrape endpoint and/or `ipc` push to the DOCA Telemetry Service (DTS) for single-endpoint multi-process aggregation. See [src/plugins/telemetry/doca/README.md](../src/plugins/telemetry/doca/README.md).
+5. **Multi-process Prometheus exporter (`prometheus_mp`)**: EXPERIMENTAL native aggregation of all processes of a multi-process run behind one scrape endpoint, without an external service. See [src/plugins/telemetry/prometheus_mp/README.md](../src/plugins/telemetry/prometheus_mp/README.md) and "Multi-process aggregation" below.
+6. **DOCA exporter**: EXPERIMENTAL DOCA/CollectX telemetry exporter. Drives one or more delivery backends (`NIXL_TELEMETRY_DOCA_BACKENDS`, default `scrape`): a local Prometheus scrape endpoint and/or `ipc` push to the DOCA Telemetry Service (DTS) for single-endpoint multi-process aggregation. See [src/plugins/telemetry/doca/README.md](../src/plugins/telemetry/doca/README.md).
 
 ### Event Structure
 
@@ -147,7 +148,55 @@ Telemetry is configured by environment variables:
 - When telemetry is requested but no output sink is configured (neither `NIXL_TELEMETRY_EXPORTER` nor `NIXL_TELEMETRY_DIR`), it falls back to the collect-only NOP exporter: events are collected in-process so `getXferTelemetry()` / `get_xfer_telemetry()` works, but nothing is written out.
 - If telemetry is enabled but no exporter is set, or the exporter name is empty, then the sink depends on `NIXL_TELEMETRY_DIR` as explained below (falling back to NOP when it is unset).
 - Set `NIXL_TELEMETRY_EXPORTER=NOP` to explicitly keep telemetry active (events are collected and `getXferTelemetry()` works) while discarding all output. It needs no sink and writes nothing, so it can be used to measure the overhead of the telemetry collection path in isolation.
-- Exporters that expose a scrape endpoint (e.g. Prometheus) bind one port per process. Under multi-process runs (e.g. tensor/data parallelism) every rank tries to bind the same port; only one wins. Losing that race is benign and non-fatal: the affected process logs a single warning and runs without a telemetry sink instead of failing agent construction. See [src/plugins/telemetry/prometheus/README.md](../src/plugins/telemetry/prometheus/README.md).
+- Exporters that expose a scrape endpoint (e.g. Prometheus) bind one port per process. Under multi-process runs (e.g. tensor/data parallelism) every rank tries to bind the same port; only one wins. With the single-process `prometheus` exporter, losing that race is benign and non-fatal: the affected process logs a single warning and runs without a telemetry sink instead of failing agent construction (see [src/plugins/telemetry/prometheus/README.md](../src/plugins/telemetry/prometheus/README.md)). To instead export **every** rank behind one endpoint, use the `prometheus_mp` exporter (see "Multi-process aggregation" below).
+
+## Multi-process aggregation (`prometheus_mp`)
+
+The `prometheus_mp` exporter aggregates the telemetry of all processes of a
+multi-process NIXL run behind a **single** Prometheus scrape endpoint, natively
+(no DOCA/DTS). Every process writes its own metric state to a per-process
+memory-mapped file in a shared directory; the processes race to bind the scrape
+port, the winner serves `/metrics` by reading and republishing all live processes'
+files on each scrape (labeled per process), and the losers run as writers only. A
+bind collision is benign, so no rank is dropped. Full details:
+[src/plugins/telemetry/prometheus_mp/README.md](../src/plugins/telemetry/prometheus_mp/README.md).
+
+### Configuration
+
+| Variable | Description | Default |
+| -------- | ----------- | ------- |
+| `NIXL_TELEMETRY_EXPORTER` | Set to `prometheus_mp` to select this exporter | - |
+| `NIXL_TELEMETRY_MULTIPROC_DIR` | Shared directory for per-process store files (all ranks to be aggregated must use the same path). **Required.** | - |
+| `NIXL_TELEMETRY_PROMETHEUS_PORT` | Scrape port (shared with the `prometheus` exporter) | `9090` |
+| `NIXL_TELEMETRY_PROMETHEUS_LOCAL` | Bind `127.0.0.1` instead of `0.0.0.0` | `false` |
+| `NIXL_TELEMETRY_RANK_ENV` | Name of the env var holding the rank for the optional `local_rank` label; no label if that env var is unset | `LOCAL_RANK` |
+| `NIXL_TELEMETRY_MP_STALE_TTL` | Seconds after a dead process's last update before its store is stale and reaped | `30` |
+
+Series are labeled by `hostname`, `agent_name`, `pid` (guarantees cross-process
+uniqueness), `agent_instance` (distinguishes multiple same-name agents within one
+process), and optionally `local_rank` (the local/per-GPU rank, not Dynamo's
+data-parallel `dp_rank`). `agent_errors_total` additionally includes the bounded
+`status` label. Metric names, types, and semantics are identical to the
+single-process `prometheus` exporter.
+
+`NIXL_TELEMETRY_MULTIPROC_DIR` follows Dynamo's `PROMETHEUS_MULTIPROC_DIR`
+convention: a shared **local** folder (not NFS), one per pod / process-family,
+treated as ephemeral (e.g. a per-pod Kubernetes `emptyDir`). Because NIXL is a
+library loaded independently per rank -- with no parent to propagate the path as in
+Dynamo -- the launcher/operator must set the **same** directory for every rank, so
+it is required rather than auto-defaulted.
+
+### Scope & limitations
+
+`prometheus_mp` is purpose-built for NIXL's telemetry model, **not** a generic
+Prometheus multiprocess store, and it does not reuse Python `prometheus_client`'s
+multiprocess format. The metric set is fixed at compile time (positional slots;
+names are not stored) and per-process label values are captured once at startup and
+never change -- events carry only a numeric value, with no per-observation labels.
+It therefore **cannot represent a metric with a dynamic / high-cardinality label**
+that varies per observation. No NIXL metric needs that today; if one is ever added,
+a different (keyed) store would be required. For aggregation via an external
+service, use the DOCA/CollectX exporter instead.
 
 ## Cyclic Buffer
 

src/plugins/telemetry/meson.build

@@ -14,6 +14,7 @@
 # limitations under the License.
 
 subdir('prometheus')
+subdir('prometheus_mp')
 
 # DOCA telemetry exporter: an optional DOCA pkg-config dependency, gated exactly
 # like the GPUNETIO backend (src/plugins/meson.build). Built by default when the

src/plugins/telemetry/prometheus_mp/README.md

@@ -0,0 +1,134 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# NIXL multi-process Prometheus Telemetry exporter plug-in (`prometheus_mp`)
+
+`prometheus_mp` exposes the telemetry of **all** processes of a multi-process
+NIXL run (e.g. tensor/data parallelism) behind a **single** Prometheus scrape
+endpoint, without any DOCA/DTS dependency.
+
+It complements the single-process [`prometheus`](../prometheus/README.md) exporter
+(which binds one port per process, so only one rank's metrics are scraped) and the
+DOCA/CollectX exporter (which aggregates via an external DTS service). Use
+`prometheus_mp` when you want all ranks aggregated natively with no extra
+infrastructure. General NIXL telemetry background: [docs/telemetry.md](../../../../docs/telemetry.md).
+
+## Dependencies
+
+Same as the `prometheus` plug-in: the bundled prometheus-cpp subproject and
+`libcurl` (`libcurl4-openssl-dev` / `libcurl-devel`).
+
+## How it works
+
+- **Every process writes its own metric state** to a per-process memory-mapped
+  file in a shared directory (`NIXL_TELEMETRY_MULTIPROC_DIR`). Updates are
+  lock-free; there is no serialization.
+- **Bind-race owner election.** On startup each process tries to bind the scrape
+  port. The one that wins ("owner") runs the HTTP endpoint plus a collector that,
+  on each scrape, reads every live process's file and republishes them as one
+  exposition. The processes that lose the race run in **writer-only** mode (no
+  HTTP server). A bind collision is therefore benign -- every process gets a valid
+  telemetry sink; no rank is dropped and no scary error is logged.
+- **Per-process series.** Each process is exported as its own series (cumulative
+  counters and last-operation gauges), never summed across processes, so
+  per-process values stay correct and monotonic.
+- **Stale handling.** On clean shutdown a process removes its own store file. If
+  it instead crashes or is killed -- and so cannot clean up after itself -- the
+  owner drops its series once the process is gone (verified by pid + `/proc` start
+  time) or its data ages past the TTL, and reaps the file.
+
+## Configuration
+
+```bash
+export NIXL_TELEMETRY_ENABLE="y"
+export NIXL_TELEMETRY_EXPORTER="prometheus_mp" # selects libtelemetry_exporter_prometheus_mp.so
+export NIXL_TELEMETRY_MULTIPROC_DIR="/tmp/nixl_metrics" # REQUIRED: shared by all ranks in the pod
+```
+
+This mirrors Dynamo's `PROMETHEUS_MULTIPROC_DIR` convention (a shared folder that
+every related process writes into, one leader exports): all ranks that should be
+aggregated together must point `NIXL_TELEMETRY_MULTIPROC_DIR` at the **same**
+directory. Unlike Dynamo -- which auto-creates a temp dir in the parent and lets
+child engine processes inherit it -- NIXL is a library loaded independently in each
+rank, so there is no parent to propagate the path; the launcher/operator must set
+the same directory for every rank (hence it is required, not auto-defaulted, so a
+per-process temp dir can never silently break aggregation).
+
+Recommended, following Dynamo's model: a shared **local** folder, one per pod /
+process-family, treated as ephemeral (e.g. a per-pod Kubernetes `emptyDir`, or a
+temp dir cleaned between runs). It must be a local filesystem -- **not** a network
+filesystem (NFS/CIFS), where mmap `MAP_SHARED` cross-process visibility is not
+guaranteed (the same restriction Dynamo's multiprocess dir has). tmpfs (e.g. a
+Memory-medium `emptyDir` or `/dev/shm`) works and avoids any disk writeback, but is
+optional -- a plain local dir is fine, since updates hit the page cache and the
+per-process store files are ~one page each.
+
+### Optional configuration
+
+```bash
+# Scrape port (default 9090) and bind scope -- shared with the prometheus plug-in.
+export NIXL_TELEMETRY_PROMETHEUS_PORT="<port_num>"
+export NIXL_TELEMETRY_PROMETHEUS_LOCAL="y" # bind 127.0.0.1 instead of 0.0.0.0
+
+# Optional local_rank label: names the env var that holds the rank (default LOCAL_RANK).
+# If that env var is unset, no local_rank label is emitted (series stay unique via pid).
+export NIXL_TELEMETRY_RANK_ENV="LOCAL_RANK"
+
+# Seconds after a dead process's last update before its store is considered stale
+# and reaped (default 30). A live process is always published regardless of age.
+export NIXL_TELEMETRY_MP_STALE_TTL="30"
+```
+
+## Metric labels
+
+Every series is labeled by:
+
+- `hostname` -- host where the agent runs.
+- `agent_name` -- the agent name given at initialization.
+- `pid` -- the producing process id. This guarantees each process is a distinct
+  series even if agent names collide; it is deliberately **not** named `instance`
+  (a reserved Prometheus target label).
+- `agent_instance` -- a per-process counter distinguishing multiple agents created
+  in the same process (which share `pid`, `hostname`, and `agent_name`), so their
+  series never collide. `0` for the common single-agent-per-process case.
+- `local_rank` -- **optional**, present only when a rank env var (see
+  `NIXL_TELEMETRY_RANK_ENV`) is set. This is the local/per-GPU (TP) rank, distinct
+  from Dynamo's data-parallel `dp_rank`.
+- `status` -- only on `agent_errors_total`, bounded by the fixed `AGENT_ERR_*` set.
+
+The metric names, types, counter/gauge semantics, and events are identical to the
+single-process [`prometheus`](../prometheus/README.md) exporter (same shared
+descriptor).
+
+## Design scope & limitations
+
+This exporter is **purpose-built for NIXL's telemetry model, not a generic
+Prometheus multiprocess store** (in particular it is not compatible with, and does
+not reuse, Python `prometheus_client`'s multiprocess format):
+
+- The metric set is fixed at compile time; slots are positional, so metric names
+  are never stored in the files.
+- Per-process label values (`hostname`, `agent_name`, `pid`, `local_rank`) are
+  captured once at startup and never change. Events carry only a numeric value --
+  there are **no per-observation labels**.
+- Consequently the store **cannot represent a metric with a dynamic /
+  high-cardinality label** whose value varies per observation. No NIXL metric has
+  such a label today; if one is ever added, this exporter would need a different
+  (keyed) store.
+
+This is the native, dependency-free path. For aggregation through an external
+telemetry service, use the DOCA/CollectX exporter (IPC to DTS) instead.

src/plugins/telemetry/prometheus_mp/meson.build

@@ -0,0 +1,87 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Multiprocess metric-state store: a small, prometheus-cpp-free component (mmap
+# per-process store) shared by the prometheus_mp plugin and its unit tests. Built
+# unconditionally so the CTest coverage runs even where prometheus-cpp is absent;
+# the plugin shared library (which needs prometheus-cpp) is added in a later step.
+prometheus_mp_inc_dirs = include_directories('.')
+
+telemetry_mp_store_lib = static_library(
+    'nixl_telemetry_mp_store',
+    'mp_store.cpp',
+    include_directories: [nixl_inc_dirs, utils_inc_dirs, prometheus_mp_inc_dirs],
+    dependencies: [nixl_common_dep],
+    install: false,
+    pic: true,
+)
+
+nixl_telemetry_mp_store_dep = declare_dependency(
+    link_with: telemetry_mp_store_lib,
+    include_directories: prometheus_mp_inc_dirs,
+    dependencies: [nixl_common_dep],
+)
+
+# Scrape-time collector: converts peer store snapshots into prometheus-cpp metric
+# families. Needs prometheus-cpp (core), so it is gated on it being available
+# (same condition as the prometheus plugin, set in the sibling subdir above).
+if prometheus_found
+    telemetry_mp_collector_lib = static_library(
+        'nixl_telemetry_mp_collector',
+        'mp_collector.cpp',
+        include_directories: [nixl_inc_dirs, utils_inc_dirs, prometheus_mp_inc_dirs],
+        dependencies: [nixl_common_dep, prometheus_core_dep, nixl_telemetry_mp_store_dep],
+        install: false,
+        pic: true,
+    )
+
+    nixl_telemetry_mp_collector_dep = declare_dependency(
+        link_with: telemetry_mp_collector_lib,
+        include_directories: prometheus_mp_inc_dirs,
+        dependencies: [nixl_common_dep, prometheus_core_dep, nixl_telemetry_mp_store_dep],
+    )
+
+    # Exporter with bind-race election. Built as a static library too so it can be
+    # unit-tested directly (in addition to being packaged into the plugin .so).
+    telemetry_mp_exporter_lib = static_library(
+        'nixl_telemetry_mp_exporter',
+        'prometheus_mp_exporter.cpp',
+        include_directories: [nixl_inc_dirs, utils_inc_dirs, prometheus_mp_inc_dirs],
+        dependencies: [nixl_common_dep, prometheus_core_dep, prometheus_pull_dep,
+                       nixl_telemetry_mp_store_dep, nixl_telemetry_mp_collector_dep],
+        install: false,
+        pic: true,
+    )
+
+    nixl_telemetry_mp_exporter_dep = declare_dependency(
+        link_with: telemetry_mp_exporter_lib,
+        include_directories: prometheus_mp_inc_dirs,
+        dependencies: [nixl_common_dep, prometheus_core_dep, prometheus_pull_dep,
+                       nixl_telemetry_mp_store_dep, nixl_telemetry_mp_collector_dep],
+    )
+
+    # Installable plugin: selected via NIXL_TELEMETRY_EXPORTER=prometheus_mp.
+    prometheus_mp_exporter_plugin = shared_library(
+        'libtelemetry_exporter_prometheus_mp',
+        'prometheus_mp_plugin.cpp',
+        include_directories: [nixl_inc_dirs, utils_inc_dirs, prometheus_mp_inc_dirs],
+        dependencies: [nixl_infra, nixl_common_dep, absl_log_dep, prometheus_core_dep,
+                       prometheus_pull_dep, nixl_telemetry_mp_exporter_dep],
+        install: true,
+        install_dir: plugin_install_dir,
+        name_prefix: '',
+    )
+    message('Building Prometheus multiprocess telemetry exporter plugin')
+endif