telemetry: native multi-process Prometheus aggregation (prometheus_mp)#1920
telemetry: native multi-process Prometheus aggregation (prometheus_mp)#1920e-eygin wants to merge 18 commits into
Conversation
First component of the native multi-process Prometheus aggregation (prometheus_mp) plugin. mp_store is a per-process, fixed-slot memory-mapped store holding current metric state (cumulative counters and last-operation gauges) indexed by nixl_telemetry_event_type_t, for scrape-time aggregation by a single exporter process. It is a raw mmap of a fixed POD layout (no serialization): writers update slots in place with lock-free __atomic ops; a reader validates the magic/schema/size header and returns a consistent snapshot. Per-process identity (pid + /proc start_time, agent name, hostname, optional dp_rank) lives in the header for liveness and labeling. Built as a small prometheus-cpp-free static library with CTest coverage. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
Second component of the prometheus_mp plugin. nixlMultiprocessCollector is a prometheus::Collectable that, on each scrape, globs the shared telemetry directory for peer store files, reads a snapshot of each, drops (and optionally reaps) stale ones, and returns per-process metric families. Series are emitted per (metric, process) with no cross-process aggregation: cumulative counters and last-operation gauges keyed by nixl_telemetry_event_type_t, plus agent_errors_total with a status label, labeled by hostname, agent_name and (when present) dp_rank. Staleness combines pid liveness (kill(pid,0) + /proc start_time to guard PID reuse) with a last-update TTL. The family-building and liveness logic are split into pure functions for unit testing; a shared store file-naming helper is added to mp_store. Collector library and tests are gated on prometheus-cpp. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
The collector's per-process series were disambiguated only by agent_name (plus optional dp_rank). If two processes share an agent name and have no dp_rank, they would emit an identical label set -> a duplicate Prometheus series and a rejected scrape. Add an unconditional pid label so every live process is a distinct series regardless of how callers name agents. pid (not the reserved "instance" target label) also avoids the exported_instance renaming trap. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
Third component of the prometheus_mp plugin: the exporter that ties the store and collector together, selectable via NIXL_TELEMETRY_EXPORTER=prometheus_mp. Every process writes its own metric state to a per-process store in NIXL_TELEMETRY_MULTIPROC_DIR (unique file per pid/instance). On construction each process races to bind the scrape port: the winner runs a prometheus-cpp Exposer plus a nixlMultiprocessCollector that aggregates all peers on each scrape; losers fall back to writer-only mode. A bind collision is caught internally and never rethrown as nixlTelemetryBindFailed, so every process -- owner or writer -- gets a valid telemetry sink and all ranks are exported behind the single owner endpoint. Config: NIXL_TELEMETRY_MULTIPROC_DIR (required), NIXL_TELEMETRY_RANK_ENV (optional dp_rank source, default LOCAL_RANK), NIXL_TELEMETRY_MP_STALE_TTL; reuses NIXL_TELEMETRY_PROMETHEUS_PORT / _LOCAL. The exporter is built both as the installable plugin .so and as a static library for direct unit testing (owner mode, writer-mode-on-collision, missing-dir). Gated on prometheus-cpp. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
Fourth component of the prometheus_mp plugin: an end-to-end test that proves cross-process aggregation with real processes. The parent becomes the bind-race owner and serves the endpoint; N children fork (while the parent is still single-threaded) and run as writers against the shared dir. After a single HTTP scrape of the owner port, all N+1 processes appear as distinct per-process series. One child is then killed and reaped, and a second scrape confirms its series is dropped and its store file removed (stale TTL 0). Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
Handle processes that appear or die mid-scrape more robustly: - Reader treats a zero-magic store (a process still initializing its file, or an orphan from a process killed mid-creation) as a quiet skip instead of a "bad magic" WARN; a genuinely wrong non-zero magic still warns. - Collector now reaps unparseable store files (zero/bad magic, wrong schema, truncated) once they are older than max(stale TTL, 2s). The floor protects a store a live process is actively creating from being deleted out from under it, even when the TTL is 0. This closes a slow file leak: a process killed during store creation left a zero-magic file that was correctly skipped every scrape but never cleaned up. Cleanup is performed by the bind-race owner during Collect(), since a killed process cannot clean up after itself. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
Describe the native single-endpoint multi-process aggregation exporter in docs/telemetry.md: add it to the architecture components, update the multi-process scrape note to point ranks needing full aggregation at it, and add a "Multi-process aggregation" section covering its model, configuration (NIXL_TELEMETRY_MULTIPROC_DIR/RANK_ENV/MP_STALE_TTL), labels, and the explicit limitation that it is hardcoded to NIXL's fixed metric model and cannot carry dynamic per-observation labels. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
Document the multi-process Prometheus exporter plug-in alongside the sibling prometheus and doca plugin READMEs: how bind-race owner election and the per-process mmap stores work, configuration (NIXL_TELEMETRY_MULTIPROC_DIR and the optional RANK_ENV / MP_STALE_TTL / port vars), the per-process labels (hostname, agent_name, pid, optional dp_rank), and the explicit scope note that it is purpose-built for NIXL's fixed metric model (no dynamic per-observation labels) rather than a generic Prometheus multiprocess store. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
Clarify that NIXL_TELEMETRY_MULTIPROC_DIR follows Dynamo's PROMETHEUS_MULTIPROC_DIR convention: a shared local folder (not NFS), one per pod/process-family, treated as ephemeral. Note the key difference from Dynamo: NIXL is loaded independently per rank with no parent to propagate the path, so the launcher/operator must set the same directory for every rank (hence it is required, not auto-defaulted). tmpfs is optional, not required. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
|
👋 Hi e-eygin! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds an experimental ChangesPrometheus MP exporter
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant Process
participant nixlTelemetryPrometheusMpExporter
participant mpStoreWriter
participant PrometheusExposer
participant nixlMultiprocessCollector
Process->>nixlTelemetryPrometheusMpExporter: construct exporter
nixlTelemetryPrometheusMpExporter->>mpStoreWriter: create per-process mmap store
nixlTelemetryPrometheusMpExporter->>PrometheusExposer: attempt port bind
PrometheusExposer-->>nixlTelemetryPrometheusMpExporter: owner or writer-only mode
Process->>nixlTelemetryPrometheusMpExporter: exportEvent
nixlTelemetryPrometheusMpExporter->>mpStoreWriter: write metric update
PrometheusExposer->>nixlMultiprocessCollector: scrape /metrics
nixlMultiprocessCollector->>mpStoreWriter: read live snapshots
nixlMultiprocessCollector-->>PrometheusExposer: return MetricFamily list
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
CI pre-commit (codespell) flagged the local nowNs() helper (read as "knowns/nouns") and "unparseable". - Replace the duplicated per-file nowNs() helpers with the existing nixlTime::getNs() (CLOCK_MONOTONIC, host-wide so comparable across processes; also skew-free vs wall clock for the staleness delta). The last-update timestamp is only ever compared against another getNs() reading, so switching from system_clock to steady_clock is safe; the orphan-file mtime path is independent and still uses wall-clock time(). - Spell "unparseable" as "unparsable" in comments. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
|
@coderabbitai help |
ChatThere are 3 ways to chat with CodeRabbit:
CodeRabbit commands
Other keywords and placeholders
Status, support, documentation and community
|
|
@coderabbitai resume |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/plugins/telemetry/prometheus_mp/mp_collector.cpp`:
- Around line 138-191: The reads in buildMetricFamilies are indexing fixed-size
snapshot arrays directly from telemetry_metric_event_types and
telemetry_error_event_types, so add bounds protection before accessing
snap.counters or snap.gauges. Update the buildMetricFamilies logic to use a
checked access path (such as .at() or an explicit range guard) for the slot
derived from the enum, and make sure the error-family loop uses the same
protection. Keep the existing telemetryMetricDescriptor, counterMetric, and
gaugeMetric flow unchanged otherwise.
- Around line 200-235: The Collect() scan still relies on the range-based for
over std::filesystem::directory_iterator, so an implicit iterator increment can
throw even though the constructor uses std::error_code. Update
nixlMultiprocessCollector::Collect() to handle iterator advancement errors
explicitly, either by switching to a manual directory_iterator loop that uses
the non-throwing increment overload with an error_code or by wrapping the
iteration in a try/catch that safely returns an empty result on
filesystem_error. Keep the existing dir_ scan, readStoreSnapshot, and reapStale_
logic intact while making the traversal resilient to mid-scan directory changes.
In `@src/plugins/telemetry/prometheus_mp/mp_store.cpp`:
- Around line 22-32: mp_store.cpp is using std::min and std::istream_iterator
without directly including their standard headers. Update the file’s include
list near the top to add both <algorithm> and <iterator> so MpStore’s
implementation does not rely on transitive includes.
In `@src/plugins/telemetry/prometheus_mp/mp_store.h`:
- Around line 41-47: The slot-count derivation in MP_STORE_SLOT_COUNT is fragile
because it assumes AGENT_TELEMETRY_EVENTS_DROPPED remains the last enum value.
Add a compile-time safeguard in mp_store.h, near the MP_STORE_SLOT_COUNT
definition, using a static_assert that validates the telemetry event enums fit
within the array bounds. Make the check cover the relevant
nixl_telemetry_event_type_t values used by mp_collector.cpp’s
buildMetricFamilies so future enum extensions fail at compile time instead of
causing out-of-bounds access.
In `@src/plugins/telemetry/prometheus_mp/prometheus_mp_exporter.cpp`:
- Around line 40-49: The telemetry constant declarations in
prometheus_mp_exporter.cpp use constexpr char[] while the matching constants in
mp_store.h use inline constexpr std::string_view, so make them consistent.
Update the identifiers such as defaultRankEnvName, prometheusPortVar,
prometheusLocalVar, multiprocDirVar, rankEnvVar, and staleTtlVar to use the same
string_view style as the sibling component, keeping the existing names and
usages intact.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 249ccc98-927b-47c3-95a4-ea65420aac52
📒 Files selected for processing (16)
docs/telemetry.mdsrc/plugins/telemetry/meson.buildsrc/plugins/telemetry/prometheus_mp/README.mdsrc/plugins/telemetry/prometheus_mp/meson.buildsrc/plugins/telemetry/prometheus_mp/mp_collector.cppsrc/plugins/telemetry/prometheus_mp/mp_collector.hsrc/plugins/telemetry/prometheus_mp/mp_store.cppsrc/plugins/telemetry/prometheus_mp/mp_store.hsrc/plugins/telemetry/prometheus_mp/prometheus_mp_exporter.cppsrc/plugins/telemetry/prometheus_mp/prometheus_mp_exporter.hsrc/plugins/telemetry/prometheus_mp/prometheus_mp_plugin.cpptest/gtest/meson.buildtest/gtest/telemetry_mp_collector_test.cpptest/gtest/telemetry_mp_e2e_test.cpptest/gtest/telemetry_mp_exporter_test.cpptest/gtest/telemetry_mp_store_test.cpp
✅ Action performedReviews resumed. |
… includes) - mp_store.h: add a compile-time static_assert that MP_STORE_SLOT_COUNT covers every telemetry event type the collector indexes, so extending the enum past AGENT_TELEMETRY_EVENTS_DROPPED fails the build instead of causing an out-of-bounds counter/gauge access. - mp_collector.cpp: iterate the shared directory with the non-throwing increment(ec) instead of the range-for's throwing operator++, since peer writers and this collector's own reaping mutate the directory concurrently and a mid-iteration filesystem error would otherwise escape Collect(). - mp_store.cpp: add explicit <algorithm> and <iterator> includes (std::min, std::istream[buf]_iterator) rather than relying on transitive includes. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
|
/build |
|
/ok to test 3bf2139 |
The optional per-process rank label was named dp_rank, but its value is sourced from the local/per-GPU (tensor-parallel) rank env (default LOCAL_RANK), not the data-parallel rank. That misdescribed the value and collided with Dynamo's own data-parallel dp_rank series. Rename the emitted label and the internal identifiers (store field, writer param, exporter helper) to local_rank. The NIXL_TELEMETRY_RANK_ENV var and its LOCAL_RANK default are unchanged; the label is still optional and emitted only when the env is set. Docs and tests updated. Part of NIX-1614. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
The prometheus_mp store writer only unmapped its file on destruction, so an agent destroyed while its process keeps running left a store behind. Because liveness is pid-based, the collector kept publishing that dead agent's frozen series indefinitely. Have ~mpStoreWriter() best-effort remove its own file: clean shutdown is the deterministic "producer gone" signal, while crash/kill (destructor never runs) still falls back to the owner's liveness/TTL reaping. Adjust the store round-trip tests to read while the writer is alive (the real cross-process pattern) and add a test for the new cleanup. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
Multiple agents created in one process with the same agent name shared every series label (hostname, agent_name, pid, local_rank), producing duplicate Prometheus series that make the whole scrape fail. The store filename already carried a per-process instance counter, but it never reached the exposed labels. Persist the instance in the store header and emit it as an agent_instance label, so same-name same-process agents get distinct series (agent_instance=0 for the common single-agent case). Schema version is unchanged: prometheus_mp has not shipped, so no on-disk format compatibility is at stake. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
…tance NIXL_TELEMETRY_MP_STALE_TTL is measured from a dead process's last update, not from the moment of death; a live process is always published regardless of age. Reword both docs to say so. Also add the agent_instance label to the telemetry.md label list to match the prometheus_mp exporter. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/plugins/telemetry/prometheus_mp/prometheus_mp_exporter.cpp (1)
105-146: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winDocument the "construct after fork" invariant.
mpStoreWriterconstruction ties the store file to the current process's pid (getpid()) and a per-process instance counter. If a caller ever constructs this exporter beforefork()ing (rather than once per process, post-fork), parent and child would share the same store file/pid label, and whichever writer is destructed first (mp_store.cpp's destructor now removes the file) would silently delete metrics visibility for the other, still-live process. This appears to be relied upon by the forking end-to-end test described in the PR. Consider adding a short comment here (or in the class doc) stating this construction-after-fork requirement explicitly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/plugins/telemetry/prometheus_mp/prometheus_mp_exporter.cpp` around lines 105 - 146, Add a concise comment near nixlTelemetryPrometheusMpExporter construction, or in its class documentation, explicitly stating that each exporter must be constructed once per process after fork. Mention that mpStoreWriter binds its store file to the current PID and per-process instance, so pre-fork construction can cause parent and child to share a file and interfere during destruction.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/plugins/telemetry/prometheus_mp/prometheus_mp_exporter.cpp`:
- Around line 105-146: Add a concise comment near
nixlTelemetryPrometheusMpExporter construction, or in its class documentation,
explicitly stating that each exporter must be constructed once per process after
fork. Mention that mpStoreWriter binds its store file to the current PID and
per-process instance, so pre-fork construction can cause parent and child to
share a file and interfere during destruction.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 23c90a9c-c01f-4b2b-bba8-e0915fd5e09b
📒 Files selected for processing (7)
src/plugins/telemetry/prometheus_mp/README.mdsrc/plugins/telemetry/prometheus_mp/mp_collector.cppsrc/plugins/telemetry/prometheus_mp/mp_store.cppsrc/plugins/telemetry/prometheus_mp/mp_store.hsrc/plugins/telemetry/prometheus_mp/prometheus_mp_exporter.cpptest/gtest/telemetry_mp_collector_test.cpptest/gtest/telemetry_mp_store_test.cpp
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/telemetry.md`:
- Around line 160-164: Update the telemetry documentation near the series-label
description to explicitly document the `status` label on every
`agent_errors_total` series, matching
`src/plugins/telemetry/prometheus_mp/README.md` and the collector behavior while
distinguishing it from the common labels.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 83929b5c-a234-497f-aecb-eb2160e08d6e
📒 Files selected for processing (2)
docs/telemetry.mdsrc/plugins/telemetry/prometheus_mp/README.md
The prometheus_mp label list omitted that agent_errors_total additionally carries the bounded status label, unlike the plugin README. Align the two. Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
|
/build |
|
/ok to test ad437a8 |
…tion Resolve test/gtest/meson.build: keep both the prometheus_mp store test and the newly-merged transfer-duration histogram test in gtest_sources. docs/telemetry.md auto-merged (histogram and multiprocess sections are independent). Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
…tion Resolve docs/telemetry.md: keep both new exporter list entries (prometheus_mp and the DOCA IPC->DTS description from ai-dynamo#1897), renumbered. test/gtest/meson.build auto-merged (ai-dynamo#1919 activation test alongside the mp/histogram tests). Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
|
/build |
|
/ok to test aa3f6bd |
|
🤖 CI Triage Agent — TL;DR: The elastic EP fault-tolerance test ( Full analysisSummary: Stage 203 "Run DL EP elastic tests" (UCX master variant) failed — when rank 2 was intentionally killed mid-dispatch/combine, the surviving ranks hit a CUDA illegal-memory-access and exited with code 1 instead of gracefully detecting the failure. Root cause: UCX-master-specific regression in the endpoint teardown path. In the failing (master) run, immediately after Implicated commit: File: Suggested fix: In the forced UCP endpoint close path added in #1846, ensure endpoint/CUDA-IPC teardown does not free or destroy CUDA resources (device_ep / streams / events) that a dead peer may still reference — e.g. gate the forced close behind the existing combine-timeout fault-detection so the graceful "detected rank failures / cleaning up" path (which works on v1.22.x) runs instead of an immediate teardown, or set the correct CUDA device/context before Related: PR #1846 (#1846); this failure is on PR #1920's CI run but appears to be a UCX-master regression independent of #1920. |
|
🤖 CI Triage Agent — TL;DR: The "Run Nixlbench tests" stage (ID 356) was killed (exit 143/SIGTERM) after hanging for ~18 minutes in the UCCL backend; the hang — not a time limit — is the root cause, and it is unrelated to PR #1920 (a telemetry-only change), so this is a flaky/pre-existing UCCL teardown/connection deadlock. Full analysisSummary: Root cause: A hang, not a slow test. The last application output was at Implicated commit: unknown (not introduced by PR #1920 / commit 9f7abf6). Hang is in the UCCL backend; last-touched by Pravein Govindan Kannan in File: Suggested fix: Do not raise the stage time limit — that would only mask the hang. Instead: (1) re-run to confirm this is the known flaky UCCL hang (PR #1920 is telemetry-only and shouldn't gate on it); (2) add a per-test Related: PR #1724 (UCCL crash Fix + GDB trap); triggering PR #1920 (telemetry prometheus_mp, unrelated). |
|
🤖 CI Triage Agent — TL;DR: Both "Compiling NIXL EP Docker Image for DL" stages failed because Full analysisSummary: The Docker image build (stages 150 and 168) failed at the Root cause: Implicated commit: None — not a code regression. (The apt step itself was last touched by File: Suggested fix:
Related: none Note: the
|
|
🤖 CI Triage Agent — TL;DR: The "Run DL EP elastic tests" stage failed because the Full analysisSummary: Elastic EP test Root cause: In the fault-injection phase, rank 2 receives SIGTERM as planned ( Implicated commit: 22cdab3 — "PLUGINS/UCX: Added option to forcibly close UCP endpoints. (#1846)", Raul Akhmetshin, 2026-07-15 (most recent change, directly touches UCP endpoint close/teardown behavior exercised by this fault-tolerance test). File: UCX plugin endpoint-close path (introduced by #1846) — e.g. Suggested fix: In the forcibly-close-UCP-endpoints path added by #1846, guard the CUDA-IPC endpoint teardown so it does not call Related: PR #1846 (#1846); possibly related test-hardening PR #1909. |
|
🤖 CI Triage Agent — TL;DR: The "Run Nixlbench tests" stage was ABORTED (exit 143) not because it was slow, but because the UCCL backend hung during the startup of the UCCL WRITE DRAM→VRAM two-worker test — the node stopped producing output at 07:43:19 and went unreachable, and Jenkins killed it ~23 minutes later. Investigate the UCCL engine connection/teardown path, not the CI time limit. Full analysisSummary: Jenkins stage Root cause: A hang in the UCCL backend during two-worker (ASIO) benchmark setup. The final application log line is at Implicated commit: Not conclusively one commit from the log alone. Most relevant suspects are the UCCL backend changes: File: Suggested fix: Do not raise the Jenkins/SLURM time limit — that treats the symptom. Instead: (1) add a per-invocation timeout wrapper around each Related: UCCL history: PR #1271 ( |
2 participants
What?
Adds
prometheus_mp, a new native telemetry exporter plug-in that exposes the telemetry of all processes of a multi-process NIXL run (tensor/data/pipeline parallelism) behind a single Prometheus scrape endpoint — with no external dependency (no DOCA/DTS, and not built on Python'sprometheus_client).Select it with
NIXL_TELEMETRY_EXPORTER=prometheus_mpand point every rank at one shared directory viaNIXL_TELEMETRY_MULTIPROC_DIR.Components (each landed as its own reviewable commit):
mp_store— a per-process, fixed-slot, memory-mapped metric-state store. Each process writes its own current counter/gauge values in place with lock-free__atomicops. No serialization; the file is the state.mp_collector— aprometheus::Collectablethat, on each scrape, globs the shared dir, reads a consistent snapshot of every live peer, and emits one series per (metric, process). Includes liveness/stale filtering and orphan reaping.prometheus_mp_exporter+ plugin — bind-race owner election: every process writes its own store, the one that binds the scrape port becomes the owner (runs theExposer+ collector), the rest self-demote to writer-only. Built both as the installable plugin.soand as a static lib for unit testing.src/plugins/telemetry/prometheus_mp/README.mdand a multi-process section indocs/telemetry.md.Why?
The existing native Prometheus exporter opens one HTTP scrape endpoint per process. Under multi-process parallelism every rank creates its own NIXL agent and tries to bind the same telemetry port; only one process can bind it, so the telemetry of every other rank is lost by design. This was reported for a Dynamo + SGLang TP=8 deployment on Kubernetes (GitHub issue #1838): one rank serves metrics and the rest log a bind failure and run without a sink.
The existing exporter can only make that failure benign (it logs a warning and the rank continues without telemetry) — it cannot recover the lost ranks. This PR makes every rank's telemetry available behind a single endpoint, natively.
Two deployment-shape constraints drove a native, dependency-free design rather than routing aggregation through an external service:
This complements (does not replace) the DOCA/CollectX exporter, which aggregates through an external telemetry service — that remains the right choice when such a service is already deployed;
prometheus_mpis the standalone, native alternative.Public tracking issue: #1838
How?
Architecture
Every process writes its own store; exactly one — the bind-race owner — reads them all and serves. Losing the bind race is benign: the bind exception is caught internally and never propagated, so a writer is a fully valid telemetry sink and no rank is dropped.
Crash- and race-safety (the core of the design)
The store is a raw mmap of a fixed POD layout, so it must survive concurrent readers, new processes appearing mid-scrape, and processes killed mid-update:
__atomicop.SIGKILLcannot interrupt it, so readers never observe a torn value; a process killed mid-batch merely leaves a store a few increments behind (metrics are point-in-time snapshots, not a durable ledger).magicvalue last (release/acquire). A reader either sees a fully initialized header or skips the file; a process killed mid-creation leaves a zero-magic file that is skipped quietly and later reaped.Series & labels
Per-process series (no cross-process summing, so counters stay monotonic), labeled
hostname,agent_name,pid(guarantees cross-process uniqueness even if agent names collide; deliberately not the reservedinstancelabel),agent_instance(distinguishes multiple same-name agents created in one process;0in the common single-agent case), and optionallocal_rank(the local/per-GPU (TP) rank, only when a rank env var is set). Metric names, types, and semantics are identical to the single-processprometheusexporter.Cleanup & lifecycle
On clean shutdown a process removes its own store. A crashed or killed process cannot, so the owner reaps lazily during scrape: valid-but-dead stores via liveness (
kill(pid,0)+/procstart time, which is PID-reuse safe) once their last update is past the TTL; unparseable orphans by file mtime pastmax(TTL, 2s)(the floor protects a store a live process is mid-creating). Whole-run cleanup is left to the deployment (e.g. a per-pod emptyDir), the same model container-based Prometheus multiprocess setups already use.Configuration
NIXL_TELEMETRY_EXPORTER=prometheus_mpNIXL_TELEMETRY_MULTIPROC_DIRNIXL_TELEMETRY_PROMETHEUS_PORT/_LOCALNIXL_TELEMETRY_RANK_ENVlocal_ranklabelLOCAL_RANKNIXL_TELEMETRY_MP_STALE_TTLNIXL_TELEMETRY_MULTIPROC_DIRmust be a local filesystem (not NFS): mmapMAP_SHAREDcross-process visibility depends on a local page cache._STALE_TTLaffects only already-dead processes (a live process is always published regardless); set it at least as large as your Prometheus scrape interval so a rank's final values are captured.Scope / limitations (intentional)
Purpose-built for NIXL's fixed, low-cardinality, per-process label model — it cannot represent a metric with a dynamic/high-cardinality label that varies per observation (none exist today; adding one would require a different, keyed store). Proactive owner failover (a writer promoting itself if the owner dies) is out of scope: until a new agent re-binds the freed port, writers keep writing but nothing serves.
Testing
Runs under CTest/meson: store, collector, and exporter unit tests plus the forking e2e. All
prometheus_mptests pass; changed lines are clang-format-19 clean and every commit is DCO-signed.Summary by CodeRabbit
prometheus_mp) that aggregates metrics into a single/metricsendpoint with scrape owner election.