build(lttng): add shared LTTng-UST consumption helper#8907
Conversation
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
cfreeamd
left a comment
cfreeamd
left a comment
There was a problem hiding this comment.
🤖 AI-assisted pre-review (for @cfreeamd)
Scoped to projects/rocr-runtime/ only. The single rocr-runtime change is the one-line default added to projects/rocr-runtime/CMakeLists.txt; the rest of the PR is the shared/lttng/ vendoring which is outside ROCr review scope. One question on the new CMake variable — see inline.
| # LTTng-UST package dep default OFF while ROCR_ENABLE_LTTNG_UST defaults | ||
| # OFF; the subdir CMakeLists overrides this to ON if the option is | ||
| # explicitly enabled. | ||
| set(HSA_DEP_LTTNG_UST OFF CACHE INTERNAL "") |
There was a problem hiding this comment.
🤖 AI-assisted pre-review (for @cfreeamd)
[Question] This sets HSA_DEP_LTTNG_UST OFF but nothing in this PR (or current develop) consumes it. The sibling HSA_DEP_ROCPROFILER_REGISTER right above exists specifically to gate the CPACK block below it (if (HSA_DEP_ROCPROFILER_REGISTER) → appends rocprofiler-register to the Debian/RPM package deps); there is no matching if (HSA_DEP_LTTNG_UST) block here. The comment also references ROCR_ENABLE_LTTNG_UST and "the subdir CMakeLists overrides this to ON", but neither the ROCR_ENABLE_LTTNG_UST option nor any reader of HSA_DEP_LTTNG_UST appears in the diff or the tree.
Is the consumer (the ROCR_ENABLE_LTTNG_UST option + the subdir override + presumably a CPACK dep for lttng-ust) intended to land in a follow-up PR? If so, this default is harmless as a forward-declaration, but the comment describes machinery that doesn't exist yet and will confuse a reader who greps for it. Consider either landing the consumer in the same PR or trimming the comment to describe only what's present.
There was a problem hiding this comment.
As an aside, all of the bespoke CPack machinery in our projects is now obsolete.
stellaraccident
left a comment
stellaraccident
left a comment
There was a problem hiding this comment.
Hey Ben, is there a strong reason to do this different from every other third-party vendored dep?
Generally, we add "sysdeps" in TheRock and set them up for packaging. Then code the incoming project (in this case rocr) to take a project level dep on it and find it via usual CMake lookups.
Go ahead and follow the pattern for all of the existing ones: https://github.com/ROCm/TheRock/tree/main/third-party/sysdeps/linux
There are quite a few of them in there already including ones that are working with make-based pkgconfig deps. If you find the corresponding PR where a similar one was added, it should be easy to duplicate.
Note that we typically do not track submodules for these deps: we mirror source tarballs to permanent S3 storage and fetch them on demand when the dep is needed. Among many reasons for this, we've had real problems over time keeping deps alive for the life of how long we have to be able to build old branches, so we always mirror the sources to something we control.
|
When updating TheRock in a replacement PR, please also update the inventory: https://github.com/ROCm/TheRock/blob/main/docs/development/dependencies.md I would also recommend defining a flag for this feature in TheRock in the same opening PR: https://github.com/ROCm/TheRock/blob/main/docs/development/flags.md Then use that to enable lttng integration and the dep in TheRock's ROCR project definition. This will let you parallelize some of your patch submission because you can leave the flag off for most of your patches and only enable it once you have a critical mass that works. |
Add shared CMake glue for consuming LTTng-UST when emitting LTTng tracepoints from ROCm components (the HIP/HSA curated tracepoint providers land in later commits of this series). LTTng-UST and its userspace-rcu dependency are provided as TheRock bundled sysdeps (enable with -DTHEROCK_ENABLE_LTTNG=ON) or by a system install, and discovered via pkg-config -- they are NOT vendored in this repo. shared/lttng/cmake/RocmLttng.cmake runs pkg_check_modules for lttng-ust>=2.13 and defines the imported target PkgConfig::LTTNG_UST, failing with actionable guidance when LTTng is enabled but unavailable. This replaces the earlier approach of vendoring lttng-ust + userspace-rcu as git submodules with an in-tree ExternalProject build, per review feedback to use TheRock's standard sysdeps mechanism (ROCm/TheRock#6750). ISSUE ID : #8906
57f672a to
7b132a4
Compare
bwelton
changed the title
|
@stellaraccident thanks for the review — agreed on all points. I've reworked this to follow TheRock's sysdeps pattern instead of vendoring submodules here. Companion TheRock PR: ROCm/TheRock#6750 — adds This PR now just adds Also addressed @cfreeamd's inline note — the unused |
3 participants
Series overview (PR 1 of the LTTng tracepoint series)
This is the first PR in a stacked series that adds built-in, near-zero-cost
LTTng-UST tracepoints for the HIP and HSA runtimes, curated per public API with
typed arguments and return values. The series ships one reviewable unit at a
time: foundation/tooling first, then one tracing category at a time for HIP
and HSA (e.g. "HIP memory API tracing", "HSA signal API tracing"), then the
category-based selection layer. The full roadmap is at the bottom.
Because the value of the whole series rests on it being cheap enough to ship
always-on-capable in production libraries, this first PR carries the complete
performance evaluation for the entire effort (three workloads on
MI325X/gfx942), so reviewers can weigh the cost/benefit up front. Later PRs
reference these numbers rather than repeating them.
Motivation
ROCm today traces HIP/HSA API activity primarily through rocprofiler-sdk
(rocprofv3). That is the right tool for rich, tool-driven profiling, but it is a
heavyweight, opt-in, out-of-process mechanism. There is no lightweight,
always-available, low-overhead way to get a structured, typed timeline of
runtime API calls from a production process — the kind of thing you want when
debugging a customer issue in the field, correlating runtime activity with
kernel/OS events, or attaching to a long-running service without restarting it
under a profiler.
LTTng-UST fills that gap:
LTTng providers on one unified timeline (
babeltrace2, Trace Compass).libraries can ship enabled by default; a trace session is started only when
wanted, with per-category granularity.
value with their real C types, not opaque blobs.
Design rationale
Three design decisions define the series:
Combined per-API event, schema v1. Each curated API gets exactly ONE
tracepoint event that is fired twice per call — once on entry, once on exit —
discriminated by a
phasefield (0=enter, 1=exit). IN arguments ride theenter record; audited OUT pointers + status ride the exit record. This halves
the event-definition count vs a separate enter/exit pair and keeps a single
stable schema per API. (The series ships schema v1 from the first tracepoint
commit; there is no legacy schema in the history.)
No-link dynamic provider. The tracepoint provider is loaded dynamically;
the HIP/HSA libraries do not hard-link
liblttng-ust. If LTTng is notpresent at runtime, the provider simply isn't activated and the per-call cost
is a single predictable branch.
Category-based selection. The ~554 HIP and ~217 HSA curated APIs are
grouped into review-friendly categories (memory, streams, events,
kernel_launch, graph, signals, queues, …). A session can enable exactly the
categories it wants via a generated per-provider manifest and a small helper,
so you pay only for what you trace.
Performance
All numbers: Banff MI325X / gfx942, kernel 5.15, container build. Both
baselines are built from source on the same develop-HEAD toolchain,
differing only by LTTng: existing rocm-systems = pre-LTTng commit (LTTng not
compiled in); lttng off = these libraries with LTTng compiled in but no
active session. rocprofiler-sdk conditions (
null_*= raw buffered tracer,rocprofv3_*= full CSV tracer) run against a from-source, registration-enabledbuild at the same base commit — so the only variable measured is the tracer
itself. Validity: 0 discarded LTTng events on every run; every
rocprofiler-sdk condition independently confirmed
records>0(record-countevidence retained). rocprofiler-sdk HIP-record collection required fixing a
base-library build flag (
HIP_ENABLE_ROCPROFILER_REGISTER=ON) and a null-tracerbuffer-flush + a vLLM harness parsing bug, so the SDK numbers below are real,
apples-to-apples measurements, not measurements of a silent no-op.
1. graph_bench — light, realistic (~45K events/run, 12 reps/condition)
Shipping LTTng compiled-in but inactive (lttng off) is within 0.3% of not having
it at all. Active tracing is ≤1.10× even for the full firehose — at or below
rocprofiler-sdk.
2. Dispatch-rate microbenchmark — worst case (tight API loop, zero GPU work)
hip_rate= 20M back-to-back trivial HIP calls;hsa_rate= 40M trivial HSAcalls. 100% of wall time is tracer overhead — the ceiling. (lower s is better)
Consistent ordering on both runtimes: LTTng < rocprofiler-sdk null <
rocprofv3. At max call rate LTTng is ~1.5× cheaper than the null tracer and
~6× cheaper than rocprofv3, because rocprofv3's per-event CSV formatting/I/O
dominates.
3. vLLM — real GPU-bound inference (Qwen3-30B-A3B-FP8, ~7.2M events/run)
On a real GPU-bound workload, every tracing mechanism is within ~±4% of
baseline — statistically at the 3-rep noise floor (existing rocm-systems vs
lttng off alone differ 2.7%). The large synthetic overheads vanish once each API
call does real GPU work.
Cross-workload summary (heaviest config per mechanism)
Bottom line: LTTng-off is free; LTTng-on is at/near the noise floor on
realistic and GPU-bound workloads, and materially cheaper than rocprofiler-sdk in
high-API-rate / low-GPU-work regimes. Ranking (LTTng ≲ null < rocprofv3) holds
across all three workload shapes.
This PR
Foundation only — no tracepoints yet. It adds the shared CMake glue for
consuming LTTng-UST; the HIP/HSA providers and per-category tracepoints land in
later PRs of the series.
shared/lttng/cmake/RocmLttng.cmake— runspkg_check_modules(lttng-ust>=2.13)and defines the imported targetPkgConfig::LTTNG_UST; fails with actionable guidance (enable in TheRock /install dev package / disable the option) when LTTng is enabled but
unavailable. It does not mutate
PKG_CONFIG_PATH— discovery relies onCMAKE_PREFIX_PATH/PKG_CONFIG_PATHfrom the TheRock super-build (the sysdep)or the system.
shared/lttng/README.md— documents the consumption model, how components optin, where LTTng-UST comes from (TheRock sysdep vs system), and how to build
without LTTng.
LTTng-UST + userspace-rcu are provided as TheRock bundled sysdeps (companion
PR ROCm/TheRock#6750), built with private
rocm_sysdeps_SONAMEs + symbolversioning and gated behind
THEROCK_ENABLE_LTTNG(off by default), or by asystem install. No submodules, no vendored source, no
ExternalProjectin thisrepo. This also drops the bespoke per-project
HSA_DEP_LTTNG_USTCPACK variable— packaging deps now flow through TheRock's
RUNTIME_DEPS/bundling.lttng-ustsysdep,-DTHEROCK_ENABLE_LTTNG=ON(off by default), private underlib/rocm_sysdeps/— ROCm/TheRock#6750liblttng-ust-dev/lttng-ust-devel, via pkg-configTesting
This PR: CMake parses
RocmLttng.cmake;pkg_check_modulesresolveslttng-ust>=2.13against a TheRock-style sysdep prefix and definesPkgConfig::LTTNG_UST; the not-found path emits the guidance error. Docs/configonly — no unit test applicable.
Whole series (validated end-to-end before stacking):
tracepoints, an LTTng session enabling
rocm_hip/rocm_hsa, workload run,trace read back with
babeltrace2— events present with the combined-eventschema (phase field, typed args + return).
lttng_curated_codegen.pyoutput; the libclang verifier confirms every curatedYAML signature matches the real HIP/HSA headers (CI:
lttng-curated-gates.yml).explicitly exempted (enforced against the built
.so).39×(1+3)), 0 discarded events, all rocprofiler-sdk conditions
record-count-verified.
Full series roadmap
Each category PR adds that category's APIs to the curated YAML, regenerates the
two provider headers, and migrates only that category's wrappers to the
combined-event form — so each is independently reviewable as "here are these APIs
and their typed tracepoint arguments."
Tracking
ISSUE ID : #8906