feat: add multi-GPU support for CUDA attach by yunwei37 · Pull Request #559 · eunomia-bpf/bpftime

yunwei37 · 2026-03-11T14:53:37Z

Summary

Add gpu_device_manager that enumerates all CUDA devices at init, caches per-device SM architectures, and provides device-aware lookup APIs
Per-device PTX compilation, module loading, and patched kernel tracking for correct multi-GPU launch interception
Device-aware run_attach_entry_on_gpu() with new device_ordinal parameter (backward compatible, defaults to auto-detect)
Multi-device CUDAContext support in runtime with init_multi_gpu_contexts()
Fix cuCtxCreate calls for CUDA 13 compatibility (cuCtxCreate_v4 4-parameter signature)
Add gpu_device_manager unit tests
Add multi-GPU vector addition example (example/gpu/multi-gpu/)

Test plan

All 131 assertions in 23 test cases pass (including new gpu_device_manager tests)
Full project build succeeds with -DBPFTIME_ENABLE_CUDA_ATTACH=ON
Multi-GPU example verified on 8x NVIDIA B300 SXM6 AC (sm_103)
Single-GPU backward compatibility preserved (all new parameters default to device 0)
BPFTIME_SM_ARCH env var override works across all devices

🤖 Generated with Claude Code

Add gpu_device_manager that enumerates all CUDA devices at init time, caches per-device SM architectures, and provides device-aware APIs. Key changes: - Per-device SM arch detection and PTX compilation - Per-device module loading with separate module pools - Per-device patched kernel tracking for correct launch interception - Device-aware run_attach_entry_on_gpu() with device_ordinal parameter - Multi-device CUDAContext support in runtime - Fix cuCtxCreate calls for CUDA 13 compatibility - Add gpu_device_manager unit tests (tested on 8x NVIDIA B300 SXM6) - Add multi-GPU vector addition example Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… monitoring Add per-GPU block timing using gridDim.x (helper 508) to identify which GPU each block belongs to from inside the GPU. Update README to emphasize bpftime's unique value: zero-modification black-box monitoring, cross-GPU shared maps via UVA, and programmable GPU-internal probes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

CUDA 12.x uses cuCtxCreate_v2 (3 args) while CUDA 13+ uses cuCtxCreate_v4 (4 args). Add #if CUDA_VERSION >= 13000 guard. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Add helper 512 (bpf_get_device_ordinal): reads per-module deviceOrdinal constant set by bpftime during CUmodule loading. Each GPU gets its own CUmodule with a unique ordinal, providing reliable GPU identification from inside eBPF probes regardless of workload distribution. - Fix C1: device_count() now returns devices_.size() instead of the raw CUDA count, avoiding mismatch when cuDeviceGet fails for some devices. - Fix C2: run_attach_entry_on_gpu now uses per-device shared_mem_device_ptr instead of always using the primary device's pointer. - Fix start_ts collision: Use compound key (device_ordinal << 20 | block_id) to prevent cross-GPU block_id collisions in shared maps. - Update example to use device ordinal instead of gridDim.x for per-GPU stats, making it work with any workload distribution. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…cy (#560) * feat: skip compile/load of unmodified PTX in CUDA attach path Optimize GPU attach latency by only compiling and loading PTX files that were actually modified by the pass pipeline. Previously all extracted PTX files flowed through compile+load even when unmodified. On a llama.cpp 1B workload (RTX 5090, CUDA 12.9): - First cold fatbin: 27.9s → 5.6s (-80%) - PTX compile: 21.2s → 0.17s (-99.2%) - Fatbins 3..48 mean: 210ms → 123ms (-41.5%) Changes: - nv_attach_impl.cpp: rename should_add_trampoline to ptx_modified for clarity; propagate per-PTX modified flag from pass pipeline - nv_attach_fatbin_record.cpp: filter patched PTX to only modified entries before compile_ptxs() and module-load loop; add thread-safe cache access for ptx_pool and module_pool; add per-fatbin load_mutex; add timing instrumentation for extract/patch/compile/load stages - nv_attach_impl_frida_setup.cpp: add extract timing; downgrade expected "symbol not in patched module" messages to DEBUG - nv_attach_impl.hpp: add module_pool_mutex_ and ptx_pool_mutex_ - nv_attach_fatbin_record.hpp: add load_mutex for per-fatbin serialization - vm/compat/CMakeLists.txt: enable llvm-vm when CUDA attach is on Closes #552 * feat: add pass-internal early return for non-target PTX to reduce patch latency Add ptx_may_contain_target_kernel() fast-path check in ptxpass_core that skips heavy PTX parsing when the target kernel name is absent. Applied at both pass level (kprobe_entry, kretprobe, kprobe_memcapture) and framework level (hack_fatbin JSON serialization skip). On llama.cpp 1B (RTX 5090, CUDA 12.9): - Cold first fatbin patch: 5428ms -> 487ms (-91%) - Cold first fatbin total: 5605ms -> 661ms (-88%) - Combined with PR1: 27.9s -> 0.66s (-97.6%) Also hoists eBPF instruction word packing out of inner kernel loop and adds unit test for the new helper. * refactor: separate PTX from JSON in pass interface to eliminate serialization overhead Change process_input() ABI to receive PTX as raw pointer instead of JSON-encoded. Meta-only JSON now contains kernel name, eBPF instructions, and map symbols (~100 bytes vs 50-200KB per PTX). Key changes: - New process_input(ptx, ptx_len, meta_json, meta_len, out, out_len) - Cache key uses sha256(raw_ptx) + kernel + sha256(ebpf) instead of sha256(full_json) - Remove framework-level ptx_may_contain_target_kernel() pre-filter (no longer needed since pass calls are now cheap) - Fix 1GB per-call buffer allocation, size from PTX length instead - Update all 3 passes, unit tests, and README-passes.md On llama.cpp 1B (RTX 5090, CUDA 12.9): - Cold first fatbin: 661ms -> 275ms (-58%) - Cold patch: 487ms -> 106ms (-78%) - Combined from baseline: 27.9s -> 0.275s (-99.0%, 102x) * chore: revert unnecessary changes from optimization PR - Revert README-passes.md: docs-only change not required for the optimization - Revert ebpf_inst_words hoist: minor micro-optimization churn, not needed Keeps the diff minimal and focused on the actual latency optimization. * chore: revert unnecessary string_view and formatting changes Revert non-essential refactoring to minimize the PR diff: - Restore const std::string& signatures for validate_input, contains_entry_function, contains_ret_instruction, validate_ptx_version, find_kernel_body, pass_runtime_request_from_string - Remove ptx_owned workaround in find_kernel_body that was only needed for the string_view conversion - Restore original indentation for effective_module_pool ternary - Restore original formatting for lambda capture and ebpf_inst_words New function ptx_may_contain_target_kernel keeps string_view since it's a new addition, not a signature change. * chore: revert remaining unnecessary changes to minimize PR diff * Revert "chore: revert remaining unnecessary changes to minimize PR diff" This reverts commit a5bab96. * chore: revert non-essential comment, log message, and include changes - Restore original SPDLOG_WARN level and messages in frida_setup - Restore original comments in core.hpp - Remove unnecessary #include <cstddef> - Restore original validate_input declaration formatting * chore: simplify PR by inlining trivial helper functions Remove unnecessary abstractions that wrapped 1-2 line operations: - Inline runtime_request_ptx_view(), populate_runtime_request_ptx(), ptx_may_contain_target_kernel() at call sites - Inline build_patch_cache_key() and estimate_pass_output_buffer_size() - Restore NLOHMANN_DEFINE macros for RuntimeResponse (remove custom to_json/from_json that only omitted output_ptx for unmodified) - Remove ptx_may_contain_target_kernel from core.cpp (inlined in passes) - Update tests to match simplified serialization * chore: restore original variable names and remove unnecessary defensive code - Restore should_add_trampoline variable name (rename was unnecessary churn) - Restore original SPDLOG_DEBUG message text - Remove defensive empty-PTX error check (not required for optimization) * chore: change GPU attach timing logs from INFO to DEBUG --------- Co-authored-by: LinuxDev9002 <linuxdev8883@example.com>

Reconcile multi-GPU CUDA attach (#559/#560) with master's dynamic-attach (#542) and PTX-pass changes. Notable resolutions: - nv_attach_impl: keep the 6-arg process_input ABI (#560, passes PTX separately from the meta JSON) and call it from master's growing 1MiB->64MiB retry buffer loop; keep both the multi-GPU device-manager accessors and master's late-bootstrap API; take #559's find_patched_kernel_function(device_ordinal) plus master's record_patched_launch. - ptxpass_kprobe_entry: keep the meta_json null-check (6-arg) and master's save_strategy register-guard, using the 6-arg output_len. - bpf_attach_ctx: take master's epoch-retry init restructuring and insert init_multi_gpu_contexts(); add only the multi-GPU-specific members (cuda_device_contexts, init_multi_gpu_contexts, get_cuda_context_for_device) without re-declaring master's existing CUDA members. - Fix a stale reference left by the auto-merge: clear_patched_state_for_next_session() now clears patched_kernel_by_device_ (the per-device map) instead of the removed patched_kernel_by_name. - CMakeLists: build both new sources (nv_gpu_device_manager.cpp + nv_elf_introspect.cpp). Compiles cleanly under static analysis locally; needs GPU CI + author review to validate runtime behavior of the merged patching pipeline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR extends bpftime’s CUDA attach pipeline to support multi-GPU environments by introducing device enumeration/state tracking, per-device PTX/module handling, and device-aware attach execution. It also updates the PTX pass interface to avoid embedding full PTX text in JSON metadata and adds a new multi-GPU example plus unit tests.

Changes:

Add gpu_device_manager to enumerate CUDA devices, cache per-device SM arch/module pools, and expose device-aware lookups.
Make CUDA attach runtime/device paths multi-GPU aware (per-device patched-kernel tracking, device ordinal propagation into trampoline, device-aware run_attach_entry_on_gpu()).
Update PTX pass ABI to accept PTX text separately from metadata JSON; add tests and a multi-GPU example.

Reviewed changes

Copilot reviewed 29 out of 30 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
vm/compat/CMakeLists.txt	Ensures LLVM JIT support is built when CUDA attach is enabled.
runtime/src/handler/map_handler.cpp	Updates CUDA context creation path for CUDA 13 signature compatibility.
runtime/src/attach/bpf_attach_ctx.cpp	Initializes multi-GPU contexts during attach-context initialization.
runtime/src/attach/bpf_attach_ctx_cuda.cpp	Adds per-device CUDAContext creation + multi-GPU shared-mem pointer initialization.
runtime/include/bpf_attach_ctx.hpp	Extends CUDAContext with device ordinal + adds device-aware APIs/members.
example/gpu/multi-gpu/README.md	Documents the new multi-GPU monitoring example and how it works.
example/gpu/multi-gpu/multi_gpu_vec_add.cu	Adds an imbalanced multi-GPU CUDA workload used as a target binary.
example/gpu/multi-gpu/multi_gpu_probe.c	Adds a userspace libbpf loader/dashboard for the example probe.
example/gpu/multi-gpu/multi_gpu_probe.bpf.c	Adds the GPU-side eBPF probe (per-GPU ordinal + histograms/stats).
example/gpu/multi-gpu/Makefile	Adds build rules for the new example (probe + optional CUDA target).
attach/nv_attach_impl/trampoline/default_trampoline.cu	Adds a device-ordinal constant and helper 512 implementation.
attach/nv_attach_impl/trampoline_ptx.h	Mirrors trampoline PTX updates (deviceOrdinal const + helper 512).
attach/nv_attach_impl/test/test_ptxpass.cpp	Updates PTX pass tests for the new request format + legacy parsing.
attach/nv_attach_impl/test/test_gpu_device_manager.cpp	Adds unit tests for device enumeration and SM-arch override behavior.
attach/nv_attach_impl/test/CMakeLists.txt	Registers the new device-manager test in the nv_attach test target.
attach/nv_attach_impl/pass/ptxpass_kretprobe/main.cpp	Updates pass ABI to accept separate PTX text + metadata JSON.
attach/nv_attach_impl/pass/ptxpass_kprobe_memcapture/main.cpp	Updates pass ABI to accept separate PTX text + metadata JSON.
attach/nv_attach_impl/pass/ptxpass_kprobe_entry/main.cpp	Updates pass ABI to accept separate PTX text + metadata JSON.
attach/nv_attach_impl/pass/ptxpass_core/include/ptxpass/core.hpp	Moves PTX payload to `RuntimeRequest.full_ptx` + improves JSON emit defaults/back-compat parsing.
attach/nv_attach_impl/nv_gpu_device_manager.hpp	Declares multi-GPU device info/manager types.
attach/nv_attach_impl/nv_gpu_device_manager.cpp	Implements device enumeration + per-device SM arch/module pool setup.
attach/nv_attach_impl/nv_attach_utils.hpp	Adds per-device SM arch query API.
attach/nv_attach_impl/nv_attach_utils.cpp	Implements per-device SM arch query via CUDA driver attributes.
attach/nv_attach_impl/nv_attach_impl.hpp	Adds device-aware APIs/state (device manager, per-device patched kernel map, new pass ABI).
attach/nv_attach_impl/nv_attach_impl.cpp	Implements per-device attach execution, module const initialization, and patched-kernel tracking.
attach/nv_attach_impl/nv_attach_impl_frida_setup.cpp	Records patched kernels with the current device ordinal; adds timing diagnostics.
attach/nv_attach_impl/nv_attach_fatbin_record.hpp	Adds per-device loading state and thread-safety primitives for fatbin PTX load/compile.
attach/nv_attach_impl/nv_attach_fatbin_record.cpp	Implements per-device patch/compile/load with caching and timing diagnostics.
attach/nv_attach_impl/CMakeLists.txt	Adds `nv_gpu_device_manager.cpp` to the nv_attach_impl library build.
.claude/scheduled_tasks.lock	Adds a Claude tool lock file artifact.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

 		shm_holder.global_shared_memory.set_enable_mock(false);
 		if (!device) {
 			cuDeviceGet(&device, 0);
+#if CUDA_VERSION >= 13000
+			cuCtxCreate(&context, nullptr, 0, device);
+#else
 			cuCtxCreate(&context, 0, device);
+#endif


 		shm_holder.global_shared_memory.set_enable_mock(false);
 		if (!device) {
 			cuDeviceGet(&device, 0);
+#if CUDA_VERSION >= 13000
+			cuCtxCreate(&context, nullptr, 0, device);
+#else
 			cuCtxCreate(&context, 0, device);
+#endif


+	SPDLOG_INFO("Initializing multi-GPU contexts for {} devices",
+		    device_count);
+
+	// Set device 0’s pointer from the already-created primary context
+	if (cuda_ctx) {
+		dev_manager.get_device(0).shared_mem_device_ptr =
+			cuda_ctx->cuda_shared_mem_device_pointer;
+	}
+
+	// For additional devices, the CommSharedMem is in UVA (unified virtual
+	// addressing) host memory registered with cudaHostRegister, so it is
+	// accessible from all devices. We just need to record the device
+	// pointer for each.
+	for (int i = 1; i < device_count; i++) {
+		SPDLOG_INFO("Setting up shared mem pointer for device {}", i);
+		// The UVA device pointer is the same for all devices when using
+		// cudaHostGetDevicePointer on registered host memory
+		dev_manager.get_device(i).shared_mem_device_ptr =
+			cuda_ctx->cuda_shared_mem_device_pointer;
+	}


 	std::optional<attach::nv_attach_impl *> find_nv_attach_impl() const;
+	/// Get the CUDAContext for a specific device, or nullptr if not
+	/// available
+	cuda::CUDAContext *get_cuda_context_for_device(int device_ordinal);


+void gpu_device_manager::initialize()
+{
+	CUresult err = cuInit(0);
+	if (err != CUDA_SUCCESS) {
+		SPDLOG_WARN(
+			"Failed to initialize CUDA driver ({}), assuming 0 devices",
+			(int)err);
+		count_ = 0;
+		return;


+	// Resolve device ordinal: -1 means use current context or default
+	if (device_ordinal < 0) {
+		device_ordinal = get_current_device_ordinal();
+	}
+
+	// Get SM architecture for the target device
+	std::string sm_arch;
+	if (device_manager_.device_count() > 0) {
+		sm_arch = device_manager_.get_device(device_ordinal).sm_arch;
+	} else {
+		sm_arch = get_gpu_sm_arch();
+	}


+#include <signal.h>
+#include <stdio.h>
+#include <time.h>
+#include <stdint.h>
+#include <sys/resource.h>


+#include "catch2/catch_test_macros.hpp"
+#include "nv_gpu_device_manager.hpp"
+#include <cuda.h>
+#include <spdlog/spdlog.h>
+


@@ -0,0 +1 @@
+{"sessionId":"ee8581c2-9462-42fc-b87d-4a2733ae93ec","pid":96985,"procStart":"40945278","acquiredAt":1779651162615}


+	std::lock_guard<std::mutex> load_guard(load_mutex);
+	// Skip if already loaded for this device
+	if (devices_loaded.count(device_ordinal) > 0)
 		return;


yunwei37 · 2026-07-16T23:02:58Z

CI triage update (2026-07-16): the PR status still points at an older failing GPU check for mem_trace (test-gpu-examples). Earlier artifact-download symptoms looked transient, but the currently reported failure is on an old GPU run and the branch is behind/conflicting. I triggered a fresh workflow run where possible; without a current head run and with the GPU self-hosted runner instability seen on other PRs, I am not making speculative code changes on this branch.

yunwei37 · 2026-07-16T23:03:07Z

CI triage update:

The observed failure was an artifact/download mismatch: the downstream job could not find build/attach/nv_attach_impl from the downloaded artifact, while the matching producer path succeeded in another job.
I do not see evidence in the available logs that this is caused by the PR source code itself.
This PR is now significantly behind master and has conflicts, so a clean rerun should happen after rebasing/resolving conflicts.

Action needed: rebase/resolve conflicts, then rerun the artifact-producing workflow. No code change pushed to this PR.

pull-request-size Bot added the size/XXL label Mar 11, 2026

yusheng and others added 3 commits March 11, 2026 15:14

fix: add CUDA version guard for cuCtxCreate in nv_attach_impl

79e15d0

CUDA 12.x uses cuCtxCreate_v2 (3 args) while CUDA 13+ uses cuCtxCreate_v4 (4 args). Add #if CUDA_VERSION >= 13000 guard. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

yunwei37 requested review from Officeyutong and Sy0307 March 18, 2026 17:14

github-actions Bot mentioned this pull request Mar 23, 2026

Weekly Org Report (2026-03-09..2026-03-15) eunomia-bpf/eunomia.dev#80

Open

github-actions Bot mentioned this pull request Apr 1, 2026

Monthly Org Report (2026-03-01..2026-03-31) eunomia-bpf/eunomia.dev#82

Open

Copilot AI review requested due to automatic review settings May 24, 2026 23:02

Copilot started reviewing on behalf of yunwei37 May 24, 2026 23:03 View session

Copilot AI reviewed May 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: add multi-GPU support for CUDA attach - #559

feat: add multi-GPU support for CUDA attach#559
yunwei37 wants to merge 6 commits into
masterfrom
feat/multi-gpu-attach

yunwei37 commented Mar 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

yunwei37 commented Jul 16, 2026

Uh oh!

yunwei37 commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1 @@
		{"sessionId":"ee8581c2-9462-42fc-b87d-4a2733ae93ec","pid":96985,"procStart":"40945278","acquiredAt":1779651162615} No newline at end of file

Uh oh!

Uh oh!

Conversation

yunwei37 commented Mar 11, 2026

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

yunwei37 commented Jul 16, 2026

Uh oh!

yunwei37 commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants