Skip to content

Fix CUDA watcher thread segfault at exit due to shared memory unmapped before thread stops - #548

Draft
yunwei37 with Copilot wants to merge 2 commits into
masterfrom
copilot/fix-cuda-agent-segfault
Draft

Fix CUDA watcher thread segfault at exit due to shared memory unmapped before thread stops#548
yunwei37 with Copilot wants to merge 2 commits into
masterfrom
copilot/fix-cuda-agent-segfault

Conversation

Copilot AI commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

The CUDA watcher thread (start_cuda_watcher_thread) polls ctx->cuda_shared_mem->flag1 in shared memory. On exit, __destruct_shm() (.fini_array, priority 65535) calls bpftime_destroy_global_shm() which unmaps the Boost IPC segment — including CommSharedMem — before the thread stops, causing a SIGSEGV. ~bpf_attach_ctx() never fires because bpf_attach_ctx lives in a union with an empty destructor, and the existing atexit handler's ordering relative to .fini_array is not guaranteed.

Changes

  • bpf_attach_ctx_cuda.cpp: Expose bpftime::bpftime_stop_cuda_watcher_thread() — a non-anonymous-namespace wrapper around the existing stop_cuda_watcher_thread_at_exit(). Idempotent; safe to call from both atexit and __destruct_shm.

  • bpftime_shm_internal.cpp: Call bpftime_stop_cuda_watcher_thread() in __destruct_shm() before bpftime_destroy_global_shm(), guaranteeing the thread is joined before the segment is unmapped regardless of atexit/fini_array ordering:

    static __attribute__((destructor(65535))) void __destruct_shm()
    {
        // ...
    #ifdef BPFTIME_ENABLE_CUDA_ATTACH
        bpftime::bpftime_stop_cuda_watcher_thread(); // NEW: join before unmap
    #endif
        bpftime_destroy_global_shm();
    }
  • bpf_attach_ctx.cpp: Guard cuda_ctx->... in ~bpf_attach_ctx() with a null check to prevent dereference on a partially-constructed object.

Original prompt

This section details on the original issue you should resolve

<issue_title>[BUG] CUDA agent segfault at exit caused by watcher thread</issue_title>
<issue_description>Issue description:

When running GPU workloads (e.g. benchmark/gpu/workload/vec_add.cu) with the bpftime CUDA agent attached, the process sometimes segfaults on exit.
The crash happens in the CUDA watcher thread, which polls ctx->cuda_shared_mem->flag1 in bpf_attach_ctx::start_cuda_watcher_thread().

Root Cause:

The agent process opens the global shared memory with bpftime_initialize_global_shm(shm_open_type::SHM_OPEN_ONLY).
In runtime/src/bpftime_shm_internal.cpp, the global destructor __destruct_shm() unconditionally calls bpftime_destroy_global_shm().

bpftime_shm::~bpftime_shm() unmaps the Boost shared memory segment and calls cudaHostUnregister(base_addr) for the whole segment, which includes the cuda::CommSharedMem holding flag1/flag2.

The CUDA watcher thread is started in bpf_attach_ctx::start_cuda_watcher_thread() and is detached; because bpf_attach_ctx is stored in a union (bpf_attach_ctx_holder) whose destructor is empty, ~bpf_attach_ctx() never runs at process exit, so cuda_watcher_should_stop is never set to true.</issue_description>

Comments on the Issue (you are @copilot in this section)


🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

Co-authored-by: yunwei37 <34985212+yunwei37@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix segfault issue in CUDA agent watcher thread Fix CUDA watcher thread segfault at exit due to shared memory unmapped before thread stops Feb 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] CUDA agent segfault at exit caused by watcher thread

2 participants