[codex] fix gpu example attach races

[yunwei37]

Summary

Fix GPU example startup/teardown races in the existing runtime and syscall-server path. This PR keeps the change scoped to runtime/shared-memory coordination; it does not add the PTX pass subprocess runner from #583.

What changed:

• Move syscall-server CUDA initialization out of syscall_context construction and run it after global shared memory is created.
• Guard CUDA initialization with an atomic initializing_cuda flag so intercepted syscalls fall back to the real libc/syscall path during CUDA startup.
• Retry cuda_comm_shared_mem lookup for SHM_OPEN_ONLY clients before registering CUDA host memory, covering the window where the shared segment exists but the CUDA communication object is not visible yet.
• Stop and clear the CUDA watcher thread state before bpf_attach_ctx teardown, including the global atexit pointer/stop flag, to avoid stale watcher-thread state at process exit.

Why

Several GPU examples exposed races around startup and process teardown:

• The syscall server could initialize CUDA before shared memory/CUDA communication memory was ready.
• Clients could attach to the shared memory segment before cuda_comm_shared_mem had been constructed.
• CUDA initialization itself could trigger intercepted syscalls and recurse into bpftime startup/mocking.
• Direct-run style examples could leave stale CUDA watcher-thread globals during teardown.

Together these showed up as flaky GPU example startup, dynamic attach failures, or exit-time aborts.

Scope

This PR changes only runtime/syscall-server/shared-memory code:

• runtime/syscall-server/*
• runtime/src/bpftime_shm_internal.cpp
• runtime/src/attach/bpf_attach_ctx*.cpp
• runtime/include/bpf_attach_ctx.hpp

The PTX pass isolation runner is intentionally split out into #583 and is not required for this PR to work.

Validation

• CI is green for [codex] fix gpu example attach races #582, including runtime, attach, tools, integrated examples, PTX pass executable checks, and GPU example workflow.
• GPU examples passed in CI, including atomizer, kernel_trace, gpu_shared_map, threadscheduling, threadscheduling_dynamic_hook, cudagraph, threadhist-gpu-kernel-shared-map, directly_run_on_gpu, kernelretsnoop, threadhist, cuda-counter, launchlate, launchlate-kernel-gpu-shared-map, mem_trace, host_map_test, and cutlass.
• Non-GPU integrated examples also pass in CI. Local no-GPU smoke covered malloc, minimal, queue_demo, stack_demo, bloom_filter_demo, get_stack_id_example, and lpm_trie_demo.

[Copilot]

Pull request overview

This PR fixes several startup/teardown race conditions in the CUDA/GPU attach path and makes PTX pass execution more robust by running pass logic in a dedicated bpftime_ptxpass_runner subprocess (with LD_PRELOAD/LD_AUDIT cleared).

Changes:

• Move syscall-server CUDA initialization to occur after global shared memory setup, and add retries for cuda_comm_shared_mem discovery in SHM_OPEN_ONLY clients.
• Harden CUDA watcher thread teardown by clearing global watcher pointers before attach context teardown.
• Run PTX pass print_config/process_input via a new bpftime_ptxpass_runner executable instead of in-process dlmopen/dlsym.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
runtime/syscall-server/syscall_server_utils.cpp	Defers CUDA init until after global shm init in syscall-server startup.
runtime/syscall-server/syscall_context.hpp	Adds initialize_cuda() API under BPFTIME_ENABLE_CUDA_ATTACH.
runtime/syscall-server/syscall_context.cpp	Moves CUDA init logic into initialize_cuda() with RAII flag reset.
runtime/src/bpftime_shm_internal.cpp	Adds retry loop for cuda_comm_shared_mem lookup; guards CUDA host registration.
runtime/src/attach/bpf_attach_ctx.cpp	Uses centralized CUDA watcher stop routine in destructor.
runtime/src/attach/bpf_attach_ctx_cuda.cpp	Implements stop_cuda_watcher_thread() that clears global watcher state before join.
runtime/include/bpf_attach_ctx.hpp	Declares stop_cuda_watcher_thread() under CUDA attach build.
attach/nv_attach_impl/pass/ptxpass_runner.cpp	New subprocess runner that dlopens a pass library and isolates stdout for JSON output.
attach/nv_attach_impl/nv_attach_impl.hpp	Removes in-process pass function pointers/handles from pass config struct.
attach/nv_attach_impl/nv_attach_impl.cpp	Uses subprocess runner for pass config/process; removes boost::asio thread-pool patching.
attach/nv_attach_impl/CMakeLists.txt	Builds bpftime_ptxpass_runner and exports runner path via generated header.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.