Support full OTel profiler malloc/free uprobes

[yunwei37]

Summary

• Preserve BPF program type and perf-event bpf_cookie metadata for daemon-created cilium/ebpf perf-event uprobe links.
• Add kernel-user map support needed by the upstream OTel generic probe shape (PROG_ARRAY, LRU_HASH, LPM_TRIE) and ignore kernel-owned perf sampling fds when the daemon is only mirroring bpftime-supported uprobes.
• Focus example/otel-ebpf-profiler on the complete upstream opentelemetry-ebpf-profiler malloc/free workflow. The local minimal BPF counter/loader was removed.
• Add a full OTel stack-collection benchmark that runs the upstream OTel collector, validates exported victim profile data, and reports victim throughput for native-kernel OTel profiling and daemon-mirror OTel profiling.
• Clarify that upstream OTel currently runs through daemon mirror compatibility mode: the static Go collector issues raw syscalls, so LD_PRELOAD=libbpftime-syscall-server.so cannot intercept it; exported profiles come from the native kernel OTel pipeline while bpftime mirrors handlers into the agent-preloaded target.
• Fix runtime issues found by this workflow: user-mode pt_regs fields for x86_64 uprobes, stable per-CPU selection across one userspace BPF execution/tail-call chain, fd-range checks for high kernel object IDs, larger example daemon fd table, normal agent link detach, and prog-array lookup fd leaks.

Validation

• bash -n example/otel-ebpf-profiler/scripts/common.sh example/otel-ebpf-profiler/scripts/run-full-otel-profiler.sh example/otel-ebpf-profiler/scripts/build-upstream-otel.sh example/otel-ebpf-profiler/scripts/benchmark.sh
• cmake --build build --target bpftime-agent bpftime_daemon bpftime-syscall-server -j$(nproc)
• make -C example/otel-ebpf-profiler clean all
• Full upstream OTel daemon mirror smoke:
  OTEL_EBPF_PROFILER_DIR=/tmp/opentelemetry-ebpf-profiler RUN_SECONDS=5 OTEL_COLLECTOR_STARTUP_DELAY=5 OTEL_FLUSH_DELAY=6 example/otel-ebpf-profiler/scripts/run-full-otel-profiler.sh
• Full OTel stack-collection benchmark smoke:
  OTEL_EBPF_PROFILER_DIR=/tmp/opentelemetry-ebpf-profiler RUN_SECONDS=6 REPEATS=1 OTEL_COLLECTOR_STARTUP_DELAY=5 OTEL_FLUSH_DELAY=6 example/otel-ebpf-profiler/scripts/benchmark.sh
• Full OTel stack-collection benchmark:
  OTEL_EBPF_PROFILER_DIR=/tmp/opentelemetry-ebpf-profiler RUN_SECONDS=10 REPEATS=3 OTEL_COLLECTOR_STARTUP_DELAY=5 OTEL_FLUSH_DELAY=6 example/otel-ebpf-profiler/scripts/benchmark.sh
  • baseline: 160.19M malloc/free iterations/s avg
  • otel-kernel-stack-collector: 295.42K iterations/s avg
  • otel-daemon-mirror-stack-collector: 154.98M iterations/s avg
  • each profiler run checked collector logs for exported victim profile data
• git diff --check

Performance note

The benchmark now measures only the complete upstream OTel profiler stack-collection path. It does not use a local BPF counter or minimal cilium loader.

On this host, native-kernel OTel profiling of malloc/free uprobes is very expensive for the tight malloc/free loop: about 295K iterations/s versus 160M iterations/s baseline. In daemon mirror mode, the victim with libbpftime-agent.so measured about 155M iterations/s while the upstream collector still exported victim profiles. This result should be interpreted as daemon-mirror compatibility behavior for the current static Go OTel collector, not as proof that upstream OTel's collector consumes bpftime userspace ringbuf output.

Review

• Independent AI review (Claude CLI) found one blocking fd-leak issue in the kernel-user prog-array syscall lookup path.
• Fixed in e5498f1 and re-reviewed; the reviewer reported no remaining blocking issues.
• Follow-up Claude review for removing misleading benchmark modes found no blocking issues: Support full OTel profiler malloc/free uprobes #586 (comment)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[yunwei37]

Independent AI review (Claude CLI) after e5498f1: previous blocker in prog_array_kernel_user_impl::elem_lookup is resolved. Syscall lookups now use elem_lookup_userspace() to return the kernel prog id without opening a prog fd; helper/tail-call lookups still use the fd path. No remaining blocking issues were reported. Residual non-blocking risks are integration-test coverage and broader kernel-user map coverage.

[yunwei37]

CI monitor update for e5498f1: all non-GPU checks have completed without failures (140 success; deploy skipped; CodeQL neutral). The only checks still not complete are the two GPU/self-hosted jobs, build-bpftime-gpu and test-ptxpass, both queued since 2026-06-21 18:48 UTC with no steps started, indicating runner availability rather than a code failure. GitHub reports no required checks for this branch.

[yunwei37]

Independent Claude review for 3833ace (Remove misleading OTel benchmark modes): no blocking issues found.

Checked scope:

• benchmark.sh no longer has RUN_FULL_OTEL, daemon-mirror-full-otel-collector, or kernel-full-otel-collector benchmark modes.
• README now states benchmark intentionally excludes the upstream static Go OTel collector and points to run-full-otel-profiler.sh only as a daemon-mirror smoke test.
• otelcol-malloc-free.yaml comment now says it is not used by the benchmark.
• No dangling references found; the remaining benchmark and smoke script are self-contained.

[yunwei37]

Independent Claude review for a45fec3 (Focus OTel example on full profiler stacks): no blocking issues found.

Reviewed scope:

• local BPF counter/loader files are removed (main.go, go.mod, go.sum, bpf/malloc_free.bpf.c),
• example now uses otelcol-ebpf-profiler as the only profiler path,
• benchmark validates that OTel debug exporter emits Profiles containing the victim process,
• benchmark reports completed malloc/free iterations and throughput.

Non-blocking note from review: the benchmark validates victim profile export, not specific resolved malloc/free frame symbols, which is intentional because symbol resolution can vary across hosts.

[yunwei37]

CI triage update (2026-07-16): the current blockers are build-bpftime-gpu and test-ptxpass, both queued on [self-hosted, Linux, X64, gpu] and previously cancelled after waiting. I do not see a PR-local failure in the accessible logs; this is blocked on GPU self-hosted runner availability.

[yunwei37]

CI triage update:

• The remaining failing/blocking checks are build-bpftime-gpu and test-ptxpass.
• Both jobs target the self-hosted [Linux, X64, gpu] runner and are stuck in the same queued/cancelled pattern seen on fix(runtime): handle shared memory exhaustion #597/fix(runtime): isolate maps with colliding public names #598 and the new feat(runtime): support bpf_get_func_ip for uprobes #601 validation run.
• Available logs do not show a PR-specific code failure.

Action needed: restore or free the GPU self-hosted runner, then rerun the GPU/PTX workflows. No code change pushed to this PR.