Skip per-tick sampling of profiler-internal threads

[eregon]

CpuAndWallTimeWorker and IdleSamplingHelper threads are now sampled only once per profiling period (via the on-serialize flush) instead of every 10ms tick, reducing profiling overhead.

What does this PR do?
^

Motivation:
Reduce profiler overhead

Change log entry

Yes, Changed: Profiling: Reduce profiler overhead by sampling profiler-internal threads only once per minute.

Additional Notes:

How to test the change?

[datadog-datadog-prod-us1-2]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

Test for memory leaks | test-memcheck     

Check Pull Request CI Status | all-jobs-are-green     

❄️ 1 New flaky test detected

Datadog::Profiling::Collectors::ThreadContext profiler-internal thread skipping still records one sample per profile period via serialize from rspec  

expected: 1
     got: 0

(compared using ==)

Failure/Error: expect(t1_samples.size).to eq(1)

  expected: 1
       got: 0

...

New test introduced in this PR is flaky.

View in Flaky Test Management

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 90.01% (-0.03%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ce3c80f | Docs | Datadog PR Page | Give us feedback!}

[eregon]

In the benchmarks/profiling_sample_overhead.rb benchmark we still have ~1000 samples for the profiling ovearhead (Datadog::Profiling::Sampling, sampling), even though we didn't sample any thread, but it still took CPU time (~40µs).
What should we do about those? Maybe we should group them by reporting period too to reduce the number of samples?

[eregon]

Before:

{duration: 10.02499,
 samples: 998,
 trigger_simulated_signal_delivery_attempts: 996,
 cpu_sampling_time_ns_total: 30466000,
 cpu_sampling_overhead: "0.30%",
 serialization_time_ns_total: 4696000,
 serialization_overhead: "0.05%",
 sampling_rate: 99.5512214974778,
 sample_every_n_ms: 10.045080160320643,
 inactive_thread_samples_skipped: 3983}

After:

{duration: 10.020218,
 samples: 998,
 trigger_simulated_signal_delivery_attempts: 997,
 cpu_sampling_time_ns_total: 24977000,
 cpu_sampling_overhead: "0.25%",
 serialization_time_ns_total: 2432000,
 serialization_overhead: "0.02%",
 sampling_rate: 99.59863148685987,
 sample_every_n_ms: 10.04029859719439,
 inactive_thread_samples_skipped: 2988,
 profiler_thread_samples_skipped: 1996}

cpu_sampling_overhead and serialization_overhead drop and all regular samples are skipped for the 2 profiler-internal threads.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b3d4ae0f42

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Avoid charging sampler sleep as overhead

When no_signals_workaround_enabled is true, run_sampling_trigger_loop calls grab_gvl_and_sample() from this same worker thread, making CpuAndWallTimeWorker the current_thread in thread_context_collector_sample. Because this line now marks that current thread as internal, the new skip path returns before updating its previous-sample timestamps, and the immediately following record_sampling_overhead(state, current_thread_context) consumes the whole interval since the last tick as profiler overhead; no-signal profiles will therefore attribute the worker's sleep/wait time to overhead instead of recording one internal-thread sample per period.

Useful? React with 👍 / 👎.

[eregon]

Alas, this was a nice and simple change but it won't be so simple anymore 😓

[eregon]

Addressed in 4909597 and e2c83f2.
It's much simpler if we are OK to attribute sampling overhead to the profiler-internal threads in the case they are the current thread for thread_context_collector_sample().

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-06-25 15:49:45

Comparing candidate commit e2c83f2 in PR branch dont-sample-profiler-internal-threads with baseline commit 75ccdd1 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'