feat(otlp): added tls server config option for http and grpc by lucastemb · Pull Request #2220 · DataDog/saluki

lucastemb · 2026-07-28T16:38:19Z

Summary

Adds TLS support for both OTLP HTTP and gRPC servers.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

How did you test this PR?

Unit tests

References

Closes: Add a reusable TLS config for the OTLP source shared between gRPC and HTTP #2014

pr-commenter · 2026-07-28T17:00:00Z

Binary Size Analysis (Agent Data Plane)

Baseline: 28b8637 · Comparison: e4ef0f4 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 40.91 MiB (baseline) vs 41.65 MiB (comparison)
Size Change: +757.30 KiB (+1.81%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
`core`	+193.79 KiB	17288
`tokio`	+180.18 KiB	5315
`hyper`	+95.57 KiB	621
`saluki_components::common::datadog`	+50.31 KiB	591
`h2`	+44.80 KiB	1037
`[sections]`	+37.80 KiB	10
`hyper_util`	-31.21 KiB	205
`saluki_common::resource_tracking::groups`	+29.73 KiB	84
`serde_core`	+29.63 KiB	1210
`serde_json`	+27.11 KiB	536
`agent_data_plane_config_system::system::translate`	+25.22 KiB	1
`anon.96d806bad0362156919432e87b84ba85.154.llvm.17940444502793867768`	+17.27 KiB	1
`anon.90ffda7748c861fe2f6f03211060ab76.488.llvm.2966725881188392761`	-16.92 KiB	1
`serde`	-16.30 KiB	61
`prost`	+15.34 KiB	437
`datadog_agent_config::generated::witness`	-14.84 KiB	1
`std`	+12.66 KiB	777
`anon.23dfa8ebac1b10fc2833d8517ecb12b6.596.llvm.12314602514886898010`	+12.19 KiB	1
`anon.90ffda7748c861fe2f6f03211060ab76.639.llvm.2966725881188392761`	-12.02 KiB	1
`saluki_components::transforms::dogstatsd_mapper`	+11.39 KiB	14

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +2.7%  +746Ki  +2.1%  +453Ki    [61316 Others]
  [NEW] +60.6Ki  [NEW] +60.4Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::hf62768f370e7cd6b
  [NEW] +42.9Ki  [NEW] +42.8Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h8e3b77267deeca7d
  [NEW] +41.8Ki  [NEW] +41.6Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h7e8668382102cacc
  [NEW] +40.5Ki  [NEW] +40.2Ki    _<saluki_components::common::datadog::config::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::config::ForwarderConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h263d8230b71bdad9
  [NEW] +36.9Ki  [NEW] +36.6Ki    _<saluki_components::common::datadog::obfuscation::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::obfuscation::ObfuscationConfig>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::he637993877827eea
  [NEW] +35.0Ki  [NEW] +34.9Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::hdf178f6f32c70dd5
  [NEW] +33.2Ki  [NEW] +33.0Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::hc1e2150eb6822b04
  [NEW] +32.0Ki  [NEW] +31.8Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::ha68b45d9fb4741ec
  [NEW] +30.4Ki  [NEW] +30.2Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hc515be606bb7300f
  [NEW] +28.3Ki  [NEW] +28.1Ki    saluki_env::workload::collectors::containerd::NamespaceWatcher::build_initial_metadata_operations::_{{closure}}::hee96a57ba9a2961a
  [DEL] -28.7Ki  [DEL] -28.6Ki    saluki_env::workload::collectors::containerd::NamespaceWatcher::build_initial_metadata_operations::_{{closure}}::hf8693ed9b0a23feb
  [DEL] -29.9Ki  [DEL] -29.7Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h14f8748dfe79d0fb
  [DEL] -31.4Ki  [DEL] -31.3Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::hd4c2b27009bc6445
  [DEL] -32.5Ki  [DEL] -32.3Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::h97868b07fa692b3b
  [DEL] -32.9Ki  [DEL] -32.7Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::hf64ce4d9ba36780e
  [DEL] -33.7Ki  [DEL] -33.5Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::hc1b5025fade25a7b
  [DEL] -36.9Ki  [DEL] -36.6Ki    _<saluki_components::common::datadog::obfuscation::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::obfuscation::ObfuscationConfig>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h04547875805b5e90
  [DEL] -42.5Ki  [DEL] -42.3Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::hfb0d81d4de2ee6dc
  [DEL] -43.2Ki  [DEL] -43.1Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::ha5ca38efb1a9f33f
  [DEL] -59.4Ki  [DEL] -59.2Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h6f46e768e49cd654
  +1.8%  +757Ki  +1.3%  +463Ki    TOTAL

pr-commenter · 2026-07-28T17:20:39Z

Regression Detector (Agent Data Plane)

Run ID: 2c32a934-71b4-453e-afd3-2da291a9a80d
Baseline: 28b86379 · Comparison: e4ef0f49 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (3)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_idle	memory	⚪ +0.94	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +0.43	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.39	metrics profiles logs

Bounds Checks: ✅ Passed (3)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 46 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 67.6 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 31.7 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

datadog-official

The TLS implementation is incomplete for production receiver modes: proxy/relay mode still accepts plaintext, while gRPC TLS neither advertises HTTP/2 nor isolates handshakes from the accept loop. These paths can make configured TLS ineffective or prevent legitimate clients from connecting, so the implementation needs targeted fixes before shipping.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 93e1a74 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93e1a747d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-28T18:31:43Z

+#[derive(Clone, Debug, Default, Deserialize, PartialEq, Eq, Serialize)]
+pub struct TlsConfig {


Default omitted TLS fields during deserialization

When a user supplies the normal TLS configuration containing only cert_file and key_file, deserialization fails because every String in TlsConfig is required; the #[serde(default)] attributes on the surrounding protocol structs do not apply inside a present tls object. Add serde defaults to this struct or its fields so optional ca_file and client_ca_file values remain empty and basic server TLS can start.

Useful? React with 👍 / 👎.

Fixed in e4ef0f4

chatgpt-codex-connector · 2026-07-28T18:31:43Z

+        match self.grpc_tls_config {
+            Some(tls_config) => {
+                let tls_acceptor = TlsAcceptor::from(Arc::new(tls_config));


Advertise h2 on the gRPC TLS listener

When gRPC TLS is enabled, build_server_config leaves alpn_protocols empty and this custom acceptor uses that configuration unchanged. Standard tonic gRPC clients require ALPN negotiation to select h2 and reject the resulting connection with H2NotNegotiated; tonic's built-in server TLS path explicitly adds h2. Add the h2 ALPN protocol before constructing this acceptor.

Useful? React with 👍 / 👎.

Fixed in e4ef0f4

chatgpt-codex-connector · 2026-07-28T18:31:43Z

+                            match listener.accept().await {
+                                Ok((stream, _)) => match acceptor.accept(stream).await {
+                                    Ok(stream) => return Some((Ok::<_, io::Error>(stream), (listener, acceptor))),


Avoid serializing gRPC TLS handshakes

When any client opens the TLS-enabled gRPC port and does not complete its handshake, this await blocks the unfold stream before it can accept another socket, with no timeout. A single stalled or malicious connection can therefore prevent every subsequent gRPC client from connecting; accept and process handshakes concurrently, or at minimum enforce a handshake timeout.

Useful? React with 👍 / 👎.

Fixed in e4ef0f4

chatgpt-codex-connector · 2026-07-28T18:31:43Z

+            http_tls_config: None,
+            grpc_tls_config: None,


Apply TLS configuration in OTLP proxy mode

When OTLP proxy mode is enabled, bin/agent-data-plane/src/cli/run.rs:810-845 constructs OtlpRelayConfiguration rather than the native source, but the relay calls OtlpServerBuilder::new without with_tls_configs. Because these newly added builder defaults are plaintext, the configured HTTP and gRPC TLS keys are silently ignored in proxy mode and both receiver ports remain unencrypted; build and pass the receiver TLS configurations from the relay path as well.

Useful? React with 👍 / 👎.

Deferred since future of proxy mode uncertain at the time of writing.

thieman · 2026-07-28T20:03:10Z

+| `otlp_allow_context_heap_allocs`                                | Allow heap allocations for OTLP contexts            |                |
+| `otlp_cached_contexts_limit`                                    | Max cached OTLP metric contexts                     |                |
+| `otlp_cached_tagsets_limit`                                     | Max cached OTLP tagsets                             |                |
+| `otlp_config.receiver.protocols.grpc.tls.ca_file`               | OTLP gRPC TLS compatibility certificate authorities |                |


Is the goal to fully implement the options from the OTEL side? Seems like we're missing quite a bit if so? https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md

I believe so. I scoped this PR to the options that are present here, but I am confused about what the authoritative and comprehensive list is.

thieman · 2026-07-28T20:12:09Z

+            None => {
+                let grpc_incoming = tonic::transport::server::TcpIncoming::from(grpc_listener);
+                thread_pool_handle
+                    .spawn_traced_named("otlp-grpc-server", grpc_server.serve_with_incoming(grpc_incoming));


Nit: can this thread_pool_handle statement be out of the match? Seems identical in both cases?

thieman · 2026-07-28T20:13:54Z

+        .clone_key();
+
+    // The Collector loads `ca_file` into `tls.Config.RootCAs`. A pure inbound receiver does not consult those roots,
+    // but loading and validating the file preserves the Collector's configuration behavior.


We don't actually do anything meaningful with the ca_file? What's the point, then? Per the OTEL docs, sounds like we should be using this to validate client certificates?

Yes. As far as I can tell, the Core Agent behaves the same as well. It's effectively a no-op in both since the receiver is a server accepting inbound connections and that config option is only consulted when its a client validating a server.

feat(otlp): added tls server config option for http and grpc

93e1a74

dd-octo-sts Bot added area/components Sources, transforms, and destinations. area/docs Reference documentation. source/otlp OTLP source. relay/otlp labels Jul 28, 2026

This comment has been minimized.

Sign in to view

lucastemb marked this pull request as ready for review July 28, 2026 18:26

lucastemb requested a review from a team as a code owner July 28, 2026 18:26

datadog-official Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread lib/saluki-components/src/common/otlp/mod.rs

Comment thread lib/saluki-components/src/common/otlp/mod.rs Outdated

Comment thread lib/saluki-components/src/common/otlp/mod.rs Outdated

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

fix(otlp): added integration test and addressed codex comments

e4ef0f4

dd-octo-sts Bot added the area/test All things testing: unit/integration, correctness, SMP regression, etc. label Jul 28, 2026

thieman reviewed Jul 28, 2026

View reviewed changes

		#[derive(Clone, Debug, Default, Deserialize, PartialEq, Eq, Serialize)]
		pub struct TlsConfig {

Uh oh!

Conversation

lucastemb commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type

How did you test this PR?

References

Uh oh!

This comment has been minimized.

pr-commenter Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis (Agent Data Plane)

✅ Binary size difference within threshold

Uh oh!

pr-commenter Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ✅ No significant changes detected

Uh oh!

datadog-official Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thieman Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lucastemb commented Jul 28, 2026 •

edited

Loading

pr-commenter Bot commented Jul 28, 2026 •

edited

Loading

pr-commenter Bot commented Jul 28, 2026 •

edited

Loading

thieman Jul 28, 2026 •

edited

Loading