Skip to content

feat(otlp): added support for datadog_metric.as_type=rate for sums - #2201

Open
lucastemb wants to merge 4 commits into
mainfrom
lt/rate-otlp-delta
Open

feat(otlp): added support for datadog_metric.as_type=rate for sums#2201
lucastemb wants to merge 4 commits into
mainfrom
lt/rate-otlp-delta

Conversation

@lucastemb

@lucastemb lucastemb commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

When a metric is emitted with the attribute as_type=rate, we should strive to emit delta Sums as a Rate.

as_type=rate will be treated as case insensitive

Additional plumbing is added to make additional metric types such as Count and Gauge no-ops while making unknown types emit an error.

Lastly, after investigating #2076, we discovered that with_delta_interval sits behind a feature gate with the default false with no way to toggle it to true, meaning, that the duration will stay zero since we have no way to compute the interval.

Consequently, we will emit a warning the first time that a metric is observed notifying no delta interval is available.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

Unit tests.

The issue calls for correctness tests, but it should be deferred and opened as a subsequent issue once we do more work around interval-inference in the future.

References

@dd-octo-sts dd-octo-sts Bot added area/components Sources, transforms, and destinations. source/otlp OTLP source. labels Jul 23, 2026
@pr-commenter

pr-commenter Bot commented Jul 23, 2026

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: 36f7279 · Comparison: 5636c9e · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.09 MiB (baseline) vs 41.00 MiB (comparison)
Size Change: -96.94 KiB (-0.23%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
core -97.26 KiB 1517
figment -49.00 KiB 175
serde +38.01 KiB 53
hyper +30.83 KiB 51
saluki_components::common::datadog -29.23 KiB 88
hyper_util -22.85 KiB 15
axum +18.82 KiB 47
saluki_components::sources::dogstatsd +14.89 KiB 38
tracing +12.44 KiB 26
&mut serde_json -10.21 KiB 26
anyhow -9.70 KiB 246
std -9.64 KiB 74
quick_cache -9.26 KiB 27
prost +8.92 KiB 76
saluki_components::transforms::trace_obfuscation +8.56 KiB 16
otlp_protos::otlp_include::opentelemetry -8.14 KiB 27
saluki_common::cache::Cache<K,V,W,H> +7.80 KiB 8
alloc +7.79 KiB 298
saluki_core::pooling::elastic +7.62 KiB 5
tokio -7.62 KiB 308
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +29.6Ki  [NEW] +29.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h5bcc4ba05a5c79a8
  +283% +17.0Ki  +288% +17.0Ki    h2::proto::connection::DynConnection<B>::recv_frame::hb1d1642e492b62c5
  [NEW] +15.5Ki  [NEW] +15.4Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::map_to_dd_format::h8f13bd45746d8b5b
  [NEW] +13.9Ki  [NEW] +13.7Ki    saluki_components::common::datadog::apm::_::_<impl serde_core::de::Deserialize for saluki_components::common::datadog::apm::ApmConfiguration>::deserialize::h8279d5536d30f1e2
  [NEW] +13.0Ki  [NEW] +12.8Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h5711d2fa74aaa67f
  [NEW] +12.2Ki  [NEW] +12.1Ki    saluki_components::sources::dogstatsd::replay::writer::run_capture_loop::he083f64fbd1fc69a
  +248% +12.0Ki  +252% +12.0Ki    prost::message::Message::encode::h04108a619fa4f4e4
 +21e2% +11.0Ki +35e2% +11.0Ki    _<saluki_components::transforms::trace_obfuscation::TraceObfuscation as saluki_core::components::transforms::SynchronousTransform>::transform_buffer::h9b9b66a17d4a7470
  [NEW] +10.5Ki  [NEW] +10.4Ki    _<hyper::proto::h2::server::Server<T,S,B,E> as core::future::future::Future>::poll::h328dd619d20d0cf7
  [NEW] +10.2Ki  [NEW] +10.0Ki    _<hyper::proto::h2::server::Server<T,S,B,E> as core::future::future::Future>::poll::hd437171930e8aa63
 -83.2% -10.4Ki -84.3% -10.4Ki    _<hyper_util::server::conn::auto::Connection<I,S,E> as core::future::future::Future>::poll::h79d6ebf5a295e073
 -92.6% -10.8Ki -93.6% -10.8Ki    saluki_components::sources::otlp::metrics::cache::PointsCache::from_config::h404f5cb7a28885c3
 -98.1% -11.9Ki -98.9% -11.9Ki    std::sys::backtrace::__rust_begin_short_backtrace::h636265569ecef8fc
  [DEL] -13.3Ki  [DEL] -13.1Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hc2a432f2009d0c17
 -28.2% -14.1Ki -28.3% -14.1Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::hf64ce4d9ba36780e
  [DEL] -14.6Ki  [DEL] -14.5Ki    figment::figment::Figment::extract::h1cc98fe0a058adb7
  [DEL] -14.8Ki  [DEL] -14.6Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::map_to_dd_format::hd42d532662b40175
  [DEL] -17.5Ki  [DEL] -17.4Ki    saluki_components::transforms::trace_obfuscation::TraceObfuscation::obfuscate_span::hfb6846f70c2f8055
  [DEL] -29.6Ki  [DEL] -29.4Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::hd4c2b27009bc6445
  [DEL] -36.7Ki  [DEL] -36.4Ki    _<saluki_components::common::datadog::obfuscation::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::obfuscation::ObfuscationConfig>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h04547875805b5e90
  -0.9% -68.3Ki  -0.5% -31.1Ki    [7397 Others]
  -0.2% -96.9Ki  -0.2% -59.8Ki    TOTAL

@lucastemb
lucastemb marked this pull request as ready for review July 23, 2026 19:55
@lucastemb
lucastemb requested a review from a team as a code owner July 23, 2026 19:56

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The new as_type=rate path constructs every converted delta Sum with a zero interval. Both Datadog encoders divide the value by that interval and discard the resulting non-finite value, so a valid delta Sum such as 42 is sent as a rate series with no points and is silently lost.

📊 Validated against 1 scenarios · Open Bits AI session

🤖 Datadog Autotest · Commit 94add5d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

let values = match data_type {
DataType::Gauge => MetricValues::gauge((timestamp_s, value)),
DataType::Count => MetricValues::counter((timestamp_s, value)),
DataType::Rate => MetricValues::rate((timestamp_s, value), Duration::ZERO),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Zero rate interval drops converted delta Sums

All default-config delta Sums requesting as_type=rate are silently omitted from Datadog metric payloads, causing users to lose those measurements.

Assertion details
  • Input: An OTLP delta Sum with datadog.metric.as_type=rate and value 42, when interval inference is unavailable (the default configuration).
  • Expected: The converted metric should preserve the value in an encodable rate representation, or use a defined fallback that does not discard the point.
  • Actual: The diff creates MetricValues::Rate(..., Duration::ZERO). The v1 and v2 encoders divide the value by interval.as_secs_f64(), producing infinity, and their non-finite filter removes the point; the payload retains rate type and interval 0 but has no points.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same rationale as below.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 94add5d755

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

let values = match data_type {
DataType::Gauge => MetricValues::gauge((timestamp_s, value)),
DataType::Count => MetricValues::counter((timestamp_s, value)),
DataType::Rate => MetricValues::rate((timestamp_s, value), Duration::ZERO),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid zero-interval rate metrics

For any OTLP delta Sum datapoint with datadog.metric.as_type=rate, this creates a MetricValues::Rate with Duration::ZERO; I checked the Datadog metrics encoders and they divide rate values by the stored interval before filtering finite points, so every finite value becomes inf/NaN and is dropped from the payload instead of being emitted as a rate. Use the inferred delta interval when available, and a non-zero/no-scale representation for the no-interval case.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Fixed in 5636c9e

@pr-commenter

pr-commenter Bot commented Jul 23, 2026

Copy link
Copy Markdown

Regression Detector (Agent Data Plane)

Run ID: f4e967c3-4510-4729-923d-23975b13487b
Baseline: 36f7279b · Comparison: 5636c9ed · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (3)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
quality_gates_rss_dsd_low memory ⚪ +0.22 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ +0.04 metrics profiles logs
quality_gates_rss_idle memory ⚪ -0.13 metrics profiles logs
Bounds Checks: ✅ Passed (3)
experiment check replicates observed links
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 45.2 MiB ≤ 50 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 67.2 MiB ≤ 75 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 31.2 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

@dd-octo-sts dd-octo-sts Bot added the encoder/datadog-metrics Datadog Metrics encoder. label Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/components Sources, transforms, and destinations. encoder/datadog-metrics Datadog Metrics encoder. source/otlp OTLP source.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support datadog.metric.as_type=rate for OTLP delta sums

1 participant