feat(otlp): update OTLP cumulative metric caching and increased testing

[lucastemb]

Summary

Cache keys for cumulative OTLP metrics now use a ContextKey hash. Previously, the cache retained canonical string keys that could grow without bounds.

Additionally, this PR introduces unit testing around existing origin-aware OTLP cumulative metric cache behavior.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

Unit tests since it covers the cache behavior.

References

• Test origin-aware OTLP cumulative metric cache keys. #2065

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 032b9ac · Comparison: 455f644 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 40.83 MiB (baseline) vs 40.90 MiB (comparison)
Size Change: +75.28 KiB (+0.18%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
otlp_protos::otlp_include::opentelemetry	+60.97 KiB	86
core	-58.05 KiB	1865
prost	-47.31 KiB	136
tokio	+41.62 KiB	964
saluki_components::common::datadog	+34.20 KiB	107
figment	+29.90 KiB	62
[sections]	+19.23 KiB	8
serde	-13.84 KiB	20
std	-13.70 KiB	71
hyper_util	-12.92 KiB	6
tracing	-12.56 KiB	37
axum	-12.07 KiB	48
tokio_rustls	-11.35 KiB	9
hyper	+10.54 KiB	51
quick_cache	+8.96 KiB	51
rustls	+8.28 KiB	8
saluki_components::transforms::trace_obfuscation	+8.11 KiB	26
serde_core	+7.74 KiB	99
saluki_components::encoders::datadog	+7.59 KiB	52
saluki_io::net::util	-7.31 KiB	11

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +33.4Ki  [NEW] +33.1Ki    _<saluki_components::common::datadog::config::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::config::ForwarderConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h215858157e52a210
   +58% +18.5Ki   +58% +18.5Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::h4eec0f2810c68dc2
  +270% +16.3Ki  +274% +16.3Ki    h2::proto::connection::DynConnection<B>::recv_frame::h0d14cf3b5c6935a8
  [NEW] +14.4Ki  [NEW] +14.3Ki    h2::server::Connection<T,B>::poll_closed::h7613993231370815
  [NEW] +13.4Ki  [NEW] +13.3Ki    h2::server::Connection<T,B>::poll_closed::h835edce0e04c993f
  [NEW] +13.2Ki  [NEW] +13.1Ki    h2::server::Connection<T,B>::poll_closed::h8c033e12e916a451
  [NEW] +12.1Ki  [NEW] +11.9Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::h2291bdd8b058a751
 +21e2% +10.7Ki +34e2% +10.7Ki    _<saluki_components::transforms::trace_obfuscation::TraceObfuscation as saluki_core::components::transforms::SynchronousTransform>::transform_buffer::h4694da026393f90f
  [NEW] +9.90Ki  [NEW] +9.74Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::h249e6a1863e304e8
  [NEW] +9.61Ki  [NEW] +9.46Ki    _<figment::value::magic::RelativePathBuf as figment::value::magic::Magic>::deserialize_from::h82581f4e7815d2ab
  [NEW] +9.12Ki  [NEW] +8.97Ki    _<figment::value::magic::Tagged<T> as figment::value::magic::Magic>::deserialize_from::h871281d86388e420
  [NEW] +8.63Ki  [NEW] +8.53Ki    rustls::conn::ConnectionCore<Data>::process_new_packets::h004d0d77443e49c4
 -74.8% -8.89Ki -75.4% -8.89Ki    prost::message::Message::encode_to_vec::h6374245488921d75
 -92.5% -9.13Ki -93.2% -9.13Ki    prost::message::Message::decode::h3049809fba850c16
 -92.0% -9.36Ki -92.7% -9.36Ki    prost::message::Message::decode::h3ae9ce85c9ae5e13
 -87.3% -9.86Ki -88.0% -9.86Ki    std::sys::backtrace::__rust_begin_short_backtrace::h3d689d6ace8b6cf2
 -69.2% -11.8Ki -69.5% -11.8Ki    prost::message::Message::encode::h28ad969e25341bd5
  [DEL] -14.3Ki  [DEL] -14.1Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_any::h2e163da9da896fb8
  [DEL] -19.6Ki  [DEL] -19.5Ki    saluki_components::transforms::apm_stats::ApmStats::process_trace::h0c7467fe110ac0a8
  [DEL] -24.3Ki  [DEL] -24.2Ki    saluki_components::transforms::trace_obfuscation::TraceObfuscation::obfuscate_span::hca56db06df8fb52f
  +0.2% +13.4Ki  -0.8% -57.0Ki    [10107 Others]
  +0.2% +75.3Ki  +0.0% +4.03Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: b9eb2253-e9dc-4c68-83b3-77cb8a4233b5
Baseline: 032b9ac5 · Comparison: 455f6449 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_low	memory	⚪ +0.09	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.07	metrics profiles logs
quality_gates_rss_idle	memory	⚪ +0.01	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.09	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ -0.21	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 133 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 45.6 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 67.3 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 194 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 31.4 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.