feat(agent-data-plane): emit liveness signals - #2213
Conversation
Add a periodic liveness source that emits Core Agent-compatible running metrics and service checks through the existing metric and service-check pipelines.\n\nKeep the liveness baseline pipelines available whenever ADP is enabled, and fetch/cache Core Agent system host tags without suppressing signals when tag retrieval fails.
Defer liveness IPC client construction until the source tick so host tag failures cannot block topology startup. Bound and cancel tag lookups and dispatches, cache successful system tags, and skip missed ticks.\n\nRoute liveness metrics directly to the Datadog metrics encoder so optional host-tag enrichment cannot suppress liveness signals.
This comment has been minimized.
This comment has been minimized.
Keep liveness tied to a running topology rather than allowing it to create a liveness-only topology. Require the metric and service-check baselines whenever a data pipeline runs so proxy-only OTLP topologies can emit liveness, while no-pipeline configurations retain the established early exit.
|
[GPT-5.6 Terra (OpenAI)] Investigated the automated CI summaries. The subsequent |
Binary Size Analysis (Agent Data Plane)Baseline: 49ff2f8 · Comparison: d94bbb2 · diff ✅ Binary size difference within thresholdChanges by Module
Detailed Symbol Changes |
Regression Detector (Agent Data Plane)Run ID: Optimization Goals: ✅ No significant changes detectedFine details of change detection per experiment (3)Experiments configured
Bounds Checks: ✅ Passed (3)
ExplanationA change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression ( |
|
|
||
| // If no data pipelines are enabled, then there's nothing for us to do. | ||
| // Without a data pipeline, there is no useful topology to run. In particular, liveness alone must not keep ADP | ||
| // alive. |
There was a problem hiding this comment.
This comment was fine before, revert it
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Restored the original no-data-pipelines comment in c90a1a2c7b.
| // Notably, we _don't_ need either of these if all we're doing is running the OTLP pipeline in proxy mode, which | ||
| // is the only reason we're differentiating here. | ||
| // Liveness requires the metrics and service-check baseline pipelines for every running topology, including an | ||
| // OTLP proxy-only topology. |
There was a problem hiding this comment.
Again, this comment was fine, revert it
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Restored the original baseline explanation in c90a1a2c7b and retained a concise connected-liveness exception. Reverting the proxy-mode sentence verbatim would be inaccurate because connected OTLP proxy-only topologies now deliberately provision metric and service-check output paths for liveness.
Remove liveness-specific IPC host-tag retrieval, caching, retries, and dispatch timeouts. Route its metric through the common enrichment pipeline so host tags remain a pipeline concern, while service checks continue through the standard encoder path.\n\nKeep connected topologies provisioned for liveness, including OTLP proxy-only, but leave standalone OTLP proxy mode free of liveness and otherwise-unneeded output paths.
|
|
||
| /// Periodically emits Agent Data Plane liveness signals. | ||
| pub struct LivenessConfiguration { | ||
| hostname: MetaString, |
There was a problem hiding this comment.
Is hostname here necessary? I would have thought this would be added by the downstream metrics/service_checks components
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Kept the hostname. metrics_enrich intentionally does not add metric hosts, and this service check routes directly to dd_service_checks_encode without a host-enrichment transform. Removing it would leave the metric without a host dimension and omit host_name from the service-check payload.
| let service_check = | ||
| Event::ServiceCheck(ServiceCheck::new(UP_SERVICE_CHECK_NAME, CheckStatus::Ok).with_hostname(hostname)); | ||
|
|
||
| (metric, service_check) |
There was a problem hiding this comment.
These are going to be around for the lifetime of the process, can we just make them once and then have them be static or something? It's not a lot, but we should minimize allocations here
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Fixed in c90a1a2c7b: Liveness::new constructs both payloads once, and each timer tick only clones the prebuilt events for dispatch.
| } | ||
|
|
||
| #[test] | ||
| fn emits_liveness_events_without_direct_system_host_tags() { |
There was a problem hiding this comment.
Yes this component should not put any host tags on, but that should be happening downstream. Think we want to remove this test.
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Replaced the implementation-specific assertion about absent source tags in c90a1a2c7b with payload-contract tests for metric name/value/host/version and service-check name/status/hostname.
Construct the liveness metric and service check with the source, then clone them for each dispatch tick. Preserve explicit hostnames because the shared metric path and direct service-check encoder do not add them.\n\nRestore the original topology comments while documenting the connected liveness exception for proxy-only OTLP deployments, and replace source-tag implementation assertions with payload contract coverage.
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod liveness_topology_tests { |
There was a problem hiding this comment.
Extraneous test, remove
There was a problem hiding this comment.
[GPT-5.6 Terra (OpenAI)] Removed the constant-only test in d94bbb262d.
Remove the constant-only liveness topology test module after review feedback.\n\nLiveness routing remains covered by the production topology configuration and existing component tests.
![datadog-prod-us1-5[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The new liveness source is wired only after the required metric and service-check baselines are selected, so connected OTLP proxy-only topologies can emit both signals while standalone proxy mode and the no-pipeline exit path remain unchanged. Payload inspection covered hostname, version tagging, status encoding, and downstream encoder contracts; local compilation was unavailable because the uncached hyper-http-proxy git dependency could not be fetched.
🤖 Datadog Autotest · Commit d94bbb2 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d94bbb262d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let metric_context = Context::from_static_name(RUNNING_METRIC_NAME) | ||
| .with_host(hostname.clone()) | ||
| .with_tags(TagSet::from_iter([Tag::from(format!("version:{version}"))])); | ||
| let metric = Event::Metric(Metric::gauge(metric_context, 1.0)); |
There was a problem hiding this comment.
Metric::gauge(metric_context, 1.0) creates a point without a timestamp, and prebuilding then cloning it preserves that state for every 15-second tick. Because this source connects directly through metrics_enrich rather than the aggregation transform that timestamps non-timestamped values, the Datadog metrics encoders serialize the point timestamp as 0, so the running gauge is reported at the Unix epoch instead of indicating current liveness. Construct the metric with the current Unix timestamp on each emission.
Useful? React with 👍 / 👎.
1 participant
Tracking
Summary
datadog.agent_data_plane.runninganddatadog.agent_data_plane.upthrough the existing Datadog encoders and forwarder.metrics_enrichpath, so common host-tag enrichment applies exactly as it does to other metrics.Validation
cargo nextest run -p agent-data-plane --bin agent-data-plane(160 passed)cargo check -p agent-data-plane --testsmake fmtorigin/main...HEAD(no blockers or important issues)make test-integration-macos-run CASE=adp-no-pipelines-exit— requires provisioned Core Agent plus release ADP and Panoramic binaries, unavailable locally; the exact CI regression is fixed by the restored no-pipeline exit condition and will rerun in CI.make check-all— local FIPS feature checks requirecmake, which is unavailable in this environment.