feat(dogstatsd): add dogstatsd top

[thieman]

Summary

This PR adds the retained-context diagnostic exposed by the Datadog Agent as dogstatsd top to the agent-data-plane binary.

It adds:

• agent-data-plane dogstatsd top for an online privileged-API snapshot followed by local analysis.
• agent-data-plane dogstatsd dump-contexts for dump-only workflows.
• agent-data-plane dogstatsd top --path <artifact> for offline and off-host analysis.
• A privileged API route that coordinates a snapshot through the DogStatsD aggregate owner and returns only the completed artifact path.
• Bidirectional Datadog Agent artifact interoperability, deterministic Agent-compatible reporting, secure atomic artifact publication, focused lifecycle/concurrency coverage, and recorded high-cardinality measurements.

The diagnostic reports contexts currently retained by DogStatsD aggregation. It does not report packet rate, sample frequency, or contexts observed during a collection window.

Tracking

• Jira: DADP-153 — Add dogstatsd top command to show top DogStatsD contexts

Reviewer TL;DR

The intended review is primarily about the approach and contracts rather than every test helper or injected failure seam.

The central design is:

1. dsd_agg remains the sole owner of AggregationState.
2. The API requests a snapshot over a bounded channel serviced in the aggregate task's existing select! loop.
3. The owner clones only shared context handles and small type/unit metadata, then resumes ingestion.
4. JSON serialization, zstd compression, synchronization, and atomic publication happen off the aggregate owner.
5. The API returns a JSON path, never the potentially large context set.
6. The CLI reads the artifact with a streaming decoder and renders the Agent-compatible report.

The most useful review order is:

1. Operator contract: docs/agent-data-plane/dogstatsd-top.md
2. CLI surface: bin/agent-data-plane/src/cli/dogstatsd/top.rs
3. Report semantics: bin/agent-data-plane/src/dogstatsd_contexts/report.rs
4. Aggregate ownership boundary: lib/saluki-components/src/transforms/aggregate/mod.rs
5. API and coordination: bin/agent-data-plane/src/dogstatsd_contexts/api.rs
6. Artifact format/publication: bin/agent-data-plane/src/dogstatsd_contexts/artifact.rs
7. Production wiring: bin/agent-data-plane/src/cli/run.rs and internal/control_surfaces.rs
8. Black-box verification: dogstatsd-top and dogstatsd-top-windows

Suggested review questions:

• Is AggregationState the right authoritative retained-state boundary?
• Are the owner pause, allocation, cancellation, and shutdown semantics acceptable?
• Does the user-visible report preserve the Agent contract where intended?
• Is relying on the shared privileged-API access-control boundary appropriate for context data that can contain sensitive names, hosts, and tags?
• Are the explicitly documented differences from the Agent desirable?

Motivation

High-cardinality retained contexts can account for significant Agent memory. Traffic-window statistics answer "what arrived recently?" but not "what is still retained for aggregation?"

Saluki already had dogstatsd stats, including cardinality analysis, but it is not a valid source for this command:

• dsd_stats branches directly from raw dsd_in.metrics.
• It observes a requested traffic window rather than retained aggregation state.
• Its key omits host and origin-tag identity, so it can collapse contexts that the aggregator retains separately.

This implementation therefore snapshots the context map owned by the actual post-mapper, post-prefix-filter, post-tag-filterlist dsd_agg transform.

User interface

Command tree

agent-data-plane dogstatsd
├── top
│   ├── --path, -p
│   ├── --num-metrics, -m       default: 10
│   ├── --num-tags, -t          default: 5
└── dump-contexts

Online dump and analysis

$ agent-data-plane dogstatsd top
Wrote /opt/datadog-agent/run/dogstatsd_contexts.json.zstd
   Contexts	Metric name	(number of unique values for each tag)
          2	requests.count	(2 instance, 1 env, 1 service)
          1	queue.depth	(1 queue)

The command:

1. creates the same privileged API client used by existing ADP commands;
2. requests a new snapshot and completed artifact;
3. prints and flushes Wrote <path>;
4. opens that path and renders the report;
5. leaves the artifact on disk.

Printing/flushing the path before parsing is intentional: if later local analysis fails, the operator still knows where the completed diagnostic artifact is.

Dump only

$ agent-data-plane dogstatsd dump-contexts
Wrote /opt/datadog-agent/run/dogstatsd_contexts.json.zstd

This is useful when an operator wants to copy the artifact away from a pressured host or analyze it later.

Offline analysis

$ agent-data-plane dogstatsd top --path /tmp/copied-contexts.json.zstd
   Contexts	Metric name	(number of unique values for each tag)
          2	requests.count	(2 instance, 1 env, 1 service)

Offline mode does not construct an API client or contact the running ADP process.

Input validation

Compared with the Agent command, the ADP CLI intentionally rejects rather than copying unsafe edge behavior:

• negative limits are rejected by unsigned option parsing;
• extra positional arguments are rejected;
• the Agent's historical --mum-tags typo is intentionally unsupported; use --num-tags.

Zero remains valid and follows the Agent's existing "do not summarize only one remainder" rule.

Privileged API contract

Request

POST /agent/dogstatsd-contexts-dump HTTP/1.1
Content-Length: 0

The route is registered only on ADP's TLS-enabled privileged API and only when the DogStatsD pipeline is enabled.

The CLI uses the existing DataPlaneAPIClient::from_config path shared by other ADP privileged endpoints. This PR adds no endpoint-specific authentication, certificate handling, TLS configuration, or HTTP-client infrastructure; the route relies entirely on the existing privileged API access-control boundary.

Successful response

HTTP/1.1 200 OK
Content-Type: application/json

"/opt/datadog-agent/run/dogstatsd_contexts.json.zstd"

The response contains only the path. Context records are never returned or buffered in HTTP.

This preserves the Agent's shared-filesystem operational model. Remote callers that cannot see ADP's filesystem must securely copy the completed artifact and use offline mode; this PR does not add a download endpoint.

Error responses

Status	Meaning
503	No aggregate owner is available, or the owner stopped before responding
504	Snapshot acquisition exceeded the 30-second bound
500	Run-path validation, serialization, compression, filesystem, publication, task, or result-path failure

Public bodies are fixed and redacted. Metric names, hosts, tags, credentials, raw serde errors, and filesystem internals are not reflected to API callers.

Snapshot and ownership model

AggregationState remains exclusively owned by the aggregate task. The new public boundary is a cloneable AggregateContextSnapshotHandle containing the request sender; the owner-side receiver is taken exactly once when the transform is built.

A request contains a one-shot response channel. The aggregate loop:

1. skips work if the requester has already canceled;
2. iterates the retained context map while still on the owner task;
3. creates one AggregateContextSnapshotEntry per retained context;
4. sends the immutable vector to the API task;
5. resumes normal event/flush processing.

Each entry contains:

• an Arc-backed Context clone, preserving name, host, client tags, and origin tags without deep-copying their contents;
• a small metric-kind enum;
• a cloned MetaString unit.

Serialization does not occur on the owner task.

Consistency

The current DogStatsD topology supplies one aggregate owner, so the snapshot is internally point-in-time for that owner.

The API coordinator accepts multiple handles and requests them concurrently. If future topology changes add multiple aggregation owners, each owner snapshot remains internally consistent while the combined result is a documented rolling snapshot rather than a globally atomic barrier.

Memory and pause behavior

Snapshot memory is O(contexts) and is included in firm aggregate memory accounting:

aggregate_context_limit * size_of::<AggregateContextSnapshotEntry>()

At one million contexts, the structural entry vector is 40 MB in the current build. The API reuses the first owner vector when combining results, so the current one-owner case does not allocate a second million-entry buffer.

The public handle documents that ownership transfers to the caller, which must account for the vector while retaining it.

What constitutes a reported context

The source is the set currently retained in Saluki's AggregationState:

• ordinary contexts remain while values are pending aggregation and leave after their closed values flush;
• idle counters remain while zero-filling semantics retain them and leave after configured counter expiry;
• fully timestamped metrics that use passthrough do not appear;
• the non-timestamped part of a mixed metric does appear;
• rejected contexts beyond the context limit do not appear.

This intentionally follows Saluki's authoritative aggregation lifecycle. It does not force the Agent context resolver's exact internal TTL model onto a different architecture.

Artifact contract

Location

<run_path>/dogstatsd_contexts.json.zstd

A successful dump replaces the prior canonical artifact. ADP does not automatically remove the latest artifact.

Record schema

The artifact is an unversioned stream of top-level JSON objects, one per context, compatible with the Agent's ContextDebugRepr:

{"Name":"requests.count","Host":"node-a","Type":"Counter","TaggerTags":["pod_name:web-a"],"MetricTags":["env:prod","instance:a"],"NoIndex":false,"Source":1}

Field	ADP mapping
Name	Context metric name
Host	Context host or ""
Type	Agent spelling for the retained metric kind; millisecond histograms use Historate
TaggerTags	Origin tags
MetricTags	Client/instrumented tags
NoIndex	false; Saluki has no corresponding retained flag
Source	1, Agent MetricSourceDogstatsd

The writer borrows names and tags from the snapshot while serializing. It does not build a second raw JSON representation in memory.

Compression and reading

ADP writes zstd. The reader detects zstd by magic bytes rather than filename suffix and also accepts plain Agent NDJSON. Decoding is buffered and streaming.

A compressed artifact renamed without .zstd remains readable by ADP. ADP-produced artifacts retain the .zstd suffix required by the stock Agent reader.

Publication guarantees

ADP improves the Agent's direct create/truncate behavior while delegating cross-platform staging and replacement to tempfile:

1. Create a randomized same-directory NamedTempFile exclusively.
2. On Unix, normalize the staged file to exact owner-only mode 0600; on Windows, inherit the configured run_path ACL like the Agent.
3. Stream records through zstd and a buffered writer.
4. Finalize compression and buffering.
5. Synchronize the staged file.
6. Atomically replace the canonical path with NamedTempFile::persist.
7. Let tempfile remove an unpersisted staging file on drop after failure.

The previous canonical artifact remains intact on every failure before persistence. Cleanup is best-effort through tempfile; ADP no longer carries custom Windows file/security FFI, replacement code, or compound cleanup-error machinery.

Report contract

For every decoded record, the report:

1. groups by Name only;
2. increments that name's context count once;
3. unions complete strings from MetricTags;
4. ignores host, type, origin tags, NoIndex, and source while rendering.

Consequences:

• host and origin-tag differences increase context count;
• origin tags do not appear in displayed cardinality;
• repeated identical client tags count once per metric name;
• values containing additional colons remain one full tag value;
• colonless tags use the complete string as their key.

Ordering is deterministic:

• metrics: context count descending, then name ascending;
• tag keys: cardinality descending, then key ascending.

Both metric and tag limits preserve the Agent's limit + 1 rule: exactly one item that would otherwise be hidden is shown rather than summarized as one remainder. Hidden metric rows sum context counts; hidden tag text sums cardinalities.

Comparison with Datadog Agent

Area	Datadog Agent	ADP in this PR
User goal	Diagnose currently retained DogStatsD contexts	Same
Commands	dogstatsd top, dogstatsd dump-contexts	Same command names under agent-data-plane dogstatsd
Offline analysis	top --path	Same
Defaults	10 metrics, 5 tags	Same
Tag-limit flag	Historical typo --mum-tags/-t	Corrected --num-tags/-t; typo intentionally unsupported
Online API path	/agent/dogstatsd-contexts-dump on Agent command API	Same path on ADP privileged API
Access control	Agent command API bearer middleware plus IPC TLS	Same existing ADP privileged-API boundary and client behavior as other privileged routes
Authoritative state	DogStatsD sampler context resolvers	dsd_agg AggregationState
Complete snapshot	Rolling across sampler workers	One owner today; concurrent rolling coordinator supports future multiple owners
Artifact path	<run_path>/dogstatsd_contexts.json.zstd	Same
Artifact records	Unversioned PascalCase NDJSON	Exact compatible schema
API response	JSON string path only	Same response shape
Publication	Direct create/truncate of canonical file	NamedTempFile staging, sync, atomic persistence, drop cleanup
Concurrent requests	Can write the same canonical file concurrently	Serialized with an owned async mutex through blocking publication completion
Unix permissions	Derived from create mode and umask	Exact 0600
Windows permissions	Inherited run_path file security	Same inherited run_path ACL model
Compression detection when reading	.zstd suffix	Zstd magic bytes; plain NDJSON also accepted
Negative limits / extra arguments	Can panic or be ignored	Rejected
Report grouping/order/text	Name grouping, client-tag cardinality, deterministic tie-breaks, limit + 1	Preserved

The route, method, response shape, artifact path, schema, and report behavior match the Agent contract. ADP serves the route on its existing privileged API listener rather than the Agent command-port server.

Bidirectional interoperability evidence

This was tested with the installed Datadog Agent 7.80.3 and with the containerized Agent used by Panoramic.

Agent writes, ADP reads

An isolated real Agent received DogStatsD traffic and ran:

agent dogstatsd dump-contexts

Both commands then analyzed that exact Agent-produced artifact:

agent dogstatsd top --path <agent-artifact>
agent-data-plane dogstatsd top --path <agent-artifact>

The reports matched exactly.

The actual Agent 7.80.3 compressed artifact and its exact decompressed NDJSON are checked in under test/integration/cases/dogstatsd-top/, so this direction remains automated.

ADP writes, Agent reads

An isolated real ADP received DogStatsD traffic and ran:

agent-data-plane dogstatsd dump-contexts

Both commands then analyzed the ADP-produced artifact:

agent-data-plane dogstatsd top --path <adp-artifact>
agent dogstatsd top --path <adp-artifact>

The reports matched exactly.

The Linux Panoramic case repeats both directions with the real binaries. A separate Windows Panoramic case exercises the privileged ADP online/offline workflow while allowing the artifact to inherit the configured run_path ACL like the Agent.

Code map

Area	Files	Responsibility
Aggregate boundary	lib/saluki-components/src/transforms/aggregate/mod.rs	Snapshot entry types, request channel, owner-loop handling, lifecycle and memory accounting
Public aggregate exports	lib/saluki-components/src/transforms/mod.rs	Production handle/types and feature-gated cross-crate test helpers
API	bin/agent-data-plane/src/dogstatsd_contexts/api.rs	Owner coordination, timeout/status mapping, and serialized blocking publication
Artifact	bin/agent-data-plane/src/dogstatsd_contexts/artifact.rs	Agent schema, streaming reader/writer, NamedTempFile staging, sync, and atomic persistence
Report	bin/agent-data-plane/src/dogstatsd_contexts/report.rs	Grouping, cardinality, sorting, limits, exact text rendering
CLI	bin/agent-data-plane/src/cli/dogstatsd/top.rs	Options, preflight validation, online/offline/dump workflows, stdout ordering
Privileged API client	bin/agent-data-plane/src/cli/utils.rs	Existing privileged client construction plus context-dump POST/path handling
Topology wiring	bin/agent-data-plane/src/cli/run.rs	Captures the exact dsd_agg handle and constructs the handler from the configured run path
Privileged registration	bin/agent-data-plane/src/internal/control_surfaces.rs	Registers the handler only on the privileged API
Black-box tests	test/integration/cases/dogstatsd-top/, test/integration/cases/dogstatsd-top-windows/	Real processes, UDP, privileged API, online/offline CLI paths, cross-tool artifact parity, and Unix permission enforcement
Documentation	docs/agent-data-plane/dogstatsd-top.md	Operator-facing semantics, security, workflows, errors, interoperability

Verification strategy

The test suite is intentionally layered so reviewers can evaluate contracts without tracing every failure-injection helper.

1. Aggregate ownership and lifecycle

Coverage includes:

• every retained metric shape and unit mapping;
• full context identity, including host/client/origin differences;
• ordinary context removal after flush;
• idle-counter retention and expiry;
• fully timestamped passthrough exclusion;
• mixed timestamped/non-timestamped inclusion;
• real owner-loop snapshot handling and shutdown;
• canceled queued requests skipping snapshot construction;
• one-time receiver ownership;
• closed/stopped/canceled one-shot behavior;
• firm memory-bound calculation.

2. Real processing-chain identity

An in-process topology sends metrics through the actual:

DogStatsD mapper → prefix filter → metric tag filterlist → aggregate transform

The snapshot must contain only the mapped, prefixed, filtered identity; raw names, removed tags, and blocked metrics must be absent.

3. API contract and concurrency

Coverage includes:

• POST route and GET rejection;
• JSON path-only success response;
• no-owner, closed-owner, stopped-owner, and timeout status mapping;
• multi-owner concurrent request/ordered flattening;
• first-vector allocation reuse;
• serialized concurrent publications;
• cancellation before snapshot and after blocking publication begins;
• publication errors and blocking-task panics producing redacted 500 responses.

4. Artifact schema, corruption, and publication

Coverage includes:

• every Agent field and metric type;
• plain/compressed fixtures and magic-byte detection;
• unknown versus missing fields;
• malformed JSON, arrays, trailing corruption, and truncated zstd;
• sensitive malformed values redacted from error chains;
• partial record-write error context;
• replacement of a prior canonical artifact;
• persistence failure context with tempfile drop cleanup;
• exact Unix mode 0600;
• no temporary-file accumulation.

5. Privileged API client and CLI

Coverage includes:

• empty POST construction through the existing privileged API client;
• path response and status handling;
• all short/long/legacy options and validation ordering;
• exact stdout and path-flush behavior;
• online, dump-only, empty, corrupt, and offline flows.

6. Black-box Linux and Windows integration

The committed Linux Panoramic case:

1. boots the current ADP inside a real Datadog Agent image;
2. verifies DogStatsD UDP and privileged HTTPS listeners;
3. sends real UDP DogStatsD contexts, including same-name contexts distinguished by host and client tags;
4. invokes the actual online agent-data-plane dogstatsd top command;
5. invokes the privileged API directly and checks the application/json path-only response;
6. invokes dump-contexts and verifies exact output and mode 0600;
7. copies the artifact and runs offline ADP analysis;
8. runs the stock Agent CLI against the ADP artifact and requires identical reports;
9. runs both CLIs against actual Agent 7.80.3 compressed and plain artifacts and requires identical reports;
10. verifies process stability and absence of panic logs.

Local result against a freshly rebuilt image:

PASS dogstatsd-top (21.47s)
7 Panoramic assertions passed, 0 failed

The Windows Panoramic case uses the real Windows ADP binary and PowerShell to:

1. verify UDP and privileged HTTPS listeners;
2. send host-distinguished DogStatsD contexts;
3. run online top, dump-contexts, and copied-artifact offline top through the existing privileged API client path;
4. verify process stability and absence of panic logs.

Windows publication intentionally applies no artifact-specific DACL; the completed artifact inherits the configured run_path ACL, matching the Agent's os.Create behavior.

Native Windows privileged-boundary/inherited-ACL/NamedTempFile::persist result:

PASS dogstatsd-top-windows (28.95s)
7 Panoramic assertions passed, 0 failed
Full Windows integration suite: 31 passed, 0 failed

7. Repository/platform verification

• cargo nextest run -p saluki-components transforms::aggregate::tests — 26 passed.
• cargo nextest run -p agent-data-plane — 208 passed.
• cargo nextest run -p saluki-io — 113 passed.
• cargo check --workspace && cargo check --workspace --tests — passed.
• make fmt — passed with no tracked changes.
• make check-all — passed, including the FIPS feature matrix.
• make test-all — passed:
  • 2,112 normal tests
  • 21 property tests
  • 41 Miri tests
  • 8 Loom tests
• make build-windows-cross — passed for x86_64-pc-windows-msvc.
• cargo xwin check --target x86_64-pc-windows-msvc --package agent-data-plane --tests — passed.
• A simulated core.autocrlf=true checkout preserves both NDJSON fixtures as LF and byte-matches their compressed artifacts.
• The Windows target cross-check, native unit suite, and native Panoramic suite pass without direct Windows filesystem/security dependencies.

Performance evidence

One-time local Criterion sample-size-10 measurements taken while selecting the implementation (the benchmark-only scaffolding was removed after review):

Operation	Contexts	Time	Structural/artifact size
Aggregate snapshot	10,000	45.759 µs	400,000-byte entry vector
Aggregate snapshot	100,000	1.0218 ms	4,000,000-byte entry vector
Aggregate snapshot	1,000,000	20.818 ms	40,000,000-byte entry vector
Artifact publication	10,000	16.464 ms	27,053-byte zstd artifact
Artifact publication	100,000	122.96 ms	268,466-byte zstd artifact
Artifact publication	1,000,000	1.1381 s	2,332,149-byte zstd artifact

The owner pause measurement covers only snapshot construction. Artifact serialization and publication occur after the owner resumes.

These measurements are evidence, not regression thresholds. The benchmark workload shares realistic tag sets; it does not model worst-case unique tag-value memory during offline report aggregation.

Failure-handling matrix

Failure	Expected behavior
DogStatsD disabled	Route absent
Owner unavailable/stopped	503
Snapshot acquisition exceeds bound	504
Request canceled before owner handles it	Skip snapshot construction
Request canceled during blocking publication	Publication finishes or cleans up while retaining the lock
Invalid/empty/non-directory run path	Redacted 500, no CWD fallback
JSON/write/zstd/buffer/sync failure	Redacted 500; unpersisted NamedTempFile is removed on drop and the prior canonical remains
Atomic persistence failure	Redacted 500; tempfile retains cleanup ownership and the prior canonical remains
Malformed/truncated offline artifact	Path/record/line/column error without leaking record contents
Concurrent requests	Serialize complete snapshot/publication operations

Non-goals and explicit trade-offs

• This is not a replacement for time-windowed dogstatsd stats.
• The API does not return or download artifact bytes.
• The CLI and ADP must share a filesystem for online analysis.
• The artifact schema remains the Agent's unversioned diagnostic format to preserve interoperability.
• The fixed canonical path means a successful later dump replaces the previous artifact.
• There is no automatic artifact retention or cleanup policy.
• Multi-owner output would be rolling, not globally atomic.
• Snapshot construction remains O(contexts) and briefly pauses each owner; serialization does not.
• Offline report aggregation retains unique complete client-tag strings per metric name. Worst-case unique tag-value memory remains a profiling follow-up.

Residual validation and review notes

• Native Windows Panoramic verification passed through the existing privileged API client path and with inherited-ACL/NamedTempFile::persist publication (dogstatsd-top-windows: 7 assertions; full suite: 31 passed).
• The targeted Level-4 dogstatsd-top Linux integration case passed locally (PASS dogstatsd-top, 7 assertions). The complete integration and correctness suites were not run locally.
• Publication synchronizes the staged file before persistence but does not synchronize the parent directory; this guarantees atomic visibility, not full power-loss namespace durability.
• On Windows, custom run_path ACL security is an installer/operator responsibility, matching the Agent's inherited-permission model.

[thieman]

Adding this documentation file seems fine to me, but highlighting for reviewers (not you, LLM, human reviewers)

[thieman]

[GPT-5.6 Sol (OpenAI)] Acknowledged. I left the operator document in place and am treating this thread as a human-review callout rather than making a code change or resolving it on reviewers' behalf.

[datadog-datadog-prod-us1]

More details

The retained-context snapshot, authenticated API, artifact publication, and offline report paths were reviewed for cancellation, malformed input, limit, and lifecycle hazards; no reproducible diff-only regression was identified. No additional tests recommended: the PR already includes targeted coverage for the realistic edge cases found during review.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 6454800 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}