diff --git a/antithesis/scratchbook/deployment-topology.md b/antithesis/scratchbook/deployment-topology.md new file mode 100644 index 0000000000..76c4d3a3ad --- /dev/null +++ b/antithesis/scratchbook/deployment-topology.md @@ -0,0 +1,69 @@ +--- +sut_path: /home/ssm-user/src/saluki +commit: 93b9f37baf07004a9235f308305d78ef3d7fc226 +updated: 2026-07-15 +external_references: + - path: https://datadoghq.atlassian.net/browse/SMPTNG-737 + why: Antithesis triage umbrella + - path: test/antithesis/bin/launch.sh + why: authoritative fault profile and cost knobs +--- + +# Deployment topology — cheap demonstrative scenarios + +Every container and every fault axis is a cost. This document proposes minimal topologies for the target defect families and states the cost delta versus the existing `general` scenario. + +## Cost baseline + +- `general`: 3 containers (intake, agent [ADP-on SUT], workload) + node term/hang/throttle on `agent` + global cpu_mod + clock_jitter + network faults. 30 m default. +- `differential`: 4 containers, no node faults. Expensive by container count. + +The dominant cost knob is the **node-fault branching factor** on `FAULT_NODES`. Dropping it collapses the timeline tree. cpu_mod and clock_jitter are global and cheap and one of them (clock_jitter) is a real aggregation trigger, so keep both. + +## Proposal A — `aggregation` (P0, recommended first build) + +The cheapest genuinely demonstrative scenario. Reuses the converged `general` topology, drops node faults. + +- Containers: 3 — intake, agent (converged, ADP-on), workload. Same as `general`. +- `SCENARIO_FAULT_NODES=""` — **no node faults**. Keep global cpu_mod + clock_jitter. +- Duration: short (10 m is plenty; single-packet triggers converge fast). +- Drivers: existing `send_dogstatsd` + `sketchburst`, plus new marker drivers for `flush-liveness-marker`, `counter-nan-poisoning`, `counter-sum-oracle`. +- New instrumentation: one `eventually_flush_marker` harness command; one value-oracle `assert_always` in `series_observation.rs`. +- Demonstrates: the aggregation family the user named (NaN, sum, sketch, points-per-payload, clock-monotonic under clock_jitter) AND the green-by-default flush-liveness anchor. +- Cost delta vs `general`: removes the entire node-fault branching factor and 2/3 of the duration. Substantially cheaper per run, higher aggregation signal. + +Why this over a "slow intake stall" scenario: the forwarder sheds rather than wedges (sut-analysis counter-fact), so backpressure cannot demonstrate a stall. The honest cheap liveness contribution here is a regression anchor, not a live-bug hunt. + +## Proposal B — `boot-partition` (P0, freshest-bug guard) + +Directly demonstrates the recurring boot-gate stall family (SMPTNG-766, current branch). Not reusable from `general` because the converged image cannot partition ADP from its own Core Agent. + +- Containers: 3 — a standalone-ish ADP, a Core-Agent gRPC endpoint (full Agent, or a minimal gRPC stub serving hostname/registration/health), and intake. +- Faults: network faults on the ADP↔Core-Agent link, healed before judging. cpu_mod + clock_jitter global. No node term/hang. +- Duration: boot-length; short. +- Demonstrates: `boot-liveness-under-partition` — the exact family just fixed on this branch. Strong regression guard. +- Cost: cheap in compute (3 containers, short, network-fault-only). More build effort — needs the split topology and possibly a gRPC stub. + +## Proposal C — leave `differential` alone +Host-tag identity and cross-lane divergence stay in `differential`. Not cheap; out of scope for this pass. + +## Blocker found while de-risking Proposal B + +Proposal B as first scoped — split ADP and Core Agent into two containers and network-partition the boot gRPC link — is **infeasible**. ADP dials the Core Agent at a hardcoded `https://127.0.0.1:{cmd_port}` (`lib/datadog-agent/commons/src/ipc/config.rs:209`); the only alternative transport is vsock (host/hypervisor CID). No config points ADP at a Core Agent in another container over TCP. The link is always loopback inside one container, where Antithesis inter-container network faults do not reach. The SMPTNG-766 boot-gate family is therefore not provoked by an inter-container partition — it is provoked by timing faults (cpu_mod, clock_jitter, node-hang) starving the co-located Core Agent past ADP's bounded ~10-attempt retry budget, ingredients that already live in `general`. + +The genuinely partition-able cross-container link is ADP → **intake** (the HTTP forwarder). Partitioning it does not wedge ADP (forwarder sheds), but it does let us assert recovery liveness: ADP survives, and load resumes flowing out once the partition heals. + +## Decision — built the `liveness` scenario + +Goal per the user: demonstrate ADP's liveness is not harmed, cheaply. Built a lean `liveness` scenario: + +- Containers: 3 — intake, agent (converged ADP-on SUT), workload. Mirrors `general`. +- `SCENARIO_FAULT_NODES=""` — no node faults. The launcher's always-on network faults partition the ADP→intake link and heal; global cpu_mod + clock_jitter stay on. This is the cheap adversary. +- Duration: short. +- Liveness claims, both green-by-default `always`: (1) `eventually_adp_alive` — ADP boots and stays reachable (reused unchanged); (2) `eventually_flush_marker` — a marker counter sent on the DogStatsD socket eventually reaches intake at `GET /antithesis/metrics/adp`, proving load in ⇒ load out end-to-end after faults quiesce. + +The `general` intake records the ADP-on agent's POSTs under the `adp` lane (single-listener `else` branch, `intake/src/bin/intake.rs:96`), so the marker-presence probe reads `/antithesis/metrics/adp` directly. + +## Open questions +- Confirm the short duration gives Antithesis enough timelines for a network partition to inject and heal before the `eventually_` fault-quiet judging window. +- Adding node termination on `agent` would additionally demonstrate restart liveness, at the cost of the node-fault branching factor. Deferred to keep this scenario cheap. diff --git a/antithesis/scratchbook/existing-assertions.md b/antithesis/scratchbook/existing-assertions.md new file mode 100644 index 0000000000..333e9d1780 --- /dev/null +++ b/antithesis/scratchbook/existing-assertions.md @@ -0,0 +1,70 @@ +--- +sut_path: /home/ssm-user/src/saluki +commit: 93b9f37baf07004a9235f308305d78ef3d7fc226 +updated: 2026-07-15 +external_references: + - path: https://datadoghq.atlassian.net/browse/SMPTNG-737 + why: Antithesis triage umbrella +--- + +# Existing Antithesis SDK assertions + +The codebase is already instrumented on both sides. SUT-side assertions use the `saluki_antithesis` facade (`lib/saluki-antithesis/src/lib.rs`) — a thin, default-off macro wrapper over the SDK, with numeric helpers `always_ge!`/`always_le!`. Intake/harness-side use the raw SDK. Only relevant entries for the liveness/aggregation targets are listed. + +## SUT-side — boot liveness (covers stall family #1) + +| File:line | Type | Message | +|---|---|---| +| `bin/agent-data-plane/src/main.rs:91` | reachable | agent-data-plane completed bootstrap | +| `bin/agent-data-plane/src/main.rs:153,197` | always_or_unreachable | bootstrap-stage guards | +| `internal/env/host.rs:45` | always_or_unreachable | ADP resolved the Core Agent hostname at boot | +| `internal/env/mod.rs:75,91,100` | always | host / workload / autodiscovery providers connected | +| `internal/remote_agent.rs` | (classify) | initial registration permanent-vs-transient split | + +These already catch the boot-gate stall. A fatal boot `exit(1)` trips no Antithesis platform crash property, so these `always`/`always_or_unreachable` anchors plus the harness `eventually_adp_alive` are the only things that catch it. + +## SUT-side — aggregation state invariants (covers half of the aggregation target) + +| File:line | Type | Message | +|---|---|---| +| `transforms/aggregate/mod.rs:612` | always_le | context map ≤ context_limit | +| `transforms/aggregate/mod.rs:646` | always_ge | flush wall-clock never moves backward (SMPTNG-767) | +| `transforms/aggregate/mod.rs:654` | always_le | zero-value bucket span ≤ 10,000 | +| `transforms/aggregate/mod.rs:672` | sometimes | (flush liveness marker) | +| `transforms/aggregate/mod.rs:693` | always_le | counter-expiry add no overflow | +| `transforms/aggregate/config.rs:68,69` | always_ge/le | histogram percentile quantile in [0,1] | +| `lib/ddsketch/src/agent/sketch.rs:231` | always | DDSketch sample is finite at insert | +| `lib/ddsketch/src/agent/sketch.rs:241` | always_le | DDSketch min ≤ max after insert | +| `lib/ddsketch/src/agent/sketch.rs:784,785` | reachable/always_le | bin collapse reached; bin count within bin_limit | + +## SUT-side — data path / stall mechanics + +| File:line | Type | Message | +|---|---|---| +| `sources/dogstatsd/mod.rs:726,1240` | always_le | receive-buffer / size bounds | +| `sources/dogstatsd/mod.rs:1958-2001` | unreachable | dispatch output missing / failed mid-buffer | +| `topology/interconnect/dispatcher.rs:93` | sometimes | dispatch backpressure observed | +| `common/datadog/io.rs:719,749` | sometimes | forwarder send outcomes | +| `net/util/retry/queue/mod.rs:288,300` | sometimes/always_le | retry queue dropped-oldest / bound | +| `pooling/elastic.rs:282`, `fixed.rs:189` | unreachable | pool semaphore closed | +| `core/runtime/restart.rs:159` | always_or_unreachable | restart-strategy guard | +| `core/health/mod.rs:638` | always_or_unreachable | health component guard | + +## Intake / harness-side + +| File:line | Type | Message | +|---|---|---| +| `harness/src/bin/eventually_adp_alive.rs:78` | assert_always | ADP booted: API reachable and DogStatsD socket present | +| `intake/src/series_observation.rs:61` | assert_reachable | intake.first_series_observed (coarse boot-vs-flushed signal) | +| `intake/src/properties/payload/point.rs:25` | assert_always | Pyld20 value not NaN | +| `intake/src/properties/payload/point.rs:46` | assert_always | Pyld21 timestamp future bound | +| `intake/src/properties/payload/metric_payload.rs:16,27` | assert_always | Pyld07 decode, Pyld08 point count | +| `intake/src/properties/payload/envelope.rs:19` | assert_always | Pyld01-03 envelope | +| `scenarios/general/src/bin/parallel_driver_send_dogstatsd.rs` | reachable/sometimes | workload ran / sent a batch | +| `scenarios/general/src/bin/parallel_driver_sketchburst.rs:56` | reachable | sketchburst swept the sketch bins | +| `scenarios/general/src/bin/parallel_driver_poll_stats.rs:67,76` | reachable/sometimes | stats poll / overlap | + +## What is MISSING for the target properties + +- **Flush-stall liveness ("load in, no load out").** No per-run unique marker probe. `series_observation.rs:61` is only a coarse first-series-ever signal. The intake `Context` (`capture.rs:80`) records identity only, keyed in a dedup `BTreeMap` with `first_seen` — the plumbing to check "a specific marker arrived within a window" exists, but no command uses it. This is the sound signal and it is unbuilt. +- **Aggregation-value oracle.** All intake payload asserts are structural (shape/domain). None reads the flushed point VALUE against an expected sum for a known input. `Context` stores no value, so a "counter sum equals N increments" check needs a small addition in `series_observation.rs evaluate_series`, keyed on a per-run marker name. diff --git a/antithesis/scratchbook/liveness-assertability.md b/antithesis/scratchbook/liveness-assertability.md new file mode 100644 index 0000000000..f8f7b4da12 --- /dev/null +++ b/antithesis/scratchbook/liveness-assertability.md @@ -0,0 +1,38 @@ +--- +sut_path: /home/ssm-user/src/saluki +commit: 93b9f37baf07004a9235f308305d78ef3d7fc226 +updated: 2026-07-16 +external_references: + - path: https://github.com/DataDog/saluki/pull/2112#issuecomment-4993034650 + why: the contested boot-retry fix; states the exact gap this note resolves + - path: https://datadoghq.atlassian.net/browse/SMPTNG-766 + why: boot-fatal on transient Core-Agent gRPC error, the fix under review +--- + +# Finding: "ADP is online and able to service load" is not soundly assertable + +## The question this answers + +PR #2112 makes ADP's boot-time Core-Agent gRPC calls (hostname, registration, health) retry in the background instead of blocking boot and crashing on timeout. A reviewer asks why background-retry over block-then-crash. The unresolved half of the PR is whether the background-retry path **harms liveness** — whether ADP can proceed past boot into a state where it accepts load but cannot service it. This research set out to build a cheap Antithesis scenario demonstrating liveness is unharmed. The conclusion is that no such sound assertion exists. That is the finding, not a failure to build one. + +## Why no sound assertion exists + +1. **Reachable is not servicing.** The only liveness assertion available, `eventually_adp_alive`, checks a TCP connect to the API on :5100 and the presence of the DogStatsD socket file. The API server and the socket listener are separate tasks from the data pipeline. Both bind and answer while the pipeline is wedged or half-initialized. So the assertion proves the listeners came up, exactly the "boot faults resolved" half the PR already demonstrates. It cannot see whether load is serviced end to end. + +2. **Servicing load is not an invariant under the fault model.** A liveness property is only soundly assertable when the good thing is guaranteed to eventually happen. "Load goes out" is not guaranteed. The forwarder sheds — it drops to the retry queue rather than blocking (`common/datadog/io.rs:588`, `:640`) — so under an always-on network partition, ADP correctly does not flush. A missing flush is legitimate behavior, not a wedge. An `always`/`eventually` "load went out" assertion therefore false-REDs on healthy ADP whenever a partition or a clock_jitter/cpu_mod-delayed flush is in play. This is the throughput-liveness-gauge pitfall. There is no green-by-default form of it. + +3. **The failing link is loopback, so the fault cannot be isolated.** ADP dials the Core Agent at a hardcoded `https://127.0.0.1:{cmd_port}` (`lib/datadog-agent/commons/src/ipc/config.rs:209`). The Core-Agent↔ADP boot gRPC lives inside one converged container. Antithesis inter-container network faults do not reach it. You can fault the whole container — pausing Core Agent and ADP together — or the irrelevant ADP→intake link, but you cannot inject a partition on the exact link that causes the known problem, as an independent variable, in a compose topology. So even a scenario that could observe servicing could not attribute a result to the boot Core↔ADP fault. + +4. **The boot edge is ambiguous by construction.** Background-retry means boot "completes" and the listeners bind before a backgrounded dependency has necessarily resolved. Part of the boot sequence blocks and part is backgrounded, and per the PR author it is unclear why. There is no single observable edge that means "fully initialized and servicing," so there is nothing to anchor a readiness assertion on. + +## What follows + +- `eventually_adp_alive` is the ceiling of what is soundly assertable here: the listeners recover. That is a genuine liveness property because listener-binding IS guaranteed to recover in the fault-quiet window. +- "ADP is servicing load under boot-time Core↔ADP faults" is not soundly assertable with the current architecture. Not for want of a scenario — the architecture provides no sound observable, and the fault model makes servicing conditional rather than invariant. +- This does not say the background-retry fix is wrong. It says Antithesis cannot adjudicate the liveness question the reviewer raised. The block-then-crash alternative has the advantage that it removes the ambiguous half-alive state entirely, which is the state we cannot assert against. + +## What would change this + +Any of these would create a sound assertion, at a cost: +- A single observable "fully initialized and servicing" edge in ADP (collapse the split blocking/backgrounded boot into one readiness gate), giving a real edge to anchor an `always`-recovers assertion on. +- A servicing signal that is invariant under the fault model — e.g. an ADP self-report that ingest→aggregate→encode advanced within a window, read SUT-side, independent of whether the forwarder reached a sink. This asserts internal forward progress without depending on load actually leaving, sidestepping the shed-vs-wedge ambiguity. Worth scoping if the liveness question must be answered. diff --git a/antithesis/scratchbook/property-catalog.md b/antithesis/scratchbook/property-catalog.md new file mode 100644 index 0000000000..744a1296b7 --- /dev/null +++ b/antithesis/scratchbook/property-catalog.md @@ -0,0 +1,60 @@ +--- +sut_path: /home/ssm-user/src/saluki +commit: 93b9f37baf07004a9235f308305d78ef3d7fc226 +updated: 2026-07-15 +external_references: + - path: https://datadoghq.atlassian.net/browse/SMPTNG-737 + why: Antithesis triage umbrella; ticket set 738-767 + - path: https://datadoghq.atlassian.net/wiki/spaces/DADP/pages/6497671050 + why: load-independence liveness doctrine + - path: git log (full history) + why: fix-commit triggers for each property +--- + +# Property catalog — liveness and aggregation + +Priority: P0 = cheapest to demonstrate and highest signal, P2 = valuable but needs richer topology or faults. Each property names its Antithesis assertion type, its trigger, its home (SUT-side vs intake-side), and the tickets/commits it guards. + +## Liveness + +### `boot-liveness-under-partition` (P0) +ADP eventually becomes reachable and binds the DogStatsD socket even when the Core-Agent gRPC stream is partitioned during boot. **Always** (green-by-default; a fatal `exit(1)` trips no platform crash property, so an `always` anchor is the only catch). Trigger: network partition ADP↔Core-Agent across the boot window. Home: SUT-side boot anchors (`internal/env/host.rs:45`, `env/mod.rs:75-100`, `remote_agent.rs`) already exist; harness `eventually_adp_alive` catches the exit. Guards SMPTNG-766 and the current branch. **Needs a split topology** (ADP and a Core-Agent gRPC stub in separate containers) to be partitionable — the converged image cannot partition ADP from its Core Agent. +Open questions: is a lightweight Core-Agent gRPC stub cheaper to stand up than reusing the full Agent? does the partition need to straddle a specific boot sub-step (hostname vs registration vs health check) to be a strong test? + +### `flush-liveness-marker` (P0, green-by-default anchor) +Once ADP is up and a uniquely-marked metric is sent, that exact context eventually appears at intake within a bounded window. **Always**, in an `eventually_` command so fault-quiet timing applies. Trigger: driver emits `saluki.antithesis.livemark.:1|c`; a harness command polls `GET /antithesis/metrics/adp` and asserts the marker context is present. Home: intake-side — only intake proves bytes left ADP. Sound because it reads ONE context's presence (immune to dedup-count false-RED at saturation) and the nonce makes it per-run (immune to global-counter false-GREEN on re-flush). No current-code bug reproduces via it — the forwarder sheds rather than wedges (see sut-analysis counter-fact) — so this is a **regression anchor** against a future wedge (e.g. reintroducing a blocking downstream), not a live-bug hunt. Refines `series_observation.rs:61`. +Open questions: what window bound separates "healthy flush" from "wedge" given the 15 s flush interval plus cpu_mod/clock_jitter skew? + +### `component-death-wedge` (P2) +If a single pipeline component task dies, ingest must not silently wedge forever with the UDS still bound. **Sometimes** the pipeline recovers, or **Always** ADP tears down (root supervisor is `OneForOne` with 0 restarts, so death SHOULD bring the process down, not hang). Trigger: node-hang fault on the SUT at a moment that kills one component. Home: SUT-side + the flush marker. Higher cost — needs precise fault timing. + +## Aggregation correctness + +### `counter-nan-poisoning` (P0) +A single `|g NaN` (or `|c NaN`) sample must not poison a window's aggregate sum. `merge_scalar_sum` (`value/mod.rs:493`) has no NaN guard. **Always** value-not-NaN, already present intake-side as Pyld20 (`point.rs:25`). Trigger: one non-finite DogStatsD sample. No faults. Guards `e955c5a44e`, SMPTNG-753, SMPTNG-764. + +### `counter-sum-oracle` (P0) +A per-run-unique counter fed N deterministic increments in one window flushes a point whose value equals the known sum. **Always**, intake-side. Trigger: driver sends N increments to `saluki.antithesis.sumcheck.`. Home: **new** assertion in `series_observation.rs evaluate_series` keyed on the marker name — reads the decoded `MetricSeries` point value (the one place the post-aggregation wire value is visible). Composes with `flush-liveness-marker`. This is the missing aggregation-VALUE oracle. No faults. +Open questions: how to bound "one window" from the driver so the sum is deterministic across the 10 s window / 15 s flush? + +### `flush-clock-monotonic` (P1) +Flushed payload timestamps never move backward, even under a backward clock step. **Always_ge**, already present SUT-side (`aggregate/mod.rs:646`). Trigger: global clock_jitter fault (near-free). Guards SMPTNG-767 (in progress). Also visible intake-side as Pyld21 timestamp bounds. + +### `sketch-bin-bounded` (P1) +DDSketch bin count stays within `bin_limit` and mass is conserved under a bin-collapse sweep. **Always_le** already present (`ddsketch/agent/sketch.rs:785`) + `reachable` collapse. Trigger: `sketchburst` driver (exists) sweeps past `bin_limit` via `|d`. Guards `bc3cc747e5`. No faults. + +### `v2-points-per-payload` (P1) +The v2 `/api/v2/series` encoder honors `serializer_max_series_points_per_payload`. **Always_le** point count, intake-side Pyld08 (`metric_payload.rs:27`). Trigger: high points-per-series volume. Guards SMPTNG-765 (OPEN — v2 encoder ignored the limit). No faults. + +### `subsecond-window-safe` (P2) +A sub-second aggregate window must not divide-by-zero panic. **Always_or_unreachable**. Trigger: `aggregate` window config < 1 s. Guards `738b6874a9`. Needs a config knob the generator can set. + +### `host-tag-identity` (P2) +A `host:` tag is treated as a host dimension, not a normal tag, so context identity matches the Agent. Best as a **differential** property (needs the A/B lane). Guards `380ebef07d`. Not cheap — belongs in `differential`. + +## Assumptions +- clock_jitter and cpu_mod are global, symmetric, and near-free; keep them in every cheap scenario. +- Node termination/hang/throttle are the expensive branching axes; a cheap scenario omits them unless the property requires a kill/hang. + +## Open questions (catalog-level) +- Should the P0 cheap scenario bundle `flush-liveness-marker` + `counter-nan-poisoning` + `counter-sum-oracle` (no faults, one lean topology) or should boot-liveness-under-partition be built first as the freshest-bug guard? This is the pivotal build decision — see deployment-topology.md. diff --git a/antithesis/scratchbook/sut-analysis.md b/antithesis/scratchbook/sut-analysis.md new file mode 100644 index 0000000000..396ce5d8bf --- /dev/null +++ b/antithesis/scratchbook/sut-analysis.md @@ -0,0 +1,92 @@ +--- +sut_path: /home/ssm-user/src/saluki +commit: 93b9f37baf07004a9235f308305d78ef3d7fc226 +updated: 2026-07-15 +external_references: + - path: https://datadoghq.atlassian.net/browse/SMPTNG-737 + why: Antithesis triage umbrella; authoritative ADP/Antithesis ticket set (738-767) + - path: https://datadoghq.atlassian.net/wiki/spaces/DADP/pages/6497671050 + why: "What Comes After DogStatsD?" states the load-independence liveness doctrine + - path: https://datadoghq.atlassian.net/wiki/spaces/~602449d8f3d296006864db68/pages/6980436239 + why: "Agent Time Should be Monotonic, Is Not" — backward-clock aggregation write-up (SMPTNG-767) + - path: git log (full history) + why: mined fix commits for boot/stall/aggregation defect triggers +--- + +# SUT analysis — cheap-to-demonstrate liveness and aggregation + +Scope of this pass: not the whole SUT. It targets two defect families the user named — aggregation bugs and "load comes in but no load goes out" flush-stall liveness — and asks which scenarios demonstrate them most cheaply. "Cheap to demonstrate" means few containers, a short window, and a deterministic trigger, not merely a small compute bill. + +## The SUT + +`agent-data-plane` (ADP) is the primary artifact. In the Antithesis harness it runs converged: the Datadog Agent image with the data plane ON (`DD_DATA_PLANE_ENABLED=true`), so the embedded ADP owns DogStatsD. Load enters over a Unix datagram/stream socket, is decoded, enriched, aggregated, serialized as v2 series and sketches, then POSTed to an HTTP intake. + +## Data path: ingest → intake flush + +Metrics lane wiring: `bin/agent-data-plane/src/cli/run.rs:763` — +`dsd_in.metrics → dsd_enrich → dsd_prefix_filter → dsd_tag_filterlist → dsd_agg → dsd_post_agg_filter → metrics_enrich → dd_metrics_encode → dd_out` + +1. UDS bind — `sources/dogstatsd/mod.rs:895`, bind at `:956` +2. Accept/recv loop — `mod.rs:1298` accept, `:1402` `stream.receive` +3. Decode/frame into a `FixedSizeEventBuffer` (cap 1024, `saluki-core/src/topology/mod.rs:33`) +4. Source dispatch, **awaited inline** — `mod.rs:1489` (buffer full) and `:1555` (100 ms tick) `dispatch_events(...).await` +5. Enrich/filter transforms — `run.rs:763` +6. Aggregation — `transforms/aggregate/mod.rs:381` `state.insert`, per-type merge `saluki-core/.../metric/value/mod.rs:433` +7. Flush on window close, **awaited** — `aggregate/mod.rs:324` `state.flush(...).await`, 15 s interval / 10 s window +8. Post-agg filter → host enrich +9. v2 encode — `encoders/datadog/metrics/mod.rs:616`; builder over `mpsc::channel(8)` +10. Forwarder — `forwarders/datadog/mod.rs:165` `send_transaction` +11. Transaction fan-out — `common/datadog/io.rs:365`, per-endpoint `channel(8)` +12. HTTP send + retry circuit breaker — `io.rs:499`; open circuit re-enqueues to the retry queue `io.rs:638` +13. Wire — intake `POST /api/v2/series` at `test/antithesis/intake/src/http/datadog/metrics.rs:60`; contexts recorded `capture.rs:147` + +Interconnects are bounded tokio mpsc, capacity 128 (`saluki-core/src/topology/mod.rs:34`). + +## Pivotal counter-fact: the forwarder sheds, it does not wedge + +The obvious "load in, no load out" scenario — point ADP at a dead or slow intake and watch it back up — **does not reproduce a stall.** The endpoint loop's retry-queue push DROPS when full (`io.rs:588`, `push_low_priority` `io.rs:640`) and its input-recv arm is always enabled, so it keeps draining even when intake is down or the concurrency limit is saturated. Series are shed to the retry queue, not blocked. A backpressure-only liveness scenario is a dud. Any scenario design that relies on slow-intake backpressure to force a wedge is wrong. + +The stalls that ARE real, in order of how cheaply they can be demonstrated: + +1. **Boot gate — Core-Agent gRPC partition.** Steps 1-13 never start until boot completes. `get_hostname` retries only a bounded ~10 attempts (`lib/datadog-agent/commons/src/ipc/client/mod.rs`); registration and the post-connect health check were each historically added outside the retry scheme and made fatal. A partition outlasting the budget → ADP `exit(1)`, no load ever goes out. This is the freshest, most recurring family (current branch `blt/fix_hostname_fetch_retry`, SMPTNG-766). Deterministic trigger, boot-length window. +2. **First-connection TLS/entropy.** The first `dd_out` POST is ADP's first outbound TLS handshake. AWS-LC CPU-jitter entropy once SIGABRTed under Antithesis determinism (SMPTNG-762, platform-patched). Regression guard now. +3. **Killed, non-restarted component.** Root supervisor is `OneForOne` with **0 restarts** (`run.rs:212`). If aggregate/encode/forward dies, its bounded input channel fills and the awaited upstream send (steps 4, 7) blocks forever — ingest wedges while the UDS stays bound. Real wedge, but needs a fault that kills exactly one component task. +4. **Sustained backpressure.** Least reliable — see the counter-fact. Do not rely on it. + +## Aggregation failure-prone areas + +Single deterministic packets or one config value provoke these — no fault injection needed: + +- Counter NaN poisoning — `merge_scalar_sum` (`value/mod.rs:493`) has no NaN guard; one `|g NaN` sample poisons the window sum. Surfaces intake-side as Pyld20. +- `NaN`/`±Inf` forwarded as JSON `null` — `e955c5a44e`, deser accepts `"NaN".parse::()`. +- Sub-second aggregate window → divide-by-zero panic — `738b6874a9`. +- Sketch bin explosion at minimum sample rate → downstream builder panic — `bc3cc747e5`; the `sketchburst` driver already sweeps past `bin_limit`. +- `host:` tag treated as a normal tag → wrong context identity — `380ebef07d`. +- Timestamped `|T` count reinflated by `@rate` → count divergence — `f0a3687dcc`. +- v2 encoder ignored `max_series_points_per_payload` — `0025407262` / SMPTNG-765, still open. +- Non-monotonic flush wall-clock under clock jitter → out-of-order/duplicate timestamps — SMPTNG-767, in progress. clock_jitter is a global, near-free fault that triggers this. +- Type/rate change mid-window silently discarded — `value/mod.rs:436`, `:450`. + +## Concurrency model + +Actors over locks: components are tasks joined by bounded mpsc channels. Source dispatch and aggregate flush are `.await`ed inline in single-task `select!` loops, so downstream backpressure parks the upstream loop — the mechanism behind the killed-component wedge (stall #3). The intake capture state is an `Arc>` dedup map. + +## Cost model for Antithesis runs + +Per `test/antithesis/bin/launch.sh`, cost per run scales with: container count (state space per timeline), fault surface (node termination/hang/throttle on `FAULT_NODES`, plus global cpu_mod + clock_jitter, plus network faults), and duration (default 30 m). The two existing scenarios: + +- `general` — 3 containers, node faults on `agent` + both globals. The fault-heavy one. +- `differential` — 4 containers, no node faults. The expensive one by container count. + +Cheapest demonstrative profile: 3 containers, **no node faults**, keep only cpu_mod + clock_jitter (near-free globals, and clock_jitter is a real aggregation trigger), short duration. Drops the entire node-fault branching factor. + +## Assumptions + +- ADP dials the Core Agent at a hardcoded `https://127.0.0.1:{cmd_port}` (`lib/datadog-agent/commons/src/ipc/config.rs:209`), vsock the only alternative. There is no way to point ADP at a Core Agent in another container over TCP, so the ADP↔Core-Agent boot gRPC link cannot be network-partitioned by splitting containers. Boot-gate stall #1 is reachable only via timing faults on the converged container, already present in `general`. +- `standalone` ADP mode is vestigial (AGENTS.md) and skips the Core-Agent boot gRPC, so it cannot exercise stall #1. +- The one cross-container link a network fault can reach is ADP → intake. Partitioning it exercises recovery liveness (ADP survives, load resumes on heal), not a wedge. + +## Open questions + +- Can stall #3 (killed component) be triggered deterministically enough to be a cheap scenario, or only under node-hang faults? +- Does an intake-side per-run marker liveness probe belong in a no-fault scenario as a green-by-default regression anchor, given no current-code bug reproduces via it? diff --git a/test/antithesis/AGENTS.md b/test/antithesis/AGENTS.md index f158bd4668..fe9a4494cd 100644 --- a/test/antithesis/AGENTS.md +++ b/test/antithesis/AGENTS.md @@ -11,9 +11,12 @@ to build and submit Antithesis runs. Do not hand-type `snouty launch`. **launch.sh** `test/antithesis/bin/launch.sh ` builds the images, renders the compose -with concrete tags, and submits to the `persistent_storage` webhook with the node, -cpu, and clock fault profile the scenario's `launch.env` sets. See the script -header for env overrides such as DURATION and WEBHOOK. +with concrete tags, and submits to the unified `run_test` webhook under +`antithesis.source=datadog_agent` with the node, cpu, and clock fault profile the +scenario's `launch.env` sets. `datadog_agent` is both this project's fixed source, +which selects the "Datadog Agent" customizations, and its property-history bucket. +See the script header for env overrides such as DURATION, WEBHOOK, SOURCE, and the +new `SIMULTANEOUS_FAULTS` / `FORCE_DISABLE_ALL_FAULTS` toggles. **snouty validate** diff --git a/test/antithesis/bin/launch.sh b/test/antithesis/bin/launch.sh index 928beebe96..357d6c2652 100755 --- a/test/antithesis/bin/launch.sh +++ b/test/antithesis/bin/launch.sh @@ -15,7 +15,7 @@ # SCENARIO_DESCRIPTION human description; the git commit is appended # SCENARIO_FAULT_NODES space-separated SUT container names to node-fault; # empty means no node termination/hang/throttle at all -# SCENARIO_WEBHOOK optional; tenant webhook, default persistent_storage +# SCENARIO_WEBHOOK optional; tenant webhook, default run_test # # Required environment, read by snouty: # ANTITHESIS_TENANT tenant name @@ -27,8 +27,12 @@ # TEST_NAME= default SCENARIO_TEST_NAME # DESCRIPTION= default SCENARIO_DESCRIPTION; commit is appended # FAULT_NODES= default SCENARIO_FAULT_NODES -# WEBHOOK= default SCENARIO_WEBHOOK or persistent_storage -# SOURCE= property-history key; default is the git branch +# WEBHOOK= default SCENARIO_WEBHOOK or run_test +# SOURCE= antithesis.source, default datadog_agent +# SIMULTANEOUS_FAULTS= default false. Same node faults on all +# FAULT_NODES at once, no-op without FAULT_NODES +# FORCE_DISABLE_ALL_FAULTS= default false. Master switch over every +# fault above, runs with none # DRY_RUN=1 print the exact command and exit without submitting set -euo pipefail @@ -61,39 +65,51 @@ if [[ -n "$(git -C "$REPO_ROOT" status --porcelain 2>/dev/null)" ]]; then fi export ANTITHESIS_IMAGE_TAG="$GIT_SHA" -WEBHOOK="${WEBHOOK:-${SCENARIO_WEBHOOK:-persistent_storage}}" +WEBHOOK="${WEBHOOK:-${SCENARIO_WEBHOOK:-run_test}}" DURATION="${DURATION:-30}" TEST_NAME="${TEST_NAME:-${SCENARIO_TEST_NAME:?launch.env must set SCENARIO_TEST_NAME}}" DESCRIPTION="${DESCRIPTION:-$SCENARIO_DESCRIPTION} (commit ${GIT_SHA})" # May be empty: an empty node list means no node faults for this scenario. FAULT_NODES="${FAULT_NODES-$SCENARIO_FAULT_NODES}" +# run_test global fault toggles. Defaults match the endpoint defaults so a plain +# shot behaves as it did before the move off persistent_storage. +SIMULTANEOUS_FAULTS="${SIMULTANEOUS_FAULTS:-false}" +FORCE_DISABLE_ALL_FAULTS="${FORCE_DISABLE_ALL_FAULTS:-false}" -# Property-history key. Passing --source makes the run tracked, not ephemeral, -# so findings are produced and each property's history is grouped by this key. -# Default to the branch so history follows the branch; without it snouty runs -# ephemeral and no findings are available to triage. -SOURCE="${SOURCE:-$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)}" +# antithesis.source. On the unified run_test endpoint this one identifier both +# selects the datadog_agent customization block -- title "Datadog Agent" and the +# 20 GB VM bump -- and groups this project's property history under a single +# datadog_agent bucket. It also tracks the run rather than running it ephemeral. +# Without it snouty runs ephemeral and surfaces no findings to triage. Fixed to +# datadog_agent for every shot so all runs share one history and the right +# customizations. +SOURCE="${SOURCE:-datadog_agent}" -# Pinned fault profile, submitted to the persistent_storage endpoint. cpu_mod and +# Pinned fault profile, submitted to the run_test endpoint. cpu_mod and # clock_jitter are global and symmetric, so every scenario gets them; clock_jitter # is what exercises the AWS-LC CPU-jitter entropy path. Network faults stay on -# everywhere and heal before judging. +# everywhere and heal before judging. force_disable_all_faults rides at its false +# default so the shot is self-describing. Flip it to true to run one fault-free +# shot without editing anything else. # # Node termination, hang, and throttle apply only to the containers in FAULT_NODES, # so a scenario gets none by leaving it empty. The differential A/B does: node # termination, hang, or throttle on one SUT but not the other -- or on both at # different times -- drops data unevenly and manufactures a divergence that is a # fault artifact, not an ADP defect. All three are unfair for the same reason, so -# all three are gated together on FAULT_NODES. +# all three are gated together on FAULT_NODES. simultaneous_faults rides with them +# because it only means anything when node faults hit two or more containers. FAULTS=( --param custom.cpu_mod=true --param custom.clock_jitter=true + --param custom.force_disable_all_faults="$FORCE_DISABLE_ALL_FAULTS" ) if [[ -n "$FAULT_NODES" ]]; then FAULTS+=( --param custom.include_for_node_termination="$FAULT_NODES" --param custom.include_for_node_hang="$FAULT_NODES" --param custom.include_for_node_throttle="$FAULT_NODES" + --param custom.simultaneous_faults="$SIMULTANEOUS_FAULTS" ) fi