DataDog · blt · Jul 17, 2026 · Jul 16, 2026
@@ -0,0 +1,69 @@
+---
+sut_path: /home/ssm-user/src/saluki
+commit: 93b9f37baf07004a9235f308305d78ef3d7fc226
+updated: 2026-07-15
+external_references:
+  - path: https://datadoghq.atlassian.net/browse/SMPTNG-737
+    why: Antithesis triage umbrella
+  - path: test/antithesis/bin/launch.sh
+    why: authoritative fault profile and cost knobs
+---
+
+# Deployment topology — cheap demonstrative scenarios
+
+Every container and every fault axis is a cost. This document proposes minimal topologies for the target defect families and states the cost delta versus the existing `general` scenario.
+
+## Cost baseline
+
+- `general`: 3 containers (intake, agent [ADP-on SUT], workload) + node term/hang/throttle on `agent` + global cpu_mod + clock_jitter + network faults. 30 m default.
+- `differential`: 4 containers, no node faults. Expensive by container count.
+
+The dominant cost knob is the **node-fault branching factor** on `FAULT_NODES`. Dropping it collapses the timeline tree. cpu_mod and clock_jitter are global and cheap and one of them (clock_jitter) is a real aggregation trigger, so keep both.
+
+## Proposal A — `aggregation` (P0, recommended first build)
+
+The cheapest genuinely demonstrative scenario. Reuses the converged `general` topology, drops node faults.
+
+- Containers: 3 — intake, agent (converged, ADP-on), workload. Same as `general`.
+- `SCENARIO_FAULT_NODES=""` — **no node faults**. Keep global cpu_mod + clock_jitter.
+- Duration: short (10 m is plenty; single-packet triggers converge fast).
+- Drivers: existing `send_dogstatsd` + `sketchburst`, plus new marker drivers for `flush-liveness-marker`, `counter-nan-poisoning`, `counter-sum-oracle`.
+- New instrumentation: one `eventually_flush_marker` harness command; one value-oracle `assert_always` in `series_observation.rs`.
+- Demonstrates: the aggregation family the user named (NaN, sum, sketch, points-per-payload, clock-monotonic under clock_jitter) AND the green-by-default flush-liveness anchor.
+- Cost delta vs `general`: removes the entire node-fault branching factor and 2/3 of the duration. Substantially cheaper per run, higher aggregation signal.
+
+Why this over a "slow intake stall" scenario: the forwarder sheds rather than wedges (sut-analysis counter-fact), so backpressure cannot demonstrate a stall. The honest cheap liveness contribution here is a regression anchor, not a live-bug hunt.
+
+## Proposal B — `boot-partition` (P0, freshest-bug guard)
+
+Directly demonstrates the recurring boot-gate stall family (SMPTNG-766, current branch). Not reusable from `general` because the converged image cannot partition ADP from its own Core Agent.
+
+- Containers: 3 — a standalone-ish ADP, a Core-Agent gRPC endpoint (full Agent, or a minimal gRPC stub serving hostname/registration/health), and intake.
+- Faults: network faults on the ADP↔Core-Agent link, healed before judging. cpu_mod + clock_jitter global. No node term/hang.
+- Duration: boot-length; short.
+- Demonstrates: `boot-liveness-under-partition` — the exact family just fixed on this branch. Strong regression guard.
+- Cost: cheap in compute (3 containers, short, network-fault-only). More build effort — needs the split topology and possibly a gRPC stub.
+
+## Proposal C — leave `differential` alone
+Host-tag identity and cross-lane divergence stay in `differential`. Not cheap; out of scope for this pass.
+
+## Blocker found while de-risking Proposal B
+
+Proposal B as first scoped — split ADP and Core Agent into two containers and network-partition the boot gRPC link — is **infeasible**. ADP dials the Core Agent at a hardcoded `https://127.0.0.1:{cmd_port}` (`lib/datadog-agent/commons/src/ipc/config.rs:209`); the only alternative transport is vsock (host/hypervisor CID). No config points ADP at a Core Agent in another container over TCP. The link is always loopback inside one container, where Antithesis inter-container network faults do not reach. The SMPTNG-766 boot-gate family is therefore not provoked by an inter-container partition — it is provoked by timing faults (cpu_mod, clock_jitter, node-hang) starving the co-located Core Agent past ADP's bounded ~10-attempt retry budget, ingredients that already live in `general`.
+
+The genuinely partition-able cross-container link is ADP → **intake** (the HTTP forwarder). Partitioning it does not wedge ADP (forwarder sheds), but it does let us assert recovery liveness: ADP survives, and load resumes flowing out once the partition heals.
+
+## Decision — built the `liveness` scenario
+
+Goal per the user: demonstrate ADP's liveness is not harmed, cheaply. Built a lean `liveness` scenario:
+
+- Containers: 3 — intake, agent (converged ADP-on SUT), workload. Mirrors `general`.
+- `SCENARIO_FAULT_NODES=""` — no node faults. The launcher's always-on network faults partition the ADP→intake link and heal; global cpu_mod + clock_jitter stay on. This is the cheap adversary.
+- Duration: short.
+- Liveness claims, both green-by-default `always`: (1) `eventually_adp_alive` — ADP boots and stays reachable (reused unchanged); (2) `eventually_flush_marker` — a marker counter sent on the DogStatsD socket eventually reaches intake at `GET /antithesis/metrics/adp`, proving load in ⇒ load out end-to-end after faults quiesce.
+
+The `general` intake records the ADP-on agent's POSTs under the `adp` lane (single-listener `else` branch, `intake/src/bin/intake.rs:96`), so the marker-presence probe reads `/antithesis/metrics/adp` directly.
+
+## Open questions
+- Confirm the short duration gives Antithesis enough timelines for a network partition to inject and heal before the `eventually_` fault-quiet judging window.
+- Adding node termination on `agent` would additionally demonstrate restart liveness, at the cost of the node-fault branching factor. Deferred to keep this scenario cheap.
@@ -0,0 +1,70 @@
+---
+sut_path: /home/ssm-user/src/saluki
+commit: 93b9f37baf07004a9235f308305d78ef3d7fc226
+updated: 2026-07-15
+external_references:
+  - path: https://datadoghq.atlassian.net/browse/SMPTNG-737
+    why: Antithesis triage umbrella
+---
+
+# Existing Antithesis SDK assertions
+
+The codebase is already instrumented on both sides. SUT-side assertions use the `saluki_antithesis` facade (`lib/saluki-antithesis/src/lib.rs`) — a thin, default-off macro wrapper over the SDK, with numeric helpers `always_ge!`/`always_le!`. Intake/harness-side use the raw SDK. Only relevant entries for the liveness/aggregation targets are listed.
+
+## SUT-side — boot liveness (covers stall family #1)
+
+| File:line | Type | Message |
+|---|---|---|
+| `bin/agent-data-plane/src/main.rs:91` | reachable | agent-data-plane completed bootstrap |
+| `bin/agent-data-plane/src/main.rs:153,197` | always_or_unreachable | bootstrap-stage guards |
+| `internal/env/host.rs:45` | always_or_unreachable | ADP resolved the Core Agent hostname at boot |
+| `internal/env/mod.rs:75,91,100` | always | host / workload / autodiscovery providers connected |
+| `internal/remote_agent.rs` | (classify) | initial registration permanent-vs-transient split |
+
+These already catch the boot-gate stall. A fatal boot `exit(1)` trips no Antithesis platform crash property, so these `always`/`always_or_unreachable` anchors plus the harness `eventually_adp_alive` are the only things that catch it.
+
+## SUT-side — aggregation state invariants (covers half of the aggregation target)
+
+| File:line | Type | Message |
+|---|---|---|
+| `transforms/aggregate/mod.rs:612` | always_le | context map ≤ context_limit |
+| `transforms/aggregate/mod.rs:646` | always_ge | flush wall-clock never moves backward (SMPTNG-767) |
+| `transforms/aggregate/mod.rs:654` | always_le | zero-value bucket span ≤ 10,000 |
+| `transforms/aggregate/mod.rs:672` | sometimes | (flush liveness marker) |
+| `transforms/aggregate/mod.rs:693` | always_le | counter-expiry add no overflow |
+| `transforms/aggregate/config.rs:68,69` | always_ge/le | histogram percentile quantile in [0,1] |
+| `lib/ddsketch/src/agent/sketch.rs:231` | always | DDSketch sample is finite at insert |
+| `lib/ddsketch/src/agent/sketch.rs:241` | always_le | DDSketch min ≤ max after insert |
+| `lib/ddsketch/src/agent/sketch.rs:784,785` | reachable/always_le | bin collapse reached; bin count within bin_limit |
+
+## SUT-side — data path / stall mechanics
+
+| File:line | Type | Message |
+|---|---|---|
+| `sources/dogstatsd/mod.rs:726,1240` | always_le | receive-buffer / size bounds |
+| `sources/dogstatsd/mod.rs:1958-2001` | unreachable | dispatch output missing / failed mid-buffer |
+| `topology/interconnect/dispatcher.rs:93` | sometimes | dispatch backpressure observed |
+| `common/datadog/io.rs:719,749` | sometimes | forwarder send outcomes |
+| `net/util/retry/queue/mod.rs:288,300` | sometimes/always_le | retry queue dropped-oldest / bound |
+| `pooling/elastic.rs:282`, `fixed.rs:189` | unreachable | pool semaphore closed |
+| `core/runtime/restart.rs:159` | always_or_unreachable | restart-strategy guard |
+| `core/health/mod.rs:638` | always_or_unreachable | health component guard |
+
+## Intake / harness-side
+
+| File:line | Type | Message |
+|---|---|---|
+| `harness/src/bin/eventually_adp_alive.rs:78` | assert_always | ADP booted: API reachable and DogStatsD socket present |
+| `intake/src/series_observation.rs:61` | assert_reachable | intake.first_series_observed (coarse boot-vs-flushed signal) |
+| `intake/src/properties/payload/point.rs:25` | assert_always | Pyld20 value not NaN |
+| `intake/src/properties/payload/point.rs:46` | assert_always | Pyld21 timestamp future bound |
+| `intake/src/properties/payload/metric_payload.rs:16,27` | assert_always | Pyld07 decode, Pyld08 point count |
+| `intake/src/properties/payload/envelope.rs:19` | assert_always | Pyld01-03 envelope |
+| `scenarios/general/src/bin/parallel_driver_send_dogstatsd.rs` | reachable/sometimes | workload ran / sent a batch |
+| `scenarios/general/src/bin/parallel_driver_sketchburst.rs:56` | reachable | sketchburst swept the sketch bins |
+| `scenarios/general/src/bin/parallel_driver_poll_stats.rs:67,76` | reachable/sometimes | stats poll / overlap |
+
+## What is MISSING for the target properties
+
+- **Flush-stall liveness ("load in, no load out").** No per-run unique marker probe. `series_observation.rs:61` is only a coarse first-series-ever signal. The intake `Context` (`capture.rs:80`) records identity only, keyed in a dedup `BTreeMap` with `first_seen` — the plumbing to check "a specific marker arrived within a window" exists, but no command uses it. This is the sound signal and it is unbuilt.
+- **Aggregation-value oracle.** All intake payload asserts are structural (shape/domain). None reads the flushed point VALUE against an expected sum for a known input. `Context` stores no value, so a "counter sum equals N increments" check needs a small addition in `series_observation.rs evaluate_series`, keyed on a per-run marker name.
@@ -0,0 +1,38 @@
+---
+sut_path: /home/ssm-user/src/saluki
+commit: 93b9f37baf07004a9235f308305d78ef3d7fc226
+updated: 2026-07-16
+external_references:
+  - path: https://github.com/DataDog/saluki/pull/2112#issuecomment-4993034650
+    why: the contested boot-retry fix; states the exact gap this note resolves
+  - path: https://datadoghq.atlassian.net/browse/SMPTNG-766
+    why: boot-fatal on transient Core-Agent gRPC error, the fix under review
+---
+
+# Finding: "ADP is online and able to service load" is not soundly assertable
+
+## The question this answers
+
+PR #2112 makes ADP's boot-time Core-Agent gRPC calls (hostname, registration, health) retry in the background instead of blocking boot and crashing on timeout. A reviewer asks why background-retry over block-then-crash. The unresolved half of the PR is whether the background-retry path **harms liveness** — whether ADP can proceed past boot into a state where it accepts load but cannot service it. This research set out to build a cheap Antithesis scenario demonstrating liveness is unharmed. The conclusion is that no such sound assertion exists. That is the finding, not a failure to build one.
+
+## Why no sound assertion exists
+
+1. **Reachable is not servicing.** The only liveness assertion available, `eventually_adp_alive`, checks a TCP connect to the API on :5100 and the presence of the DogStatsD socket file. The API server and the socket listener are separate tasks from the data pipeline. Both bind and answer while the pipeline is wedged or half-initialized. So the assertion proves the listeners came up, exactly the "boot faults resolved" half the PR already demonstrates. It cannot see whether load is serviced end to end.
+
+2. **Servicing load is not an invariant under the fault model.** A liveness property is only soundly assertable when the good thing is guaranteed to eventually happen. "Load goes out" is not guaranteed. The forwarder sheds — it drops to the retry queue rather than blocking (`common/datadog/io.rs:588`, `:640`) — so under an always-on network partition, ADP correctly does not flush. A missing flush is legitimate behavior, not a wedge. An `always`/`eventually` "load went out" assertion therefore false-REDs on healthy ADP whenever a partition or a clock_jitter/cpu_mod-delayed flush is in play. This is the throughput-liveness-gauge pitfall. There is no green-by-default form of it.
+
+3. **The failing link is loopback, so the fault cannot be isolated.** ADP dials the Core Agent at a hardcoded `https://127.0.0.1:{cmd_port}` (`lib/datadog-agent/commons/src/ipc/config.rs:209`). The Core-Agent↔ADP boot gRPC lives inside one converged container. Antithesis inter-container network faults do not reach it. You can fault the whole container — pausing Core Agent and ADP together — or the irrelevant ADP→intake link, but you cannot inject a partition on the exact link that causes the known problem, as an independent variable, in a compose topology. So even a scenario that could observe servicing could not attribute a result to the boot Core↔ADP fault.
+
+4. **The boot edge is ambiguous by construction.** Background-retry means boot "completes" and the listeners bind before a backgrounded dependency has necessarily resolved. Part of the boot sequence blocks and part is backgrounded, and per the PR author it is unclear why. There is no single observable edge that means "fully initialized and servicing," so there is nothing to anchor a readiness assertion on.
+
+## What follows
+
+- `eventually_adp_alive` is the ceiling of what is soundly assertable here: the listeners recover. That is a genuine liveness property because listener-binding IS guaranteed to recover in the fault-quiet window.
+- "ADP is servicing load under boot-time Core↔ADP faults" is not soundly assertable with the current architecture. Not for want of a scenario — the architecture provides no sound observable, and the fault model makes servicing conditional rather than invariant.
+- This does not say the background-retry fix is wrong. It says Antithesis cannot adjudicate the liveness question the reviewer raised. The block-then-crash alternative has the advantage that it removes the ambiguous half-alive state entirely, which is the state we cannot assert against.
+
+## What would change this
+
+Any of these would create a sound assertion, at a cost:
+- A single observable "fully initialized and servicing" edge in ADP (collapse the split blocking/backgrounded boot into one readiness gate), giving a real edge to anchor an `always`-recovers assertion on.
+- A servicing signal that is invariant under the fault model — e.g. an ADP self-report that ingest→aggregate→encode advanced within a window, read SUT-side, independent of whether the forwarder reached a sink. This asserts internal forward progress without depending on load actually leaving, sidestepping the shed-vs-wedge ambiguity. Worth scoping if the liveness question must be answered.