chore(antithesis): Alter dogstatsd generation to is_malformed regime - #2215
Conversation
Our previous approach to generating dogstatsd load for use in the Antithesis rig was very Datadog/lading inspired: generate something that is correct-by-construction and then bias the result to be 'interesting'. This works well but the biasing systematically avoids conditions that are not-malformed -- that is, accepted by the SUT -- but are demonstrative of goofy behavior. This commit changes the generator to be a hybrid of that approach with a state search approach. We define a predicate `is_malformed` that confirms whether or not a payload will be accepted by the SUT and then payloads which are not malformed, however strange, are transmitted.
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
dd-octo-sts
Bot
added
the
area/test
|
Binary Size Analysis (Agent Data Plane)Baseline: e34c659 · Comparison: e4ff7fa · diff ✅ Binary size difference within thresholdChanges by Module
Detailed Symbol Changes |
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The generator redesign did not reveal a clear diff-only behavioral regression. Adversarial cases covered delimiter-bearing fields, packed values, optional fields, framing, fallback sizing, and malformed-input repair; harness execution was blocked by the sandbox's unavailable hyper-http-proxy Git dependency.
🤖 Datadog Autotest · Commit e4ff7fa · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
Regression Detector (Agent Data Plane)Run ID: Optimization Goals: ✅ No significant changes detectedFine details of change detection per experiment (3)Experiments configured
Bounds Checks: ✅ Passed (3)
ExplanationA change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression ( |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e4ff7fa8ae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// forwards. A packed datagram must be entirely well-formed. | ||
| #[test] | ||
| fn write_line_stays_within_its_limit(seed: u64, limit_bytes: u16, vibe in any_vibe()) { | ||
| fn every_payload_is_well_formed(seed: u64) { |
There was a problem hiding this comment.
Rename this proptest and the newly added never_panics proptest to include the literal property_test_ prefix. The filters in Makefile:573-579 exclude these names from the release-mode test-property target and instead run them only in the debug-mode test target, so the new randomized coverage is assigned to the wrong CI job.
AGENTS.md reference: AGENTS.md:L39-L41
Useful? React with 👍 / 👎.
1 participant
Summary
Our previous approach to generating dogstatsd load for use in the
Antithesis rig was very Datadog/lading inspired: generate something
that is correct-by-construction and then bias the result to be
'interesting'. This works well but the biasing systematically avoids
conditions that are not-malformed -- that is, accepted by the SUT --
but are demonstrative of goofy behavior. This commit changes the
generator to be a hybrid of that approach with a state search approach.
We define a predicate
is_malformedthat confirms whether or nota payload will be accepted by the SUT and then payloads which are not
malformed, however strange, are transmitted.
Change Type
How did you test this PR?
References