chore(antithesis): non-utf8 bytes in datagrams represent ~1% of generation - #2217
Conversation
dd-octo-sts
Bot
added
the
area/test
This comment has been minimized.
This comment has been minimized.
Binary Size Analysis (Agent Data Plane)Baseline: 557ae84 · Comparison: 7471989 · diff ✅ Binary size difference within thresholdChanges by Module
Detailed Symbol Changes |
blt
changed the title
![datadog-prod-us1-6[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The new injection is bounded to one byte per datagram, preserves the existing size and forwarding invariants, and excludes event and service-check lines from metric-name mutation. No diff-only behavioral regression was identified; runtime confirmation was blocked by unavailable Rust dependencies in the sandbox.
🤖 Datadog Autotest · Commit 43cf97d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
Regression Detector (Agent Data Plane)Run ID: Optimization Goals: ✅ No significant changes detectedFine details of change detection per experiment (5)Experiments configured
Bounds Checks: ✅ Passed (5)
ExplanationA change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression ( |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 43cf97d287
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
34de63f to
a827899
Compare
43cf97d to
86e0959
Compare
a827899 to
e45d94e
Compare
86e0959 to
793eced
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 793eced78f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
e45d94e to
cb1f9df
Compare
2a19b2e to
74d04b8
Compare
cb1f9df to
639f6b7
Compare
74d04b8 to
fdea3d5
Compare
39f3b2a to
a8467bf
Compare
fdea3d5 to
52ca146
Compare
a8467bf to
75b7b70
Compare
7fc724f to
8d1cf98
Compare
966965a to
2c757ed
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d1cf982d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
8d1cf98 to
5f2c894
Compare
3906c44 to
085e47a
Compare
5f2c894 to
0f55ac4
Compare
There was a problem hiding this comment.
Same deal here as the other PR: we should fix this, so things pass... and I'll look at exclude these crates out-of-band.
tobz
left a comment
tobz
left a comment
There was a problem hiding this comment.
Woops, did not mean to approve yet. 😅
0f55ac4 to
667dd4b
Compare
9f4124e to
c920eaa
Compare
667dd4b to
377fcf0
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 377fcf0f54
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let context = if non_utf8 { | ||
| Context::mint_non_utf8_within(kind, rng, budget) | ||
| } else { | ||
| Context::mint_within(kind, rng, budget) | ||
| } | ||
| .with_context(|| { |
There was a problem hiding this comment.
When any of these six initial mints exhausts its random probe retries, Context::mint_within or mint_non_utf8_within returns None even though the datagram budget is valid, but .with_context(...) misclassifies that transient exhaustion as a configuration contradiction. The /contexts route converts the error to HTTP 500, and the new Fetched::Refused path then aborts the driver invocation instead of retrying, so ordinary unlucky generation can fail an Antithesis command; retry the complete seed operation or distinguish probe exhaustion from a genuinely unaffordable budget.
Useful? React with 👍 / 👎.
c920eaa to
604b4cf
Compare
377fcf0 to
8644d54
Compare
The essential property we want to determine is whether ADP-on/ADP-off emits payloads to intake API that the intake API rejects. This is especially of interest for inputs that we know are ultimately rejected by intake API -- non-utf8 bytes in the wrong spot -- but are _not_ rejected by the SUT. The old mechanism had a feral/clean 'vibe' which served for a while but was confusing to debug. Is feral malformed? Is feral well-formed but wild? Anyway I got tired of it. There's now a predicate which defines whether a payload is well-formed or not -- that is, accepted by ADP-off Datadog Agent -- independent of whether intake API ultimately accepts the payloads that ingress inspires. Later I will build a generator that only emits malformed ingress but that is only hinted at in this work.
604b4cf to
557ae84
Compare
8644d54 to
7471989
Compare
There was a problem hiding this comment.
A selected corrupt pull prepends the same invalid identity to every datagram, so a 10,000-datagram invocation can poison consecutive backend flushes rather than limiting rejection to roughly 1%. Initial pool seeding can also turn rare generator exhaustion into an immediate command failure.
🤖 Datadog Autotest · Commit 7471989 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| let mut stats = DatagramStats::default(); | ||
| let mut line = Vec::new(); | ||
| loop { | ||
| if let Some(lead) = pull.lead { |
There was a problem hiding this comment.
Long carrying invocations can recreate the near-total intake rejection rate this PR is intended to prevent, reducing useful differential-test coverage.
Assertion details
- Input: A pull containing a non-UTF-8 metric-name context with a large configured
datagram_count. - Expected:
Non-UTF-8 datagrams should be distributed so approximately 1% of generated traffic exercises the behavior without contaminating every downstream flush during one long invocation. - Actual:
Pull::newrecords one invalid identity andwrite_datagramprepends it wheneverleadis set. Because the driver reuses that pull for every datagram in the invocation, a selected pull corrupts the entire consecutive batch—up to 10,000 datagrams—not a dispersed 1% sample. Strict downstream name decoding can therefore reject every aggregation payload produced during that streak. Fixing this requires choosing corruption per datagram or budgeting it at the downstream-flush level while reusing a stable pooled corrupt identity; the tests that currently require every datagram from a carrying pull to be corrupt must be updated accordingly.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| } else { | ||
| Context::mint_within(kind, rng, budget) | ||
| } | ||
| .with_context(|| { |
There was a problem hiding this comment.
Ordinary random generation can fail an Antithesis command before any SUT behavior is exercised, producing false failures and lost load.
Assertion details
- Input: The first
/contextsrequest when any clean or non-UTF-8 seed exhausts its bounded random mint attempts despite a valid budget. - Expected:
A valid datagram budget should survive transient mint exhaustion and eventually seed both halves; only a proven unaffordable budget should become a permanent refusal. - Actual: Each of the six initial seed calls returns
Noneboth when the byte budget is impossible and when its bounded random probe loop merely exhausts..with_context(...)converts both cases into the same error;/contextsreturns HTTP 500, and the newFetched::Refusedpath immediately aborts the driver rather than retrying. The fix must distinguish permanent budget failure from transient probe exhaustion, retry only the latter, and preserve the permanent configuration error path across the mint API, pool, and fetch handling.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
3 participants
Summary
ADP and Datadog Agent deal with non-utf8 bytes differently. The way that
intake accepts non-utf8 bytes varies depending on where in the payload they
appear and for which protocol version. Previously I created datagrams that were
too non-utf8 heavy, meaning most payloads could be rejected by intake. This
commit changes the generator to emit non-utf8 bytes into about 1% of datagrams.
Change Type
How did you test this PR?
References