chore(antithesis): Bound contexts in runs - #2216
Conversation
dd-octo-sts
Bot
added
the
area/test
Binary Size Analysis (Agent Data Plane)Baseline: 8326377 · Comparison: 604b4cf · diff ✅ Binary size difference within thresholdChanges by Module
Detailed Symbol Changes |
This comment has been minimized.
This comment has been minimized.
![datadog-datadog-prod-us1-2[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sampling three independent per-kind maxima permits the intake to retain up to 3,000,000 contexts, rather than the stated 1,000,000 total bound, so long runs can still exceed the memory and cardinality budget this change is intended to enforce.
🤖 Datadog Autotest · Commit 34de63f · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
Regression Detector (Agent Data Plane)Run ID: Optimization Goals: ✅ No significant changes detectedFine details of change detection per experiment (5)Experiments configured
Bounds Checks: ✅ Passed (5)
ExplanationA change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression ( |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34de63f221
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
e4ff7fa to
f89d2be
Compare
34de63f to
a827899
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a82789926a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
a827899 to
e45d94e
Compare
f89d2be to
07e1995
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e45d94e437
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
07e1995 to
b943695
Compare
e45d94e to
cb1f9df
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cb1f9df376
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
b943695 to
a6dd29d
Compare
cb1f9df to
639f6b7
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 639f6b7fee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
639f6b7 to
39f3b2a
Compare
212be47 to
57f3d76
Compare
f0d03a5 to
83d5808
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 83d5808c54
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
dc91ee2 to
966965a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 966965a016
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
966965a to
2c757ed
Compare
0e9a907 to
4bcdae0
Compare
2c757ed to
3906c44
Compare
c33ffb4 to
eaf9e7e
Compare
3906c44 to
085e47a
Compare
tobz
left a comment
tobz
left a comment
There was a problem hiding this comment.
This all seems pretty reasonable to me. 👍🏻
eaf9e7e to
3088cdc
Compare
085e47a to
f7989ab
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f7989abc95
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| loop { | ||
| if let Some(contexts) = try_fetch_contexts(&client, &url) { | ||
| return Some(contexts); |
There was a problem hiding this comment.
When the intake processes a GET /contexts request but a network fault drops or corrupts its response, this loop retries the same mutating endpoint without a request identifier. Each retry calls Pool::serve again and can mint another n contexts, so one driver invocation can consume a large portion of the shared cap with identities it never renders; after enough retries, later drivers recur over a large mostly-unobserved pool instead of producing the configured variation. Cache responses by an invocation/request ID or otherwise make retries reuse the original allocation.
AGENTS.md reference: test/antithesis/AGENTS.md:L86-L90
Useful? React with 👍 / 👎.
3088cdc to
71a9d5c
Compare
f7989ab to
9f4124e
Compare
9f4124e to
c920eaa
Compare
There was a problem hiding this comment.
More details
The new shared context protocol is internally consistent under static adversarial review: request bounds, startup retries, binary round trips, pooled identity caps, and datagram budget arithmetic show no clear diff-only regression. Targeted Rust tests could not execute because Cargo requires the uncached hyper-http-proxy Git dependency and sandbox Git access returned HTTP 403.
🤖 Datadog Autotest · Commit 9f4124e · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c920eaaa43
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| else { | ||
| break; | ||
| }; | ||
| if seen.insert(digest(&context)) { |
There was a problem hiding this comment.
Fresh evidence after the earlier duplicate-identity comment is that the new seen set hashes the raw Context: when two metric mints have the same name, kind, and tags in different orders or with duplicate tags, both hashes differ and consume cap slots, while capture::tagset_with_host canonicalizes those tags into a BTreeSet, making them one observed identity. This occurs increasingly as a large metric pool draws from the small common name/tag vocabulary, so the run can stop minting with substantially fewer distinct observed contexts than metric_contexts; deduplicate using the parser/output-normalized metric identity instead.
AGENTS.md reference: test/antithesis/AGENTS.md:L86-L90
Useful? React with 👍 / 👎.
| // kind would stop minting early and the run would explore fewer identities than configured. | ||
| // Remint on a duplicate instead of pushing it. | ||
| let mut minted = None; | ||
| if pool.len() < cap { |
There was a problem hiding this comment.
When metric_contexts samples 1 and the first metric has one of the deliberately generated 351- or 400-byte names, Context::mint_within accepts it because the Agent parser forwards it, but capture::metric_name_kept rejects every resulting series at the 350-byte intake limit. This condition then prevents any further metric minting, so all later metric requests recur over the permanently unobservable identity and the timeline records no metric context for the intended multi-point comparison; retain at least one backend-observable identity in tight pools while mixing drop-boundary contexts separately.
AGENTS.md reference: test/antithesis/AGENTS.md:L86-L90
Useful? React with 👍 / 👎.
| MetricType::Timing, | ||
| MetricType::Histogram, |
There was a problem hiding this comment.
Fresh evidence beyond the earlier shared-total-cap comment is that each sampled timing or histogram input expands after aggregation into the default max, median, avg, count, and 95percentile output series (lib/saluki-components/src/transforms/aggregate/config.rs lines 113-119). The pool nevertheless charges either type as one metric_contexts slot, so a metric-heavy pool near the one-million ceiling can produce several million distinct observed series despite the advertised run-wide bound; account for this fan-out when allocating the metric cap or define the cap over the expanded output identities.
Useful? React with 👍 / 👎.
This commit introduces a new protocol to the intake node to allow for context bounding in runs. The purposes are twofold. First, while it's interesting to explore timelines in which there are too many contexts for either lane to handle these are less relevant to me _now_, so I've capped total contexts at 1M. Second, in order for timeseries equality to _work_ I need to be certain that I have at least _some_ timeseries with more than one point emitted into them. Previously drivers were emitting essentially random contexts and it was very unlikely for more than one point to be emitted per context. Oops. Drivers now request N contexts from the intake service for each sub-kind of dogstatsd they emit, meaning all contexts are sourced from a single spot in a topology and not from pure randomness.
c920eaa to
604b4cf
Compare
## Summary <!-- Please provide a brief summary about what this PR does. This should help the reviewers give feedback faster and with higher quality. --> This commit introduces a new protocol to the intake node to allow for context bounding in runs. The purposes are twofold. First, while it's interesting to explore timelines in which there are too many contexts for either lane to handle these are less relevant to me _now_, so I've capped total contexts at 1M. Second, in order for timeseries equality to _work_ I need to be certain that I have at least _some_ timeseries with more than one point emitted into them. Previously drivers were emitting essentially random contexts and it was very unlikely for more than one point to be emitted per context. Oops. Drivers now request N contexts from the intake service for each sub-kind of dogstatsd they emit, meaning all contexts are sourced from a single spot in a topology and not from pure randomness. ## Change Type - [ ] Bug fix - [ ] New feature - [x] Non-functional (chore, refactoring, docs) - [ ] Performance ## How did you test this PR? <!-- Please how you tested these changes here --> ## References <!-- Please list any issues closed by this PR. --> <!-- - Closes: <issue link> --> <!-- Any other issues or PRs relevant to this PR? Feel free to list them here. --> 557ae84
3 participants
Summary
This commit introduces a new protocol to the intake node to allow for
context bounding in runs. The purposes are twofold. First, while it's
interesting to explore timelines in which there are too many contexts
for either lane to handle these are less relevant to me now, so I've
capped total contexts at 1M. Second, in order for timeseries equality
to work I need to be certain that I have at least some timeseries
with more than one point emitted into them. Previously drivers were
emitting essentially random contexts and it was very unlikely for more
than one point to be emitted per context. Oops.
Drivers now request N contexts from the intake service for each sub-kind
of dogstatsd they emit, meaning all contexts are sourced from a single
spot in a topology and not from pure randomness.
Change Type
How did you test this PR?
References