-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Add baseline Dynamic Tests for CNM #54938
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
AlexandreYang
wants to merge
55
commits into
main
Choose a base branch
from
alex/baseline_dynamic_test_for_cnm
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 2 commits
Commits
Show all changes
55 commits
Select commit
Hold shift + click to select a range
778ceec
empty
AlexandreYang cbab65c
Add baseline CNM dynamic tests
AlexandreYang c5e261b
Address baseline dynamic test review
AlexandreYang 47147ad
Harden baseline dynamic test wiring
AlexandreYang 8022154
[review-fix 3] Refresh generated Bazel metadata
AlexandreYang 412f580
[review-fix 4] Satisfy Network Path lint
AlexandreYang 757b8ce
Focus baseline Dynamic Tests E2E coverage
AlexandreYang 235cc7a
[review-fix 1] Preserve traceroute defaults and hide baseline flag
AlexandreYang 9aac167
Split host traffic Dynamic Tests suites
AlexandreYang 25ed7aa
[review-fix 1] Clarify effective Dynamic Tests state
AlexandreYang f093369
[review-fix 1] Preserve baseline selections across windows
AlexandreYang 46cde90
[review-fix 2] Retry one-shots under backpressure
AlexandreYang 3cb688c
[review-fix 3] Clarify platform and E2E assumptions
AlexandreYang 99403ac
[review-fix 4] Publish config and remove external E2E pull
AlexandreYang ec8e923
[review-fix 5] Keep baseline rollout setting internal
AlexandreYang 239d15e
Use recurring scheduling for baseline path tests
AlexandreYang 9a43c6d
Simplify baseline dynamic test configuration
AlexandreYang acff989
Simplify baseline selector design
AlexandreYang b1d70ae
Remove unnecessary baseline saturation handling
AlexandreYang 8d3c2b3
Simplify baseline Dynamic Tests activation
AlexandreYang c1cd703
Require explicit traceroute for baseline tests
AlexandreYang 1dc2cfb
Simplify baseline Dynamic Tests activation
AlexandreYang d749ade
Extract baseline selector package
AlexandreYang 86e2828
Reuse path test hash in baseline selector
AlexandreYang 0e3b8fd
Rename collector baseline tests
AlexandreYang fb3f6e1
Simplify baseline Dynamic Tests telemetry
AlexandreYang 282cde5
Simplify baseline ranking signals
AlexandreYang 610aa4d
Simplify baseline path selection
AlexandreYang 2d1d8c7
Simplify baseline candidate selection
AlexandreYang a98bf96
Organize baseline collector logic
AlexandreYang 34fde89
Group baseline connection signals
AlexandreYang c9d8cc3
Revert unrelated event forwarding refactor
AlexandreYang e9898c4
Simplify CNM baseline release note
AlexandreYang 136f119
Simplify CNM baseline scheduling setup
AlexandreYang d3c3e68
Unify network path test scheduling
AlexandreYang ca027f0
Restore network path subnet lookup
AlexandreYang c5ec567
Separate baseline path scheduling
AlexandreYang ab2d7ab
Revert "Separate baseline path scheduling"
AlexandreYang 79f5be4
Clarify network path scheduling modes
AlexandreYang aecda3c
Clarify standard path scheduling helper
AlexandreYang 5461d8c
Group network path scheduling helpers
AlexandreYang 8325dd8
Explain baseline candidate selection
AlexandreYang c650133
Move baseline ranking derivation to collector
AlexandreYang 8525a1e
Clarify baseline scheduling mode
AlexandreYang b225e69
Generalize network path connection signals
AlexandreYang adfc8be
[review-fix 1] Preserve baseline RC provenance
AlexandreYang 9ac5b16
[review-fix 2] Keep RC filters standard-only
AlexandreYang 8afc6af
Revert "[review-fix 2] Keep RC filters standard-only"
AlexandreYang db7b106
[review-fix 1] Isolate baseline filters from Dynamic RC
AlexandreYang 97ee4b7
Revert "[review-fix 1] Isolate baseline filters from Dynamic RC"
AlexandreYang 6c21596
Clarify baseline and Dynamic RC compatibility
AlexandreYang 00e11c5
Place baseline RC contract at scheduling boundary
AlexandreYang dc92f11
Document RC provenance at baseline selection
AlexandreYang 1304876
[review-fix 1] Restore resolver symlinks verbatim
AlexandreYang 3f6a287
[review-fix 2] Fall back from stale resolver links
AlexandreYang File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,195 @@ | ||
| // Unless explicitly stated otherwise all files in this repository are licensed | ||
| // under the Apache License Version 2.0. | ||
| // This product includes software developed at Datadog (https://www.datadoghq.com/). | ||
| // Copyright 2026-present Datadog, Inc. | ||
|
|
||
| package npcollectorimpl | ||
|
|
||
| import ( | ||
| "encoding/binary" | ||
| "math" | ||
| "sort" | ||
|
|
||
| "github.com/cespare/xxhash/v2" | ||
|
|
||
| "github.com/DataDog/datadog-agent/comp/networkpath/npcollector/impl/common" | ||
| npmodel "github.com/DataDog/datadog-agent/comp/networkpath/npcollector/model" | ||
| ) | ||
|
|
||
| const ( | ||
| baselineDiagnosticCandidates = 128 | ||
| baselineHealthyCandidates = 128 | ||
| baselineSelectionsPerWindow = 3 | ||
| ) | ||
|
|
||
| type baselineCandidate struct { | ||
| pathtest common.Pathtest | ||
| hash uint64 | ||
| timeout bool | ||
| count uint64 | ||
| rttVar uint64 | ||
| } | ||
|
|
||
| type baselinePool struct { | ||
| capacity int | ||
| items map[uint64]*baselineCandidate | ||
| } | ||
|
|
||
| func newBaselinePool(capacity int) baselinePool { | ||
| return baselinePool{capacity: capacity, items: make(map[uint64]*baselineCandidate, capacity)} | ||
| } | ||
|
|
||
| // baselinePathtestHash is deliberately local to the bounded selector. The | ||
| // selector hashes every eligible connection, so use the Agent's existing | ||
| // allocation-efficient xxhash dependency instead of the store's general hash. | ||
| func baselinePathtestHash(digest *xxhash.Digest, p common.Pathtest) uint64 { | ||
| digest.Reset() | ||
| writeBaselineHashString(digest, string(p.Origin)) | ||
| writeBaselineHashString(digest, p.Namespace) | ||
| writeBaselineHashString(digest, p.Hostname) | ||
| var port [2]byte | ||
| binary.LittleEndian.PutUint16(port[:], p.Port) | ||
| _, _ = digest.Write(port[:]) | ||
| writeBaselineHashString(digest, string(p.Protocol)) | ||
| writeBaselineHashString(digest, p.SourceContainerID) | ||
| return digest.Sum64() | ||
| } | ||
|
|
||
| func writeBaselineHashString(digest *xxhash.Digest, value string) { | ||
| var length [8]byte | ||
| binary.LittleEndian.PutUint64(length[:], uint64(len(value))) | ||
| _, _ = digest.Write(length[:]) | ||
| _, _ = digest.WriteString(value) | ||
| } | ||
|
|
||
| func saturatingAdd(a, b uint64) (uint64, bool) { | ||
| if math.MaxUint64-a < b { | ||
| return math.MaxUint64, true | ||
| } | ||
| return a + b, false | ||
| } | ||
|
|
||
| func (p *baselinePool) remove(hash uint64) { delete(p.items, hash) } | ||
|
|
||
| func (p *baselinePool) weakest(diagnostic bool) *baselineCandidate { | ||
| var weakest *baselineCandidate | ||
| for _, candidate := range p.items { | ||
| if weakest == nil || baselineBetter(weakest, candidate, diagnostic) { | ||
| weakest = candidate | ||
| } | ||
| } | ||
| return weakest | ||
| } | ||
|
|
||
| func (p *baselinePool) add(hash uint64, pathtest common.Pathtest, timeout bool, weight, rttVar uint64, diagnostic bool) (replaced, discarded, saturated bool) { | ||
| if candidate, found := p.items[hash]; found { | ||
| candidate.timeout = candidate.timeout || timeout | ||
| candidate.count, saturated = saturatingAdd(candidate.count, weight) | ||
| candidate.rttVar = max(candidate.rttVar, rttVar) | ||
| return false, false, saturated | ||
| } | ||
|
|
||
| if len(p.items) < p.capacity { | ||
| p.items[hash] = &baselineCandidate{pathtest: pathtest, hash: hash, timeout: timeout, count: weight, rttVar: rttVar} | ||
| return false, false, false | ||
| } | ||
|
|
||
| weakest := p.weakest(diagnostic) | ||
| // Timeout/RTO is the primary diagnostic class. Do not let a non-timeout | ||
| // candidate evict one when the diagnostic pool contains only timeouts. | ||
| if diagnostic && weakest.timeout && !timeout { | ||
| return false, true, false | ||
| } | ||
| delete(p.items, weakest.hash) | ||
| estimate, overflow := saturatingAdd(weakest.count, weight) | ||
| // Reuse the evicted entry. High-cardinality snapshots should not allocate a | ||
| // candidate object for every connection that passes through a bounded pool. | ||
| *weakest = baselineCandidate{ | ||
| pathtest: pathtest, | ||
| hash: hash, | ||
| timeout: timeout, | ||
| count: estimate, | ||
| rttVar: rttVar, | ||
| } | ||
| p.items[hash] = weakest | ||
| return true, false, overflow | ||
| } | ||
|
|
||
| func baselineBetter(a, b *baselineCandidate, diagnostic bool) bool { | ||
| if diagnostic && a.timeout != b.timeout { | ||
| return a.timeout | ||
| } | ||
| if a.count != b.count { | ||
| return a.count > b.count | ||
| } | ||
| if diagnostic && a.rttVar != b.rttVar { | ||
| return a.rttVar > b.rttVar | ||
| } | ||
| return a.hash < b.hash | ||
| } | ||
|
|
||
| func (p *baselinePool) sorted(diagnostic bool) []*baselineCandidate { | ||
| result := make([]*baselineCandidate, 0, len(p.items)) | ||
| for _, candidate := range p.items { | ||
| result = append(result, candidate) | ||
| } | ||
| sort.Slice(result, func(i, j int) bool { return baselineBetter(result[i], result[j], diagnostic) }) | ||
| return result | ||
| } | ||
|
|
||
| type baselineSelector struct { | ||
| diagnostic baselinePool | ||
| healthy baselinePool | ||
| hashDigest xxhash.Digest | ||
| } | ||
|
|
||
| func newBaselineSelector() *baselineSelector { | ||
| return &baselineSelector{ | ||
| diagnostic: newBaselinePool(baselineDiagnosticCandidates), | ||
| healthy: newBaselinePool(baselineHealthyCandidates), | ||
| } | ||
| } | ||
|
|
||
| type baselineAdmission struct { | ||
| replaced bool | ||
| discarded bool | ||
| saturated bool | ||
| } | ||
|
|
||
| func (s *baselineSelector) add(pathtest common.Pathtest, conn npmodel.NetworkPathConnection) baselineAdmission { | ||
| hash := baselinePathtestHash(&s.hashDigest, pathtest) | ||
| diagnostic := conn.TCPTimeout || conn.TCPRTO || conn.Retransmits > 0 | ||
| if diagnostic { | ||
| s.healthy.remove(hash) | ||
| replaced, discarded, saturated := s.diagnostic.add(hash, pathtest, conn.TCPTimeout || conn.TCPRTO, conn.Retransmits, conn.RTTVar, true) | ||
| return baselineAdmission{replaced: replaced, discarded: discarded, saturated: saturated || conn.NumericSaturated} | ||
| } | ||
| if _, found := s.diagnostic.items[hash]; found { | ||
| _, _, saturated := s.diagnostic.add(hash, pathtest, false, 0, conn.RTTVar, true) | ||
| return baselineAdmission{saturated: saturated || conn.NumericSaturated} | ||
| } | ||
| replaced, discarded, saturated := s.healthy.add(hash, pathtest, false, conn.Bytes, 0, false) | ||
| return baselineAdmission{replaced: replaced, discarded: discarded, saturated: saturated || conn.NumericSaturated} | ||
| } | ||
|
|
||
| func (s *baselineSelector) selectPathtests() []common.Pathtest { | ||
| selected := make([]common.Pathtest, 0, baselineSelectionsPerWindow) | ||
| for _, candidate := range s.diagnostic.sorted(true) { | ||
| selected = append(selected, candidate.pathtest) | ||
| if len(selected) == baselineSelectionsPerWindow { | ||
| return selected | ||
| } | ||
| } | ||
| for _, candidate := range s.healthy.sorted(false) { | ||
| selected = append(selected, candidate.pathtest) | ||
| if len(selected) == baselineSelectionsPerWindow { | ||
| break | ||
| } | ||
| } | ||
| return selected | ||
| } | ||
|
|
||
| func (s *baselineSelector) reset() { | ||
| s.diagnostic = newBaselinePool(baselineDiagnosticCandidates) | ||
| s.healthy = newBaselinePool(baselineHealthyCandidates) | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.