Commit f1ae1d2
[Debugger] Bound Dynamic Instrumentation probe expression evaluation with a time budget (#8806)
## Summary of changes
- Adds a cooperative **evaluation time budget** to Dynamic
Instrumentation (DI) probe expressions, so checkpointed generated
evaluation work stops when expiry is observed; regex additionally
receives a hard timeout.
- Introduces `EvaluationBudget` (a value type passed by `ref` through
compiled evaluation paths), `CompiledExpressionDelegate<T>` (delegate
signature now carries `ref EvaluationBudget`), and
`EvaluationTimeBudgetExceededException`.
- The parser injects lightweight budget checkpoints into the generated
expression tree at the points that can dominate runtime: the expression
root, generated collection and dump loops, selected string/comparison
operations, regex, and `instanceof` type resolution.
- A single active-time budget is shared across templates → condition →
metric → span decorations → capture expressions. It is paused before
deferred capture work and resumed after capture-expression compilation,
so intervening processing and compilation do not consume the remaining
allowance.
- Hand-rolls the enumerable `any`/`all`/`filter` loops (instead of LINQ)
so a checkpoint can be placed inside each iteration; uses a real `Regex`
timeout for pattern matching.
- Removes budget checkpoints from two paths where they add cost without
value: the `SafeEquals` binary path and member access, which is already
restricted to fields and auto-property backing fields.
## Reason for change
- DI probe expressions are authored remotely and compiled to delegates
that execute **in the customer's process, on the customer's thread**. A
pathological or accidentally expensive expression (large collections,
nested filters, complex predicates, catastrophic regex) could add
unbounded latency and degrade the host application.
- There was no evaluation time budget. This change makes supported
expensive paths self-limiting: once expiry is observed at a cooperative
checkpoint, evaluation aborts and surfaces an evaluation error; regex
also receives a hard timeout.
## Implementation details
### The budget (`EvaluationBudget`)
- Mutable value type passed by `ref` through compiled delegates and
evaluation helpers. Its state is deliberately copied when persisted
between the main and deferred-capture phases and when bridged through
type-resolution callbacks; the budget itself does not require a heap
allocation.
- `Create(maxMs)` records a deadline as `Stopwatch.GetTimestamp() +
duration` (with an overflow guard so very large values clamp to
`long.MaxValue`).
- `ThrowIfExceeded()` is the hot amortized checkpoint used in loops and
selected operations: reading the clock on every operation would be too
expensive, so it samples the clock once every
`OperationsBeforeTimeCheck` (32) calls.
- The hot checkpoint is marked `[MethodImpl(AggressiveInlining)]`; the
throw/clock helpers are `[NoInlining]` to keep the inlined path tiny.
- `ThrowIfExceededImmediately()` samples the clock at expression
roots/fallbacks and expensive type-resolution boundaries.
- `TimedOut` is sticky: once the deadline is hit, all later checkpoints
throw immediately.
- `GetRemainingTimeout()` converts the remaining budget into a
`TimeSpan` and is handed to `Regex.IsMatch(...)`. Regex can block for a
long time inside a single call, so it gets a real hard timeout; a
`RegexMatchTimeoutException` is converted into the budget exception (and
marks the budget timed out).
- `Pause()` stores the remaining stopwatch ticks instead of an absolute
deadline; `Resume()` rebases the deadline from that remaining
active-evaluation allowance.
```mermaid
flowchart TD
Start["ThrowIfExceeded() — inlined into the hot path"] --> T{TimedOut?}
T -- "yes (sticky)" --> Throw["ThrowTimedOut() [NoInlining]"]
T -- no --> Dec["--operationsUntilTimeCheck"]
Dec --> C{"> 0 ?"}
C -- "yes (31 of 32 calls)" --> Ret["return — no clock read"]
C -- "no (every 32nd call)" --> Clock["ThrowIfTimeExceeded() [NoInlining]<br/>reset counter to 32, read Stopwatch"]
Clock --> D{"now >= deadline?"}
D -- no --> Ret
D -- yes --> Mark["MarkTimedOut() + throw EvaluationTimeBudgetExceededException"]
```
### Threading the budget through compiled expressions
- `CompiledExpressionDelegate<T>` adds a trailing `ref EvaluationBudget
budget` parameter; `CompiledExpression<T>.BudgetedDelegate` is the
compiled instance.
- The parser creates an `evaluationBudget` `ref` parameter for the
generated lambda and emits ordinary
`EvaluationBudget.ThrowIfExceeded(ref evaluationBudget)` checks
(`BudgetCheck()`) at strategic spots, with immediate checks at roots and
type-resolution boundaries.
- **One active-time budget per probe hit:**
`ProbeExpressionEvaluator.Evaluate` creates a budget after initial
expression compilation and passes the same `ref` to every
sub-expression. If deferred capture expressions are present, it pauses
and stores the budget on the (`ref struct`) result.
`EvaluateCaptureExpressions` compiles those expressions while the budget
is paused, then resumes and evaluates them with the same remaining
active-time allowance.
```mermaid
flowchart TD
A["Probe hit → compile/cache initial expressions"] --> B["CreateBudget(): deadline = now + configured max"]
B --> C["Templates(..., ref budget)"]
C --> D["Condition(..., ref budget)"]
D --> E["Metric(..., ref budget)"]
E --> F["Span decorations(..., ref budget)"]
F --> G["Pause budget and store remaining active time in result"]
G --> H["Compile deferred capture expressions while paused"]
H --> I["Resume and evaluate captures with remaining active-time budget"]
```
### Where checkpoints are injected (`BudgetCheck()`)
- **Root:** an immediate check at the top of every compiled expression
(including the fallback delegate) guarantees even a trivial expression
observes an already-exceeded budget.
- **Collection loops:** `any` / `all` / `filter` are hand-built loops
(`BuildEnumerableLoop`) with checkpoints before enumeration and at the
top of each iteration; enumerator disposal uses `Expression.TryFinally`.
The bounded capture-filter path checks once per item and passes the same
budget into the predicate so nested checkpointed operations share it.
- **String operations** that scale with input length: `Substring`,
`Contains` / `StartsWith` / `EndsWith`, `IsEmpty` (string and collection
length), and string lexicographic comparisons.
- **Dumps:** generated collection/dictionary dump loops.
- **Regex:** real `Regex.IsMatch` timeout via `GetRemainingTimeout()`.
- **`instanceof`:** immediate checks around loaded-assembly scans and
type-resolution callbacks.
### Checkpoints intentionally removed
- **`SafeEquals`:** equality only dispatches to an allowlisted set of
`Equals` implementations that are bounded and fast, so a per-comparison
checkpoint added overhead with no protective value.
- **Member access:** no per-access checkpoint is needed because the
current resolver emits fields or compiler-generated auto-property
backing fields, rejects side-effecting getters, and guards static
initialization. Removing the wrapping block also eliminated a stray
rendering artifact in the expression snapshots, so the affected
snapshots revert to their clean pre-budget form (no semantic change to
results/errors).
### Notes / risks
- **Behavior change:** expressions that exceed the budget now throw
`EvaluationTimeBudgetExceededException`, surfaced as an evaluation
error. As before, a condition that errors defaults to `true`.
- **Config:**
`DD_INTERNAL_DYNAMIC_INSTRUMENTATION_MAX_EVALUATION_TIME_MS` defaults to
50 ms and accepts values from 10–1000 ms; invalid or missing values use
the default. The configured value is propagated through probe processors
into their evaluators.
- **Trimming:** `Datadog.Trace.Trimming.xml` gains a
`System.Linq.Expressions.TryExpression` entry (auto-generated) because
the new enumerable loops use `Expression.TryFinally`.
- **Hot path:** ordinary checkpoints are aggressively inlined and sample
the clock only every 32 calls; roots and expensive type-resolution
boundaries check immediately, and throw/clock helpers remain
non-inlined.
## Test coverage
- `DebuggerExpressionLanguageTests` covers timeout propagation,
immediate root/fallback checks, pause/resume and shared capture-budget
behavior, regex timeout handling, and budgeted type resolution;
expression snapshots were regenerated for the new loop structure while
sanitizing budget plumbing from their rendered form.
- `DebuggerSettingsTests` covers the configured range/default behavior,
and `ProbeProcessorTests` covers propagation and updates of the
configured evaluation limit.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>1 parent 90443cf commit f1ae1d2
66 files changed
Lines changed: 1779 additions & 627 deletions
File tree
- tracer
- src
- Datadog.Trace.Trimming/build
- Datadog.Trace
- Configuration
- Debugger
- ExceptionAutoInstrumentation
- Expressions
- Generated
- net461/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator
- net6.0/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator
- netcoreapp3.1/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator
- netstandard2.0/Datadog.Trace.SourceGenerators/ConfigurationKeysGenerator
- test
- Datadog.Trace.Debugger.IntegrationTests
- Datadog.Trace.Tests/Debugger
- ProbeExpressionsResources/Approvals
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
232 | 232 | | |
233 | 233 | | |
234 | 234 | | |
235 | | - | |
236 | 235 | | |
237 | 236 | | |
238 | 237 | | |
239 | 238 | | |
240 | 239 | | |
241 | 240 | | |
242 | 241 | | |
243 | | - | |
244 | 242 | | |
245 | 243 | | |
246 | 244 | | |
| |||
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
376 | 376 | | |
377 | 377 | | |
378 | 378 | | |
| 379 | + | |
379 | 380 | | |
380 | 381 | | |
381 | 382 | | |
| |||
Lines changed: 10 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
831 | 831 | | |
832 | 832 | | |
833 | 833 | | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
834 | 844 | | |
835 | 845 | | |
836 | 846 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| 30 | + | |
| 31 | + | |
29 | 32 | | |
30 | 33 | | |
31 | 34 | | |
| |||
55 | 58 | | |
56 | 59 | | |
57 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
58 | 68 | | |
59 | 69 | | |
60 | 70 | | |
| |||
168 | 178 | | |
169 | 179 | | |
170 | 180 | | |
| 181 | + | |
| 182 | + | |
171 | 183 | | |
172 | 184 | | |
173 | 185 | | |
| |||
Lines changed: 3 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
312 | 312 | | |
313 | 313 | | |
314 | 314 | | |
315 | | - | |
| 315 | + | |
316 | 316 | | |
317 | 317 | | |
318 | 318 | | |
| |||
351 | 351 | | |
352 | 352 | | |
353 | 353 | | |
354 | | - | |
| 354 | + | |
355 | 355 | | |
356 | 356 | | |
357 | 357 | | |
| |||
692 | 692 | | |
693 | 693 | | |
694 | 694 | | |
695 | | - | |
| 695 | + | |
696 | 696 | | |
697 | 697 | | |
698 | 698 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
256 | | - | |
| 256 | + | |
257 | 257 | | |
258 | 258 | | |
259 | 259 | | |
| |||
Lines changed: 11 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
19 | 20 | | |
20 | | - | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | | - | |
| 27 | + | |
27 | 28 | | |
28 | | - | |
| 29 | + | |
29 | 30 | | |
30 | | - | |
| 31 | + | |
31 | 32 | | |
32 | | - | |
| 33 | + | |
33 | 34 | | |
34 | 35 | | |
Lines changed: 19 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
Lines changed: 171 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
Lines changed: 16 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
0 commit comments