Tracer Version(s)
4.11.0 (also present on main as of 2026-07-27; the same shape exists in every release since the change landed in 4.5)
Python Version(s)
Python 3.12.12
Pip Version(s)
N/A, virtualenv
Bug Report
BaseWrappingContext._pop_storage raises AttributeError: 'NoneType' object has no attribute 'pop' whenever a wrapping context receives return or exit without having received enter for that invocation.
This is reachable from ddtrace alone, with no application framework involved: _UniversalWrappingContext reads its context list twice per call — once on entry, once on exit — and register() can mutate that list in between.
# ddtrace/internal/wrapping/context.py (4.11.0)
def __enter__(self): # line 539
...
for context in self._contexts: # line 545 <- list read here
context.__enter__()
def __return__(self, value): # line 567
for context in self._contexts[::-1]: # line 568 <- list re-read HERE
context.__return__(value)
def __exit__(self, exc_type, exc_value, traceback): # line 553
...
for context in self._contexts[::-1]: # line 562 <- and here
context.__exit__(exc_type, exc_value, traceback)
register() (line 513) appends to self._contexts with no lock and no in-flight-call tracking. A context registered between those two reads is never entered, so its _storage ContextVar still holds its default=None when the exit path calls _pop_storage on it:
def _pop_storage(self) -> dict[str, t.Any]: # line 319
storage = t.cast(dict[str, t.Any], self._storage.get()) # line 320 -> None
self._storage.set(storage.pop("__dd_wrapping_context_prev__")) # line 321 -> AttributeError
return storage
t.cast is a typing-only no-op, so there is no runtime check. get() and set() (lines 336–341) share the same unguarded pattern.
Why the guard is missing
#17407 ("fix(internal): crash in ContextVar", merged 2026-04-09) replaced a Token-based reset() with storing the previous value under dd_wrapping_context_prev and restoring it via set(). That was the right call for the SIGSEGV it fixed — the Token → tok_ctx reference cycle was real.
But reset() was also the only runtime validation that enter and exit were paired: it raises ValueError if the context changed since the token was created. That PR dropped the check deliberately, reasoning it was unnecessary "because BaseWrappingContext fully controls the enter/exit lifecycle."
That assumption doesn't hold, and it's ddtrace itself that breaks it — _UniversalWrappingContext.register() can add a context to a function that has a call in flight. So the invariant #17407 relies on is violated by a sibling class in the same module, and because the check is gone the violation surfaces as an opaque AttributeError from library internals rather than a diagnosable error.
There's prior art for this interaction being fragile: #15272 (lazy wrapping for Code Origin) was reverted by #15776 for "Context already registered" errors and re-landed as #15963 with a fix permitting duplicate registration. The registration path was patched; the invocation lifecycle wasn't.
Impact
Two failure modes, both observed in production on 4.11.0:
On the success path — the wrapped function returns normally and ddtrace throws while unwinding. Side effects have already committed, but the caller sees a failure. For an HTTP handler that's a 500 on a request that actually succeeded; for a retrying job runner it risks double-applying non-idempotent work.
On the exception path — the AttributeError replaces the application's own exception. See "Masking" below; this one is arguably worse than the crash, since it destroys the real error.
The crash is nondeterministic and races with instrumentation registration, so it correlates with neither a specific build nor a specific code path. In our case it surfaced as a single tracer-framed error group spanning six unrelated endpoints plus background worker activities.
Exposure scales with (wrapped functions) × (threads) × (request rate). tracer.wrap became a Code Origin instrumentation target in 4.9.0, and Code Origin defaults to enabled as of 4.9.0 — so any application with a large number of @tracer.wrap()-decorated functions and a threaded server is exposed without opting in. Worth noting the public docs still present DD_CODE_ORIGIN_FOR_SPANS_ENABLED as opt-in (export DD_CODE_ORIGIN_FOR_SPANS_ENABLED=true) and don't mention the new default; that mismatch is probably why this hasn't been reported before.
Masking
Masking: same setup, but the wrapped function raises
The application's ValueError never reaches the caller — it is replaced by the tracer's AttributeError:
def boom(x):
in_call.set()
released.wait(5)
raise ValueError("the real error")
# ... identical mid-call registration of CtxB ...
# Expected: ValueError: the real error
# Actual: AttributeError: 'NoneType' object has no attribute 'pop'
Reproduction Code
import threading
from ddtrace.internal.wrapping.context import WrappingContext
released, in_call = threading.Event(), threading.Event()
def plain_fn(x):
in_call.set()
released.wait(5)
return x * 2
class CtxA(WrappingContext): pass
class CtxB(WrappingContext): pass
CtxA(plain_fn).wrap() # installed before any call
err = []
def call():
try:
plain_fn(21)
except BaseException as e:
err.append(e)
t = threading.Thread(target=call); t.start()
in_call.wait(5)
CtxB(plain_fn).wrap() # registered while the call is IN FLIGHT
released.set(); t.join()
if err:
raise err[0]
print("OK")
Error Logs
Traceback (most recent call last):
File "minimal.py", line 9, in plain_fn
return x * 2
File ".../ddtrace/internal/wrapping/context.py", line 569, in __return__
context.__return__(value)
File ".../ddtrace/internal/wrapping/context.py", line 325, in __return__
self._pop_storage()
File ".../ddtrace/internal/wrapping/context.py", line 321, in _pop_storage
self._storage.set(storage.pop("__dd_wrapping_context_prev__"))
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'pop'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "minimal.py", line 19, in call
plain_fn(21)
File "minimal.py", line 6, in plain_fn
def plain_fn(x):
File ".../ddtrace/internal/wrapping/context.py", line 551, in _exit
self.__exit__(*sys.exc_info())
File ".../ddtrace/internal/wrapping/context.py", line 563, in __exit__
context.__exit__(exc_type, exc_value, traceback)
File ".../ddtrace/internal/wrapping/context.py", line 334, in __exit__
self._pop_storage()
File ".../ddtrace/internal/wrapping/context.py", line 321, in _pop_storage
self._storage.set(storage.pop("__dd_wrapping_context_prev__"))
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'pop'
Note the double _pop_storage: return fails first, then the bytecode CONTEXT_FOOT exception handler calls _exit, which fails the same way. Production tracebacks show this exact two-link shape.
Libraries in Use
No response
Operating System
No response
Tracer Version(s)
4.11.0 (also present on main as of 2026-07-27; the same shape exists in every release since the change landed in 4.5)
Python Version(s)
Python 3.12.12
Pip Version(s)
N/A, virtualenv
Bug Report
BaseWrappingContext._pop_storage raises AttributeError: 'NoneType' object has no attribute 'pop' whenever a wrapping context receives return or exit without having received enter for that invocation.
This is reachable from ddtrace alone, with no application framework involved: _UniversalWrappingContext reads its context list twice per call — once on entry, once on exit — and register() can mutate that list in between.
register() (line 513) appends to self._contexts with no lock and no in-flight-call tracking. A context registered between those two reads is never entered, so its _storage ContextVar still holds its default=None when the exit path calls _pop_storage on it:
t.cast is a typing-only no-op, so there is no runtime check. get() and set() (lines 336–341) share the same unguarded pattern.
Why the guard is missing
#17407 ("fix(internal): crash in ContextVar", merged 2026-04-09) replaced a Token-based reset() with storing the previous value under dd_wrapping_context_prev and restoring it via set(). That was the right call for the SIGSEGV it fixed — the Token → tok_ctx reference cycle was real.
But reset() was also the only runtime validation that enter and exit were paired: it raises ValueError if the context changed since the token was created. That PR dropped the check deliberately, reasoning it was unnecessary "because BaseWrappingContext fully controls the enter/exit lifecycle."
That assumption doesn't hold, and it's ddtrace itself that breaks it — _UniversalWrappingContext.register() can add a context to a function that has a call in flight. So the invariant #17407 relies on is violated by a sibling class in the same module, and because the check is gone the violation surfaces as an opaque AttributeError from library internals rather than a diagnosable error.
There's prior art for this interaction being fragile: #15272 (lazy wrapping for Code Origin) was reverted by #15776 for "Context already registered" errors and re-landed as #15963 with a fix permitting duplicate registration. The registration path was patched; the invocation lifecycle wasn't.
Impact
Two failure modes, both observed in production on 4.11.0:
On the success path — the wrapped function returns normally and ddtrace throws while unwinding. Side effects have already committed, but the caller sees a failure. For an HTTP handler that's a 500 on a request that actually succeeded; for a retrying job runner it risks double-applying non-idempotent work.
On the exception path — the AttributeError replaces the application's own exception. See "Masking" below; this one is arguably worse than the crash, since it destroys the real error.
The crash is nondeterministic and races with instrumentation registration, so it correlates with neither a specific build nor a specific code path. In our case it surfaced as a single tracer-framed error group spanning six unrelated endpoints plus background worker activities.
Exposure scales with (wrapped functions) × (threads) × (request rate). tracer.wrap became a Code Origin instrumentation target in 4.9.0, and Code Origin defaults to enabled as of 4.9.0 — so any application with a large number of @tracer.wrap()-decorated functions and a threaded server is exposed without opting in. Worth noting the public docs still present DD_CODE_ORIGIN_FOR_SPANS_ENABLED as opt-in (export DD_CODE_ORIGIN_FOR_SPANS_ENABLED=true) and don't mention the new default; that mismatch is probably why this hasn't been reported before.
Masking
Masking: same setup, but the wrapped function raises
The application's ValueError never reaches the caller — it is replaced by the tracer's AttributeError:
Reproduction Code
Error Logs
Note the double _pop_storage: return fails first, then the bytecode CONTEXT_FOOT exception handler calls _exit, which fails the same way. Production tracebacks show this exact two-link shape.
Libraries in Use
No response
Operating System
No response