Skip to content

fix(iast): guard taint queries without a context#18996

Open
KowalskiThomas wants to merge 1 commit into
mainfrom
dd/kowalski/iast-fix-crash-taint-query-no-context
Open

fix(iast): guard taint queries without a context#18996
KowalskiThomas wants to merge 1 commit into
mainfrom
dd/kowalski/iast-fix-crash-taint-query-no-context

Conversation

@KowalskiThomas

@KowalskiThomas KowalskiThomas commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Description

This fixes a recurring crash where IAST dereferences a Python type object that has already been freed by the cyclic GC.

Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV)
#0   0x00006553759b6e73 PyType_IsSubtype (/usr/src/python/Objects/typeobject.c:2137)
#1   0x0000000000000000 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381)
#2   0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66)
#3   0x0000000000000000 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51)
#4   0x00007f20d5851de1 PyObject_TypeCheck (/opt/python/cp312-cp312/include/python3.12/object.h:381)
#5   0x00007f20d5851de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:66)
#6   0x00007f20d5851de1 is_text(_object const*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/utils/string_utils.h:51)
#7   0x00007f20d5851de1 TaintEngineContext::get_tainted_object_map(_object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:141)
#8   0x0000000000000000 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732)
#9   0x0000000000000000 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183)
#10  0x0000000000000000 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121)
#11  0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357)
#12  0x0000000000000000 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137)
#13  0x0000000000000000 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105)
#14  0x0000000000000000 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430)
#15  0x00007f20d5852a70 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:732)
#16  0x00007f20d5852a70 std::__shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > >, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1183)
#17  0x00007f20d5852a70 std::shared_ptr<absl::lts_20250127::node_hash_map<unsigned long, std::pair<long, std::shared_ptr<TaintedObject> >, absl::lts_20250127::hash_internal::Hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<long, std::shared_ptr<TaintedObject> > > > > >::~shared_ptr() (/opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:121)
#18  0x00007f20d5852a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/context/taint_engine_context.cpp:357)
#19  0x00007f20d5852a70 call_impl<bool, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&, 0, pybind11::detail::void_type> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2137)
#20  0x00007f20d5852a70 call<bool, pybind11::detail::void_type, pyexport_taint_engine_context(pybind11::module&)::<lambda(pybind11::object)>&> (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/cast.h:2105)
#21  0x00007f20d5852a70 operator() (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:430)
#22  0x00007f20d5852a70 pybind11::cpp_function::initialize<pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object)#5}, bool, pybind11::object, pybind11::name, pybind11::scope, pybind11::sibling>(pyexport_taint_engine_context(pybind11::module_&)::{lambda(pybind11::object&&)#5}, bool (*)(pybind11::object), pybind11::name const, pybind11::scope&, pybind11::sibling)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .cold] (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:400)
#23  0x00007f20d5809d00 pybind11::cpp_function::dispatcher(_object*, _object*, _object*) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/appsec/_iast/_taint_tracking/_vendor/pybind11/include/pybind11/pybind11.h:1063)
#24  0x00006553759af7cf cfunction_call (/usr/src/python/Objects/methodobject.c:551)
#25  0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245)
#26  0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719)
#27  0x0000655375a8fa91 _PyObject_VectorcallTstate (/usr/src/python/./Include/internal/pycore_call.h:93)
#28  0x0000655375984894 partial_vectorcall (/usr/src/python/./Modules/_functoolsmodule.c:269)
#29  0x00006553759a0ad4 object_vacall (/usr/src/python/Objects/call.c:850)
#30  0x00006553759fdd3e PyObject_CallFunctionObjArgs (/usr/src/python/Objects/call.c:961)
#31  0x00007f20ded79d0a WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3024)
#32  0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245)
#33  0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719)
#34  0x00006553759a3a86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238)
#35  0x00007f20e038fdc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869)
#36  0x00007f20e03905a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188)
#37  0x0000000000000000 __Pyx_PyObject_Call (/project/uvloop/loop.c:191431)
#38  0x00007f20c37988eb __Pyx_PyObject_Call (/project/uvloop/loop.c:191431)
#39  0x00007f20c37988eb __Pyx_PyObject_FastCallDict (/project/uvloop/loop.c:191552)
#40  0x00007f20c3874a69 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66873)
#41  0x00007f20c387896b __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975)
#42  0x00007f20c3872e52 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927)
#43  0x00007f20c3874c88 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335)
#44  0x0000000000000000 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33)
#45  0x00007f20c3890311 uv__queue_empty (/project/build/libuv-x86_64/src/queue.h:33)
#46  0x00007f20c3890311 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68)
#47  0x00007f20c388d647 uv_run (/project/build/libuv-x86_64/src/unix/core.c:440)
#48  0x00007f20c37aedb5 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18471)
#49  0x00007f20c3816e50 __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876)
#50  0x0000000000000000 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528)
#51  0x00007f20c3827cf0 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528)
#52  0x00007f20c3827cf0 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331)
#53  0x00006553759a147c PyObject_VectorcallMethod (/usr/src/python/Objects/call.c:887)
#54  0x0000000000000000 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700)
#55  0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798)
#56  0x00007f20c382bd60 Py_DECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:700)
#57  0x00007f20c382bd60 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:798)
#58  0x00007f20c382bd60 __pyx_pf_6uvloop_4loop_4Loop_44run_until_complete (/project/uvloop/loop.c:33769)
#59  0x0000000000000000 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797)
#60  0x00007f20c382d591 Py_XDECREF (/opt/_internal/cpython-3.12.11/include/python3.12/object.h:797)
#61  0x00007f20c382d591 __pyx_pw_6uvloop_4loop_4Loop_45run_until_complete (/project/uvloop/loop.c:33322)
#62  0x00006553759a08f8 PyObject_Vectorcall (/usr/src/python/Objects/call.c:325)
#63  0x00006553759c7357 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:2719)
#64  0x00006553759a3a86 gen_send_ex2 (/usr/src/python/Objects/genobject.c:238)
#65  0x00007f20e038fdc7 task_step_impl (/usr/src/python/Modules/_asynciomodule.c:2869)
#66  0x00007f20e03905a2 task_step (/usr/src/python/Modules/_asynciomodule.c:3188)
#67  0x00006553759a05de _PyObject_MakeTpCall (/usr/src/python/Objects/call.c:245)
#68  0x000065537592377a context_run (/usr/src/python/Python/context.c:668)
#69  0x0000655375a12beb cfunction_vectorcall_FASTCALL_KEYWORDS (/usr/src/python/Objects/methodobject.c:439)
#70  0x0000655375919275 _PyEval_EvalFrameDefault (/usr/src/python/Python/bytecodes.c:3227)
#71  0x0000655375a4a679 PyEval_EvalCode (/usr/src/python/Python/ceval.c:579)
#72  0x0000655375a682bc run_eval_code_obj (/usr/src/python/Python/pythonrun.c:1723)
#73  0x0000655375a68234 run_mod (/usr/src/python/Python/pythonrun.c:1744)
#74  0x0000655375a67df1 pyrun_file (/usr/src/python/Python/pythonrun.c:1643)
#75  0x0000655375a67c37 _PyRun_SimpleFileObject (/usr/src/python/Python/pythonrun.c:433)
#76  0x0000655375a67a57 _PyRun_AnyFileObject (/usr/src/python/Python/pythonrun.c:78)
#77  0x0000655375a71fb0 Py_RunMain (/usr/src/python/Modules/main.c:713)
#78  0x0000655375a71b3d Py_BytesMain (/usr/src/python/Modules/main.c:768)
#79  0x00007f20e1519d90 __libc_start_call_main (sysdeps/nptl/libc_start_call_main.h:58)
#80  0x0000000000000000 call_init (csu/libc-start.c:128)
#81  0x00007f20e1519e40 call_init (csu/libc-start.c:128)
#82  0x00007f20e1519e40 __libc_start_main_alias_2 (csu/libc-start.c:379)
#83  0x00006553759ea075 _start 

Root cause (as identified by LLM):

  • is_pyobject_tainted is forwarded to the native is_in_taint_map(obj, context_id)
  • When no IAST request context is active, context_id is None, so the native binding takes the multi-slot fallback get_tainted_object_map(obj), whose very first step is is_text(obj)
  • is_text validates the object pointer (non-null, aligned, non-null ob_type) but cannot detect a freed type
  • PyUnicode_CheckPyObject_TypeCheckPyType_IsSubtype then walks the dangling type and faults.

This happens when the query is invoked from an object finalizer / async-generator teardown run by the GC (uvloop's run_until_complete is in the reported stack), i.e. mid-lifecycle rather than at interpreter exit, so the existing shutting_down guard (added in #18534, only set from the atexit handler) does not cover it.

No other code seems to be affected by this specific issue.

Changes

  • is_pyobject_tainted now early-returns False when is_iast_request_enabled() is false, matching get_tainted_ranges.
  • The get_ranges public wrapper (_taint_tracking/__init__.py): short-circuit to [] when the resolved context_id is None, so no-request callers can't reach the same is_text-on-a-freed-type native path.
  • Added regression tests: no-context query does not consult native; a __del__/GC finalizer querying taint with no active context is safe and returns False; raw get_ranges returns empty without a context.

@datadog-official

This comment was marked as outdated.

@datadog-prod-us1-4

This comment was marked as outdated.

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/appsec/_iast/_taint_tracking/__init__.py                        @DataDog/asm-python
ddtrace/appsec/_iast/_taint_tracking/_taint_objects_base.py             @DataDog/asm-python
releasenotes/notes/iast-fix-crash-taint-query-without-context-b3d9f1c47a2e08d6.yaml  @DataDog/apm-python
tests/appsec/iast/taint_tracking/test_context.py                        @DataDog/asm-python

@KowalskiThomas KowalskiThomas force-pushed the dd/kowalski/iast-fix-crash-taint-query-no-context branch 2 times, most recently from 87b751c to 2f8ccea Compare July 12, 2026 07:33
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 12, 2026

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 4974 circular imports that already exist on the base branch and have not been changed by this PR.

Show existing cycles (showing 5 of 4974 shortest)
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.botocore -> ddtrace.internal.datastreams
ddtrace.internal.core -> ddtrace._trace.span -> ddtrace.internal.core
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.aiokafka -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.google_cloud_pubsub -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kafka -> ddtrace.internal.datastreams

To see all cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json

@KowalskiThomas KowalskiThomas force-pushed the dd/kowalski/iast-fix-crash-taint-query-no-context branch 2 times, most recently from c6f337a to 0bff86b Compare July 12, 2026 07:37
@KowalskiThomas KowalskiThomas added AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos identified-by:crashtracking Identified by Crash Tracking labels Jul 12, 2026
@KowalskiThomas KowalskiThomas marked this pull request as ready for review July 12, 2026 07:39
@KowalskiThomas KowalskiThomas requested review from a team as code owners July 12, 2026 07:39
@datadog-official

datadog-official Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 15 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | appsec/appsec_iast_default 1/6   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-py | appsec/appsec_iast_default 2/6   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-py | appsec/appsec_iast_default 3/6   View in Datadog   GitLab

View all 15 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a7f70f6 | Docs | Datadog PR Page | Give us feedback!

is_pyobject_tainted and the raw get_ranges wrapper forwarded to the
native no-context multi-slot resolver, whose first step is is_text(obj).
Invoked from a GC/finalizer with no active request, that dereferences a
type already freed by the cyclic GC and crashes (SIGSEGV). Both wrappers
now short-circuit in Python when no IAST request context is active,
mirroring get_tainted_ranges.
@KowalskiThomas KowalskiThomas force-pushed the dd/kowalski/iast-fix-crash-taint-query-no-context branch from 0bff86b to a7f70f6 Compare July 12, 2026 07:41
@pr-commenter

pr-commenter Bot commented Jul 12, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-12 07:54:54

Comparing candidate commit a7f70f6 in PR branch dd/kowalski/iast-fix-crash-taint-query-no-context with baseline commit 8e44831 in branch main.

Found 9 performance improvements and 8 performance regressions! Performance is the same for 67 metrics, 0 unstable metrics.

scenario:iast_aspects-re_expand_aspect

  • 🟥 execution_time [+317.212µs; +347.282µs] or [+8.742%; +9.571%]

scenario:iast_aspects-re_finditer_noaspect

  • 🟩 execution_time [-35.668µs; -32.268µs] or [-10.145%; -9.178%]

scenario:iast_aspects-re_fullmatch_aspect

  • 🟥 execution_time [+31.397µs; +36.132µs] or [+10.382%; +11.948%]

scenario:iast_aspects-re_fullmatch_noaspect

  • 🟩 execution_time [-35.670µs; -32.799µs] or [-10.923%; -10.044%]

scenario:iast_aspects-re_group_aspect

  • 🟩 execution_time [-82.327µs; -78.783µs] or [-15.871%; -15.187%]

scenario:iast_aspects-re_group_noaspect

  • 🟩 execution_time [-86.252µs; -80.842µs] or [-16.322%; -15.299%]

scenario:iast_aspects-re_groups_aspect

  • 🟥 execution_time [+64.694µs; +69.693µs] or [+12.146%; +13.085%]

scenario:iast_aspects-re_groups_noaspect

  • 🟩 execution_time [-82.896µs; -78.777µs] or [-15.332%; -14.570%]

scenario:iast_aspects-re_match_aspect

  • 🟥 execution_time [+31.584µs; +35.606µs] or [+10.372%; +11.693%]

scenario:iast_aspects-re_match_noaspect

  • 🟩 execution_time [-34.186µs; -31.133µs] or [-10.412%; -9.482%]

scenario:iast_aspects-re_search_aspect

  • 🟥 execution_time [+32.417µs; +36.394µs] or [+11.246%; +12.626%]

scenario:iast_aspects-re_search_noaspect

  • 🟩 execution_time [-36.197µs; -33.623µs] or [-11.678%; -10.848%]

scenario:iast_aspects-re_sub_aspect

  • 🟥 execution_time [+78.206µs; +81.978µs] or [+19.706%; +20.657%]

scenario:iast_aspects-re_sub_noaspect

  • 🟩 execution_time [-60.526µs; -56.604µs] or [-13.897%; -12.996%]

scenario:iast_aspects-re_subn_aspect

  • 🟥 execution_time [+60.529µs; +65.715µs] or [+14.338%; +15.566%]

scenario:iast_aspects-re_subn_noaspect

  • 🟩 execution_time [-63.642µs; -59.386µs] or [-14.132%; -13.187%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+84.773µs; +92.846µs] or [+19.264%; +21.099%]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos Bits AI identified-by:crashtracking Identified by Crash Tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant