fix(llmobs): stop capturing LangGraph engine state in tool span metadata - #19711
fix(llmobs): stop capturing LangGraph engine state in tool span metadata#19711jessicagamio wants to merge 1 commit into
Conversation
Codeowners resolved asResolved from the full PR diff against |
Circular import analysis
|
Dependency direction analysis
|
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 8aff4bb | Docs | Datadog PR Page | Give us feedback! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8caccd308f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| LangGraph namespaces its live execution state under dunder-prefixed ``configurable`` keys | ||
| (send/read closures, scratchpad counters, replay state, the checkpointer) and keeps callback | ||
| managers under ``callbacks``. None of it is meaningful on a span, and holding it means |
There was a problem hiding this comment.
Use plain literals in this private docstring
This helper lives in a private _integrations module, so its docstring is not Sphinx-rendered; repository convention explicitly forbids rST double-backtick literals in this context. Replace the markup around configurable and callbacks with plain prose.
AGENTS.md reference: AGENTS.md:L42-L50
Useful? React with 👍 / 👎.
8caccd3 to
760a7d1
Compare
_llmobs_set_meta_tags_from_tool stored a tool's RunnableConfig verbatim. Under LangGraph that config carries the live pregel runtime -- send/read closures, scratchpad counters, replay state, the checkpointer, callback managers -- none of which is JSON serializable, so the agentless JSON encoder raised TypeError and dropped every span in the trace. Measured on the MLOS-835 reproducer: 41 leaf values, 4 levels deep, 6 of them unserializable, 8 of 12 configurable keys were private __pregel_* internals. Filtering on the framework's own private-key convention (rather than an allowlist) leaves 16 leaves and 0 unserializable, while keeping thread_id, checkpoint_id, checkpoint_ns, checkpoint_map, tags, metadata and recursion_limit. Verified against a plain main encoder with no default= guard: 8/8 runs reproduced the bug without this change, 7/7 delivered with it. Addresses MLOB-7962. Escalations MLOS-834, MLOS-835. Known gap: the tests pin LangGraph's current shape, so they go stale rather than red if it moves state out from behind the __ prefix. A live-LangGraph test in tests/contrib/langgraph/ would assert the invariant instead.
760a7d1 to
8aff4bb
Compare
Description
LangGraph packs live execution objects into the
RunnableConfigit hands to a tool — closures, counters, a checkpointer, callback managers. The LangChain integration stored that config as-is in tool span metadata:Those objects are not JSON serializable, so agentless trace encoding raised
TypeErrorand dropped every span in the trace — silently, with the exception surfacing in the customer's application rather than in ddtrace.This drops
callbacksand LangGraph's privateconfigurablekeys before storing, and passes the rest throughload_data_value. The useful identifiers stay:thread_id,checkpoint_ns,checkpoint_map,recursion_limit, and thelanggraph_*node/step/trigger metadata.Fixes MLOB-7962.
Testing
A/B against a LangGraph
create_agentreproduction, 8 runs per arm, trace encoder untouched in both. Only runs where the model actually invoked the tool counted as valid:Verified in the UI that the delivered metadata is still a queryable nested object, with the LangGraph identifiers intact and no private keys present.
Tests in
tests/contrib/langchain/test_langchain_llmobs.py:test_tool_span_metadata_is_json_serializable— the regression test. Drives_llmobs_set_meta_tags_from_tooland asserts the metadata survivesjson.dumps; fails without this change.test_format_tool_config_drops_framework_internalstest_format_tool_config_result_is_json_serializabletest_format_tool_config_passes_through_non_dictSuite: same pre-existing failures before and after (VCR cassettes, API keys), +4 passing.
Risks
Low — the change only removes keys from a metadata field. Traces that previously failed to encode now encode.
Known limitation: the tests use a fixture shaped like LangGraph's current config, so they would keep passing if LangGraph moved execution state out from behind its private-key prefix. A test in
tests/contrib/langgraph/driving a real graph would assert the invariant instead. Happy to add here or as a follow-up.Additional Notes
tool_infois left alone — it is aname/descriptionstring pair and already serializable.