Skip to content

fix(llmobs): stop capturing LangGraph engine state in tool span metadata - #19711

Open
jessicagamio wants to merge 1 commit into
mainfrom
jessica.gamio/fix-MLOB-7962-langchain-tool-config
Open

fix(llmobs): stop capturing LangGraph engine state in tool span metadata#19711
jessicagamio wants to merge 1 commit into
mainfrom
jessica.gamio/fix-MLOB-7962-langchain-tool-config

Conversation

@jessicagamio

@jessicagamio jessicagamio commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Description

LangGraph packs live execution objects into the RunnableConfig it hands to a tool — closures, counters, a checkpointer, callback managers. The LangChain integration stored that config as-is in tool span metadata:

metadata["tool_config"] = tool_inputs.get("config")

Those objects are not JSON serializable, so agentless trace encoding raised TypeError and dropped every span in the trace — silently, with the exception surfacing in the customer's application rather than in ddtrace.

This drops callbacks and LangGraph's private configurable keys before storing, and passes the rest through load_data_value. The useful identifiers stay: thread_id, checkpoint_ns, checkpoint_map, recursion_limit, and the langgraph_* node/step/trigger metadata.

Fixes MLOB-7962.

Testing

A/B against a LangGraph create_agent reproduction, 8 runs per arm, trace encoder untouched in both. Only runs where the model actually invoked the tool counted as valid:

Valid runs Reproduced Fixed Traces delivered
Before 8 8/8 0 0
After 7 0 7/7 7

Verified in the UI that the delivered metadata is still a queryable nested object, with the LangGraph identifiers intact and no private keys present.

Tests in tests/contrib/langchain/test_langchain_llmobs.py:

  • test_tool_span_metadata_is_json_serializable — the regression test. Drives _llmobs_set_meta_tags_from_tool and asserts the metadata survives json.dumps; fails without this change.
  • test_format_tool_config_drops_framework_internals
  • test_format_tool_config_result_is_json_serializable
  • test_format_tool_config_passes_through_non_dict

Suite: same pre-existing failures before and after (VCR cassettes, API keys), +4 passing.

Risks

Low — the change only removes keys from a metadata field. Traces that previously failed to encode now encode.

Known limitation: the tests use a fixture shaped like LangGraph's current config, so they would keep passing if LangGraph moved execution state out from behind its private-key prefix. A test in tests/contrib/langgraph/ driving a real graph would assert the invariant instead. Happy to add here or as a follow-up.

Additional Notes

tool_info is left alone — it is a name/description string pair and already serializable.

@jessicagamio
jessicagamio requested review from a team as code owners August 14, 2026 19:13
@jessicagamio
jessicagamio requested a review from sabrenner August 14, 2026 19:13
@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/llmobs/_integrations/langchain.py                               @DataDog/ml-observability
releasenotes/notes/langchain-tool-config-framework-internals-655e1a16d03992b3.yaml  @DataDog/apm-python
tests/contrib/langchain/test_langchain_llmobs.py                        @DataDog/ml-observability

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 14, 2026

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 5 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 14, 2026

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 254 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 254 highest severity)
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.internal.ci_visibility.git_client -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=132)
ddtrace.internal.ci_visibility.recorder -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=132)
ddtrace.llmobs._integrations.openai_agents -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.internal.ci_visibility.api._base -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8aff4bb | Docs | Datadog PR Page | Give us feedback!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8caccd308f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +137 to +139
LangGraph namespaces its live execution state under dunder-prefixed ``configurable`` keys
(send/read closures, scratchpad counters, replay state, the checkpointer) and keeps callback
managers under ``callbacks``. None of it is meaningful on a span, and holding it means

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use plain literals in this private docstring

This helper lives in a private _integrations module, so its docstring is not Sphinx-rendered; repository convention explicitly forbids rST double-backtick literals in this context. Replace the markup around configurable and callbacks with plain prose.

AGENTS.md reference: AGENTS.md:L42-L50

Useful? React with 👍 / 👎.

@jessicagamio
jessicagamio force-pushed the jessica.gamio/fix-MLOB-7962-langchain-tool-config branch from 8caccd3 to 760a7d1 Compare August 14, 2026 20:28
_llmobs_set_meta_tags_from_tool stored a tool's RunnableConfig verbatim. Under
LangGraph that config carries the live pregel runtime -- send/read closures,
scratchpad counters, replay state, the checkpointer, callback managers -- none of
which is JSON serializable, so the agentless JSON encoder raised TypeError and
dropped every span in the trace.

Measured on the MLOS-835 reproducer: 41 leaf values, 4 levels deep, 6 of them
unserializable, 8 of 12 configurable keys were private __pregel_* internals.
Filtering on the framework's own private-key convention (rather than an allowlist)
leaves 16 leaves and 0 unserializable, while keeping thread_id, checkpoint_id,
checkpoint_ns, checkpoint_map, tags, metadata and recursion_limit.

Verified against a plain main encoder with no default= guard: 8/8 runs reproduced
the bug without this change, 7/7 delivered with it.

Addresses MLOB-7962. Escalations MLOS-834, MLOS-835.

Known gap: the tests pin LangGraph's current shape, so they go stale rather than
red if it moves state out from behind the __ prefix. A live-LangGraph test in
tests/contrib/langgraph/ would assert the invariant instead.
@jessicagamio
jessicagamio force-pushed the jessica.gamio/fix-MLOB-7962-langchain-tool-config branch from 760a7d1 to 8aff4bb Compare August 14, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant