[bug] ADK auto-instrumentation not displaying LLM input / output on some spans correctly

[ryoung562]

Some LLM spans don't show proper input/output messages

In some llm spans the input/output shows correctly:

loom recording:

https://www.loom.com/share/25083529ff32414ba17878e058b97dd9

Link to adk traces:

https://app.arize.com/organizations/QWNjb3VudE9yZ2FuaXphdGlvbjoyODIzNDpETlJ6/spaces/U3BhY2U6Mjk0NDI6ZVJrVQ==/models/TW9kZWw6NTM2ODU0MDY0MjpWR2ZS?queryFilterA=&selectedTab=llmTracing&timeZoneA=America%2FLos_Angeles&startA=1759993200000&endA=1760644799999&envA=tracing&modelType=generative_llm

[dosubot]

The most common reasons for missing LLM input/output in ADK auto-instrumentation are:

• The LLM span is missing the expected input/output attributes because the underlying LlmRequest or LlmResponse objects are absent, malformed, or fail serialization (for example, due to complex objects that can't be JSON-encoded) reference.
• TraceConfig or environment variables (like OPENINFERENCE_HIDE_INPUTS, OPENINFERENCE_HIDE_OUTPUTS, etc.) are set to hide or redact input/output, which causes those fields to be omitted from the span reference.
• Global state conflicts, such as initializing the instrumentor multiple times or running multiple instrumented agents in the same environment, can cause serialization failures or missing attributes reference.
• Some span types (like agent wrapper or RunnableSequence spans) may not propagate input/output if the expected structure (like a messages key) is missing, or if context isn't properly propagated reference.

To troubleshoot: compare the span attributes between affected and unaffected spans, check your TraceConfig and environment variables, ensure LlmRequest/LlmResponse objects are present and serializable, and confirm the instrumentor is only initialized once per process. If you want to distinguish instrumentation issues from ingestion/UI problems, use a ConsoleSpanExporter to inspect raw trace payloads before they reach the UI reference.

If the issue only appears on certain span types, you may need to restructure your inputs or use a custom processor to propagate input/output to those spans reference. For serialization issues, try converting outputs to dicts before they reach the instrumentor.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[ryoung562]

here are 2 examples from google adk-sample repo that have openinference adk auto-instrumentation enabled. the first one is travel concierge (muli-agent example) where instrumentation seems to work but have the input/output issues.

here is the repo for the travel concierge example:
https://github.com/google/adk-samples/tree/main/python/agents/travel-concierge

there is second example (RAG agent), which seems to error out with our auto-instrumentation enabled, so I can't get this to run successful. see screenshot and attached error log.

rag_agent_error_log.txt

here is the RAG adk example:
https://github.com/google/adk-samples/tree/main/python/agents/RAG

[satyadevai]

@ryoung562 I analyzed this ticket and below are my observations:

• The same agent workflow works correctly in Phoenix auto-instrumentation. The spans contain the same attributes, but they are unflattened in Phoenix Arize spans, whereas AX spans have flattened attributes.

• Google ADK spans include additional attribute keys such as gcp, genai etc. These keys contain many values, which might be related to message length limitations.

• I’ve observed the same behavior in BeeAI as well (related issue: [bug] Beeai - LLM spans are not properly capturing tools in llm attribute #2140 (comment)).

This seems like it could be an issue in arize AX, though I’m not certain about the exact root cause yet.

I’ve attached attributes & screenshots for reference.

Phoenix Attributes for all LLM Calls:

ADKPhoenix.json

AX Attributes for all LLM Calls

ADKAX.json