[BUG] CloudWatchAgentSpanCollector fails to collect openinference.instrumentation.langchain spans for LangGraph agents deployed via CDK

[nikhilswagle]

Description

When using OnDemandEvaluationDatasetRunner with CloudWatchAgentSpanCollector to evaluate a LangGraph agent deployed on AgentCore Runtime via CDK (not AgentCore CLI), the built-in evaluators fail with:

ValidationException: Provided input has no spans with supported scope. 
Currently supported scopes are ['strands.telemetry.tracer', 
'opentelemetry.instrumentation.langchain', 'openinference.instrumentation.langchain']

Environment

• bedrock-agentcore SDK: 1.7.0
• openinference-instrumentation-langchain: 0.1.63
• openinference-instrumentation-bedrock: (latest)
• opentelemetry-instrumentation-langchain: 0.60.0
• aws-opentelemetry-distro: (latest)
• langgraph: 1.1.10
• langchain-aws: 1.4.5 (ChatBedrockConverse)
• Python: 3.12 (ARM64 on AgentCore Runtime)
• Region: us-east-1
• Deployment method: AWS CDK (not AgentCore CLI)
• Transaction Search: enabled at 100% sampling

What works

• Spans with openinference.instrumentation.langchain scope ARE present in aws/spans log group (verified via CloudWatch Insights manual query — 35 results found)
• Spans with openinference.instrumentation.bedrock scope also present
• session.id attribute correctly set on spans in aws/spans
• Simple scenarios (greeting — no tool calls, 1 LLM call) evaluate successfully: GoalSuccessRate=1.0, Correctness=1.0, Helpfulness=0.83
• Agent invocations succeed for all scenarios

What fails

• All scenarios involving tool calls (MCP Gateway → Lambda) fail with "no spans with supported scope"
• The CloudWatchAgentSpanCollector._fetch_spans() queries both aws/spans and the Runtime log group, but the collected spans passed to the Evaluate API don't include the openinference.instrumentation.langchain scoped ones

Steps to reproduce

1. Deploy a LangGraph agent on AgentCore Runtime via CDK with ChatBedrockConverse for LLM calls and langchain-mcp-adapters for tool calls
2. Add openinference-instrumentation-langchain, openinference-instrumentation-bedrock, and aws-opentelemetry-distro to the agent bundle
3. Set OTEL env vars: AGENT_OBSERVABILITY_ENABLED=true, OTEL_PYTHON_DISTRO=aws_distro, OTEL_PYTHON_CONFIGURATOR=aws_configurator, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
4. Set entrypoint to ["opentelemetry-instrument", "main.py"]
5. Explicitly initialize instrumentors in main.py:

   from openinference.instrumentation.langchain import LangChainInstrumentor
   LangChainInstrumentor().instrument()
   from openinference.instrumentation.bedrock import BedrockInstrumentor
   BedrockInstrumentor().instrument()

6. Enable Transaction Search at 100% sampling
7. Run OnDemandEvaluationDatasetRunner with CloudWatchAgentSpanCollector

Evidence

Spans exist in aws/spans with correct scope

CloudWatch Insights query on aws/spans:

fields @timestamp, @message 
| filter @message like "list-vendors-3e2c9f9f" 
| limit 5

Returns 5 results with spans like:

{
  "scope": {
    "name": "openinference.instrumentation.langchain",
    "version": "0.1.63"
  },
  "name": "nexusflow-dynamodb-query___list_vendors",
  "attributes": {
    "openinference.span.kind": "TOOL",
    "session.id": "list-vendors-3e2c9f9f-63a2-4191-9f16-ea1d711aaa31",
    "tool.name": "nexusflow-dynamodb-query___list_vendors"
  }
}

SDK's own query format also finds spans

Running the exact query from CloudWatchAgentSpanCollector._fetch_spans():

fields @timestamp, @message
| filter attributes.session.id = "list-vendors-dece0b51-ac1d-461f-b8e0-6ab724f60bd6"
| filter ispresent(scope.name)
| filter ispresent(traceId)
| filter ispresent(spanId)
| sort @timestamp asc

Returns 35 results.

But the SDK returns 0 spans

Despite the manual query returning 35 results, CloudWatchAgentSpanCollector.collect() returns 0 spans for the same session ID, causing all evaluators to fail.

Suspected root cause

The CloudWatchAgentSpanCollector._helper.query_log_group() method may not be correctly parsing the CloudWatch Insights query results from aws/spans, or there is a time window mismatch between when the SDK queries and when the spans become available for Insights queries.

Additionally, multi-step LangGraph scenarios create multiple trace contexts (due to asyncio.run() in tool invocations breaking trace propagation), which may cause the span collector to miss spans that belong to the same session but have different trace IDs.

Workaround

Simple scenarios without tool calls (e.g., greeting with 1 LLM call) evaluate successfully, suggesting the issue is specific to multi-step graph execution with MCP tool calls.

[nikhilswagle]

Additional Finding (2026-05-11): Evaluate API Span Parser Bug

Summary

After building a custom span collector that bypasses CloudWatchAgentSpanCollector and
successfully retrieves all spans from CloudWatch Insights, we confirmed the root cause
is in the Evaluate API's span parser, not the span collector.

Investigation

We built a custom span collector that queries CloudWatch Insights directly using:

fields @timestamp, @message
| filter @message like "<session_id>"
| filter ispresent(scope.name)
| filter ispresent(traceId)
| filter ispresent(spanId)
| sort @timestamp asc

This successfully retrieves all spans (35 total from aws/spans, 22 from Runtime log group).
After filtering to supported scopes (openinference.instrumentation.langchain), we get
11 structural spans + 11 corresponding log events = 22 entries.

The catch-22

When passing these spans to the Evaluate API:

1. With tool log event included →

   SpanEventParsingException: Failed to parse event body for span with ID: 9843f34807c1f689
   

2. Without tool log event →

   LogEventMissingException: Session span data is incomplete. Span with ID: 9843f34807c1f689
   and name: nexusflow-dynamodb-query___list_vendors is missing a corresponding log event.
   

Root cause

The Runtime log group stores tool invocation log events with deeply nested, multi-level
escaped JSON in the body.output.messages field. The Evaluate API's span parser cannot
handle this format:

{
  "scope": {"name": "openinference.instrumentation.langchain"},
  "spanId": "9843f34807c1f689",
  "body": {
    "output": {
      "messages": [
        {
          "content": "[{\"type\": \"text\", \"text\": \"\\\"{\\\\\\\"vendors\\\\\\\": [\\\\\\\"BrightStar Electronics\\\\\\\", ...]}\"}]"
        }
      ]
    }
  }
}

The 4+ levels of JSON escaping (MCP tool response → LangChain message → OpenTelemetry
log event → CloudWatch) is what the parser cannot process.

Scope of impact

• Affects ALL evaluator types: built-in, custom LLM-as-judge, and custom code-based (Lambda)
• The parsing failure occurs at the API layer before spans reach any evaluator logic
• Only tool-free scenarios (greetings, direct responses) can be evaluated via the Evaluate API
• Any agent using MCP Gateway → Lambda tool calls with LangGraph is affected

Tested evaluators (all fail with same error)

• Builtin.Helpfulness
• Builtin.GoalSuccessRate
• Custom LLM-as-judge evaluator (NexusFlowTestEvaluator-SrNQI56yWz)

Current workaround

A local LLM-as-Judge evaluation script that invokes the agent directly and evaluates the response text using Claude Opus 4.7 as the judge model. This bypasses the Evaluate API entirely.