I have a custom RAG agent deployed on AgentCore Runtime that retrieves from Bedrock Knowledge Base and generates responses using Bedrock Converse API. The agent has observability enabled and successfully produces
traces in CloudWatch.
Agent setup:
- Custom Python handler using BedrockAgentCoreApp
- Retrieves from Knowledge Base
- Generates answers via bedrock-runtime.converse()
- Observability enabled with traces in aws/spans log group
Problem:
When running BatchEvaluationRunner with Builtin.Correctness and Builtin.GoalSuccessRate evaluators, all sessions fail with:
error.message: "Provided input has no spans to evaluate. Please ensure that spans provided in the request are valid and have model/tool/agent invocation details for the provided scope"
The evaluators find the traces but report "no spans to evaluate" despite having all required attributes.
Question:
Do built-in AgentCore evaluators only support Bedrock Agents Classic, or should they work with custom AgentCore runtime agents that implement RAG with Knowledge Base + Bedrock LLM? In addition, it is stated in prerequisites that only Strands Agents or LangGraph with opentelemetry-instrumentation-langchain or openinference-instrumentation-langchain are supported. So if i am producing simple OpenTelemetry (OTEL) traces, agentcore evaluations do not work?
I have a custom RAG agent deployed on AgentCore Runtime that retrieves from Bedrock Knowledge Base and generates responses using Bedrock Converse API. The agent has observability enabled and successfully produces
traces in CloudWatch.
Agent setup:
Problem:
When running BatchEvaluationRunner with Builtin.Correctness and Builtin.GoalSuccessRate evaluators, all sessions fail with:
error.message: "Provided input has no spans to evaluate. Please ensure that spans provided in the request are valid and have model/tool/agent invocation details for the provided scope"
The evaluators find the traces but report "no spans to evaluate" despite having all required attributes.
Question:
Do built-in AgentCore evaluators only support Bedrock Agents Classic, or should they work with custom AgentCore runtime agents that implement RAG with Knowledge Base + Bedrock LLM? In addition, it is stated in prerequisites that only Strands Agents or LangGraph with opentelemetry-instrumentation-langchain or openinference-instrumentation-langchain are supported. So if i am producing simple OpenTelemetry (OTEL) traces, agentcore evaluations do not work?