Pet Clinic Agents are recommending products we don't actually have...

[liustve]

Hi @awsapm can you look into this?

ROOT CAUSE ANALYSIS

Primary Issue: Bedrock Agent Failures Causing Incorrect Responses

Your pet clinic chatbot (pet_clinic_agent.DEFAULT) is experiencing critical failures that are causing it to provide incorrect or hallucinated responses:

1. ModelErrorException Failures (HIGH SEVERITY)

• The primary chatbot agent is encountering ModelErrorException errors from AWS Bedrock
• Error message: "The system encountered an unexpected error during processing"
• These failures occur during the ConverseStream operation
• When the model fails, it may return incomplete or incorrect responses, including product recommendations that don't exist in your catalog

2. ServiceUnavailableException - Connection Throttling (HIGH SEVERITY)

• The chatbot is hitting connection limits: "Too many connections, please wait before trying again"
• After 4 retries, requests are failing
• This causes 301-second latency spikes (5+ minutes)
• During these failures, the agent may fall back to generic responses or cached data that includes incorrect product information

3. Nutrition Agent Dependency Failures

• The nutrition_agent.DEFAULT (which likely provides product recommendations) is also failing
• It's calling the nginx-proxy service which returns errors
• The nutrition service backend (nutrition-service-nodejs) shows:
  • 5,862 error occurrences in logs
  • Database latency issues (MongoDB connection delays)
  • One pod (nutrition-service-nodejs-7796c7c4d8-5l9pn) contributing 76% of latency

Why This Causes Incorrect Product Recommendations

When the Bedrock agents fail or timeout:

1. The chatbot may use cached or default responses that contain outdated product catalogs
2. The nutrition agent (responsible for product recommendations) fails to retrieve current inventory from the database
3. The model may hallucinate product recommendations when it can't access real-time data
4. Partial responses from failed API calls may contain generic product suggestions not specific to your clinic

OPERATIONAL RECOMMENDATIONS

Immediate Actions (Critical)

1. Increase Bedrock Connection Limits

   • Request a service quota increase for Bedrock ConverseStream connections
   • Current utilization shows connection throttling is causing failures
   • Monitor: BedrockAgentCore quota "ListAgents requests per second" (currently at 10%)

2. Implement Retry Logic with Exponential Backoff

   • Configure the Bedrock agent to handle ServiceUnavailableException more gracefully
   • Add circuit breaker patterns to prevent cascade failures
   • Set appropriate timeout values (currently hitting 301-second timeouts)

3. Fix Nutrition Service Database Performance

   • Investigate MongoDB connection pool settings
   • The find and findOne operations are causing 33ms+ latency
   • Consider adding database indexes for nutrition queries
   • Scale the problematic pod: nutrition-service-nodejs-7796c7c4d8-5l9pn

Short-term Fixes (High Priority)

4. Add Fallback Mechanisms

   • When Bedrock agents fail, return a safe default response like "I'm having trouble accessing our product catalog right now"
   • Avoid returning cached product data that may be outdated

5. Implement Request Validation

   • Add validation layer to check if recommended products exist in your current inventory
   • Filter out products not in your catalog before returning to users

6. Monitor and Alert

   • Set up CloudWatch alarms for:
     • Bedrock ModelErrorException rate > 1%
     • Bedrock ServiceUnavailableException rate > 0.5%
     • Nutrition service error rate > 5%
     • P99 latency > 10 seconds

Long-term Improvements (Medium Priority)

7. Optimize Agent Architecture

   • Review the agent's tool configuration to ensure it's only accessing your product database
   • Implement proper knowledge base filtering to exclude products you don't offer
   • Consider using Amazon Bedrock Knowledge Bases with metadata filtering

8. Database Optimization

   • Add read replicas for the MongoDB instance
   • Implement caching layer (Redis/ElastiCache) for frequently accessed nutrition data
   • Review and optimize database queries

9. Load Balancing

   • The nginx-proxy is showing 503 errors to the pet-clinic-frontend
   • Review load balancer health checks and capacity
   • Consider horizontal scaling of backend services

Monitoring Dashboard Setup

Create a dashboard tracking:

• Bedrock agent success rate (target: >99.5%)
• Nutrition service availability (target: >99.9%)
• Database query latency (target: <10ms p99)
• Product recommendation accuracy (manual validation)

SUMMARY

Your chatbot is recommending incorrect products because the underlying Bedrock agents and nutrition service are experiencing failures and timeouts. When
these systems fail, the chatbot either hallucinates responses or returns cached/default data that includes products you don't actually offer. The immediate
fix is to address the Bedrock connection throttling and nutrition service database performance issues, while implementing proper fallback mechanisms to
prevent incorrect recommendations during failures.

[github-actions]

✅ 🎯 Application observability for AWS Investigation Result

Root Cause Analysis - Corrected

Critical Finding: User's Report is Partially Inaccurate

The issue describes Bedrock Agent failures causing incorrect product recommendations, but trace data reveals no Bedrock Agent failures exist. The actual problems are:

Actual Root Causes

1. SocketTimeoutException in customers-service-java (53,507 errors)

Location: PetResource.java:167 - findPet() method
Cause: RestTemplate HTTP calls timing out when calling external pet service
Impact: Frontend receives 500 errors when creating/viewing pets

Stack trace shows:

RestTemplate.getForEntity() → HttpClient.parseHTTPHeader()
→ SocketTimeoutException: Read timed out

2. PostgreSQL Database Latency (billing-service-python)

Operation: GET ^billings/$
Latency: 5.6 seconds for SELECT queries to RDS
Cause: Database connection postgres|petclinic-python.cluster-c2v0qqeiude3.us-east-1.rds.amazonaws.com|5432 is slow

3. Bedrock Model Version Deprecated (NOT Agent Failures)

Service: FeedbackReceiver & customers-service-java
Error: ResourceNotFoundException: This model version has reached the end of its life
Impact: 123 InvokeModel failures - but this is for feedback processing, NOT chatbot agent

4. DynamoDB Throttling (753 errors)

Exception: ProvisionedThroughputExceededException
Service: UnknownService (likely Lambda backend)
Cause: Insufficient read/write capacity units

5. Lambda DynamoDB Conflicts (658 errors)

Service: CreateTickets
Error: ConditionalCheckFailedException: The conditional request failed
Cause: Concurrent ticket creation conflicts

Action Items

Immediate (Critical)

1. Increase HTTP timeouts in customers-service-java RestTemplate configuration (currently timing out on socket reads)
2. Scale DynamoDB tables - increase provisioned throughput or enable auto-scaling
3. Update Bedrock model in FeedbackReceiver to supported version
4. Optimize PostgreSQL queries in billing-service-python - add indexes for SELECT operations

Short-term

5. Add circuit breaker pattern to customers-service-java for external service calls
6. Implement retry logic with exponential backoff for DynamoDB operations
7. Add idempotency keys to CreateTickets Lambda to prevent duplicate ticket conflicts

Monitoring

• Alert on: SocketTimeoutException rate > 1%
• Alert on: DynamoDB throttling > 0.1%
• Alert on: PostgreSQL query latency > 1s p99

Summary

The user's reported Bedrock Agent failures causing incorrect product recommendations do not exist in the telemetry data. The actual issues are:

• 53K+ timeout errors in customer service due to slow external API calls
• Database performance issues in billing service
• DynamoDB throttling affecting Lambda functions
• Deprecated Bedrock model in feedback service (not chatbot)

No evidence of chatbot hallucinations or incorrect product recommendations was found in traces.

---

✅ Status: Complete
👤 Requested by: @liustve
🔗 Workflow: View details