Pet Clinic agents are telling users to buy pet products we don't actually offer

[liustve]

Hi @awsapm can you look into this?

ROOT CAUSE ANALYSIS

Critical Issue: Data Availability Problem in nutrition-service-nodejs

Error Rate: 7% average (up to 18% peak) over the last 24 hours
HTTP Status: 404 Not Found errors
Affected Operation: GET /nutrition/:pet_type

Telemetry Evidence:

1. Service Dependency Chain:

   • pet_clinic_agent.DEFAULT (Primary chatbot) →
   • nutrition_agent.DEFAULT (Nutrition specialist) →
   • nutrition-service-nodejs (Product database) →
   • MongoDB (test|mongodb|27017)

2. Error Pattern from Transaction Search (100% sampled data):

   • 3,279 error requests with HTTP 404 status in the last 24 hours
   • MongoDB findOne operations complete successfully (OK status)
   • But the service returns 404, indicating missing data in the database

3. Performance Impact:

   • Successful requests: 43,374 (93% success rate)
   • Failed requests: 3,279 (7% error rate)
   • When data is missing, the chatbot likely falls back to generic or hallucinated recommendations

Why This Causes Incorrect Product Recommendations:

When the nutrition agent queries for specific pet types and receives 404 errors:

1. The MongoDB query executes successfully but returns no matching records
2. The service returns HTTP 404 (Not Found)
3. The Bedrock agent (Claude 3.5 Haiku model) doesn't receive product data
4. The LLM fills the gap with generic nutritional advice or hallucinates product names
5. Users see recommendations for products that don't exist in your inventory

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

OPERATIONAL RECOMMENDATIONS

Immediate Actions:

1. Audit MongoDB Product Database

   • Connect to MongoDB instance: test|mongodb|27017
   • Verify nutrition product records exist for all supported pet types
   • Check for missing or incomplete product catalog entries
   • Validate pet_type field values match what the chatbot queries

2. Review Data Synchronization

   • Ensure product catalog is properly loaded into MongoDB
   • Check if there's a data migration or seeding process that failed
   • Verify environment-specific configuration (currently using "test" database)

3. Implement Fallback Handling

   • Configure the nutrition agent to explicitly handle 404 responses
   • Add guardrails to prevent hallucinated product recommendations
   • Return "product not available" messages instead of making up products

Monitoring Improvements:

4. Create SLO for Data Availability

   • Set up an SLO monitoring the error rate for GET /nutrition/:pet_type
   • Target: < 1% error rate (currently at 7%)
   • Alert when 404 errors exceed threshold

5. Add Custom Metrics

   • Track which pet_type values are causing 404s
   • Monitor MongoDB query success vs. empty result sets
   • Create dashboards showing product catalog completeness

6. Enhanced Logging

   • Log the specific pet_type being queried when 404 occurs
   • Capture which products the LLM recommends vs. what's in inventory
   • Track user feedback on incorrect recommendations

Configuration Adjustments:

7. Bedrock Agent Prompt Engineering

   • Update the nutrition agent's system prompt to:
     • Only recommend products from the provided API response
     • Explicitly state when product information is unavailable
     • Never invent or hallucinate product names

8. API Response Handling

   • Modify nutrition-service-nodejs to return structured error messages
   • Include available pet types in error responses
   • Provide fallback product categories when specific items are missing

Infrastructure Considerations:

9. Database Environment Review

   • Currently using "test" MongoDB database - verify this is intentional for production
   • Consider separate databases for dev/test/prod environments
   • Implement database health checks and data validation

10. Service Resilience

    • Add circuit breaker pattern for MongoDB connection failures
    • Implement caching for frequently requested nutrition data
    • Consider read replicas if database performance becomes an issue

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

SUMMARY

The chatbot is recommending non-existent products because 7% of nutrition queries return 404 errors due to missing data in MongoDB. The Bedrock LLM, lacking actual product information,
generates plausible-sounding but incorrect recommendations. Fix this by populating the MongoDB product catalog, adding proper error handling, and implementing guardrails to prevent
hallucinated responses.

Priority: HIGH - This directly impacts customer trust and could lead to order fulfillment issues.

[github-actions]

✅ 🎯 Application observability for AWS Investigation Result

Root Cause Identified

Problem: The issue is NOT in nutrition-service-nodejs. The 404 errors (7% rate, 2,926 errors) are legitimate responses when the MongoDB database doesn't contain nutrition data for specific pet types being queried.

Key Findings

Service Analysis:
• nutrition-service-nodejs is operating correctly - MongoDB queries execute successfully
• Database contains only 6 pet types: cat, dog, lizard, snake, bird, hamster (pet-nutrition-service/db-seed.js:15-20)
• Service returns 404 when pet_type doesn't match these exact values
• 37,238 successful requests (92.7%) vs 2,926 errors (7.3%)

Root Cause - Data Mismatch:
• AI agents (pet_clinic_agent.py, nutrition_agent.py:22) call nutrition service with pet_type parameter
• Agents may pass pet types not in database (e.g., "rabbit", "guinea pig", "fish", "turtle")
• Case sensitivity matters - agents use .lower() but database seed uses lowercase values
• When no match found, service correctly returns 404

Agent Behavior on 404:
• nutrition_agent.py:27 handles 404: returns error message with pet type
• nutrition_agent.py:76-87 system prompt tells agent to "recommend SPECIFIC PRODUCT NAMES"
• Without actual product data, Claude model generates plausible-sounding but fake product names

Fix Required

Option 1: Expand Database Coverage (Recommended)

Add missing pet types to pet-nutrition-service/db-seed.js:

{ pet_type: 'rabbit', facts: '...', products: '...' },
{ pet_type: 'guinea pig', facts: '...', products: '...' },
{ pet_type: 'fish', facts: '...', products: '...' },
{ pet_type: 'turtle', facts: '...', products: '...' }

Option 2: Improve Agent Error Handling

Update nutrition_agent.py:76-87 system prompt:

"CRITICAL: NEVER recommend products unless provided in the API response"
"If product data is unavailable, say 'We don't currently have specific product recommendations for [pet_type]'"
"DO NOT invent or suggest product names that weren't explicitly provided"

Option 3: Add Default/Fallback Data

Modify server.js:22-24 to return generic nutrition info instead of 404 for unsupported pet types.

Priority

HIGH - Directly impacts customer trust and order fulfillment accuracy as stated in original report.

---

✅ Status: Complete
👤 Requested by: @liustve
🔗 Workflow: View details

[liustve]

Thanks @awsapm can you make a PR for option #2?

Improve Agent Error Handling
Update nutrition_agent.py:76-87 system prompt.

[github-actions]

✅ 🎯 Application observability for AWS Investigation Result

Pull Request Created

Branch: awsapm/19557439157-2025-11-21T02-00-33-581Z
Target: main

Changes Implemented

File Modified: agent/python/nutrition_agent.py

Updates to System Prompt (lines 76-87):
• Added critical instructions to prevent hallucinated product recommendations
• Agent now explicitly told to NEVER invent product names when API returns no data
• Clear fallback message for unsupported pet types
• Maintains helpful tone while ensuring accuracy

Key Additions:

"CRITICAL: NEVER recommend specific product names unless provided in the API response."
"If the API returns an error or no products, say 'We don't currently have specific product recommendations for [pet_type]'."
"DO NOT invent or suggest product names that weren't explicitly provided by the API."

Testing Recommendation

Verify agent behavior when querying unsupported pet types (e.g., "rabbit", "guinea pig"):
• Should return friendly "no recommendations available" message
• Should NOT generate fake product names

---

PR Link: #257

---

✅ Status: Complete
👤 Requested by: @liustve
🔗 Workflow: View details