Add comprehensive monitoring and metrics for AI service

[ErnieAtLYD]

Problem

The AI service currently lacks monitoring for performance, costs, and usage patterns, making it difficult to optimize and debug in production.

Solution

Implement comprehensive monitoring and metrics collection for the AI reflection service.

Tasks

• Add response time metrics collection
• Track OpenAI API costs and usage
• Monitor cache hit/miss ratios
• Add memory usage monitoring for cache/rate limiting
• Implement error rate tracking by error type
• Add request volume and rate limiting metrics
• Set up alerting for high error rates or costs
• Create monitoring dashboard

Metrics to Track

• Response time percentiles (p50, p95, p99)
• OpenAI API call volume and costs
• Cache hit/miss ratios
• Rate limiting triggers
• Error rates by type
• Memory usage of in-memory stores

Acceptance Criteria

• Metrics are exposed in a standard format (Prometheus/OpenTelemetry)
• Historical data is retained for analysis
• Alerts trigger for concerning patterns
• Dashboard provides actionable insights

Priority

High - Essential for production monitoring and cost control

Related

Performance monitoring requirements from PR #12 code review