Feat : Add CEL Expression Metrics for Improved Observability

[harshithsaiv]

Description

This PR addresses issue #195 by introducing a comprehensive metrics system for CEL expression execution. With Prometheus-based metrics tracking, operators can now observe CEL compilation and evaluation performance, identify failures, and monitor resource consumption—all of which directly resolve the observability gaps highlighted in issue #195.

• Detailed latency histograms: Monitor the performance of CEL expressions.
• Counters: Track success and failure rates.
• Error identification: Identify specific error types during evaluation.
• Gauges: Observe the concurrent load of evaluations.
• Filtering: Metrics can be filtered by resource type, operation type, and expression purpose.

Implementation Details

1. Core Metrics Package

• Location: pkg/cel/metrics/metrics.go
• Functionality:
  • Define and register Prometheus metrics.
  • Provide helper functions for recording metric values.
  • Implement utility functions (e.g., WithCELEvaluation()) for streamlined metrics recording.

2. Timing Utilities

• Location: pkg/cel/utils/timers.go
• Functionality:
  • Provide timer objects for measuring CEL compilation metrics.
  • Ensure consistent recording of both timing and result metrics.

3. Documentation

• Location: pkg/cel/metrics/doc.go
• Functionality:
  • Common usage patterns.
  • Explanation of available labels and their meanings.
  • Examples for different usage scenarios.

[harshithsaiv]

Hi @a-hilaly,

I want to clarify that I've closely followed the structure from the Kubernetes apiserver CEL metrics implementation (source) as specified in the issue.

The implementation includes:

• Core metrics for both CEL compilation and evaluation, capturing duration and result counters.
• Enhanced observability through additional metrics for tracking active evaluations and capturing error types.
• Contextual labels that facilitate better filtering and analysis.
• Helper functions that simplify integration with the existing codebase.

While I've added some extensions beyond the basic Kubernetes implementation (such as the utils package with timer helpers), the core structure adheres to the established patterns for consistency and familiarity.

Could you please review and approve this implementation? I believe it effectively addresses the observability gap while maintaining alignment with Kubernetes metrics best practices.