Metrics Observability for Gateway

[Tharsanan1]

Metrics Observability for Gateway

Overview

We need to support metrics observability in our gateway components. The Router (Envoy proxy) already supports metrics, so we don't need to implement any code for metrics collection on the router side. However, in the GatewayController and PolicyEngine, we need to implement the necessary metrics and expose them via metrics endpoints.

I propose the following metrics to be implemented in each component.

---

Policy Engine Metrics

1. Request Processing Metrics 🎯

policy_engine_requests_total

• Labels: phase (request_headers/request_body/response_headers/response_body), route, api_name, api_version
• Type: Counter
• Use Case: Track total traffic through the policy engine, identify high-traffic routes, detect traffic patterns per API

policy_engine_request_duration_seconds

• Labels: phase, route
• Type: Histogram (buckets: 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5)
• Use Case: Measure end-to-end latency added by policy execution. Critical for SLA monitoring - if p99 latency exceeds thresholds, investigate slow policies

policy_engine_request_errors_total

• Labels: phase, error_type, route
• Type: Counter
• Use Case: Alert on error spikes, identify problematic routes. Error types: context_build_failed, policy_execution_failed, chain_not_found, stream_error

policy_engine_short_circuits_total

• Labels: route, policy_name
• Type: Counter
• Use Case: Track how often policies terminate request processing early (e.g., auth failures, rate limits). High short-circuit rate on auth = potential attack

---

2. Policy Execution Metrics 🔄

policy_engine_policy_executions_total

• Labels: policy_name, policy_version, api, route, status (executed/skipped/error)
• Type: Counter
• Use Case: Understand which policies run most frequently, detect if policies are being skipped unexpectedly

policy_engine_policy_duration_seconds

• Labels: policy_name, policy_version, api, route
• Type: Histogram
• Use Case: Identify slow policies causing latency bottlenecks. If one policy's p99 is high, it needs optimization

policy_engine_policy_skipped_total

• Labels: policy_name, api, route, reason (condition_not_met/disabled)
• Type: Counter
• Use Case: Validate conditional execution logic. If a policy is always skipped, the condition might be misconfigured

policy_engine_policies_per_chain

• Labels: route, api
• Type: Histogram (buckets: 1, 2, 5, 10, 20, 50)
• Use Case: Track policy chain complexity. Long chains increase latency risk

---

3. Configuration Management Metrics ⚙️

policy_engine_policy_chains_loaded

• Labels: mode (xds/file)
• Type: Gauge
• Use Case: Monitor total active routes/chains. Sudden drops = config reload failed

policy_engine_xds_updates_total

• Labels: status (success/failure), type (full/incremental)
• Type: Counter
• Use Case: Track xDS update reliability. High failure rate = connectivity issues with gateway-controller

policy_engine_xds_connection_state

• Labels: state (connected/disconnected/connecting)
• Type: Gauge (0/1)
• Use Case: Alert when xDS stream disconnects. Service becomes stale without config updates

---

4. Resource & Performance Metrics 💻

policy_engine_active_streams

• Type: Gauge
• Use Case: Monitor concurrent request processing. Spike = traffic surge or slow policies blocking

policy_engine_body_bytes_processed

• Labels: phase, operation (read/write)
• Type: Counter
• Use Case: Monitor data volume processed. Large bodies = memory pressure

policy_engine_context_build_duration_seconds

• Labels: type (request/response)
• Type: Histogram
• Use Case: Measure overhead of building execution context. Baseline latency cost before any policy runs

---

5. System Health Metrics 🏥

policy_engine_up

• Type: Gauge (0/1)
• Use Case: Basic liveness check for alerting

policy_engine_grpc_connections_active

• Labels: type (envoy/xds)
• Type: Gauge
• Use Case: Monitor Envoy ext_proc connections. Multiple connections = multiple Envoy instances

policy_engine_goroutines

• Type: Gauge
• Use Case: Detect goroutine leaks. Growing count over time = resource leak

policy_engine_memory_bytes

• Labels: type (heap_alloc/heap_sys/stack)
• Type: Gauge
• Use Case: Monitor memory usage patterns. Sudden spikes = memory leak or traffic surge

---

6. Error & Reliability Metrics ⚠️

policy_engine_policy_errors_total

• Labels: policy_name, error_type
• Type: Counter
• Use Case: Alert on policy-specific failures. If one policy has high error rate, it's broken

policy_engine_stream_errors_total

• Labels: error_type (send/receive/init)
• Type: Counter
• Use Case: Track gRPC streaming issues with Envoy. High error rate = network or Envoy misconfiguration

policy_engine_route_lookup_failures_total

• Type: Counter
• Use Case: Count requests to unknown routes. High count = routing config mismatch

policy_engine_panic_recoveries_total

• Labels: component
• Type: Counter
• Use Case: Track unexpected panics. Any panic = critical bug

---

Gateway Controller Metrics

1. API Configuration Management Metrics 📋

gateway_controller_api_operations_total

• Labels: operation (create/update/delete/get/list), status (success/failure), api_type (rest/websub)
• Type: Counter
• Use Case: Track configuration change velocity. High failure rate on creates = validation issues. Helps capacity planning for config changes

gateway_controller_api_operation_duration_seconds

• Labels: operation, api_type
• Type: Histogram (buckets: 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0)
• Use Case: Identify slow configuration operations. If UpdateAPI takes >5s, database writes are bottleneck. Critical for deployment CI/CD pipelines

gateway_controller_apis_total

• Labels: api_type (rest/websub), status (pending/deployed/failed)
• Type: Gauge
• Use Case: Monitor total deployed APIs. Sudden drop = mass deletion or storage failure. Capacity planning metric

gateway_controller_validation_errors_total

• Labels: operation, error_type (schema/policy/duplicate/invalid_upstream)
• Type: Counter
• Use Case: Catch configuration quality issues. High schema errors = bad client integration. Policy errors = policy misconfiguration

gateway_controller_deployment_latency_seconds

• Type: Histogram
• Use Case: End-to-end deployment time (API creation → Envoy receiving config → status=deployed). Critical SLA metric for API deployment pipelines

---

2. xDS Snapshot & Translation Metrics 🔄

gateway_controller_snapshot_generation_duration_seconds

• Labels: type (router/policy)
• Type: Histogram (buckets: 0.01, 0.05, 0.1, 0.5, 1.0, 2.5, 5.0, 10.0)
• Use Case: Measure TranslateConfigs() performance. If p99 > 5s with 1000+ APIs, snapshot generation is bottleneck. Impacts deployment latency

gateway_controller_snapshot_generation_total

• Labels: type (router/policy), status (success/failure), trigger (api_create/api_update/api_delete/policy_update/cert_update)
• Type: Counter
• Use Case: Track snapshot regeneration frequency. High count = config churn. Multiple snapshots/sec = potential performance issue

gateway_controller_snapshot_size

• Labels: resource_type (listeners/clusters/routes/endpoints/secrets)
• Type: Gauge
• Use Case: Monitor Envoy config complexity. Routes > 10,000 = potential routing table explosion. Track growth over time for capacity planning

gateway_controller_translation_errors_total

• Labels: error_type (invalid_vhost/missing_upstream/cert_not_found)
• Type: Counter
• Use Case: Detect configuration translation failures. Non-zero count = broken API deployments

gateway_controller_routes_per_api

• Type: Histogram (buckets: 1, 5, 10, 25, 50, 100, 250)
• Use Case: Understand API complexity distribution. High route count per API = performance risk

---

3. xDS Server & Client Metrics 🌐

gateway_controller_xds_clients_connected

• Labels: server (router/policy), node_id
• Type: Gauge
• Use Case: Monitor connected Envoy/Policy-Engine instances. Sudden drop = downstream service crash. Helps detect scaling issues

gateway_controller_xds_stream_requests_total

• Labels: server, type_url (listener/cluster/route/endpoint/secret/policy), operation (subscribe/unsubscribe)
• Type: Counter
• Use Case: Track xDS resource subscriptions. High unsubscribe rate = client instability

gateway_controller_xds_snapshot_ack_total

• Labels: server, node_id, status (ack/nack)
• Type: Counter
• Use Case: Critical reliability metric. NACK = Envoy rejected config (invalid resources). Must alert on NACKs - means broken routing

gateway_controller_xds_stream_duration_seconds

• Labels: server, node_id
• Type: Histogram
• Use Case: Track client connection duration. Short durations = connection instability

---

4. Storage & Database Metrics 💾

gateway_controller_database_operations_total

• Labels: operation (insert/update/delete/select), table (api_configurations/llm_provider_templates/certificates), status (success/failure)
• Type: Counter
• Use Case: Monitor database health. "database locked" errors show up here. High failure rate = SQLite contention

gateway_controller_database_operation_duration_seconds

• Labels: operation, table
• Type: Histogram (buckets: 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0)
• Use Case: Identify slow queries. SQLite with single connection can bottleneck. If p95 > 100ms, need connection pooling or migration to PostgreSQL

gateway_controller_database_size_bytes

• Labels: database (sqlite/config_store)
• Type: Gauge
• Use Case: Monitor database growth. Unbounded growth = missing cleanup. Plan for storage capacity

gateway_controller_config_store_size

• Labels: type (apis/policies/templates/certificates)
• Type: Gauge
• Use Case: Track in-memory store size. Large count = memory pressure. Compare with database count for consistency checks

gateway_controller_storage_errors_total

• Labels: operation, error_type (locked/not_found/constraint_violation)
• Type: Counter
• Use Case: Track storage failures. "database locked" = single connection bottleneck. constraint_violation = duplicate configs

---

5. Certificate Management Metrics 🔐

gateway_controller_certificates_total

• Labels: type (custom/system)
• Type: Gauge
• Use Case: Monitor certificate inventory. Track growth over time

gateway_controller_certificate_operations_total

• Labels: operation (upload/delete/load), status (success/failure)
• Type: Counter
• Use Case: Track certificate lifecycle operations. High failure rate on uploads = invalid certificates

gateway_controller_certificate_expiry_seconds

• Labels: cert_id, cert_name
• Type: Gauge
• Use Case: Critical security metric. Alert when expiry < 30 days. Prevents certificate expiration incidents

gateway_controller_sds_updates_total

• Labels: status (success/failure)
• Type: Counter
• Use Case: Track SDS snapshot updates to Envoy. Failures = Envoy not receiving cert updates

---

6. Policy Management Metrics 📜

gateway_controller_policies_total

• Labels: api_id, route
• Type: Gauge
• Use Case: Track total policy configurations. Monitor policy coverage across APIs

gateway_controller_policy_chain_length

• Labels: api_id, route
• Type: Histogram (buckets: 0, 1, 2, 5, 10, 20, 50)
• Use Case: Track policy chain complexity. Long chains = higher latency in policy-engine. Capacity planning for policy execution

gateway_controller_policy_snapshot_updates_total

• Labels: status (success/failure)
• Type: Counter
• Use Case: Track policy xDS updates. Failures = policy-engine not receiving config

gateway_controller_policy_validation_errors_total

• Labels: error_type (unknown_policy/invalid_params/condition_error)
• Type: Counter
• Use Case: Detect policy configuration issues. Unknown policy = typo or missing policy definition

---

7. Control Plane Integration Metrics 🔌

gateway_controller_control_plane_connection_state

• Labels: state (disconnected/connecting/connected/reconnecting)
• Type: Gauge (0/1)
• Use Case: Monitor WebSocket health. Disconnected = no deployment notifications sent. Must alert on prolonged disconnection

gateway_controller_control_plane_reconnections_total

• Type: Counter
• Use Case: Track reconnection frequency. High count = network instability or control plane issues

gateway_controller_control_plane_events_sent_total

• Labels: event_type (api_deployed/api_deleted), status (success/failure)
• Type: Counter
• Use Case: Track deployment notifications. Failures = control plane not aware of deployments

gateway_controller_control_plane_message_latency_seconds

• Type: Histogram
• Use Case: Track WebSocket message round-trip time. High latency = network issues

---

8. HTTP API Server Metrics 🌍

gateway_controller_http_requests_total

• Labels: method, endpoint, status_code
• Type: Counter
• Use Case: Track REST API usage patterns. High 4xx = client errors. High 5xx = server bugs

gateway_controller_http_request_duration_seconds

• Labels: method, endpoint
• Type: Histogram (buckets: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0)
• Use Case: API performance SLA. If POST /apis p95 > 1s, deployment is slow. Critical for CI/CD pipelines

gateway_controller_http_request_size_bytes

• Labels: endpoint
• Type: Histogram
• Use Case: Track request payload sizes. Large payloads = performance impact. Helps set limits

gateway_controller_http_response_size_bytes

• Labels: endpoint
• Type: Histogram
• Use Case: Track response sizes. Large GET /apis responses = pagination needed

gateway_controller_concurrent_requests

• Type: Gauge
• Use Case: Monitor request concurrency. Spikes = load surge. Helps set worker pool sizes

---

9. LLM & MCP Management Metrics 🤖

gateway_controller_llm_providers_total

• Labels: status (active/inactive)
• Type: Gauge
• Use Case: Track LLM provider inventory

gateway_controller_llm_provider_templates_total

• Type: Gauge
• Use Case: Monitor template catalog size

gateway_controller_mcp_proxies_total

• Labels: type, status
• Type: Gauge
• Use Case: Track MCP proxy deployments

gateway_controller_llm_operations_total

• Labels: operation (create/update/delete), status (success/failure)
• Type: Counter
• Use Case: Track LLM provider lifecycle operations

---

10. System Health & Resource Metrics 💻

gateway_controller_up

• Type: Gauge (0/1)
• Use Case: Basic liveness check for alerting

gateway_controller_info

• Labels: version, storage_type, build_date
• Type: Gauge (always 1)
• Use Case: Track deployed version. Helps correlate issues with releases

gateway_controller_goroutines

• Type: Gauge
• Use Case: Detect goroutine leaks. Growing count = resource leak. Should be stable under steady load

gateway_controller_memory_bytes

• Labels: type (heap_alloc/heap_sys/stack_inuse)
• Type: Gauge
• Use Case: Monitor memory usage. Unbounded growth = memory leak. Config store in-memory = memory proportional to API count

gateway_controller_snapshot_cache_size_bytes

• Type: Gauge
• Use Case: Track xDS cache memory usage. Large snapshots = memory pressure

gateway_controller_config_reload_timestamp

• Type: Gauge (Unix timestamp)
• Use Case: Track last configuration reload. Detect stale configs

---

11. Validation & Error Metrics ⚠️

gateway_controller_validation_duration_seconds

• Labels: validator_type (api/policy/llm/mcp)
• Type: Histogram
• Use Case: Measure validation overhead. Slow validation increases deployment latency

gateway_controller_errors_total

• Labels: component, error_type
• Type: Counter
• Use Case: Track errors by component. High error rate = component bugs

gateway_controller_panic_recoveries_total

• Labels: component
• Type: Counter
• Use Case: Track unexpected panics. Any panic = critical bug

---

Implementation Strategy

Technology Stack

I propose using Prometheus client libraries in the code to implement these metric collectors and expose them via standard Prometheus metrics endpoints.

Deployment Options

When users install the gateway with metrics enabled, they can:

1. Use the default metrics profile: Enable the metrics profile in the Docker Compose command to use the default setup with Prometheus for scraping and Grafana for visualization.

2. Use custom observability stack: Choose not to use the default provided strategy and instead use their own preferred scraping and visualization techniques if needed.

Please refer to this discussion — #506 — where we explore how profiles can be used to support different observability strategies.

This approach provides flexibility while also offering an out-of-the-box option for users who want to get started quickly with metrics observability.