Distributed Tracing for Gateway

[Tharsanan1]

Distributed Tracing for Gateway

Overview

This proposal outlines the distributed tracing strategy for our Gateway with external policy engine processor.

Problem Statement

We need visibility into:

• Complete request flow through the gateway
• Time spent at each component (router → policy engine → backend)
• Per-policy evaluation latency and outcomes
• Bottleneck identification across distributed components
• Error correlation and root cause analysis
• Policy decision patterns (allow/deny rates, reasons)

Proposed Solution

Architecture

### Architecture

┌─────────────┐                    ┌──────────────────┐
│   Envoy     │                    │                  │
│  Gateway    │─┐  OTLP/gRPC      │  OpenTelemetry   │    ┌──────────────────┐
└─────────────┘ │  (port 4317)    │    Collector     │───▶│ Distributed      │
                ├────────────────▶ │    (Runtime)     │    │ Tracing Impl     │
┌─────────────┐ │                  │                  │    │  (Zipkin)        │
│   Policy    │─┘                  │  - Batching      │    └──────────────────┘
│   Engine    │                    │  - Sampling      │
└─────────────┘                    │  - Enrichment    │
                                   └──────────────────┘

Why we export traces to the OpenTelemetry Collector instead of a specific or set of predefined otel backends(Jeager,Zipkin,Grapana Tempo)

• Decouples applications from observability backends (tracing, metrics, logging systems), allowing you to change or add backends without modifying or redeploying gateway components
• Centralized sampling/processing logic
• Performance optimization through batching and buffering
• Multi-backend support (can send to multiple destinations simultaneously)

What We'll Export

1. From Envoy Gateway

Root Span (per request):

• Span name: gateway.request
• Span kind: SERVER
• Key attributes:
  • HTTP method, route, status code
  • Request/response sizes
  • Client address, user agent
  • Custom: gateway.tenant_id, gateway.api_key_id

External Processor Call Span:

• Span name: policy_engine.evaluate
• Span kind: CLIENT
• Attributes:
  • RPC details (gRPC service/method)
  • Call duration, timeout settings

2. From Policy Engine

Server Span (receiving ext_proc call):

• Span name: policy_engine.process
• Span kind: SERVER
• Attributes:
  • Total policies to evaluate
  • Evaluation mode (sequential/parallel)

Per-Policy Child Spans (critical for multi-policy observability):

• Span name: policy.evaluate.{policy_name}
  • Examples: policy.evaluate.rate_limit, policy.evaluate.authentication
• Span kind: INTERNAL
• Attributes per policy:
  • policy.name: Policy identifier
  • policy.type: Category (rate_limit, auth, validation, etc.)
  • policy.result: "allow" | "deny" | "error"
  • policy.execution_order: Sequence number (1, 2, 3...)
  • policy.decision_reason: Why it passed/failed
  • Policy-specific context:
    • Rate limit: limits, remaining quota, window
    • Auth: principal, method (JWT, API key)
    • Validation: schema version, error count

What We Can Observe

Performance Analysis

• Latency breakdown: Time at gateway vs policy engine vs each policy
• Bottleneck detection: Identify slowest policies or components
• P50/P95/P99 metrics: Performance percentiles at each layer

Policy Insights

• Which policies execute most frequently
• Per-policy allow/deny rates
• Policy evaluation order and timing
• Correlation: which policies typically deny together?

Error Tracking

• Root cause identification (which policy caused failure?)
• Error patterns by endpoint, tenant, or time
• Cascading failure detection

References

• OpenTelemetry Specification
• Envoy Gateway Tracing Docs

[dgilling]

In docs/product, we should probably highlight Moesif OTel support as well for expansion opportunity.