Centralized Logging Observability for Gateway

[Tharsanan1]

Summary

This proposal outlines an approach for achieving centralized logging observability for the Gateway microservices (gateway-controller, policy-engine, router). The solution leverages the twelve-factor app methodology where all application logs are written to stdout/stderr, leaving the infrastructure layer responsible for collecting, aggregating, and storing these logs.

Current State

• All components log to stdout/stderr (container standard practice)
• Logs viewable via docker compose logs (ephemeral, not production-grade)
• No centralized aggregation, search, or retention capabilities

Proposed Solution

We can achieve centralized logging observability through various approaches:

• Grafana Loki Stack - Lightweight, cost-effective, designed for cloud-native environments
• ELK Stack (Elasticsearch, Logstash, Kibana) - Full-featured, mature solution with powerful search
• Fluentd/Fluent Bit Stack - Flexible log processor with broad integration support
• Cloud-Native Solutions - AWS CloudWatch, GCP Cloud Logging, Azure Monitor

Default Implementation: Profile-Based Services

When shipping the product, we need to provide a default logging solution with minimal friction. I propose using Docker Compose profiles to make logging components optional and easily deployable.

Example: Grafana Loki Stack Approach

The following services would be added to docker-compose.yml with the logging profile:

services:
  
  ### All our existing services for core gateway components
  
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    volumes:
      - ./observability/grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    ports:
      - "3000:3000"
    networks:
      - gateway-network
    profiles: ["tracing", "logging"]
  
  loki:
    image: grafana/loki:latest
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./observability/loki/loki-config.yaml:/etc/loki/local-config.yaml
      - loki-data:/loki
    networks:
      - gateway-network
    profiles: ["logging"]
  
  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - ./observability/promtail/promtail-config.yaml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    networks:
      - gateway-network
    depends_on:
      - loki
    profiles: ["logging"]

volumes:
  loki-data:

Usage

Users can deploy the gateway with logging enabled using:

# Start gateway components with logging stack
docker compose --profile logging up

# Or combine multiple profiles
docker compose --profile logging --profile tracing up

Key Benefits

1. Zero Impact on Gateway Core - Logging stack is completely optional and has no impact on gateway configuration or performance when not used
2. User Freedom - Users can choose not to use the default stack and integrate their own preferred logging observability strategy
3. Low Friction - Developers can enable logging with a single profile flag
4. Production Flexibility - In production environments, users can disable the bundled logging and use enterprise solutions

[Tharsanan1]

Adding the Grafana stack to our distributed docker-compose setup introduces licensing concerns, as Grafana is released under the AGPL license. To avoid these issues, we will include the Fluent Bit–based stack, which is released under the Apache 2.0 license instead.