Audit log receiver

[hilmarf]

Audit Log Receiver

Overview

This PR introduces the Audit Log Receiver for the OpenTelemetry Collector. The receiver accepts audit log data via HTTP, persists it using the storage extension, and processes logs asynchronously in the background.

Key Features

• HTTP Endpoint: Accepts POST requests at /v1/logs (supports JSON and OTLP protobuf).
• Immediate Response: Returns HTTP 202 Accepted after storing the log.
• Persistence: Uses the OpenTelemetry storage extension (file, SQL, Redis, etc.) for durability.
• Asynchronous Processing: Background goroutine processes and exports stored logs based on configurable intervals and age thresholds.
• Configurable: Supports custom endpoints, storage backends, processing intervals, and age thresholds.
• High Throughput: Designed for bursty workloads and scalable ingestion.

Architecture

• Receiver: Handles HTTP requests, generates UUIDs/timestamps, and stores logs.
• Storage: Persists logs as key-value pairs with user-defined backend.
• Background Processor: Periodically processes and exports logs older than a configured threshold.
• Retry Logic: Failed logs remain in storage for future processing attempts.

Example Configuration

extensions:
  file_storage:
    directory: ./storage
    create_directory: true

receivers:
  auditlogreceiver:
    endpoint: 0.0.0.0:4310
    storage: file_storage
    process_interval: 30s
    process_age_threshold: 30s

exporters:
  logging:
    loglevel: debug

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [auditlogreceiver]
      exporters: [logging]

Testing

• Includes test scripts for both Windows and Linux.
• Manual and automated tests for log ingestion and processing.
• Example test configuration and main program for standalone testing.

Benefits

• Immediate HTTP response for high-throughput scenarios.
• Asynchronous, reliable log processing.
• Flexible storage and configuration.
• Scalable and robust for audit log workloads.

TODO

• Implement circuit breaker for retry operations.
• Improve logging of processed logs (count only valid ones).
• Analyze persistence queue impact in exporters.