Skip to content

0019-otel-batchspanprocessor-pattern-for-event-sink.md

Latest commit

 

History

History
56 lines (41 loc) · 2.14 KB

0019-otel-batchspanprocessor-pattern-for-event-sink.md

File metadata and controls

56 lines (41 loc) · 2.14 KB
Raw
title ADR-0019: OTel BatchSpanProcessor Pattern for Event Sink
last_reviewed 2026-05-18
owner agt-maintainers

ADR-0019: OTel BatchSpanProcessor Pattern for Event Sink

Status

Accepted

Context

The Governance Event Sink needed to route structured governance events to external systems (SIEM, XDR, observability platforms, message buses) without blocking the governance decision hot path. Requirements:

  • Async delivery to avoid adding latency to policy evaluation
  • Batching to reduce network round-trips
  • Fan-out to multiple sinks simultaneously
  • Resilience to sink failures (one failing sink must not block others)
  • Bounded memory usage under backpressure

The OpenTelemetry SDK's BatchSpanProcessor solves an analogous problem for trace spans and is battle-tested across thousands of production deployments.

Decision

We adopted the OTel BatchSpanProcessor architecture for GovernanceEventProcessor:

  • Bounded queue (default 1024 events) with drop-on-full semantics
  • Background thread drains the queue on a schedule (default 2000ms)
  • Batch size cap (default 100 events per export call)
  • Export timeout (default 10000ms) per sink call
  • Fan-out: each registered GovernanceEventSink receives the same batch
  • Circuit breaker: after N consecutive failures (default 5), a sink is bypassed for a cooldown period (default 60s)

The GovernanceEventSink protocol uses structural typing (Protocol) so external packages can implement it without importing agent-os as a dependency.

Consequences

  • Zero-latency impact on governance decisions (events are enqueued, not sent inline)
  • Predictable memory usage via bounded queue
  • Graceful degradation -- circuit breaker prevents cascading failures
  • External sinks are decoupled (structural typing, no import dependency)
  • Events can be dropped under extreme backpressure (acceptable tradeoff for governance where the audit log is the source of truth)

References

  • agent-governance-python/agent-os/src/agent_os/event_sink.py
  • docs/specs/AUDIT-COMPLIANCE-1.0.md Section 11
  • PR #2362 (GovernanceEventSink SPI)
  • OpenTelemetry SDK BatchSpanProcessor specification