feat(security): [EPIC][SECURITY] SIEM Integration and Security Event Export

[crivetimihai]

Note: This PR was re-created from #3069 due to repository maintenance. Your code and branch are intact. @crivetimihai please verify everything looks good.

🔗 Related Issue

Closes #2597

Summary

This PR fully implements SIEM integration and security event export for MCP Gateway.

It introduces an asynchronous SIEM export pipeline, wires security/audit/auth event producers into that pipeline, adds admin APIs for destination management and health checks, and adds comprehensive configuration and metrics support.

What Changed

• Added mcpgateway/services/siem_export_service.py.
• Added mcpgateway/routers/siem.py with admin endpoints:
  • GET /admin/siem/health
  • GET /admin/siem/destinations
  • POST /admin/siem/destinations
  • PUT /admin/siem/destinations
  • POST /admin/siem/test/{destination_name}
• Registered SIEM router in mcpgateway/main.py.
• Initialized SIEM export service during app startup and shut it down cleanly during app teardown.
• Added SIEM-specific Prometheus metrics in mcpgateway/services/metrics.py:
  • siem_events_exported_total
  • siem_export_latency_seconds
  • siem_queue_depth
• Added SIEM export configuration and parsing in mcpgateway/config.py:
  • New SIEM env/settings flags for enablement, queue/retry/backoff, source filters, URL allowlist, redaction fields, Redis stream/group, and destinations.
  • JSON/YAML parsing for SIEM_DESTINATIONS.
  • Optional SIEM_DESTINATIONS_FILE support.
• Updated auth logging middleware (mcpgateway/middleware/auth_middleware.py) so auth events can be exported to SIEM even when DB security logging is disabled.
• Updated mcpgateway/services/security_logger.py to emit events to SIEM and support SIEM-only mode (persist=False) with in-memory failed-auth tracking.
• Updated mcpgateway/services/audit_trail_service.py to emit audit events to SIEM independently of DB audit persistence.
• Documented new SIEM configuration in .env.example.

SIEM Export Capabilities

• Asynchronous batching and flush intervals.
• Redis Streams backend with local in-memory queue fallback.
• Destination fan-out delivery.
• Event source filtering (auth, security, audit).
• Backpressure policy support (drop_oldest or block_producer).
• Retry with bounded exponential backoff and dead-letter queue handling.
• Destination validation, URL allowlist checks, and health/status tracking.
• Payload formatting support for json, cef, and leef.
• Destination types support: splunk_hec, datadog, elasticsearch, webhook, syslog.
• Sensitive field redaction before export.

Tests Added/Updated

• Added tests/unit/mcpgateway/services/test_siem_export_service.py.
• Added tests/unit/mcpgateway/routers/test_siem.py.
• Updated tests/unit/mcpgateway/services/test_security_logger.py.
• Updated tests/unit/mcpgateway/services/test_audit_trail_service.py.

How To Test

1. Static checks:

• make flake8
• make pylint
• make bandit

2. Unit tests for SIEM/security integration:

• uv run pytest tests/unit/mcpgateway/services/test_siem_export_service.py tests/unit/mcpgateway/routers/test_siem.py tests/unit/mcpgateway/services/test_security_logger.py tests/unit/mcpgateway/services/test_audit_trail_service.py

3. Manual runtime verification:

• Configure SIEM in .env (example):
  • SIEM_EXPORT_ENABLED=true
  • SIEM_EXPORT_EVENT_SOURCES=["auth","security","audit"]
  • SIEM_DESTINATIONS=[{"name":"webhook-1","type":"webhook","url":"https://example.com/siem"}]
• Start gateway: make dev
• Call admin SIEM endpoints with an authorized admin token:
  • GET /admin/siem/destinations
  • POST /admin/siem/test/webhook-1
  • GET /admin/siem/health
• Trigger auth and audit events (for example invalid token auth attempts and admin CRUD actions) and verify:
  • SIEM destination receives events.
  • Health endpoint reflects delivery counters/latency.
  • Redacted fields are not exported in clear text.

Closes #2597