Kubernetes operator for managing OpenSearch Data Prepper pipelines.
- Overview - what the operator is, key features, requirements
- Installation - Helm, Kustomize, building from source
- Quick Start - your first pipeline in 5 minutes
- Local Development - build and run with kind
- Architecture - operator internals, reconciliation workflow
- Pipeline Graph - pipelines array, in-process connectors, trace analytics
- Source Types - Kafka, HTTP, OTel, S3, Pipeline connector
- Sink Types - OpenSearch, S3, Kafka, Stdout, Pipeline connector
- Scaling - Kafka partition-based, HPA, static
- Peer Forwarder - auto-detection of stateful processors
- Source Discovery - automatic pipeline creation from Kafka/S3
- Defaults - namespace-level default settings
- Trace Analytics - a graph of 3 pipelines for tracing
- Kafka -> OpenSearch - processing logs from Kafka
- S3 Ingestion - S3 source with codecs and compression
- Kafka Auto-Discovery - automatic pipeline creation by topic
- S3 Auto-Discovery - automatic pipeline creation by prefix
- Monitoring - Prometheus metrics, ServiceMonitor, Grafana
- API: DataPrepperPipeline - full description of all fields
- API: DataPrepperSourceDiscovery - full description of all fields
- API: DataPrepperDefaults - full description of all fields
- Helm Values - Helm chart parameters
- Metrics - all operator Prometheus metrics
- Status and Conditions - phases, conditions, reconciler behavior
- Security - RBAC, Secrets, TLS, Pod Security
- High Availability - leader election, multi-replica
- Large-Scale Deployments - tuning for hundreds of pipelines
- Troubleshooting - diagnostics and common issues
- Upgrading - upgrading the operator, CRDs, and Data Prepper