Skip to content

Commit d0f3a60

Browse files
committed
Add Kubernetes secret support for secure Kafka connections
Operator Enhancements: - Add SecretReference struct to CRD for referencing K8s secrets - Support secret refs in SSL config (CA, client cert, key, key password) - Support secret refs in SASL config (username, password, keytab) - Automatically mount secrets as volumes in pipeline pods - Secrets mounted at /etc/streamforge/secrets/ with 0400 permissions CRD Changes (operator/src/crd.rs): - Add caSecret, certificateSecret, keySecret, keyPasswordSecret to SslConfig - Add usernameSecret, passwordSecret, keytabSecret to SaslConfig - Each secret ref contains name and key fields Reconciler Changes (operator/src/reconciler.rs): - New add_secret_volumes() method to collect and mount secrets - Collect secrets from source and all destinations - Create volume and volume mount for each unique secret - Support for multiple secrets across different security configs Documentation: - Add SECRETS.md - Comprehensive secret management guide - Add secure-sasl-pipeline.yaml - SASL authentication example - Add secure-tls-pipeline.yaml - Mutual TLS example - Update pipelines/README.md with security examples Benefits: ✅ Store credentials securely in etcd (encrypted) ✅ Separate config from secrets ✅ RBAC control for secret access ✅ Easy credential rotation without changing pipelines ✅ Audit trail for secret access ✅ Support for external secret managers (AWS Secrets Manager, etc.) This addresses the production security requirement for handling sensitive Kafka credentials (passwords, certificates, keytabs).
1 parent caf8319 commit d0f3a60

6 files changed

Lines changed: 736 additions & 5 deletions

File tree

examples/pipelines/README.md

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,56 @@ kubectl get streamforgepipeline -n streamforge-system
2020
kubectl get pods -n streamforge-system -l streamforge.io/pipeline=simple-mirror
2121
```
2222

23+
### secure-sasl-pipeline.yaml
24+
Pipeline with SASL authentication using Kubernetes secrets.
25+
26+
**Features:**
27+
- SASL/SCRAM authentication
28+
- Credentials stored in Kubernetes secrets (not inline)
29+
- TLS encryption with CA certificate
30+
- Production-ready security
31+
32+
**Usage:**
33+
```bash
34+
# Create secrets first
35+
kubectl create secret generic kafka-sasl-credentials \
36+
--from-literal=username=myuser \
37+
--from-literal=password=mypassword \
38+
-n streamforge-system
39+
40+
kubectl create secret generic kafka-ca-cert \
41+
--from-file=ca.crt=/path/to/ca-cert.pem \
42+
-n streamforge-system
43+
44+
# Deploy pipeline
45+
kubectl apply -f secure-sasl-pipeline.yaml
46+
```
47+
48+
### secure-tls-pipeline.yaml
49+
Pipeline with mutual TLS (mTLS) authentication.
50+
51+
**Features:**
52+
- Client certificate authentication
53+
- All certificates stored in secrets
54+
- Encrypted communication
55+
- Highest security level
56+
57+
**Usage:**
58+
```bash
59+
# Create TLS secret
60+
kubectl create secret generic kafka-tls-certs \
61+
--from-file=ca.crt=/path/to/ca-cert.pem \
62+
--from-file=client.crt=/path/to/client-cert.pem \
63+
--from-file=client.key=/path/to/client-key.pem \
64+
--from-literal=key.password=myKeyPassword \
65+
-n streamforge-system
66+
67+
# Deploy pipeline
68+
kubectl apply -f secure-tls-pipeline.yaml
69+
```
70+
71+
**📖 For detailed secret management documentation, see [SECRETS.md](./SECRETS.md)**
72+
2373
## Pipeline Specification
2474

2575
A StreamforgePipeline has the following main components:

0 commit comments

Comments
 (0)