You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Mar 26, 2025. It is now read-only.
Copy file name to clipboardExpand all lines: README.md
+40-18Lines changed: 40 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,33 +22,55 @@
22
22
23
23
# Kafka-Operator
24
24
25
-
The Banzai Cloud Kafka operator is a Kubernetes operator to automate provisioning, management, autoscaliging and operations of [Apache Kafka](https://kafka.apache.org) clusters deployed to K8s.
25
+
The Banzai Cloud Kafka operator is a Kubernetes operator to automate provisioning, management, autoscaling and operations of [Apache Kafka](https://kafka.apache.org) clusters deployed to K8s.
26
26
27
27
## Overview
28
28
29
+
Apache Kafka is an open-source distributed streaming platform.
Some of the main features of the **Kafka-operator** are:
32
34
33
35
- Provision secure and production ready Kafka clusters
34
36
- Fine grained broker configuration support
35
37
- Advanced and highly configurable External Access via LoadBalancers using Envoy
36
-
- Graceful Kafka cluster scaling (up and down)
38
+
- Graceful Kafka cluster scaling (up and down) and rebalancing
37
39
- Monitoring via Prometheus
38
40
- Encrypted communication using SSL
39
-
- Automatic reactiin and self healing based on alers (pluggable alerts, with meaningful defaults)
41
+
- Automatic reaction and self healing based on alerts (plugin system, with meaningful default alert plugins)
40
42
41
43
### Motivation
42
44
43
-
At [Banzai Cloud](https://banzaicloud.com) we are building a Kubernetes distribution, [PKE](https://github.com/banzaicloud/pke) and platform, [Pipeline](https://github.com/banzaicloud/pipeline) and operate managed Kafka clusters for our customers.
45
+
At [Banzai Cloud](https://banzaicloud.com) we are building a Kubernetes distribution, [PKE](https://github.com/banzaicloud/pke) and a hybrid-cloud container management platform, [Pipeline](https://github.com/banzaicloud/pipeline) and operate managed Kafka clusters for our customers. Apache Kafka predates Kubernetes and it has been designed mostly for `static` on-premise environments. State management, node identity, failover, etc is all part and internal to Kafka, thus making it properly work on Kubernetes and an underlying dynamic environment could be a challenge.
46
+
47
+
There are already several approaches to operate Kafka on Kubernetes - however we did not find any of them appropriate for a highly dynamic environment, nor satisfying our customer needs. The skillset to operate Kafka and Kubernetes often does not overlap, and these are clearly visible on the existing operators. We are lucky to have Apache Kafka and Kubernetes contributors among our team and when we decided to work on the Kafka operator we designed and PoC every step from both perspectives.
48
+
49
+
At the same time there is a huge interest in the Kafka community for a solution which enables Kafka on Kubernetes, both in the open source and closed source space.
44
50
45
-
There is a huge interest in the Kafka community for a solution which enables Kafka on Kubernetes.
46
-
There were several initiatives to simplify Kafka on Kubernetes:
However, none of these gave a full solution to **automate** the Kafka experience and make it consumable for the wider audience. Our motivation is to build an open source solution and a community which drives the innovation and features of this operator.
55
+
To deep dive into more details of the most popular existing solution please see this table:
| User Management via CRD | Work in progress | No |Yes|No|
67
+
| Topic management via CRD | Work in progress | No |Yes|No|
68
+
| Reacting to Alerts| Yes (Prometheus + Cruise Control | No |No|No|
69
+
| Graceful Cluster Scaling (up and down)| Yes (using Cruise Control) | No |No|Yes|
70
+
71
+
**Note: this assessment was made on May 20, 2019 - if you find it inaccurate please submit a PR
72
+
73
+
Finally, our motivation is to build an open source solution and a community which drives the innovation and features of this operator.
52
74
53
75
If you are willing to kickstart your Kafka experience using Pipeline, check out the free developer beta:
54
76
<palign="center">
@@ -59,17 +81,16 @@ If you are willing to kickstart your Kafka experience using Pipeline, check out
59
81
60
82
## Installation
61
83
62
-
The operator installs the 2.1.0 version of Kafka, and can run on Minikube v0.33.1+ and Kubernetes 1.12.0+.
84
+
The operator installs the 2.1.0 version of Apache Kafka, and can run on Minikube v0.33.1+ and Kubernetes 1.12.0+.
63
85
64
-
As a pre-requisite it needs a Kubernetes cluster (you can create one using [Pipeline](https://github.com/banzaicloud/pipeline)).
65
-
Also, Kafka requires Zookeeper so you need to first start a Zookeeper server if you don't already have one.
86
+
As a pre-requisite it needs a Kubernetes cluster (you can create one using [Pipeline](https://github.com/banzaicloud/pipeline)). Also, Kafka requires Zookeeper so you need to first have a Zookeeper cluster if you don't already have one.
66
87
67
-
> Banzai Cloud's Kafka operator is a pure Kafka Operator without Zookeeper support.
88
+
> We believe in the `separation of concerns` principle, thus the Kafka operator does not install nor manage Zookeeper. If you would like to have a fully automated and managed experience of Apache Kafka on Kubernetes please try it with [Pipeline](https://github.com/banzaicloud/pipeline).
68
89
69
90
##### Install Zookeeper
70
91
71
-
To install Zookeeper we recommend using [Pravega's Zookeeper Operator](https://github.com/pravega/zookeeper-operator).
72
-
You can deploy Zookeeper by using a Helm chart.
92
+
To install Zookeeper we recommend using the [Pravega's Zookeeper Operator](https://github.com/pravega/zookeeper-operator).
> Remember to set your Kafka cr properly to use the newly created StorageClass.
124
+
> Remember to set your Kafka CR properly to use the newly created StorageClass.
103
125
104
126
1. Set `KUBECONFIG` pointing towards your cluster
105
-
2. Run `make deploy` (deploys the operator in the `kafka` namespace to the cluster)
127
+
2. Run `make deploy` (deploys the operator in the `kafka` namespace into the cluster)
106
128
3. Set your Kafka configurations in a Kubernetes custom resource (sample: `config/samples/banzaicloud_v1alpha1_kafkacluster.yaml`) and run this command to deploy the Kafka components:
> In this case you have to install Prometheus with proper configuration if you want to Kafka-Operator reacting on alerts.
136
+
> In this case you have to install Prometheus with proper configuration if you want the Kafka-Operator to react to alerts. Again, if you need Prometheus and would like to have a fully automated and managed experience of Apache Kafka on Kubernetes please try it with [Pipeline](https://github.com/banzaicloud/pipeline).
0 commit comments