Skip to content

Commit 50c2b07

Browse files
dvaseekaractrlaltlucamurarudobrerazvancristianpetrache
authored
Merging Master to Kraft and Reverting Rack Removal (#89)
* Fix flaky test by deleting nodeports explicitly (#67) * Upgrade Kafka to 3.6.0 (#69) * Upgrade dependencies * Fix wrong port on expectEnvoyWithConfigAz2Tls test (#70) * Upgrade Kafka to 3.6.1 (#71) Co-authored-by: Petruț™ <[email protected]> * Upgrade Kafka image to use Java v21 (#72) * Added arm64 to docker build platforms (#73) * Added arm64 to docker build platforms * Regenerated headers for 2024 * Upgrading Kafka to 3.7.0 (#77) * Update codeql-analysis.yml (#78) * [INTERNAL] Create uniq leader ID per operator deployment (#76) * [INTERNAL] Get watched namespaces from env variable (#75) (cherry picked from commit de6500b) * [CORE-106517] Fix outdated config in the sample (#83) * Cross-compile koperator for arm and intel. (#84) * Adding Contour Ingress support (#82) * Allow property security-inter-broker-protocol (#85) * adding the ability to use security-inter-broker-protocol in koperator * updating util.go to remove _ for generated names * adding replace all for external listener port name * fixing other places where externallistener name is used to not have _ * adding an alternative way to identify which port to use for kafka administration and cc connection * taking out comments for pr push * fixing kafka crd * setting omitempty so it will not be required * adding generated crds * adding comments with context for new flag UsedForKafkaAdminCommunication * Use getBrokerReadOnlyConfig function to get properties and update unit test - security_inter_broker_protocol_Set * Update crds to match generated manifest --------- Co-authored-by: Cameron Wright <[email protected]> Co-authored-by: Ha Van <[email protected]> * Revert "Allow concurrent broker restarts from same AZ (broker rack) (#62)" This reverts commit 514fa07. * Fixed build issues * Fix TestGenerateBrokerConfig * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added LoadBalancer for Kind E2E test cluster * Added watch namesapces * Added tmate for debugging * Added tmate for debugging * Added tmate for debugging * Added enabled projectcontour helm install * Enabled cloud-provider-kind * Added ProjectContour cluster role * updated certificate name * updated certificate name * Run without SSL * Removing Project Contour * Adding cloud-provider-kind * Removing cloud-provider - manually adding during test * trigger test * Remove SnpshotClusterAndCompare * Increased log length for Snapshot and Compare * Re-Add Snapshot and compare * Increased log length for Snapshot and Compare * Increased log length even more * Add Uninstall Contour CRDs * Re-Add KafkaCluster_SSL Tests * Removing BanzaiCloud Helm Chart from list of repos * pushing up latest go.sum * Clean up Merge * Enabling Tmate to debug e2e Test * Revert Cert Changes * Revert "Revert Cert Changes" This reverts commit 5c5b19c. * Enable sslClientAuth * trigger test * WIP: Fix Listener Config * Clean up test case results - tc-1 * Clean up test case results - tc-2 * Updated Kraft Test Cases * Cleanup Linting Issues * Remove Tmate Debugger * Run Kraft CLuster E2E * Add kraft e2e test * Revert Test --------- Co-authored-by: ctrlaltluc <[email protected]> Co-authored-by: Adi Muraru <[email protected]> Co-authored-by: Razvan Dobre <[email protected]> Co-authored-by: Cristian-Petrut Petrache <[email protected]> Co-authored-by: Petruț™ <[email protected]> Co-authored-by: Adrian Muraru <[email protected]> Co-authored-by: Adrian <[email protected]> Co-authored-by: aguzovatii <[email protected]> Co-authored-by: cawright-rh <[email protected]> Co-authored-by: Cameron Wright <[email protected]> Co-authored-by: Ha Van <[email protected]> Co-authored-by: Daniel Vaseekaran <[email protected]>
1 parent 20be375 commit 50c2b07

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+10652
-249
lines changed

‎.github/actions/kind-create/action.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,3 +45,12 @@ runs:
4545
chmod 600 $kubeconfig_path
4646
echo "kubeconfig=$(echo $kubeconfig_path)" >> $GITHUB_OUTPUT
4747
shell: bash
48+
49+
- name: Install cloud-provider-kind
50+
id: cloud-provider-kind
51+
run: |
52+
echo "Install cloud-provider-kind"
53+
go install sigs.k8s.io/cloud-provider-kind@latest
54+
kubectl label node e2e-kind-control-plane node.kubernetes.io/exclude-from-external-load-balancers-
55+
~/go/bin/cloud-provider-kind &
56+
shell: bash

‎.github/workflows/build-push-kafka-docker.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,18 +24,21 @@ jobs:
2424
echo ::set-output name=version::${VERSION}
2525
echo ::set-output name=tags::${TAGS}
2626
echo ::set-output name=created::$(date -u +'%Y-%m-%dT%H:%M:%SZ')
27+
- name: Set up QEMU
28+
uses: docker/setup-qemu-action@v3
2729
- name: Set up Docker Buildx
28-
uses: docker/setup-buildx-action@v1
30+
uses: docker/setup-buildx-action@v3
2931
- name: Login to DockerHub
3032
if: startsWith(github.ref, 'refs/tags/')
3133
uses: docker/login-action@v1
3234
with:
3335
username: ${{ secrets.DOCKER_USERNAME }}
3436
password: ${{ secrets.DOCKER_PASSWORD }}
3537
- name: Build and push
36-
uses: docker/build-push-action@v2
38+
uses: docker/build-push-action@v5
3739
with:
3840
context: docker/kafka
41+
platforms: linux/amd64,linux/arm64
3942
push: ${{ startsWith(github.ref, 'refs/tags/') }}
4043
tags: ${{ steps.prep.outputs.tags }}
4144
labels: |

‎.github/workflows/codeql-analysis.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ jobs:
4343

4444
# Initializes the CodeQL tools for scanning.
4545
- name: Initialize CodeQL
46-
uses: github/codeql-action/init@v1
46+
uses: github/codeql-action/init@v2
4747
with:
4848
languages: ${{ matrix.language }}
4949
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -54,7 +54,7 @@ jobs:
5454
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
5555
# If this step fails, then you should remove it and run the build manually (see below)
5656
- name: Autobuild
57-
uses: github/codeql-action/autobuild@v1
57+
uses: github/codeql-action/autobuild@v2
5858

5959
# ℹ️ Command-line programs to run using the OS shell.
6060
# 📚 https://git.io/JvXDl
@@ -68,4 +68,4 @@ jobs:
6868
# make release
6969

7070
- name: Perform CodeQL Analysis
71-
uses: github/codeql-action/analyze@v1
71+
uses: github/codeql-action/analyze@v2

‎.github/workflows/e2e-test.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,12 @@ jobs:
1818
uses: actions/setup-go@v4
1919
with:
2020
go-version: 1.21
21+
22+
# Enable Tmate Session if you'd like to Debut the E2E Kind Cluster
23+
# - name: Setup tmate session
24+
# uses: mxschmitt/action-tmate@v3
25+
# with:
26+
# detached: true
2127

2228
- name: Checkout code
2329
uses: actions/checkout@v4

‎.github/workflows/helm.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ jobs:
3333

3434
- name: Add Helm repositories
3535
run: |
36-
helm repo add banzaicloud-stable "https://kubernetes-charts.banzaicloud.com"
36+
# helm repo add banzaicloud-stable "https://kubernetes-charts.banzaicloud.com"
3737
helm repo add incubator "https://charts.helm.sh/incubator"
3838
helm repo add stable "https://charts.helm.sh/stable"
3939

‎Makefile‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,20 @@ docker-build: ## Build the operator docker image.
156156
docker-push: ## Push the operator docker image.
157157
docker push ${IMG}
158158

159+
# PLATFORMS defines the target platforms for the manager image be built to provide support to multiple
160+
# architectures. (i.e. make docker-buildx IMG=myregistry/mypoperator:0.0.1). To use this option you need to:
161+
# - be able to use docker buildx. More info: https://docs.docker.com/build/buildx/
162+
# - have enabled BuildKit. More info: https://docs.docker.com/develop/develop-images/build_enhancements/
163+
# - be able to push the image to your registry (i.e. if you do not set a valid value via IMG=<myregistry/image:<tag>> then the export will fail)
164+
# To adequately provide solutions that are compatible with multiple platforms, you should consider using this option.
165+
PLATFORMS ?= linux/arm64,linux/amd64
166+
.PHONY: docker-buildx
167+
docker-buildx: ## Build and push docker image for the manager for cross-platform support
168+
- docker buildx create --name koperator-builder
169+
docker buildx use koperator-builder
170+
docker buildx build --push --platform=$(PLATFORMS) --tag ${IMG} -f Dockerfile .
171+
- docker buildx rm koperator-builder
172+
159173
bin/controller-gen: bin/controller-gen-$(CONTROLLER_GEN_VERSION) ## Symlink controller-gen-<version> into versionless controller-gen.
160174
@ln -sf controller-gen-$(CONTROLLER_GEN_VERSION) bin/controller-gen
161175

‎api/go.mod‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ require (
77
emperror.dev/errors v0.8.1
88
github.com/banzaicloud/istio-client-go v0.0.17
99
github.com/cert-manager/cert-manager v1.13.2
10-
github.com/stretchr/testify v1.8.4
10+
// github.com/stretchr/testify v1.8.4
1111
golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa
1212
gotest.tools v2.2.0+incompatible
1313
k8s.io/api v0.28.4
@@ -16,7 +16,7 @@ require (
1616
)
1717

1818
require (
19-
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
19+
// github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
2020
github.com/go-logr/logr v1.3.0 // indirect
2121
github.com/gogo/protobuf v1.3.2 // indirect
2222
github.com/google/go-cmp v0.5.9 // indirect
@@ -25,18 +25,26 @@ require (
2525
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
2626
github.com/modern-go/reflect2 v1.0.2 // indirect
2727
github.com/pkg/errors v0.9.1 // indirect
28-
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
28+
// github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
2929
go.uber.org/multierr v1.11.0 // indirect
3030
golang.org/x/net v0.18.0 // indirect
3131
golang.org/x/text v0.14.0 // indirect
3232
gopkg.in/inf.v0 v0.9.1 // indirect
3333
gopkg.in/yaml.v2 v2.4.0 // indirect
34-
gopkg.in/yaml.v3 v3.0.1 // indirect
34+
// gopkg.in/yaml.v3 v3.0.1 // indirect
3535
k8s.io/klog/v2 v2.110.1 // indirect
3636
k8s.io/utils v0.0.0-20230726121419-3b25d923346b // indirect
3737
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
3838
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
3939
)
4040

41+
require github.com/stretchr/testify v1.8.4
42+
43+
require (
44+
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
45+
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
46+
gopkg.in/yaml.v3 v3.0.1 // indirect
47+
)
48+
4149
// remove once https://github.com/cert-manager/cert-manager/issues/5953 is fixed
4250
replace github.com/Venafi/vcert/v4 => github.com/jetstack/vcert/v4 v4.9.6-0.20230127103832-3aa3dfd6613d

‎api/v1beta1/kafkacluster_types.go‎

Lines changed: 32 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,7 @@ type KafkaClusterSpec struct {
178178
RollingUpgradeConfig RollingUpgradeConfig `json:"rollingUpgradeConfig"`
179179
// Selector for broker pods that need to be recycled/reconciled
180180
TaintedBrokersSelector *metav1.LabelSelector `json:"taintedBrokersSelector,omitempty"`
181-
// +kubebuilder:validation:Enum=envoy;istioingress
181+
// +kubebuilder:validation:Enum=envoy;contour;istioingress
182182
// IngressController specifies the type of the ingress controller to be used for external listeners. The `istioingress` ingress controller type requires the `spec.istioControlPlane` field to be populated as well.
183183
IngressController string `json:"ingressController,omitempty"`
184184
// IstioControlPlane is a reference to the IstioControlPlane resource for envoy configuration. It must be specified if istio ingress is used.
@@ -190,13 +190,14 @@ type KafkaClusterSpec struct {
190190
// when false, they will be kept so the Kafka cluster remains available for those Kafka clients which are still using the previous ingress setting.
191191
// +kubebuilder:default=false
192192
// +optional
193-
RemoveUnusedIngressResources bool `json:"removeUnusedIngressResources,omitempty"`
194-
PropagateLabels bool `json:"propagateLabels,omitempty"`
195-
CruiseControlConfig CruiseControlConfig `json:"cruiseControlConfig"`
196-
EnvoyConfig EnvoyConfig `json:"envoyConfig,omitempty"`
197-
MonitoringConfig MonitoringConfig `json:"monitoringConfig,omitempty"`
198-
AlertManagerConfig *AlertManagerConfig `json:"alertManagerConfig,omitempty"`
199-
IstioIngressConfig IstioIngressConfig `json:"istioIngressConfig,omitempty"`
193+
RemoveUnusedIngressResources bool `json:"removeUnusedIngressResources,omitempty"`
194+
PropagateLabels bool `json:"propagateLabels,omitempty"`
195+
CruiseControlConfig CruiseControlConfig `json:"cruiseControlConfig"`
196+
EnvoyConfig EnvoyConfig `json:"envoyConfig,omitempty"`
197+
ContourIngressConfig ContourIngressConfig `json:"contourIngressConfig,omitempty"`
198+
MonitoringConfig MonitoringConfig `json:"monitoringConfig,omitempty"`
199+
AlertManagerConfig *AlertManagerConfig `json:"alertManagerConfig,omitempty"`
200+
IstioIngressConfig IstioIngressConfig `json:"istioIngressConfig,omitempty"`
200201
// Envs defines environment variables for Kafka broker Pods.
201202
// Adding the "+" prefix to the name prepends the value to that environment variable instead of overwriting it.
202203
// Add the "+" suffix to append.
@@ -239,15 +240,17 @@ type RollingUpgradeConfig struct {
239240
// alerts with 'rollingupgrade'
240241
FailureThreshold int `json:"failureThreshold"`
241242

242-
// ConcurrentBrokerRestartsAllowed controls how many brokers can be restarted in parallel during a rolling upgrade. If
243+
// ConcurrentBrokerRestartCountPerRack controls how many brokers can be restarted in parallel during a rolling upgrade. If
243244
// it is set to a value greater than 1, the operator will restart up to that amount of brokers in parallel, if the
244245
// brokers are within the same rack (as specified by "broker.rack" in broker read-only configs). Since using Kafka broker
245246
// racks spreads out the replicas, we know that restarting multiple brokers in the same rack will not cause more than
246247
// 1/Nth of the replicas of a topic-partition to be unavailable at the same time, where N is the number of racks used.
247248
// This is a safe way to speed up the rolling upgrade. Note that for the rack distribution explained above, Cruise Control
248-
// requires `com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareDistributionGoal` to be configured.
249+
// requires `com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareDistributionGoal` to be configured. Default value is 1.
250+
// +kubebuilder:validation:Minimum=1
251+
// +kubebuilder:default=1
249252
// +optional
250-
ConcurrentBrokerRestartsAllowed int `json:"concurrentBrokerRestartsAllowed,omitempty"`
253+
ConcurrentBrokerRestartCountPerRack int `json:"concurrentBrokerRestartCountPerRack,omitempty"`
251254
}
252255

253256
// DisruptionBudget defines the configuration for PodDisruptionBudget where the workload is managed by the kafka-operator
@@ -622,6 +625,10 @@ func (c IngressServiceSettings) GetServiceType() corev1.ServiceType {
622625
return c.ServiceType
623626
}
624627

628+
func (c ContourIngressConfig) GetBrokerFqdn(brokerId int32) string {
629+
return strings.Replace(c.BrokerFQDNTemplate, "%id", strconv.Itoa(int(brokerId)), 1)
630+
}
631+
625632
// Replace %id in brokerHostnameTemplate with actual broker id
626633
func (c EnvoyConfig) GetBrokerHostname(brokerId int32) string {
627634
return strings.Replace(c.BrokerHostnameTemplate, "%id", strconv.Itoa(int(brokerId)), 1)
@@ -702,7 +709,7 @@ type ExternalListenerConfig struct {
702709
// IngressControllerTargetPort defines the container port that the ingress controller uses for handling external traffic.
703710
// If not defined, 29092 will be used as the default IngressControllerTargetPort value.
704711
IngressControllerTargetPort *int32 `json:"ingressControllerTargetPort,omitempty"`
705-
// +kubebuilder:validation:Enum=LoadBalancer;NodePort
712+
// +kubebuilder:validation:Enum=LoadBalancer;NodePort;ClusterIP
706713
// accessMethod defines the method which the external listener is exposed through.
707714
// Two types are supported LoadBalancer and NodePort.
708715
// The recommended and default is the LoadBalancer.
@@ -725,8 +732,16 @@ type Config struct {
725732

726733
type IngressConfig struct {
727734
IngressServiceSettings `json:",inline"`
728-
IstioIngressConfig *IstioIngressConfig `json:"istioIngressConfig,omitempty"`
729-
EnvoyConfig *EnvoyConfig `json:"envoyConfig,omitempty"`
735+
IstioIngressConfig *IstioIngressConfig `json:"istioIngressConfig,omitempty"`
736+
EnvoyConfig *EnvoyConfig `json:"envoyConfig,omitempty"`
737+
ContourIngressConfig *ContourIngressConfig `json:"contourIngressConfig,omitempty"`
738+
}
739+
740+
type ContourIngressConfig struct {
741+
// TLS secret used for Contour IngressRoute resource
742+
TLSSecretName string `json:"tlsSecretName"`
743+
// Broker hostname template for Contour IngressRoute resource to generate broker hostnames.
744+
BrokerFQDNTemplate string `json:"brokerFQDNTemplate"`
730745
}
731746

732747
// InternalListenerConfig defines the internal listener config for Kafka
@@ -764,6 +779,9 @@ type CommonListenerSpec struct {
764779
// At least one of the listeners should have this flag enabled
765780
// +optional
766781
UsedForInnerBrokerCommunication bool `json:"usedForInnerBrokerCommunication"`
782+
// UsedForKafkaAdminCommunication allows for a different port to be returned when the koperator is checking for the port to use to check if kafka is operating.
783+
// +optional
784+
UsedForKafkaAdminCommunication bool `json:"usedForKafkaAdminCommunication,omitempty"`
767785
}
768786

769787
func (c *CommonListenerSpec) GetServerSSLCertSecretName() string {

‎api/v1beta1/zz_generated.deepcopy.go‎

Lines changed: 21 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)